You can probably refer to #3754 which should be highly related to your question.

why do think it must be set to 1?

Good catch :)

I am closing since this is more related to joblib

Should this be in the FAQ? I get asked this regularly (and on a webcast two days ago). It's also non-obvious to users how to decide which n_jobs to set and we could give some guidance on that.

Should this be in the FAQ? I get asked this regularly (and on a webcast two days ago). It's also non-obvious to users how to decide which n_jobs to set and we could give some guidance on that.

it seems reasonable.

Can I take this?

Should this be in the FAQ? I get asked this regularly (and on a webcast two days ago). It's also non-obvious to users how to decide which n_jobs to set and we could give some guidance on that.

Will have to solve this one. Right?

Will figure out

Has this been done now? Do we still have to specify n_jobs=1 for the inner parallelism (estimator) ?

There already seems to be 2 questions in FAQ(First and Second) which address n_jobs parameter's use cases. But, they don't explain how it should be used.

Am I supposed to add a new question explaining how to use n_jobs parameter? Also this section of Glossary seems to be addressing that part well enough.

Hi,
Is this still an issue? If so, then can someone help me understand better?

From the last PR linked above, nested parallelism is now enabled by default when you use parallel_backend("dask"), see docs. This solves the original problem that led to this issue being opened, thus I believe clarification in docs shouldn't be required anymore.

Unless we want to document this more extensively in a tutorial, I suggest we close this issue.

I'm going to close this as the original post was addressed: no need to set n_jobs to 1.

As to documenting what n_jobs should be in nested CV procedures, this is neither easy or a good first issue IMHO.

We have related docs in https://scikit-learn.org/stable/modules/computing.html#parallelism and there is also #14228 which I think will tackle most of it.

Although, this is closed I think it would be good for an update, for users who land here now:

What would be the current best practise for parallelism in nested cross validation with sklearn today?
Nov 2022: sklearn stable v. 1.1.3, or 1.2.dev0 would be the relevant recent versions
@mattvan83 @NicolasHug

Running inside a jupyter notebook, I am trying to use parallel computation on a server (120 cpu cores) like so:

with parallel_backend('loky', n_jobs=-1):
    innerCV = GridSearchCV(
        pipe,
        params,
        scoring= scoring,
        refit= refit_scorer,
        cv=10,
        verbose=1,
        )

    outerCV = cross_validate(
        innerCV,
        model_X,
        model_y,
        scoring=scoring,
        cv=10,
        return_estimator=True,
        verbose=1,
        )

The pipe is my estimator object, which itself is a sklearn.pipeline wrapping some transformers and various options for estimators specified in my params grid.

It runs without errors, however, I am not sure if it is completely optimised. Some time during the fit I see load on all CPUs but most of the time just 10 of them get to work. I assume this is due to the cv=10 I am using here (though, is it the inner or outer the gets parallelised?).

The times when all CPUs are in use might be when an estimator is tested which has some internal (numpy) parallelisation, I assume?

So, is this a tangible way today to approach nested CV parallelisation in sklearn today? ...Or would it be better to:

• specify n_jobs in inner and outer CV individually instead of using a context manager?
  • and if so, should they both get n_jobs=-1?
• be using dask backend be beneficial to loki (when on a single machine)
• go about it in a completely different way?

Any guidance welcome!
Thanks, robna.