Hi Nicolas, can I work on this one? It will be my first issue. Since there are lots of unique occurrences of n_jobs, it might take time. But I am motivated to get it done.

@arpanchowdhry note that it is labeled as moderate (not "easy") since it requires some knowledge of the implementations. I wouldn't particularly recommend this as a first issue, but if you're confident please feel free to try!

@NicolasHug I agree that this is not easy as it involves a lot of inner working knowledge. I have been making progress by reading a lot of code. But if you or someone else gets it first that is fine, it has still been good learning for me.

@NicolasHug Hi! Are you still looking for help? I'm interested in contributing to this issue. Please let me know in case you've room.

Yes feel free to submit a PR for some estimators

@vinidixit Please go ahead.

Based on the script:

import re
from collections import defaultdict
from pathlib import Path

matches = defaultdict(list)
for filename in Path('sklearn').glob('**/*.py'):
    with open(filename) as f:
        for i, line in enumerate(f.readlines()):
            match = re.search(r"n_jobs ?:", line)
            if match:
                matches[str(filename)].append(i+1)

for file, lines in matches.items():
    print('- [ ] [{FN}](https://github.com/scikit-learn/scikit-learn/blob/master/{FN})'.format(FN=file), end=" - ")
    line_number_format = '[{LN}](https://github.com/scikit-learn/scikit-learn/blob/master/{FN}#L{LN})'
    line_number_links = ', '.join([line_number_format.format(FN=file, LN=line) for line in lines])
    print(line_number_links)

I got this lists:

• sklearn/multiclass.py - 160, 483, 693
• sklearn/multioutput.py - 217, 313
• sklearn/pipeline.py - 774, 1011
• sklearn/cluster/_bicluster.py - 226, 370
• sklearn/cluster/_dbscan.py - 84, 202
• sklearn/cluster/_k_means.py - 256, 714
• sklearn/cluster/_mean_shift.py - 54, 156, 283
• sklearn/cluster/_optics.py - 140, 418
• sklearn/cluster/_spectral.py - 372
• sklearn/compose/_column_transformer.py - 98, 710
• sklearn/covariance/_graph_lasso.py - 549
• sklearn/decomposition/_dict_learning.py - 253, 467, 675, 985, 1115, 1276
• sklearn/decomposition/_kernel_pca.py - 93
• sklearn/decomposition/_online_lda.py - 216
• sklearn/decomposition/_sparse_pca.py - 67, 272
• sklearn/ensemble/_bagging.py - 497, 896
• sklearn/ensemble/_forest.py - 957, 1273, 1549, 1853, 2083
• sklearn/ensemble/_iforest.py - 89
• sklearn/ensemble/_stacking.py - 302, 547
• sklearn/ensemble/_voting.py - 114, 345
• sklearn/ensemble/partial_dependence.py - 245
• sklearn/feature_selection/_rfe.py - 397
• sklearn/gaussian_process/_gpc.py - 545
• sklearn/inspection/_partial_dependence.py - 509
• sklearn/inspection/_permutation_importance.py - 80
• sklearn/linear_model/_base.py - 402
• sklearn/linear_model/_coordinate_descent.py - 1304, 1478, 2009, 2185
• sklearn/linear_model/_least_angle.py - 1286, 1503
• sklearn/linear_model/_logistic.py - 1324, 1777
• sklearn/linear_model/_omp.py - 804
• sklearn/linear_model/_passive_aggressive.py - 69
• sklearn/linear_model/_perceptron.py - 48
• sklearn/linear_model/_stochastic_gradient.py - 791
• sklearn/linear_model/_theil_sen.py - 250
• sklearn/manifold/_isomap.py - 62
• sklearn/manifold/_locally_linear.py - 84, 258, 590
• sklearn/manifold/_mds.py - 179, 305
• sklearn/manifold/_spectral_embedding.py - 402
• sklearn/manifold/_t_sne.py - 586
• sklearn/metrics/pairwise.py - 1483, 1660, 1861
• sklearn/model_selection/_search.py - 887, 1217
• sklearn/model_selection/_validation.py - 98, 316, 666, 1007, 1165, 1438
• sklearn/neighbors/_classification.py - 83, 323
• sklearn/neighbors/_graph.py - 79, 157, 269, 408
• sklearn/neighbors/_lof.py - 119
• sklearn/neighbors/_regression.py - 87, 261
• sklearn/neighbors/_unsupervised.py - 59
• sklearn/semi_supervised/_label_propagation.py - 101, 327, 440

I can help to check them to see the progress

[Edit 13 Nov: Script rerun for some file renaming]

For example, this doc is added (chronogically) after the discussion in #10415 in this commit.

But IMO, seems like the explanation of the parameter is still a bit generic, especially if we assess it based on @NicolasHug's idea of

It would be much more helpful for users to understand where and how parallelization is actually applied.
For example for random forest: "Each tree is built in parallel" would be a simple yet significant improvement.

    n_jobs : int or None, optional (default=None)
        The number of jobs to use for the computation.
        ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context.
        ``-1`` means using all processors. See :term:`Glossary <n_jobs>`
        for more details.

Are these kind of documentation counted as good enough or do you think we need more context specific documentation? @NicolasHug

Thanks for the script @kwinata!

Are these kind of documentation counted as good enough or do you think we need more context specific documentation? @NicolasHug

That could definitely be improved: "The number of jobs to use for the computation" is very vague and doesn't say what is parallelized. Users should get a sense of how much faster their code may be when they change n_jobs.

Welcome @NicolasHug

Sure then, I would be happy to work on improving those cases!

@NicolasHug , I would like to work on sklearn/linear_model/stochastic_gradient.py, sklearn/linear_model/coordinate_descent.py and sklearn/ensemble/forest.py just for the start.
Later, I would like to pick on batches.

@kwinata , please do let me know if the above mentioned cases have already been worked.

Thank you

@PyExtreme thanks! Feel free to start new PRs (Please reference this issue when you do)

Tip: you can filter PRs by author and you'll see that @kwinata does not seem to have opened anything yet.

Yes @PyExtreme feel free to work on them. I have been busy for this past few weeks. Currently I will work on the cluster and neighbors algorithms.

@NicolasHug, I have updated the list of files containing n_jobs in the last release (see this gist). Checked files there are those containing n_jobs with a ref to the Glossary: they are the majority. I'm wondering if this could still be a good issue for a Sprint (it was used at the Scipy sprint 2019 if I understand correctly). It is not an obvious task to identify if an explanation needs to be updated or not (that's why you labelled with 'Moderate'), and in that case (unlike the random_state issue #10548) there is not even a simple addition to provide. May I suggest to provide a list of the explanations that you really feel unsatisfactory? No deadline... :) I'm not adding this one to Sprints , if you are ok with that.

As a note, there is still an open PR (#15613) referred to this issue.

Checked files there are those containing n_jobs with a ref to the Glossary: they are the majority

A link to the glossary is the bare minimum, but that's not what this issue is about. The issue is about:

The great majority of the docstrings for n_jobs is something like "number of jobs to run in parallel".

Any docstring that does not explain where and how parallelization happens should be updated.

hi ! @annejeevan and I will work on documenting n_jobs for forest models and multiclass as part of the data umbrella sprint.

take sklearn/_graph.py with @krumeto

Hi, take sklearn/gaussian_process/_gpc.py

Hi, I'll work on sklearn/model_selection/_validation.py

I opend the PR for _permutation_importance.py too.

Submitted #18633 and #18634 for sklearn/calibration.py and sklearn/multioutput.py

I would be interested in a docs page that has a list of all the algorithms that support parallelism with n_jobs. Specifically because its helpful to get a high-level glance at which algorithms might be able to utilize Dask for parallelism across a cluster. Would the parallelism page be a good place for this, or perhaps a new page? I would be happy to contribute this, just want to get an idea of what the maintainers think about it

@rikturr I'm not sure such a list would be in-scope for our docs: it would be easy to forget to update and the benefits aren't clear to me.

But you can get what you want by programmatically checking the signature of all estimators: https://scikit-learn.org/stable/modules/generated/sklearn.utils.all_estimators.html

@NicolasHug thanks for the pointer. For anyone else that wants to do it, here's a quick snippet:

from sklearn.utils import all_estimators
import inspect

has_n_jobs = []
for est in all_estimators():
    s = inspect.signature(est[1])
    if 'n_jobs' in s.parameters:
        has_n_jobs.append(est)
print(has_n_jobs)