There is something weird. On my laptop (2 cores, 4 hyperthreads):

• pytest -v sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py => 0.70s (no xdist)
• pytest -v -n 1 sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py => 1.46s (1 xdist worker)
• pytest -v -n 2 sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py => 11s to 48s (2 xdist workers)
• OMP_NUM_THREADS=2 pytest -v -n 2 sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py => 1.15s
• OMP_NUM_THREADS=4 pytest -v -n 2 sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py => between 7.8s and 34s

So this seems to be a really extreme case of over-subscription.

I have also tried to use -d --tx 2*popen//python=python instead of -n 2 and I get similar results.

I have also tried to add -k Classifier to only run the classification test and even with OMP_NUM_THREADS=4 and 2 xdist workers the run time does not go beyond 1.5s (because there is only 1 worker running the test, the other is idle.

I don't see any easy way to workaround this issue. I think we should just set the OMP_NUM_THREADS=1 / OPENBLAS_NUM_THREADS=1 / MKL_NUM_THREADS=1 environment variables in the CI configuration.

Thanks for investigating @ogrisel and @tomMoral !

I know, it's not really a big issue, more some evidence about how things could go wrong when using different parallelism mechanisms is seemingly innocuous settings. It's also interested that this happens with OpenMP in gradient boosting but not threading in BLAS.

Closing as "wont fix", with the solution being to use a smaller OMP_NUM_THREADS as indicated above.

The fact that is so catastrophic even on a small number of cores is intriguing though. @jeremiedbb @NicolasHug maybe you have an idea why this is happening more specifically for HistGradientBoostingClassifier/Regressor?

I wonder why we don't have a similarly scaled over-subscription problem with MKL or OpenBLAS thread pools.

I can reproduce everything on my laptop.

from sklearn.experimental import enable_hist_gradient_boosting  # noqa
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.ensemble._hist_gradient_boosting.utils import (
    get_equivalent_estimator)
from sklearn.datasets import load_boston, load_iris


def test_reg():

    X, y = load_boston(return_X_y=True)
    est = HistGradientBoostingRegressor()
    # est = get_equivalent_estimator(est, lib='lightgbm')
    est.fit(X, y)
    est.score(X, y)

def test_classif():

    X, y = load_iris(return_X_y=True)
    est = HistGradientBoostingClassifier(loss='categorical_crossentropy')
    # est = get_equivalent_estimator(est, lib='lightgbm')
    est.fit(X, y)
    est.score(X, y)

As noted by Olivier commenting out one of the tests will make it run fast.

Note that LightGBM estimators also have a big slow-down with -n 2, but much less (about 10s).

I observed a huge variability across oversubscribed runs, typically between 10s and 50s with -n 2 and OMP_NUM_THREADS=4 on my laptop with 2 cores 4 hyperthreads.

I am considering using a pytest.mark to label tests as "serial" and run pytest with and without -n.

Reference: pytest-dev/pytest-xdist#385 (comment)

Edit: For the CI

Regardless of oversubscription, I observed a pretty high variability in the hist estimators too. Typically when running one of the benchmarks (higgs for example), the first few runs are always significantly slower than the subsequent ones :/

It's not the case for the LGBM estimators

I am considering using a pytest.mark to label tests as "serial" and run pytest with and without -n.
Reference: pytest-dev/pytest-xdist#385 (comment)

That's a possibility indeed.

Note that I have just encountered another issue with pytest-xdist in this PR #14264.