Release 1.4.2 #28774

jeremiedbb · 2024-04-05T12:23:17Z

Extra check list

need to wait for joblib 1.4.0 release that has some fixes for numpy 2 support that 1.3.2 doesn't have.
move change log entries from 1.4.2 into 1.5
make sure we build the wheels using numpy 2 !

Only cherry-picked 87c32d3 and 01cd8d4 from the main branch because they are necessary for numpy 2 support.

Co-authored-by: Loïc Estève <loic.esteve@ymail.com>

…scikit-learn#28583)

github-actions · 2024-04-05T12:24:36Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: b5b65f7. Link to the linter CI: here}

jeremiedbb · 2024-04-05T15:41:20Z

in d88930f I'm forcing cibuildwheels to build scikit-learn against numpy 2. I checked the jobs and it's the case. cibuildwheels still tests the wheels against numpy < 2, which is a good thing because it confirms that we still support older versions of numpy even when we build against numpy 2.

For numpy 2 at test time, we have the scipy-dev nightly runs doing that for a while, and in addition I'll test the wheels locally on a couple of platforms before uploading them.

thomasjpfan · 2024-04-05T20:41:25Z

For numpy 2 at test time, we have the scipy-dev nightly runs doing that for a while, and in addition I'll test the wheels locally on a couple of platforms before uploading them.

In the wheel builder, can we have some testing with NumPy 2.0? From reading the GitHub actions docs, I think this works for installing NumPy 2.0 for the Python 3.12 tests:

CIBW_TEST_REQUIRES: pytest pandas ${{ matrix.python == 312 && 'numpy==2.0.0rc1' || '' }}

ogrisel · 2024-04-08T07:45:27Z

For numpy 2 at test time, we have the scipy-dev nightly runs doing that for a while, and in addition I'll test the wheels locally on a couple of platforms before uploading them.

I think this works for installing NumPy 2.0 for the Python 3.12 tests:
CIBW_TEST_REQUIRES: pytest pandas ${{ matrix.python == 312 && 'numpy==2.0.0rc1' || '' }}

I am fine with either option. If we go for the latter, I would like to add an inline comment to explain that this is temporary and can be deleted when building the wheels for future scikit-learn releases.

jeremiedbb · 2024-04-08T08:55:19Z

Thanks Thomas, let me try that.

I would like to add an inline comment to explain that this is temporary and can be deleted when building the wheels for future scikit-learn releases.

I added a comment, but this is very likely the last 1.4.X release anyway :)

jeremiedbb · 2024-04-08T09:17:23Z

The py312 builds are now failing. This is expected because we need joblib 1.4, which should be released soon.

thomasjpfan

Overall looks good.

I hope the ~ 3 hours Windows + Python 3.12 build is a one-off issue: https://github.com/scikit-learn/scikit-learn/actions/runs/8603767124/job/23576402935?pr=28774

jeremiedbb · 2024-04-08T20:37:57Z

It wasn't. I forgot to backport #28692 although I had labelled it :)
It's needed because the latest scipy release, the one with numpy 2 support, is built against the problematic version of openblas.

jeremiedbb · 2024-04-08T21:20:34Z

Hum, the build still hangs...

jeremiedbb · 2024-04-09T08:42:35Z

I can reproduce locally. Windows amd cpu. The test sklearn/metrics/tests/test_pairwise.py::test_parallel_pairwise_distances_diagonal[float64-euclidean] hangs

Reproducer:

import numpy as np
from sklearn.metrics import pairwise_distances

X = np.random.normal(size=(1000, 10), scale=1e10)

pairwise_distances(X)  # works
pairwise_distances(X, n_jobs=2)  # hangs

Surprisingly I can only reproduce with python 3.12. With 3.11 it works. Below are the outputs of show_versions for 3.11 and 3.12. Nothing is different but the python version.

# Python 3.11

Python dependencies:
      sklearn: 1.4.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 2.0.0rc1
        scipy: 1.13.0
       Cython: 3.0.10
       pandas: None
   matplotlib: None
       joblib: 1.4.0
threadpoolctl: 3.4.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\J\miniconda3\envs\tst\Lib\site-packages\scipy.libs\libopenblas_v0.3.26-382-gb1e8ba50--72a863714eca5a50b38260dedc0c2f3a.dll
        version: 0.3.26.dev
threading_layer: pthreads
   architecture: Zen

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\J\miniconda3\envs\tst\vcomp140.dll
        version: None

# Python 3.12

Python dependencies:
      sklearn: 1.4.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 2.0.0rc1
        scipy: 1.13.0
       Cython: 3.0.10
       pandas: None
   matplotlib: None
       joblib: 1.4.0
threadpoolctl: 3.4.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\J\miniconda3\envs\tst\Lib\site-packages\scipy.libs\libopenblas_v0.3.26-382-gb1e8ba50--72a863714eca5a50b38260dedc0c2f3a.dll
        version: 0.3.26.dev
threading_layer: pthreads
   architecture: Zen

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\J\miniconda3\envs\tst\vcomp140.dll
        version: None

lesteve · 2024-04-09T09:45:09Z

I am a bit puzzled since the wheel on main passes and uses scipy 1.13.0 as well:
https://github.com/scikit-learn/scikit-learn/actions/runs/8610015698/job/23594862032

Also #28692 had no failures in our test but was triggered by a MNE-Python report. Scipy 1.13 actually has a patch for the original bug hence the 0.3.26.dev version in the sklearn.show_versions() output.

All in all I think this is a slightly different bug maybe?

One slightly weird thing I see in your show_versions() output is that there is no BLAS info for the numpy 2.0.0rc1 wheel, this could well be a red herring though ...

Looking a bit at the Python 3.11 build log vs Python 3.12 build log, the Python 3.11 tests against numpy 1.26.4 for some reason. This one may also be another red herring.

jeremiedbb · 2024-04-09T09:48:51Z

Also #28692 had no failures in our test but was triggered by a MNE-Python report. Scipy 1.13 actually has a patch for the original bug hence the 0.3.26.dev version in the sklearn.show_versions() output.

It's not related to #28692. The reproducer has nothing to do with pairwise_distances_reduction. The issue seems to come from a nesting of python threads and blas calls (gemm)

jeremiedbb · 2024-04-09T10:27:10Z

I can reproduce without numpy 2 as well. And regardless of the Python version. Hangs with openblas 0.3.26 and doesn't with openblas 0.3.27. So it really seem to come from the same openblas issue as in #28692, just not at the same place.

Python dependencies:
      sklearn: 1.4.2
          pip: 24.0
   setuptools: 69.2.0
        numpy: 1.26.4
        scipy: 1.13.0
       Cython: 3.0.10
       pandas: None
   matplotlib: None
       joblib: 1.4.0
threadpoolctl: 3.4.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libomp
       filepath: C:\Users\J\miniconda3\envs\tst\Library\bin\libomp.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libblas
       filepath: C:\Users\J\miniconda3\envs\tst\Library\bin\libblas.dll
        version: 0.3.26
threading_layer: pthreads
   architecture: Zen

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\J\miniconda3\envs\tst\vcomp140.dll
        version: None

Python dependencies:
      sklearn: 1.4.2
          pip: 24.0
   setuptools: 69.2.0
        numpy: 1.26.4
        scipy: 1.13.0
       Cython: 3.0.10
       pandas: None
   matplotlib: None
       joblib: 1.4.0
threadpoolctl: 3.4.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libomp
       filepath: C:\Users\J\miniconda3\envs\tst\Library\bin\libomp.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libblas
       filepath: C:\Users\J\miniconda3\envs\tst\Library\bin\libblas.dll
        version: 0.3.27
threading_layer: pthreads
   architecture: Zen

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\J\miniconda3\envs\tst\vcomp140.dll
        version: None

ogrisel · 2024-04-09T11:51:51Z

Our Windows nightly build do not fail:

https://github.com/scikit-learn/scikit-learn/actions/runs/8610015698/job/23594862032#step:6:2048

but the version of openblas from numpy (v0.3.23-293-gc2f4bdbb, did not have the thread-safety bug yet) is different from the one shipped with scipy (v0.3.26-382-gb1e8ba50, should have the fix for the thread-safety bug).

EDIT: as soon as numpy 2 is released, we might experience the bug in main (possibly after a lockfile update) if numpy 2 final still ships a bad version of OpenBLAS.

ogrisel · 2024-04-09T12:50:57Z

I opened an issue upstream to ask numpy maintainers to ship the openblas fix in NumPy 2 numpy/numpy#26240

jeremiedbb · 2024-04-09T13:33:09Z

We had a meeting to discuss how to proceed here. Here are the conclusions:

The windows deadlock comes from the same OpenBLAS issue present in version 0.3.26.
- scipy 1.13 uses a more recent version of OpenBLAS (0.3.26.dev) so doesn't have the deadlock. Hence no need to backport FIX Avoid dead-lock with OpenBLAS 0.3.26 on Windows inside pairwise distances calculation #28692, keeping the release about numpy 2 support as initially planned.
- numpy 2.0.0rc1 comes with OpenBLAS 0.3.26 so has the deadlock. We gave feedback to numpy so that they build numpy 2 final against 0.3.27.
Some of the previous builds in this PR were testing the wheels against numpy 2 successfully (except from the windows one due to the deadlock). I've removed that and the last builds test the wheels against numpy latest (1.26.4). I did that to have all jobs green and be able to upload the wheels through the gh actions.

thomasjpfan · 2024-04-09T14:06:22Z

@jeremiedbb Thank you for pushing this through! 🎉

jeremiedbb · 2024-04-09T14:08:39Z

oh crap, I squashed

jeremiedbb · 2024-04-09T14:09:30Z

no big deal since we very likely won't have a 1.4.3 but still 😡

rgommers · 2024-04-11T07:39:42Z

Thanks all!

Re the openblas vendored in numpy 2.0.0rc1: we'll do a 2.0.0rc2 with openblas 0.3.27 included to fix this, probably next week.

jeremiedbb and others added 4 commits April 5, 2024 14:04

remove numpy<2 constraint

7c3da54

bump version

36908b6

TST remove sparse-np.float16 support in tests (scikit-learn#28635)

87c32d3

Co-authored-by: Loïc Estève <loic.esteve@ymail.com>

CI fix scipy-dev by using asarray() instead of array(..., copy=False) (…

01cd8d4

…scikit-learn#28583)

github-actions bot added module:preprocessing module:utils labels Apr 5, 2024

jeremiedbb added the No Changelog Needed label Apr 5, 2024

thomasjpfan reviewed Apr 8, 2024

View reviewed changes

jeremiedbb and others added 3 commits April 9, 2024 15:09

pin numpy==2.0.0rc1 to force cibuildwheels to build against numpy 2

c329eb3

MAINT Set 1.4.2 release month (scikit-learn#28790)

c9a7324

update news [cd build]

b5b65f7

jeremiedbb force-pushed the release-1.4.2 branch from 45a38f5 to b5b65f7 Compare April 9, 2024 13:10

jeremiedbb enabled auto-merge (squash) April 9, 2024 13:58

jeremiedbb merged commit d675b8d into scikit-learn:1.4.X Apr 9, 2024
44 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 1.4.2 #28774

Release 1.4.2 #28774

jeremiedbb commented Apr 5, 2024 •

edited

Loading

github-actions bot commented Apr 5, 2024 •

edited

Loading

jeremiedbb commented Apr 5, 2024

thomasjpfan commented Apr 5, 2024

ogrisel commented Apr 8, 2024 •

edited

Loading

jeremiedbb commented Apr 8, 2024

jeremiedbb commented Apr 8, 2024

thomasjpfan left a comment •

edited

Loading

jeremiedbb commented Apr 8, 2024

jeremiedbb commented Apr 8, 2024

jeremiedbb commented Apr 9, 2024 •

edited

Loading

lesteve commented Apr 9, 2024 •

edited

Loading

jeremiedbb commented Apr 9, 2024

jeremiedbb commented Apr 9, 2024 •

edited

Loading

ogrisel commented Apr 9, 2024 •

edited

Loading

ogrisel commented Apr 9, 2024

jeremiedbb commented Apr 9, 2024

thomasjpfan commented Apr 9, 2024

jeremiedbb commented Apr 9, 2024

jeremiedbb commented Apr 9, 2024

rgommers commented Apr 11, 2024

Release 1.4.2 #28774

Release 1.4.2 #28774

Conversation

jeremiedbb commented Apr 5, 2024 • edited Loading

github-actions bot commented Apr 5, 2024 • edited Loading

✔️ Linting Passed

jeremiedbb commented Apr 5, 2024

thomasjpfan commented Apr 5, 2024

ogrisel commented Apr 8, 2024 • edited Loading

jeremiedbb commented Apr 8, 2024

jeremiedbb commented Apr 8, 2024

thomasjpfan left a comment • edited Loading

Choose a reason for hiding this comment

jeremiedbb commented Apr 8, 2024

jeremiedbb commented Apr 8, 2024

jeremiedbb commented Apr 9, 2024 • edited Loading

lesteve commented Apr 9, 2024 • edited Loading

jeremiedbb commented Apr 9, 2024

jeremiedbb commented Apr 9, 2024 • edited Loading

ogrisel commented Apr 9, 2024 • edited Loading

ogrisel commented Apr 9, 2024

jeremiedbb commented Apr 9, 2024

thomasjpfan commented Apr 9, 2024

jeremiedbb commented Apr 9, 2024

jeremiedbb commented Apr 9, 2024

rgommers commented Apr 11, 2024

jeremiedbb commented Apr 5, 2024 •

edited

Loading

github-actions bot commented Apr 5, 2024 •

edited

Loading

ogrisel commented Apr 8, 2024 •

edited

Loading

thomasjpfan left a comment •

edited

Loading

jeremiedbb commented Apr 9, 2024 •

edited

Loading

lesteve commented Apr 9, 2024 •

edited

Loading

jeremiedbb commented Apr 9, 2024 •

edited

Loading

ogrisel commented Apr 9, 2024 •

edited

Loading