MAINT Add Python 3.13 wheels #29789

lesteve · 2024-09-05T15:41:30Z

One of the complications due to the fact that pandas does not have a release with Python 3.13 wheels (in contrary to numpy and scipy). For now I used pandas-dev to test the wheels.

The other complication is due to our custom Windows setup (we build a minimal Windows image to avoid relying on system OpenMP #18802) and run the tests in this image which has a few quirks (for example cibuildwheel does not know about it and install CIBW_TEST_REQUIRES inside the host).

Close #29292.

github-actions · 2024-09-05T15:43:19Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 910aeb9. Link to the linter CI: here}

glemaitre · 2024-09-09T07:20:24Z

build_tools/github/test_windows_wheels.sh

+docker container run \
+    --rm scikit-learn/minimal-windows \
+    powershell -Command "python -m pip install  -c 'import sklearn; sklearn.show_versions()'"


This trigger the following error:

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'import sklearn; sklearn.show_versions()'

Is there an issue with the quotes

Oh no there is a pip isntall in the middle.

build_tools/github/test_windows_wheels.sh

lesteve · 2024-09-09T12:13:32Z

The wheels building on Github Actions was green in commit e139fb7 see build log

lesteve · 2024-09-09T12:58:51Z

The wheels building on Linux arm Python 3.13 worked see build log.

The Python 3.13 Linux arm tests passed in a816413 see build log

glemaitre · 2024-09-16T13:13:00Z

Let me check this PR since that we merged 1.5.2 now and we can have this PR for the main branch now.

glemaitre

LGTM. Thanks @lesteve

thomasjpfan · 2024-09-18T15:11:48Z

.github/workflows/wheels.yml

+            platform_id: manylinux_x86_64
+            manylinux_image: manylinux2014
+            # TODO: remove next line when Python 3.13 is released
+            prerelease_pythons: True


Suggested change

prerelease_pythons: True

With cibuildwheel v2.20.0, the prerelease_python flag is not required anymore: https://cibuildwheel.pypa.io/en/stable/changelog/#v2200

CPython 3.13 wheels are now built by default - without the CIBW_PRERELEASE_PYTHONS flag.

(We'll still need to keep windows because of the -rc logic when building the image)

lesteve · 2024-09-19T13:16:29Z

Probably I should have not done this, but while I worked on this, I found the Dockerfile on Windows a pain to work with, especially for adding custom logic (which I needed to decide to use pandas-dev on Python 3.13) and also an additional layer to pass environment variables to.

This probably doesn't follow Docker best practices but I found constructing the Docker image in a bash script through docker run + docker exec + docker commit a lot more convenient.

My biases are:

I am not super motivated to learn how to do a if then else in Powershell and even less so to maintain the code
I have a Windows VM on my Linux machine but I am not sure I can install Docker which makes it hard to test locally ... last time I tried installing WSL (just for fun for some reason 😅) I ended up reading tweaking my BIOS to try to enable nested virtualization and I never managed to get it to work ...

Oh and the Python 3.13 free-threaded failure is due to #29864

thomasjpfan · 2024-09-19T14:12:34Z

I have a Windows VM on my Linux machine but I am not sure I can install Docker which makes it hard to test locally

I recently tried this and ended up just dual booting Windows.

This probably doesn't follow Docker best practices but I found constructing the Docker image in a bash script through docker run + docker exec + docker commit a lot more convenient.

I am okay with this. The only reason we use the minimal windows version is to make sure the vendored msvcp140.dll and vcomp140.dll works. In the past, I tried to remove the need to vendor dlls, but could not get it to work.

References:

For removing vcomp140.dll: https://cibuildwheel.pypa.io/en/stable/faq/#windows-importerror-dll-load-failed-the-specific-module-could-not-be-found
For removing msvcp140.dll: No 1.21.4 wheel available for cp310-win32 scipy/oldest-supported-numpy#45 (comment)

lesteve · 2024-09-20T14:30:57Z

A bit of a side-topic and not related to this PR, but for removing msvcp140.dll, I don't understand the details but it looks like matplotlib use static linking to avoid having to deal with it: https://github.com/matplotlib/matplotlib/pull/28687/files. Looks like they tried delvewheel and thought the static linking was more robust for some reason see nucleic/kiwi#179 (comment) ...

lesteve · 2024-09-23T08:47:41Z

Two approvals and the last commit with [cd build] is green, merging this one.

QuLogic · 2024-09-24T22:50:07Z

A bit of a side-topic and not related to this PR, but for removing msvcp140.dll, I don't understand the details but it looks like matplotlib use static linking to avoid having to deal with it: https://github.com/matplotlib/matplotlib/pull/28687/files. Looks like they tried delvewheel and thought the static linking was more robust for some reason see nucleic/kiwi#179 (comment) ...

We statically link to avoid crashes with mixed runtimes: matplotlib/matplotlib#28551

lesteve added 2 commits September 5, 2024 17:40

MAINT Add Python 3.13 wheels

b045dc3

[cd build gh]

95c9a24

lesteve marked this pull request as draft September 5, 2024 15:41

lesteve added 8 commits September 5, 2024 17:57

[cd build gh] fix

1016a21

[cd build gh] Install pandas-dev for Python 3.13

c9ef184

[cd build gh] fix for Linux

9f7cf23

Tweak

44fec24

[cd build gh] Attempt fix on Windows with debug

4098832

[cd build gh] tweak indentation

beaaa0d

[cd build gh] attempt to remove pandas from CIBW_TEST_REQUIRES

04c53f2

[cd build gh] fix?

0f0d6bb

glemaitre reviewed Sep 9, 2024

View reviewed changes

wip

650c8c2

ogrisel reviewed Sep 9, 2024

View reviewed changes

build_tools/github/test_windows_wheels.sh Outdated Show resolved Hide resolved

lesteve added 6 commits September 9, 2024 11:45

[cd build gh] Windows attempt

0beded4

[cd build gh] Windows attempt + debug

a6b130d

[cd build gh] Windows attempt + debug

b56d4e3

[cd build gh] debug

4704fe9

[cd build gh] fix

e139fb7

[cd build cirrus] Python 3.13 on Linux arm

8da3dab

lesteve added 2 commits September 9, 2024 17:46

[cd build cirrus] test Python 3.13 wheel

a816413

[cd build] Full wheels

bc1865b

lesteve added the Build / CI label Sep 9, 2024

lesteve marked this pull request as ready for review September 9, 2024 16:27

glemaitre self-requested a review September 16, 2024 13:12

glemaitre approved these changes Sep 17, 2024

View reviewed changes

glemaitre mentioned this pull request Sep 17, 2024

Publish Python 3.13 wheels on PyPI for 1.5.2 #29870

Closed

thomasjpfan reviewed Sep 18, 2024

View reviewed changes

lesteve added 5 commits September 19, 2024 10:11

[cd build gh] Tackle comment + move Dockerfile logic to bash

7f72670

[cd build gh] fix

66e3954

[cd build gh] fix

0b9426a

[cd build gh] fix

e89c736

[cd build gh] fix

2fd1018

[cd build gh] keep PYTHON_VERSION the same as everywhere else

d1ff895

lesteve mentioned this pull request Sep 20, 2024

Attempt to remove msvcp140.dll vendoring on Windows #29898

Closed

thomasjpfan approved these changes Sep 20, 2024

View reviewed changes

[cd build]

910aeb9

lesteve merged commit d4ca146 into scikit-learn:main Sep 23, 2024
55 checks passed

lesteve deleted the python3.13-wheels branch September 23, 2024 08:47

kbharat1210 pushed a commit to kbharat1210/scikit-learn that referenced this pull request Sep 25, 2024

MAINT Add Python 3.13 wheels (scikit-learn#29789)

70cf508

glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Oct 2, 2024

MAINT Add Python 3.13 wheels (scikit-learn#29789)

df72ce8

glemaitre mentioned this pull request Oct 2, 2024

REL scikit-learn 1.5.2 for Python 3.13 #29987

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT Add Python 3.13 wheels #29789

MAINT Add Python 3.13 wheels #29789

lesteve commented Sep 5, 2024 •

edited

Loading

github-actions bot commented Sep 5, 2024 •

edited

Loading

glemaitre Sep 9, 2024

glemaitre Sep 9, 2024

lesteve commented Sep 9, 2024

lesteve commented Sep 9, 2024 •

edited

Loading

glemaitre commented Sep 16, 2024

glemaitre left a comment

thomasjpfan Sep 18, 2024

lesteve commented Sep 19, 2024 •

edited

Loading

thomasjpfan commented Sep 19, 2024 •

edited

Loading

lesteve commented Sep 20, 2024 •

edited

Loading

lesteve commented Sep 23, 2024

QuLogic commented Sep 24, 2024

MAINT Add Python 3.13 wheels #29789

MAINT Add Python 3.13 wheels #29789

Conversation

lesteve commented Sep 5, 2024 • edited Loading

github-actions bot commented Sep 5, 2024 • edited Loading

✔️ Linting Passed

glemaitre Sep 9, 2024

Choose a reason for hiding this comment

glemaitre Sep 9, 2024

Choose a reason for hiding this comment

lesteve commented Sep 9, 2024

lesteve commented Sep 9, 2024 • edited Loading

glemaitre commented Sep 16, 2024

glemaitre left a comment

Choose a reason for hiding this comment

thomasjpfan Sep 18, 2024

Choose a reason for hiding this comment

lesteve commented Sep 19, 2024 • edited Loading

thomasjpfan commented Sep 19, 2024 • edited Loading

lesteve commented Sep 20, 2024 • edited Loading

lesteve commented Sep 23, 2024

QuLogic commented Sep 24, 2024

lesteve commented Sep 5, 2024 •

edited

Loading

github-actions bot commented Sep 5, 2024 •

edited

Loading

lesteve commented Sep 9, 2024 •

edited

Loading

lesteve commented Sep 19, 2024 •

edited

Loading

thomasjpfan commented Sep 19, 2024 •

edited

Loading

lesteve commented Sep 20, 2024 •

edited

Loading