Skip to content

CI Build and test Python 3.12 wheels #27027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Oct 2, 2023
Merged

CI Build and test Python 3.12 wheels #27027

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
314db4f
CI Build and test Python 3.12 wheels
lesteve Aug 7, 2023
d189546
[cd build] [azure parallel]
lesteve Aug 7, 2023
3252a61
[cd build] Tweak Windows docker image for Python 3.12
lesteve Aug 7, 2023
c18cd23
Upgrade pip needed for Python 3.12
lesteve Aug 8, 2023
7a2f3f1
[cd build] Use development numpy and scipy versions only for Python 3…
lesteve Aug 8, 2023
860b8d5
[cd build] Tweak windows script
lesteve Aug 8, 2023
5de909c
[cd build] Tweak
lesteve Aug 8, 2023
893d0b4
[cd build] Temporarily remove pandas install
lesteve Aug 8, 2023
52c6e07
[cd build] Fix
lesteve Aug 8, 2023
e076756
[cd build]
lesteve Aug 8, 2023
ab63ef2
[cd build]
lesteve Aug 8, 2023
3c8912f
[cd build] Tweak
lesteve Aug 8, 2023
2056347
[cd build] temporarily remove lingering pandas
lesteve Aug 8, 2023
5a71325
[cd build] use powershell
lesteve Aug 8, 2023
2b85fa0
[cd build] add ARG to Dockerfile
lesteve Aug 8, 2023
5f00fa0
Revert "[cd build] Temporarily remove pandas install"
lesteve Sep 8, 2023
8b273f6
[cd build] apply suggestions
lesteve Sep 8, 2023
97aae36
[cd build] Windows fix?
lesteve Sep 8, 2023
ebc39f5
Merge branch 'main' of https://github.com/scikit-learn/scikit-learn i…
lesteve Sep 8, 2023
22df811
[cd build]
lesteve Sep 8, 2023
ead0731
Merge branch 'main' of https://github.com/scikit-learn/scikit-learn i…
lesteve Sep 11, 2023
5437089
[cd build]
lesteve Sep 11, 2023
5fb173c
tweak indentation for better readability
lesteve Sep 12, 2023
29207f7
[cd build] cleanup
lesteve Sep 12, 2023
7dd8b2b
Tweak Windows Dockerfile
lesteve Sep 12, 2023
4a71c79
[cd build]
lesteve Sep 12, 2023
f018745
[cd build] fix Windows
lesteve Sep 12, 2023
51055e2
[cd build] no need to use prereleases for numpy and scipy
lesteve Sep 19, 2023
d22eb54
Merge branch 'main' of https://github.com/scikit-learn/scikit-learn i…
lesteve Sep 19, 2023
965318f
[cd build]
lesteve Sep 19, 2023
ad0b5c0
[cd build] trigger CI
lesteve Sep 19, 2023
05a7aaf
[cd build gh] Windows fix
lesteve Sep 19, 2023
58c1ec0
Merge branch 'main' of https://github.com/scikit-learn/scikit-learn i…
lesteve Sep 20, 2023
c7d31bd
[cd build] trigger CI
lesteve Sep 20, 2023
7577932
pandas has a release with Python 3.12 wheel
lesteve Sep 25, 2023
ae225d3
fix
lesteve Sep 25, 2023
5b0e18d
Skip test on Python 3.11 arm
lesteve Sep 25, 2023
1d79d89
[cd build] trigger CI
lesteve Sep 25, 2023
bc92239
Update metadata for Python 3.12
lesteve Sep 28, 2023
dd0d4f8
Update build_tools/cirrus/arm_wheel.yml
lesteve Oct 2, 2023
12078b2
Add comment
lesteve Oct 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion .github/workflows/wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,11 @@ jobs:
- os: windows-latest
python: 311
platform_id: win_amd64
- os: windows-latest
python: 312
platform_id: win_amd64
# TODO: remove when Python 3.12 is released
prerelease: "True"

# Linux 64 bit manylinux2014
- os: ubuntu-latest
Expand All @@ -88,6 +93,12 @@ jobs:
python: 311
platform_id: manylinux_x86_64
manylinux_image: manylinux2014
- os: ubuntu-latest
python: 312
platform_id: manylinux_x86_64
manylinux_image: manylinux2014
# TODO: remove when Python 3.12 is released
prerelease: "True"

# MacOS x86_64
- os: macos-latest
Expand All @@ -104,7 +115,10 @@ jobs:
platform_id: macosx_x86_64

# MacOS arm64
# The latest Python version is built and tested on CirrusCI
# The wheel for the latest Python version is built and tested on
# Cirrus CI but due to limited build time for free accounts on Cirrus
# CI, we build the macOS arm64 wheels for the other Python versions on
# Github Actions via cross-compilation (without running the tests).
- os: macos-latest
python: 38
platform_id: macosx_arm64
Expand All @@ -114,6 +128,9 @@ jobs:
- os: macos-latest
python: 310
platform_id: macosx_arm64
- os: macos-latest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update line 118 above to something like:

          # MacOS arm64
          # The wheel for the latest Python version is built and tested on
          # Cirrus CI but due to limited build time for free accounts on Cirrus
          # CI, we build the macOS arm64 wheels for the other Python versions on
          # Github Actions via cross-compilation (without running the tests).

python: 311
platform_id: macosx_arm64

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should not add 311 here since we already have the following entry in build_tools/cirrus/arm_wheel.yml:

CIBW_BUILD: cp311-macosx_arm64

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but they we would need a pre-release build for 312 on macos_arm64 somewhere, either on Azure or Cirrus.

@thomasjpfan what was the reason for this specific configuration setup for macos_arm64 again?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the latest macos_arm64 build to Cirrus because they limited the number of credits to 50: https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/

macOS time is 5 times more expensive than linux: https://cirrus-ci.org/pricing/#compute-credits

With the current setup, the macOS arm builds are cross compiled on GitHub Action and the latest Python build is actually tested on CirrusCI.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add this information into a comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, please add a comment on each side: in the Azure Pipelines config and in the Cirrus-CI config.

steps:
- name: Checkout scikit-learn
Expand All @@ -128,6 +145,7 @@ jobs:
env:
CONFTEST_PATH: ${{ github.workspace }}/conftest.py
CONFTEST_NAME: conftest.py
CIBW_PRERELEASE_PYTHONS: ${{ matrix.prerelease }}
CIBW_ENVIRONMENT: SKLEARN_SKIP_NETWORK_TESTS=1
SKLEARN_BUILD_PARALLEL=3
CIBW_BUILD: cp${{ matrix.python }}-${{ matrix.platform_id }}
Expand Down
18 changes: 15 additions & 3 deletions build_tools/cirrus/arm_wheel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,17 @@ macos_arm64_wheel_task:
# See `maint_tools/update_tracking_issue.py` for details on the permissions the token requires.
BOT_GITHUB_TOKEN: ENCRYPTED[9b50205e2693f9e4ce9a3f0fcb897a259289062fda2f5a3b8aaa6c56d839e0854a15872f894a70fca337dd4787274e0f]
matrix:
# Only the latest Python version is built and tested on CirrusCI, the other
# macos arm64 builds are on GitHub Actions
# Only the latest Python version is built and tested on Cirrus CI, the other
# macOS arm64 builds are on GitHub Actions. The reason is that macOS time is
# 5x more expensive than Linux times on Cirrus CI and the credits are limited
# (for free accounts).
# Note that the macOS arm64 builds are cross compiled on GitHub Actions (without
# running the tests) and while the macOS arm64 build for the latest Python version
# is actually tested on Cirrus CI.
- env:
CIBW_BUILD: cp311-macosx_arm64
CIBW_BUILD: cp312-macosx_arm64
# TODO: remove when Python 3.12 is released
CIBW_PRERELEASE_PYTHONS: True
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can now be removed, since cibuildwheel v2.15.0 CPython 3.12 wheels are built by default.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good to know!

I'd rather wait for Python 3.12 to be released before removing all of them because we rely on CI_BW_PRERELEASE_PYTHONS on Windows for building the custom Docker image.


conda_script:
- curl -L --retry 10 -o ~/mambaforge.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
Expand Down Expand Up @@ -68,6 +75,11 @@ linux_arm64_wheel_task:
CIBW_TEST_SKIP: "*_aarch64"
- env:
CIBW_BUILD: cp311-manylinux_aarch64
CIBW_TEST_SKIP: "*_aarch64"
- env:
CIBW_BUILD: cp312-manylinux_aarch64
# TODO: remove when Python 3.12 is released
CIBW_PRERELEASE_PYTHONS: True

cibuildwheel_script:
- apt install -y python3 python-is-python3
Expand Down
3 changes: 3 additions & 0 deletions build_tools/github/build_minimal_windows_image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ cp $WHEEL_PATH $WHEEL_NAME
# Dot the Python version for identyfing the base Docker image
PYTHON_VERSION=$(echo ${PYTHON_VERSION:0:1}.${PYTHON_VERSION:1:2})

if [[ "$CIBW_PRERELEASE_PYTHONS" == "True" ]]; then
PYTHON_VERSION="$PYTHON_VERSION-rc"
fi
# Build a minimal Windows Docker image for testing the wheels
docker build --build-arg PYTHON_VERSION=$PYTHON_VERSION \
--build-arg WHEEL_NAME=$WHEEL_NAME \
Expand Down
1 change: 1 addition & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -593,6 +593,7 @@ def setup_package():
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
],
Expand Down