CI Build macOS arm wheels [cd build gh] #21827

thomasjpfan · 2021-11-30T03:36:48Z

Reference Issues/PRs

Fixes #19137

What does this implement/fix? Explain your changes.

This PR:

Adds builds m1 arm wheels for Python 3.8, 3.9 and 3.10
Rewrites the build matrix to list out all the entries. With the m1 wheels with platform_id: [macosx_x86_64, macosx_arm64 ] the build matrix looked a little too complicated using exclude + a product of configurations. I think listing out all the configurations is easier to reason about.
There is no testing on the CI since we are using cross compilation, thus one of us needs to test locally.

Any other comments?

I tested the Python 3.9 build on my local m1 machine and the test pass. I'll check again with the wheels built in this PR.

grlee77 · 2021-11-30T16:26:27Z

build_tools/github/build_wheels.sh

+    # https://packages.macports.org/libomp/.
+    if [[ "$CIBW_BUILD" == *-macosx_arm64 ]]; then
+        # arm64 builds must cross compile because CI is on x64
+        export PYTHON_CROSSENV=1
+        # SciPy requires 12.0 on arm to prevent kernel panics
+        # https://github.com/scipy/scipy/issues/14688
+        # We use the same deployment target to match SciPy.
+        export MACOSX_DEPLOYMENT_TARGET=12.0
+        wget https://packages.macports.org/libomp/libomp-11.0.1_0.darwin_20.arm64.tbz2 -O libomp.tbz2
+    else
+        # Currently, the oldest supported macos version is: High Sierra / 10.13.
+        # When upgrading this, be sure to update the MACOSX_DEPLOYMENT_TARGET
+        # environment variable in wheels.yml accordingly. Note that Darwin_17 ==
+        # High Sierra / 10.13.
+        export MACOSX_DEPLOYMENT_TARGET=10.13
+        wget https://packages.macports.org/libomp/libomp-11.0.1_0+universal.darwin_17.i386-x86_64.tbz2 -O libomp.tbz2


Just was working on this for scikit-image as well and noticed there were no universal2 libomp wheels on macports. It was suggested in the napari zulip that we could potentially use lipo to make a universal2 libomp from the separate x86_64 and arm64 ones, but perhaps it is not worth the effort at this point?

I do not think it's worth the effort for now. I plan to revisit if an issue opens up and there is demand for a universal2 wheel.

also, I'm not sure about needing to bump to 12.0 for the deployment target. I had left this at 11.0 on scikit-image and the preliminary wheels we built pass the test suite when they were tested (although the user testing was on MacOS 12). Did you encounter failures for scikit-learn with 11.0?

I guess given that we have SciPy as a required dependency and it only provides 12.0 wheels we should perhaps do the same.

I guess given that we have SciPy as a required dependency and it only provides 12.0 wheels we should perhaps do the same.

Yes, I don't see how it would be possible to install scikit-learn or scikit-image wheels on macos 11 if scipy only provides wheels for macos 12+ anyways. Better be consistent in the stack.

thomasjpfan · 2021-11-30T20:47:58Z

I tested the wheels from this PR on Python 3.8, 3.9 and 3.10 with a macOS arm64 M1 machine and the test suite passes.

build_tools/github/build_wheels.sh

ogrisel

Let's wait for the current build to complete, test again locally to check that the conflict resolution with main did not introduce any unintended side effects (unlikely) and then merge.

ogrisel · 2021-12-02T09:59:33Z

~~Depending on which PR is merged first between this and #21232, we will need to add an entry for Python 3.10 on macos/arm64.~~

Scratch that, 310 is already configured here. There is no dependency between the PRs.

ogrisel · 2021-12-02T11:31:07Z

The unrelated Azure Pipelines failures on Windows has already been reported here: #21798.

ogrisel · 2021-12-02T13:53:20Z

Running the tests again locally from the latest artifacts (only Python 3.10). Will merge if green.

ogrisel · 2021-12-02T15:57:19Z

All tests pass.

I have the feeling that when running with pytest-xdist (-n 8) on my Apple M1 I can get significantly more over-subscription than when using my pure conda-forge env on the same machine.

But this can be improved in the future. All tests are green, merging.

thomasjpfan · 2021-12-03T02:52:43Z

I have the feeling that when running with pytest-xdist (-n 8) on my Apple M1 I can get significantly more over-subscription than when using my pure conda-forge env on the same machine.

I needed to set OMP_NUM_THREADS=1 manually to get similar test times.

himarange · 2021-12-16T13:52:12Z

So how do we install scikit-learn on M1 chips exactly?

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Olivier Grisel <olivier.grisel@gmail.com>

CI Build macOS arm wheels [cd build gh]

b7b343e

github-actions bot added the Build / CI label Nov 30, 2021

CI Move env variables into build_wheel.sh [cd build gh]

e449a57

grlee77 reviewed Nov 30, 2021

View reviewed changes

grlee77 mentioned this pull request Nov 30, 2021

add MacOS arm64 wheels scikit-image/scikit-image#6068

Merged

thomasjpfan added this to the 1.0.2 milestone Dec 1, 2021

thomasjpfan mentioned this pull request Dec 1, 2021

CI Explicitly list out wheel builds in config [cd build gh] #21849

Merged

Merge branch 'main' into build_m1_wheels

eb68637

ogrisel reviewed Dec 2, 2021

View reviewed changes

build_tools/github/build_wheels.sh Outdated Show resolved Hide resolved

ogrisel and others added 2 commits December 2, 2021 10:40

[cd build gh]

7c18ffa

Update comment [cd build gh]

cc0938b

ogrisel approved these changes Dec 2, 2021

View reviewed changes

ogrisel merged commit 594b1f7 into scikit-learn:main Dec 2, 2021

glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Dec 24, 2021

CI Build macOS arm wheels [cd build gh] (scikit-learn#21827)

8148b58

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Olivier Grisel <olivier.grisel@gmail.com>

glemaitre pushed a commit that referenced this pull request Dec 25, 2021

CI Build macOS arm wheels [cd build gh] (#21827)

d7ac054

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Olivier Grisel <olivier.grisel@gmail.com>

paxtonfitzpatrick mentioned this pull request Feb 9, 2022

arm64/Apple Silicon compatibility ContextLab/hypertools#253

Closed

2 tasks

SamEdwardes mentioned this pull request May 17, 2022

poetry add scikit-learn does not work on m1 mac python-poetry/poetry#5628

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Build macOS arm wheels [cd build gh] #21827

CI Build macOS arm wheels [cd build gh] #21827

thomasjpfan commented Nov 30, 2021 •

edited

Loading

grlee77 Nov 30, 2021

thomasjpfan Nov 30, 2021

grlee77 Nov 30, 2021 •

edited

Loading

grlee77 Nov 30, 2021

ogrisel Dec 2, 2021

thomasjpfan commented Nov 30, 2021

ogrisel left a comment

ogrisel commented Dec 2, 2021 •

edited

Loading

ogrisel commented Dec 2, 2021

ogrisel commented Dec 2, 2021 •

edited

Loading

ogrisel commented Dec 2, 2021

thomasjpfan commented Dec 3, 2021 •

edited

Loading

himarange commented Dec 16, 2021

CI Build macOS arm wheels [cd build gh] #21827

CI Build macOS arm wheels [cd build gh] #21827

Conversation

thomasjpfan commented Nov 30, 2021 • edited Loading

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

grlee77 Nov 30, 2021

Choose a reason for hiding this comment

thomasjpfan Nov 30, 2021

Choose a reason for hiding this comment

grlee77 Nov 30, 2021 • edited Loading

Choose a reason for hiding this comment

grlee77 Nov 30, 2021

Choose a reason for hiding this comment

ogrisel Dec 2, 2021

Choose a reason for hiding this comment

thomasjpfan commented Nov 30, 2021

ogrisel left a comment

Choose a reason for hiding this comment

ogrisel commented Dec 2, 2021 • edited Loading

ogrisel commented Dec 2, 2021

ogrisel commented Dec 2, 2021 • edited Loading

ogrisel commented Dec 2, 2021

thomasjpfan commented Dec 3, 2021 • edited Loading

himarange commented Dec 16, 2021

thomasjpfan commented Nov 30, 2021 •

edited

Loading

grlee77 Nov 30, 2021 •

edited

Loading

ogrisel commented Dec 2, 2021 •

edited

Loading

ogrisel commented Dec 2, 2021 •

edited

Loading

thomasjpfan commented Dec 3, 2021 •

edited

Loading