Skip to content

Conversation

ogrisel
Copy link
Member

@ogrisel ogrisel commented Dec 23, 2020

@ogrisel
Copy link
Member Author

ogrisel commented Dec 23, 2020

I intentionally canceled the ARM64 builds on travis to spare credits as they are unrelated to this PR.

@ogrisel
Copy link
Member Author

ogrisel commented Dec 23, 2020

Hum, the macos builds are still pending...

And the following issue actions/runner-images#1256 seems to indicate that macos-10.15 is the oldest supported platform.

Maybe we could still find a version of the libomp.dylib file that was built to be compatible with 10.13 (or even older) somewhere?

@jeremiedbb
Copy link
Member

jeremiedbb commented Dec 23, 2020

Alternatively, this SO question https://stackoverflow.com/questions/6591988/best-practice-for-osx-10-x-backwards-compatability seems relevant but I can't find the documentation

@ogrisel
Copy link
Member Author

ogrisel commented Dec 23, 2020

The homebrew doc states that 10.9-10.13 is supported on a best effort basis:

https://docs.brew.sh/Installation#2

@ogrisel
Copy link
Member Author

ogrisel commented Dec 23, 2020

cibuildwheel should automatically set MACOSX_DEPLOYMENT_TARGET to 10.9 according to https://github.com/joerick/cibuildwheel/blob/master/README.md .

@thomasjpfan
Copy link
Member

thomasjpfan commented Dec 23, 2020

I wonder if we can try to get the bottle built for high_sierra (10.13) and use that in build_tools/github/build_wheels.sh:

wget https://homebrew.bintray.com/bottles/libomp-11.0.0.high_sierra.bottle.tar.gz
brew install libomp-11.0.0.high_sierra.bottle.tar.gz

while keeping gihtub ci at macos-latest.

@ogrisel
Copy link
Member Author

ogrisel commented Dec 23, 2020

Maybe we should install libomp using conda-forge instead. It's weird because in the past (on the https://github.com/MacPython/scikit-learn-wheels repo) we used homebrew to install libomp and we did not get this issue.

The version of the macos VM on Azure Pipelines is 10.14:

https://github.com/MacPython/scikit-learn-wheels/blob/master/azure-pipelines.yml#L81

and the script set the deployment target to 10.9 explicitly:

https://github.com/MacPython/scikit-learn-wheels/blob/master/azure/posix.yml#L52

libomp was installed from homebrew as well:

https://github.com/MacPython/scikit-learn-wheels/blob/master/extra_functions.sh

@ogrisel
Copy link
Member Author

ogrisel commented Dec 23, 2020

@thomasjpfan your trick seems to work. At least the build completes successfully, and I can install the resulting wheel and run the test on macos 11.1. I don't have an old macos handy unfortunately.

@joshuacwnewton
Copy link
Contributor

joshuacwnewton commented Dec 23, 2020

Repeating #19063 (comment) here for visibility.

I tested the fix (via this provided link) on Travis's osx_image: xcode9.4 (macOS 10.13) build environment. The error we were experiencing has gone away.

@ogrisel ogrisel changed the title Try to use macos-10.13 when building the wheels with [cd build] Use high sierra (macos-10.13) compatible libomp when building the wheels with [cd build] Dec 27, 2020
@ogrisel ogrisel added the To backport PR merged in master that need a backport to a release branch defined based on the milestone. label Jan 6, 2021
@ogrisel ogrisel added this to the 0.24.1 milestone Jan 6, 2021
@ogrisel ogrisel added the Blocker label Jan 6, 2021
@ogrisel
Copy link
Member Author

ogrisel commented Jan 6, 2021

@thomasjpfan @alfaro96 @jeremiedbb ok for merge on your side?

@ogrisel
Copy link
Member Author

ogrisel commented Jan 6, 2021

Actually since high sierra is 10.13, maybe we should set the MACOSX_DEPLOYMENT_TARGET to "10.13" for the [cd build] environment and make sure that the generated wheels have 10_13 instead of 10_9 in their name + metadata.

I don't think homebrew makes it possible to get 10.9 compatible version of libomp.

Copy link
Member

@alfaro96 alfaro96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @ogrisel!

LGTM (after @jeremiedbb suggestion is applied).

Co-authored-by: Jérémie du Boisberranger <34657725+jeremiedbb@users.noreply.github.com>
@ogrisel
Copy link
Member Author

ogrisel commented Jan 6, 2021

@thomasjpfan how did you find the URL of the high sierra (10.13) bottle for libomp? I am wondering if we can find a mavericks (10.9) version of the same package.

@alfaro96
Copy link
Member

alfaro96 commented Jan 6, 2021

Actually since high sierra is 10.13, maybe we should set the MACOSX_DEPLOYMENT_TARGET to "10.13" for the [cd build] environment and make sure that the generated wheels have 10_13 instead of 10_9 in their name + metadata.

I don't think homebrew makes it possible to get 10.9 compatible version of libomp.

I have checked the official libomp formulae and the oldest version is high_sierra. Therefore, maybe it is worth it to set the MACOSX_DEPLOYMENT_TARGET to "10.13".

@alfaro96
Copy link
Member

alfaro96 commented Jan 6, 2021

@thomasjpfan how did you find the URL of the high sierra (10.13) bottle for libomp? I am wondering if we can find a mavericks (10.9) version of the same package.

The URLs are available here: https://formulae.brew.sh/api/formula/libomp.json.

@ogrisel
Copy link
Member Author

ogrisel commented Jan 6, 2021

Ok thanks, let me try to set the deployment target to 10.13 then.

@ogrisel
Copy link
Member Author

ogrisel commented Jan 6, 2021

Related: matthew-brett/delocate#56

@ogrisel
Copy link
Member Author

ogrisel commented Jan 6, 2021

From the logs of the CPython 3.8 macos build:

Repairing wheel...
  
  + delocate-listdeps /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheelcoccs2t7/built_wheel/scikit_learn-1.0.dev0-cp38-cp38-macosx_10_13_x86_64.whl && delocate-wheel --require-archs x86_64 -w /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheelcoccs2t7/repaired_wheel /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheelcoccs2t7/built_wheel/scikit_learn-1.0.dev0-cp38-cp38-macosx_10_13_x86_64.whl
  /usr/local/Cellar/libomp/11.0.0/lib/libomp.dylib

and later:

pip install /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheelcoccs2t7/repaired_wheel/scikit_learn-1.0.dev0-cp38-cp38-macosx_10_13_x86_64.whl
  Processing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheelcoccs2t7/repaired_wheel/scikit_learn-1.0.dev0-cp38-cp38-macosx_10_13_x86_64.whl

So it looks like it's working as expected: distutils is taking the MACOSX_DEPLOYMENT_TARGET environment variable into account to set the wheel metadata.

@ogrisel
Copy link
Member Author

ogrisel commented Jan 6, 2021

I still do not understand why we did not face the problem when releasing 0.23.2. Maybe the default homebrew bottle for libomp was still compatible with 10.9 at the time.

@thomasjpfan
Copy link
Member

I tested the artifacts from the bbfed0c build on travis osx 10.13 here: thomasjpfan/sklearn_osx_travis_test#1

@ogrisel
Copy link
Member Author

ogrisel commented Jan 11, 2021

Thanks for testing @thomasjpfan. Shall we merge?

@thomasjpfan thomasjpfan changed the title Use high sierra (macos-10.13) compatible libomp when building the wheels with [cd build] CI Use macos-10.13 compatible libomp when building the wheels Jan 11, 2021
@thomasjpfan thomasjpfan merged commit fe52d1e into scikit-learn:master Jan 11, 2021
@ogrisel ogrisel deleted the macos-10.13-wheels branch January 12, 2021 14:33
glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Jan 18, 2021
…-learn#19064)

Co-authored-by: Jérémie du Boisberranger <34657725+jeremiedbb@users.noreply.github.com>
jeremiedbb added a commit that referenced this pull request Jan 19, 2021
Co-authored-by: Jérémie du Boisberranger <34657725+jeremiedbb@users.noreply.github.com>
grlee77 added a commit to grlee77/scikit-image that referenced this pull request Aug 19, 2021
This fix was adapted by @hmaarrfk from scikit-learn/scikit-learn#19064

Co-authored-by: Mark Harfouche <mark.harfouche@gmail.com>
hmaarrfk added a commit to scikit-image/scikit-image that referenced this pull request Sep 1, 2021
This fix was adapted by @hmaarrfk from scikit-learn/scikit-learn#19064

Co-authored-by: Mark Harfouche <mark.harfouche@gmail.com>

Co-authored-by: Mark Harfouche <mark.harfouche@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocker To backport PR merged in master that need a backport to a release branch defined based on the milestone.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants