Skip to content

Wheel support for aarch64 #17800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
odidev opened this issue Jul 1, 2020 · 10 comments
Closed

Wheel support for aarch64 #17800

odidev opened this issue Jul 1, 2020 · 10 comments
Labels
New Feature

Comments

@odidev
Copy link

odidev commented Jul 1, 2020

Summary
Installing scikit-learn on aarch64 via pip using command "pip3 install scikit-learn" tries to build wheel from source code

Problem description
scikit-learn don't have wheel for aarch64 on PyPI repository. So, while installing scikit-learn via pip on aarch64, pip builds wheel for same resulting in it takes more time to install scikit-learn. Making wheel available for aarch64 will benefit aarch64 users by minimizing scikit-learn installation time.

Expected Output
Pip should be able to download scikit-learn wheel from PyPI repository rather than building it from source code.

@scikit-learn-team, please let me know if I can help you building wheel/uploading to PyPI repository. I am curious to make scikit-learn wheel available for aarch64. It will be a great opportunity for me to work with you.
@ogrisel
Copy link
Member

ogrisel commented Jul 1, 2020

In the mean time you should be able to use the conda-forge distribution using the miniforge installer:

https://github.com/conda-forge/miniforge/#download

The scikit-learn conda-forge package has been built on the aarch64 platform:

https://anaconda.org/conda-forge/scikit-learn/

Although I have never tried it personally.

For wheels, it should be possible to adapt our wheel building configuration to also run on https://drone.io/.

@ogrisel
Copy link
Member

ogrisel commented Jul 1, 2020

Closing as a duplicate of #13073.

@ogrisel ogrisel closed this as completed Jul 1, 2020
@rth
Copy link
Member

rth commented Jul 1, 2020

Wheels for aarch64 would be great, assuming it doesn't require much maintenance effort on our part. However if there are to be put on PyPi it would make sense to start from dependencies (numpy, scipy). As long as those have to be built from source when running pip install scikit-learn having wheels for scikit-learn won't help compilation time much.

@fkromer
Copy link

fkromer commented Jul 16, 2020 •
edited
Loading

When building a dockerized Python3.7 app with docker buildx for a Raspberry Pi builing scikit-learn feels like taking forever (almost 2 hours) on a machine with Intel® Core™ i7-8550U CPU @ 1.80GHz × 8, 15,4 GiB with a quite fast SSD and finally failing:

#9 403.2 Collecting scikit-learn==0.23.1
#9 403.2   Downloading scikit-learn-0.23.1.tar.gz (7.2 MB)
#9 408.7   Installing build dependencies: started
#9 544.1   Installing build dependencies: still running...
#9 604.2   Installing build dependencies: still running...
#9 731.0   Installing build dependencies: still running...
#9 794.8   Installing build dependencies: still running...
#9 943.5   Installing build dependencies: still running...
#9 1034.   Installing build dependencies: still running...
#9 1896.   Installing build dependencies: still running...
#9 1965.   Installing build dependencies: still running...
#9 2038.   Installing build dependencies: still running...
#9 2156.   Installing build dependencies: still running...
#9 2235.   Installing build dependencies: still running...
#9 2313.   Installing build dependencies: still running...
#9 2667.   Installing build dependencies: still running...
#9 2738.   Installing build dependencies: still running...
#9 3891.   Installing build dependencies: still running...
#9 3992.   Installing build dependencies: still running...
#9 4062.   Installing build dependencies: still running...
#9 4152.   Installing build dependencies: still running...
#9 4244.   Installing build dependencies: still running...
#9 4422.   Installing build dependencies: still running...
#9 4493.   Installing build dependencies: still running...
#9 5409.   Installing build dependencies: still running...
#9 5785.   Installing build dependencies: still running...
#9 5850.   Installing build dependencies: still running...
#9 5932.   Installing build dependencies: still running...
#9 5998.   Installing build dependencies: still running...
#9 6205.   Installing build dependencies: still running...
#9 6358.   Installing build dependencies: still running...
#9 6601.   Installing build dependencies: still running...
#9 6844.   Installing build dependencies: still running...
#9 7028.   Installing build dependencies: still running...
#9 7153.   Installing build dependencies: still running...
#9 7480.   Installing build dependencies: still running...
#9 7543.   Installing build dependencies: still running...
#9 7554.   Installing build dependencies: finished with status 'error'
#9 7554.   ERROR: Command errored out with exit status 1:
#9 7554.    command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f9gvenr7/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.28.5' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation == '"'"'CPython'"'"'' 'numpy==1.14.0; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation != '"'"'CPython'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'scipy>=0.19.1'
#9 7554.        cwd: None
#9 7554.   Complete output (201 lines):
#9 7554.   Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation == "CPython"' don't match your environment
#9 7554.   Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation != "CPython"' don't match your environment
#9 7554.   Ignoring numpy: markers 'python_version >= "3.8" and platform_system != "AIX"' don't match your environment
#9 7554.   Ignoring numpy: markers 'python_version == "3.6" and platform_system == "AIX"' don't match your environment
#9 7554.   Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment
#9 7554.   Ignoring numpy: markers 'python_version >= "3.8" and platform_system == "AIX"' don't match your environment
#9 7554.   Collecting setuptools
#9 7554.     Using cached setuptools-49.2.0-py3-none-any.whl (789 kB)
#9 7554.   Collecting wheel
#9 7554.     Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
#9 7554.   Collecting Cython>=0.28.5
#9 7554.     Downloading Cython-0.29.21-py2.py3-none-any.whl (974 kB)
#9 7554.   Collecting numpy==1.14.5
#9 7554.     Downloading numpy-1.14.5.zip (4.9 MB)
#9 7554.   Collecting scipy>=0.19.1
#9 7554.     Downloading scipy-1.5.1.tar.gz (25.6 MB)
#9 7554.     Installing build dependencies: started
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: still running...
#9 7554.     Installing build dependencies: finished with status 'done'
#9 7554.     Getting requirements to build wheel: started
#9 7554.     Getting requirements to build wheel: finished with status 'done'
#9 7554.       Preparing wheel metadata: started
#9 7554.       Preparing wheel metadata: finished with status 'error'
#9 7554.       ERROR: Command errored out with exit status 1:
#9 7554.        command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpp2qj8pbg
#9 7554.            cwd: /tmp/pip-install-mm7md5h8/scipy
#9 7554.       Complete output (137 lines):
#9 7554.       setup.py:462: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-s0szkk5z'), proceeding with generating Cython sources and expanding templates
#9 7554.         ' '.join(sys.argv[1:])))
#9 7554.       Running from SciPy source directory.
#9 7554.       lapack_opt_info:
#9 7554.       lapack_mkl_info:
#9 7554.       customize UnixCCompiler
#9 7554.         libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/aarch64-linux-gnu']
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       openblas_lapack_info:
#9 7554.       customize UnixCCompiler
#9 7554.       customize UnixCCompiler
#9 7554.         libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/aarch64-linux-gnu']
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       openblas_clapack_info:
#9 7554.       customize UnixCCompiler
#9 7554.       customize UnixCCompiler
#9 7554.         libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/aarch64-linux-gnu']
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       atlas_3_10_threads_info:
#9 7554.       Setting PTATLAS=ATLAS
#9 7554.       customize UnixCCompiler
#9 7554.         libraries tatlas,tatlas not found in /usr/local/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/local/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries tatlas,tatlas not found in /usr/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries tatlas,tatlas not found in /usr/lib/aarch64-linux-gnu
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/lib/aarch64-linux-gnu
#9 7554.       <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       atlas_3_10_info:
#9 7554.       customize UnixCCompiler
#9 7554.         libraries satlas,satlas not found in /usr/local/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/local/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries satlas,satlas not found in /usr/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries satlas,satlas not found in /usr/lib/aarch64-linux-gnu
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/lib/aarch64-linux-gnu
#9 7554.       <class 'numpy.distutils.system_info.atlas_3_10_info'>
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       atlas_threads_info:
#9 7554.       Setting PTATLAS=ATLAS
#9 7554.       customize UnixCCompiler
#9 7554.         libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/local/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries ptf77blas,ptcblas,atlas not found in /usr/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries ptf77blas,ptcblas,atlas not found in /usr/lib/aarch64-linux-gnu
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/lib/aarch64-linux-gnu
#9 7554.       <class 'numpy.distutils.system_info.atlas_threads_info'>
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       atlas_info:
#9 7554.       customize UnixCCompiler
#9 7554.         libraries f77blas,cblas,atlas not found in /usr/local/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/local/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries f77blas,cblas,atlas not found in /usr/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/lib
#9 7554.       customize UnixCCompiler
#9 7554.         libraries f77blas,cblas,atlas not found in /usr/lib/aarch64-linux-gnu
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack_atlas not found in /usr/lib/aarch64-linux-gnu
#9 7554.       <class 'numpy.distutils.system_info.atlas_info'>
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       /tmp/pip-build-env-ggia1m_d/overlay/lib/python3.7/site-packages/numpy/distutils/system_info.py:624: UserWarning:
#9 7554.           Atlas (http://math-atlas.sourceforge.net/) libraries not found.
#9 7554.           Directories to search for the libraries can be specified in the
#9 7554.           numpy/distutils/site.cfg file (section [atlas]) or by setting
#9 7554.           the ATLAS environment variable.
#9 7554.         self.calc_info()
#9 7554.       lapack_info:
#9 7554.       customize UnixCCompiler
#9 7554.         libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/aarch64-linux-gnu']
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       /tmp/pip-build-env-ggia1m_d/overlay/lib/python3.7/site-packages/numpy/distutils/system_info.py:624: UserWarning:
#9 7554.           Lapack (http://www.netlib.org/lapack/) libraries not found.
#9 7554.           Directories to search for the libraries can be specified in the
#9 7554.           numpy/distutils/site.cfg file (section [lapack]) or by setting
#9 7554.           the LAPACK environment variable.
#9 7554.         self.calc_info()
#9 7554.       lapack_src_info:
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       /tmp/pip-build-env-ggia1m_d/overlay/lib/python3.7/site-packages/numpy/distutils/system_info.py:624: UserWarning:
#9 7554.           Lapack (http://www.netlib.org/lapack/) sources not found.
#9 7554.           Directories to search for the sources can be specified in the
#9 7554.           numpy/distutils/site.cfg file (section [lapack_src]) or by setting
#9 7554.           the LAPACK_SRC environment variable.
#9 7554.         self.calc_info()
#9 7554.         NOT AVAILABLE
#9 7554.   
#9 7554.       Traceback (most recent call last):
#9 7554.         File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
#9 7554.           main()
#9 7554.         File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
#9 7554.           json_out['return_val'] = hook(**hook_input['kwargs'])
#9 7554.         File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
#9 7554.           return hook(metadata_directory, config_settings)
#9 7554.         File "/tmp/pip-build-env-ggia1m_d/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
#9 7554.           self.run_setup()
#9 7554.         File "/tmp/pip-build-env-ggia1m_d/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 249, in run_setup
#9 7554.           self).run_setup(setup_script=setup_script)
#9 7554.         File "/tmp/pip-build-env-ggia1m_d/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 142, in run_setup
#9 7554.           exec(compile(code, __file__, 'exec'), locals())
#9 7554.         File "setup.py", line 583, in <module>
#9 7554.           setup_package()
#9 7554.         File "setup.py", line 579, in setup_package
#9 7554.           setup(**metadata)
#9 7554.         File "/tmp/pip-build-env-ggia1m_d/overlay/lib/python3.7/site-packages/numpy/distutils/core.py", line 135, in setup
#9 7554.           config = configuration()
#9 7554.         File "setup.py", line 477, in configuration
#9 7554.           raise NotFoundError(msg)
#9 7554.       numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
#9 7554.       ----------------------------------------
#9 7554.   ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpp2qj8pbg Check the logs for full command output.
#9 7554.   WARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.
#9 7554.   You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#9 7554.   ----------------------------------------
#9 7555. ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f9gvenr7/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.28.5' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation == '"'"'CPython'"'"'' 'numpy==1.14.0; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation != '"'"'CPython'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'scipy>=0.19.1' Check the logs for full command output.
#9 7555. WARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.
#9 7555. You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#9 ERROR: executor failed running [/bin/sh -c pip3 install -r requirements.txt]: buildkit-runc did not terminate successfully
------
 > [5/9] RUN pip3 install -r requirements.txt:
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c pip3 install -r requirements.txt]: buildkit-runc did not terminate successfully
make: *** [Makefile:17: birt] Fehler 1

@stevemandala stevemandala mentioned this issue Oct 22, 2020
Closed
@gitrc
Copy link

gitrc commented Dec 16, 2020

So...are there aarch64 wheels for scikit-learn now? I'm trying to piece together this thread and related issues and it is unclear. If anyone could update here, I'd appreciate it before I go down the rabbit hole of building and maintaining wheels myself. Thank you!

@rth
Copy link
Member

rth commented Dec 16, 2020

Right now no, but it looks like they might be available for 0.24 in the near future #18782

@alfaro96 would know more about the situation there.

@alfaro96
Copy link
Member

alfaro96 commented Dec 17, 2020 •
edited
Loading

The wheel builder for the ARM64 architecture is available on master. We need to backport #18782, #19017 and #19018 to the 0.24.X branch and if we have enough Travis credits, ARM64 wheels will be available for 0.24.0 version.

This was referenced Jun 9, 2023
Closed
Merged
This was referenced Jun 20, 2023
Merged
Closed
@CanePlayz
Copy link

Closing as a duplicate of #13073.

@ogrisel Could we perhaps reopen this issue? The other one didn't lead to wheel support for aarch64 and I'm encountering a ton of issues when trying to build scikit locally.

@ogrisel
Copy link
Member

ogrisel commented Oct 28, 2024

We do have aarch64 for linux and macOS wheels up on pypi.org:

https://pypi.org/project/scikit-learn/#files

What OS are you running? Which command are you using to install scikit-learn? What error message do you get?

@CanePlayz
Copy link

Oh sorry, I forgot to mention that I'm on Windows arm64.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
New Feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ogrisel @rth @gitrc @fkromer @alfaro96 @odidev @CanePlayz