Skip to content

CI Build the wheels for PyPy #18879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 15 commits into from
Closed

Conversation

alfaro96
Copy link
Member

What does this implement/fix? Explain your changes.

This PR build the wheels for PyPy using the cibuildwheel package.

@alfaro96 alfaro96 changed the title CI Build the wheels for PyPy [WIP] CI Build the wheels for PyPy Nov 20, 2020
@alfaro96 alfaro96 marked this pull request as draft November 20, 2020 10:53
@alfaro96
Copy link
Member Author

@rth Do you mind having a look to the failures in the tests of the Wheel builder / Build wheel for pp36-manylinux_x86_64 (pull_request) workflow?

@alfaro96
Copy link
Member Author

I think that these tests should be marked as XFAIL for PyPy.

@alfaro96
Copy link
Member Author

Also @ogrisel may have a strong opinion here.

@rth
Copy link
Member

rth commented Nov 25, 2020

Yes, looks like there are some CI failures in OpenML for PyPy. Could you please open an issue about it? but otherwise yes I think we can indeed mark them as xfail. We don't need to block wheel building for this, I'll try to have a look.

@alfaro96
Copy link
Member Author

I will open an issue to investigate this problem and open a PR marking them as XFAIL until we discover a solution.

@alfaro96 alfaro96 changed the title [WIP] CI Build the wheels for PyPy CI Build the wheels for PyPy Nov 27, 2020
@alfaro96 alfaro96 marked this pull request as ready for review November 27, 2020 08:35
@alfaro96
Copy link
Member Author

yum -y -q install atlas-devel lapack-devel gcc-gfortran openblas-devel is too slow 😢.

@ogrisel
Copy link
Member

ogrisel commented Nov 27, 2020

yum -y -q install atlas-devel lapack-devel gcc-gfortran openblas-devel is too slow cry.

I don't understand why we would need those to build scikit-learn on PyPy. Nowadays scikit-learn only depend on BLAS via the scipy Cython BLAS API. Maybe this is just a left over?

I am pretty sure we don't need the gcc-fortran compiler as we don't have any fortran code in scikit-learn.

@alfaro96
Copy link
Member Author

yum -y -q install atlas-devel lapack-devel gcc-gfortran openblas-devel is too slow cry.

I don't understand why we would need those to build scikit-learn on PyPy. Nowadays scikit-learn only depend on BLAS via the scipy Cython BLAS API. Maybe this is just a left over?

I am pretty sure we don't need the gcc-fortran compiler as we don't have any fortran code in scikit-learn.

This is the error:

numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.

if we do not install such packages. Maybe only openblas-devel is necessary?

@ogrisel
Copy link
Member

ogrisel commented Nov 27, 2020

This error is happening when building scipy from source:

Downloading scipy-1.5.4.tar.gz (25.2 MB)

We need to find a way to make sure that cibuildwheel can use the pypy wheels for both numpy and scipy. Have a look at the configuration for the pypy build on circle ci to find how we do it there and adjust the env vars and maybe the pyproject,toml accordingly to make sure that we only build against version of numpy and scipy that have public PyPy wheels, ideally on pypi.org.

@ogrisel
Copy link
Member

ogrisel commented Nov 27, 2020

Building scipy from source is much too slow. We should really avoid it.

@alfaro96
Copy link
Member Author

alfaro96 commented Nov 27, 2020

Thank you @ogrisel!

I have not realized that scipy was building from source. It is clear that we should avoid that.

However, scipy has not released wheels for PyPy in pypi yet.

@ogrisel
Copy link
Member

ogrisel commented Nov 27, 2020

However, scipy has not released wheels for PyPy yet.

We should probably not release official scikit-learn wheels for PyPy on pypi.org as long as it's not the case then. Maybe try to see in their issue tracker if they plan to do it in the future?

In the mean time we could build our wheels against a non official version of the scipy wheels just for testing. This means that we would need a dedicated [cd build pypy] commit flag for this purpose.

@alfaro96
Copy link
Member Author

alfaro96 commented Nov 27, 2020

We should probably not release official scikit-learn wheels for PyPy on pypi.org as long as it's not the case then. Maybe try to see in their issue tracker if they plan to do it in the future?

In fact, @rth was ahead 5 months ago (approximately) MacPython/scipy-wheels#87 😉.

Base automatically changed from master to main January 22, 2021 10:53
@thomasjpfan
Copy link
Member

It's been ~ 1.5 years, and SciPy has not released PyPy wheels. I think it is safe to close this PR for now.

Note, with #21645 we now run pypy nightly and can trigger it with a commit tag on PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants