@rth Do you mind having a look to the failures in the tests of the Wheel builder / Build wheel for pp36-manylinux_x86_64 (pull_request) workflow?

I think that these tests should be marked as XFAIL for PyPy.

Also @ogrisel may have a strong opinion here.

Yes, looks like there are some CI failures in OpenML for PyPy. Could you please open an issue about it? but otherwise yes I think we can indeed mark them as xfail. We don't need to block wheel building for this, I'll try to have a look.

I will open an issue to investigate this problem and open a PR marking them as XFAIL until we discover a solution.

yum -y -q install atlas-devel lapack-devel gcc-gfortran openblas-devel is too slow 😢.

yum -y -q install atlas-devel lapack-devel gcc-gfortran openblas-devel is too slow cry.

I don't understand why we would need those to build scikit-learn on PyPy. Nowadays scikit-learn only depend on BLAS via the scipy Cython BLAS API. Maybe this is just a left over?

I am pretty sure we don't need the gcc-fortran compiler as we don't have any fortran code in scikit-learn.

yum -y -q install atlas-devel lapack-devel gcc-gfortran openblas-devel is too slow cry.

I don't understand why we would need those to build scikit-learn on PyPy. Nowadays scikit-learn only depend on BLAS via the scipy Cython BLAS API. Maybe this is just a left over?

I am pretty sure we don't need the gcc-fortran compiler as we don't have any fortran code in scikit-learn.

This is the error:

numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.

if we do not install such packages. Maybe only openblas-devel is necessary?

This error is happening when building scipy from source:

Downloading scipy-1.5.4.tar.gz (25.2 MB)

We need to find a way to make sure that cibuildwheel can use the pypy wheels for both numpy and scipy. Have a look at the configuration for the pypy build on circle ci to find how we do it there and adjust the env vars and maybe the pyproject,toml accordingly to make sure that we only build against version of numpy and scipy that have public PyPy wheels, ideally on pypi.org.

Building scipy from source is much too slow. We should really avoid it.

Thank you @ogrisel!

I have not realized that scipy was building from source. It is clear that we should avoid that.

However, scipy has not released wheels for PyPy in pypi yet.

However, scipy has not released wheels for PyPy yet.

We should probably not release official scikit-learn wheels for PyPy on pypi.org as long as it's not the case then. Maybe try to see in their issue tracker if they plan to do it in the future?

In the mean time we could build our wheels against a non official version of the scipy wheels just for testing. This means that we would need a dedicated [cd build pypy] commit flag for this purpose.

We should probably not release official scikit-learn wheels for PyPy on pypi.org as long as it's not the case then. Maybe try to see in their issue tracker if they plan to do it in the future?

In fact, @rth was ahead 5 months ago (approximately) MacPython/scipy-wheels#87 😉.

It's been ~ 1.5 years, and SciPy has not released PyPy wheels. I think it is safe to close this PR for now.

Note, with #21645 we now run pypy nightly and can trigger it with a commit tag on PRs.