Skip to content

MNT use conda-forge for the pypy build on circle ci #18867

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Nov 19, 2020
Merged

MNT use conda-forge for the pypy build on circle ci #18867

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
ade17e5
MNT use conda-forge for the pypy build on circle ci
ogrisel Nov 19, 2020
fb9e321
Use a docker image on the circle ci repo
ogrisel Nov 19, 2020
6911a4c
Debug use default python docker image
ogrisel Nov 19, 2020
3dcfc54
Update build_tools/circle/build_test_pypy.sh
ogrisel Nov 19, 2020
c2040d3
sudo apt-get
ogrisel Nov 19, 2020
bd5aec4
Use absolute path for /home/ogrisel/bin/micromamba
ogrisel Nov 19, 2020
0d558d9
Create /home/ogrisel/bin folder
ogrisel Nov 19, 2020
aeab42c
relative file extraction with tar
ogrisel Nov 19, 2020
a00e9c4
Put /home/ogrisel/bin in the PATH
ogrisel Nov 19, 2020
19e2b56
Try to init micromamba outside of the bash script
ogrisel Nov 19, 2020
31ae14f
DEBUG
ogrisel Nov 19, 2020
e48fe5e
Try the miniforge3 docker image
ogrisel Nov 19, 2020
e1a3b4e
more debug
ogrisel Nov 19, 2020
63ce206
one more try
ogrisel Nov 19, 2020
297b2de
uncomment the scheduled pypy job def
ogrisel Nov 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,13 +103,14 @@ jobs:

pypy3:
docker:
- image: pypy:3.6-7.2.0
- image: condaforge/miniforge3
steps:
- restore_cache:
keys:
- pypy3-ccache-{{ .Branch }}
- pypy3-ccache
- checkout
- run: conda init bash && source ~/.bashrc
- run: ./build_tools/circle/build_test_pypy.sh
- save_cache:
key: pypy3-ccache-{{ .Branch }}-{{ .BuildNum }}
Expand Down
44 changes: 16 additions & 28 deletions build_tools/circle/build_test_pypy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,28 @@
set -x
set -e

# System build tools
apt-get -yq update
apt-get -yq install libatlas-base-dev liblapack-dev gfortran ccache libopenblas-dev
apt-get -yq install wget bzip2 build-essential ccache

pip install virtualenv
# Install pypy and all the scikit-learn dependencies from conda-forge. In
# particular, we want to install pypy compatible binary packages for numpy and
# scipy as it would be to costly to build those from source.
conda install -y mamba
mamba create -n pypy -y \
pypy numpy scipy cython \
joblib threadpoolctl pillow pytest \
sphinx numpydoc docutils

if command -v pypy3; then
virtualenv -p $(command -v pypy3) pypy-env
elif command -v pypy; then
virtualenv -p $(command -v pypy) pypy-env
fi

source pypy-env/bin/activate
eval "$(conda shell.bash hook)"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting conda require quite a few steps..

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that was really hell. circle ci must also be doing something weird when passing the bash commands to execute in the container because some setups that would work in my local docker would not work on circle ci. And I could never get micromamba setup to work at all on circle ci (while it would work well and fast in my local setup).

conda activate pypy

# Check that we are running PyPy instead of CPython in this environment.
python --version
which python
python -c "import platform; assert platform.python_implementation() == 'PyPy'"

pip install -U pip

# pins versions to install wheel from https://antocuni.github.io/pypy-wheels/manylinux2010
pip install --extra-index-url https://antocuni.github.io/pypy-wheels/manylinux2010 numpy==1.18.0 scipy==1.3.2

# Install Cython directly
pip install https://antocuni.github.io/pypy-wheels/ubuntu/Cython/Cython-0.29.14-py3-none-any.whl
pip install sphinx numpydoc docutils joblib pillow pytest

# Build and install scikit-learn in dev mode
ccache -M 512M
export CCACHE_COMPRESS=1
export PATH=/usr/lib/ccache:$PATH
Expand All @@ -35,17 +32,8 @@ export OMP_NUM_THREADS="1"
# Set parallelism to 3 to overlap IO bound tasks with CPU bound tasks on CI
# workers with 2 cores when building the compiled extensions of scikit-learn.
export SKLEARN_BUILD_PARALLEL=3

# Build and install scikit-learn in dev mode
pip install --no-build-isolation -e .

# Check that Python implementation is PyPy
python - << EOL
import platform
from sklearn.utils import IS_PYPY
assert IS_PYPY is True, "platform={}!=PyPy".format(platform.python_implementation())
EOL

python -m pytest sklearn/
python -m pytest sklearn
python -m pytest doc/sphinxext/
python -m pytest $(find doc -name '*.rst' | sort)