Skip to content

CI Experimental [nogil] build of scikit-learn #23174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
May 2, 2022
Merged

CI Experimental [nogil] build of scikit-learn #23174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
c1a15af
CI Experimental [nogil] build of scikit-learn
ogrisel Apr 21, 2022
b93f035
fix typos [nogil]
ogrisel Apr 21, 2022
787b286
Skip ccache setup if not installed [nogil]
ogrisel Apr 21, 2022
9ec5962
Move ccache setup back to itself original position [nogil]
ogrisel Apr 21, 2022
adcaf82
missing sudo [nogil]
ogrisel Apr 21, 2022
c64638c
Add the APT package sources [nogil]
ogrisel Apr 21, 2022
64a09a7
typo [nogil]
ogrisel Apr 21, 2022
16c7a24
tee -a instead of tee [nogil]
ogrisel Apr 21, 2022
2966aba
python.exe => python [nogil]
ogrisel Apr 21, 2022
34b2a5b
VIRTUALENV is just a name, not a path [nogil]
ogrisel Apr 21, 2022
e09e2ef
git clone --depth 1 [nogil]
ogrisel Apr 21, 2022
d6f6218
Make setup_ccache idempotent
ogrisel Apr 21, 2022
f5afcdc
Better output messages in setup_ccache [nogil]
ogrisel Apr 21, 2022
f94cfb8
activate venv in tests [nogil]
ogrisel Apr 21, 2022
17967d7
Try to disable xdist [nogil]
ogrisel Apr 21, 2022
4aa9467
Trigger [nogil] build
ogrisel Apr 27, 2022
eba8dbd
Merge remote-tracking branch 'origin/main' into nogil-ci
ogrisel Apr 27, 2022
715546d
enable nogil venv for doctests
ogrisel Apr 27, 2022
3a9ebd2
Trigger [nogil]
ogrisel Apr 27, 2022
7767753
Fixed a typo [nogil]
ogrisel Apr 27, 2022
8173b1c
disable coverage for [nogil]
ogrisel Apr 28, 2022
baa4d4c
Re-enable pytext-xdist for [nogil]
ogrisel Apr 28, 2022
c872dea
Document the [nogil] commit flag
ogrisel Apr 29, 2022
b7a84e9
Apply suggestions from code review [nogil]
ogrisel Apr 29, 2022
bb79bb4
More explicit comment on the [nogil] specific index
ogrisel Apr 29, 2022
465a183
Typo [ci skip]
ogrisel Apr 29, 2022
6729e0b
Trigger [nogil]
ogrisel Apr 29, 2022
4b478ad
reorg install script for [nogil]
ogrisel May 2, 2022
3598eb6
reorg install script for [nogil]
ogrisel May 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,40 @@ jobs:
# Here we make sure, that they are still run on a regular basis.
SKLEARN_SKIP_NETWORK_TESTS: '0'

- template: build_tools/azure/posix.yml
# Experimental CPython branch without the Global Interpreter Lock:
# https://github.com/colesbury/nogil/
#
# The nogil build relies on a dedicated PyPI-style index to install patched
# versions of NumPy, SciPy and Cython maintained by @colesbury and that
# include specifc fixes to make them run correctly without relying on the GIL.
#
# The goal of this CI entry is to make sure that we do not introduce any
# dependency on the GIL in scikit-learn itself. An auxiliary goal is to early
# detect any regression in the patched build dependencies to report them
# upstream. The long-term goal is to be able to stop having to maintain
# multiprocessing based workaround / hacks in joblib / loky to make multi-CPU
# computing in scikit-learn efficient by default using regular threads.
#
# If this experimental entry becomes too unstable, feel free to disable it.
parameters:
name: Linux_nogil
vmImage: ubuntu-20.04
dependsOn: [git_commit, linting]
condition: |
and(
succeeded(),
not(contains(dependencies['git_commit']['outputs']['commit.message'], '[ci skip]')),
or(eq(variables['Build.Reason'], 'Schedule'),
contains(dependencies['git_commit']['outputs']['commit.message'], '[nogil]'
)
)
)
matrix:
pylatest_pip_nogil:
DISTRIB: 'pip-nogil'
COVERAGE: 'false'

# Check compilation with intel C++ compiler (ICC)
- template: build_tools/azure/posix.yml
parameters:
Expand Down
53 changes: 45 additions & 8 deletions build_tools/azure/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ source build_tools/shared.sh

UNAMESTR=`uname`

CCACHE_LINKS_DIR="/tmp/ccache"


make_conda() {
TO_INSTALL="$@"
if [[ "$DISTRIB" == *"mamba"* ]]; then
Expand All @@ -20,14 +23,21 @@ make_conda() {
}

setup_ccache() {
echo "Setting up ccache with CCACHE_DIR=${CCACHE_DIR}"
mkdir /tmp/ccache/
which ccache
for name in gcc g++ cc c++ clang clang++ i686-linux-gnu-gcc i686-linux-gnu-c++ x86_64-linux-gnu-gcc x86_64-linux-gnu-c++ x86_64-apple-darwin13.4.0-clang x86_64-apple-darwin13.4.0-clang++; do
ln -s $(which ccache) "/tmp/ccache/${name}"
done
export PATH="/tmp/ccache/:${PATH}"
ccache -M 256M
CCACHE_BIN=`which ccache || echo ""`
if [[ "${CCACHE_BIN}" == "" ]]; then
echo "ccache not found, skipping..."
elif [[ -d "${CCACHE_LINKS_DIR}" ]]; then
echo "ccache already configured, skipping..."
else
echo "Setting up ccache with CCACHE_DIR=${CCACHE_DIR}"
mkdir ${CCACHE_LINKS_DIR}
which ccache
for name in gcc g++ cc c++ clang clang++ i686-linux-gnu-gcc i686-linux-gnu-c++ x86_64-linux-gnu-gcc x86_64-linux-gnu-c++ x86_64-apple-darwin13.4.0-clang x86_64-apple-darwin13.4.0-clang++; do
ln -s ${CCACHE_BIN} "${CCACHE_LINKS_DIR}/${name}"
done
export PATH="${CCACHE_LINKS_DIR}:${PATH}"
ccache -M 256M
fi
}

pre_python_environment_install() {
Expand All @@ -48,6 +58,12 @@ pre_python_environment_install() {
apt-get -yq update
apt-get -yq install build-essential

elif [[ "$DISTRIB" == "pip-nogil" ]]; then
echo "deb-src http://archive.ubuntu.com/ubuntu/ focal main" | sudo tee -a /etc/apt/sources.list
sudo apt-get -yq update
sudo apt-get install -yq ccache
sudo apt-get build-dep -yq python3 python3-dev

elif [[ "$BUILD_WITH_ICC" == "true" ]]; then
wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
Expand All @@ -56,6 +72,7 @@ pre_python_environment_install() {
sudo apt-get update
sudo apt-get install intel-oneapi-compiler-dpcpp-cpp-and-cpp-classic
source /opt/intel/oneapi/setvars.sh

fi
}

Expand Down Expand Up @@ -120,6 +137,26 @@ python_environment_install() {
pip install https://github.com/joblib/joblib/archive/master.zip
echo "Installing pillow master"
pip install https://github.com/python-pillow/Pillow/archive/main.zip

elif [[ "$DISTRIB" == "pip-nogil" ]]; then
setup_ccache # speed-up the build of CPython it-self
Copy link
Member

@lesteve lesteve Apr 29, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can move setup_ccache earlier e.g. in pre_python_environment_install and remove it from scikit_learn_install this way it will always be called once.

Not sure whether then you'd want the "ccache already configured, skipping..." logic

Other than this LGTM.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can move setup_ccache earlier e.g. in pre_python_environment_install and remove it from scikit_learn_install this way it will always be called once.

This is what I wanted to do initially but it does not work because on some builds (in particular the macos builds) the ccache command is installed with conda instead of apt.

ORIGINAL_FOLDER=`pwd`
cd ..
git clone --depth 1 https://github.com/colesbury/nogil
cd nogil
./configure && make -j 2
./python -m venv $ORIGINAL_FOLDER/$VIRTUALENV
cd $ORIGINAL_FOLDER
source $VIRTUALENV/bin/activate

python -m pip install -U pip
# The pip version that comes with the nogil branch of CPython
# automatically uses the custom nogil index as its highest priority
# index to fetch patched versions of libraries with native code that
# would otherwise depend on the GIL.
echo "Installing build dependencies with pip from the nogil repository: https://d1yxz45j0ypngg.cloudfront.net/"
pip install numpy scipy cython joblib threadpoolctl

fi

python -m pip install $(get_dep threadpoolctl $THREADPOOLCTL_VERSION) \
Expand Down
2 changes: 1 addition & 1 deletion build_tools/azure/test_docs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ set -e

if [[ "$DISTRIB" =~ ^conda.* ]]; then
source activate $VIRTUALENV
elif [[ "$DISTRIB" == "ubuntu" ]]; then
elif [[ "$DISTRIB" == "ubuntu" || "$DISTRIB" == "pip-nogil" ]]; then
source $VIRTUALENV/bin/activate
fi

Expand Down
2 changes: 1 addition & 1 deletion build_tools/azure/test_script.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ source build_tools/shared.sh

if [[ "$DISTRIB" =~ ^conda.* ]]; then
source activate $VIRTUALENV
elif [[ "$DISTRIB" == "ubuntu" ]] || [[ "$DISTRIB" == "debian-32" ]]; then
elif [[ "$DISTRIB" == "ubuntu" || "$DISTRIB" == "debian-32" || "$DISTRIB" == "pip-nogil" ]]; then
source $VIRTUALENV/bin/activate
fi

Expand Down
1 change: 1 addition & 0 deletions doc/developers/contributing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -552,6 +552,7 @@ message, the following actions are taken.
[cd build gh] CD is run only for GitHub Actions
[lint skip] Azure pipeline skips linting
[scipy-dev] Build & test with our dependencies (numpy, scipy, etc ...) development builds
[nogil] Build & test with the nogil experimental branches of CPython, Cython, NumPy, SciPy...
[icc-build] Build & test with the Intel C compiler (ICC)
[pypy] Build & test with PyPy
[doc skip] Docs are not built
Expand Down