Skip to content

CI Adds CPython 3.10 testing [python-dev] #20942

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 18 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions .github/workflows/cpython-nightly.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
name: CPython Nightly

on:
schedule:
# Nightly build at 1:00 A.M.
- cron: "0 1 * * *"
pull_request:
# Manual run
workflow_dispatch:

jobs:
get_latest_commit:
name: Check build trigger
runs-on: ubuntu-latest
if: "github.repository == 'scikit-learn/scikit-learn'"
outputs:
commit_msg: ${{ steps.get_commit.outputs.commit_msg }}

steps:
- name: Checkout scikit-learn
uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}

- id: get_commit
name: Get commit
run: |
echo "::set-output name=commit_msg::$(git log --no-merges -1 --oneline)"

nightly-cpython:
name: Nightly CPython-${{ matrix.python-version }}
needs: get_latest_commit
if: "contains(needs.get_latest_commit.outputs.commit_msg, '[python-dev]') || github.event_name == 'schedule' || github.event_name == 'workflow_run'"
runs-on: ubuntu-20.04
strategy:
matrix:
include:
- python-version: '3.10.0-alpha - 3.10.x'
install-numpy: 'wheel'
install-scipy: 'git'
env:
INSTALL_NUMPY: ${{ matrix.install-numpy }}
INSTALL_SCIPY: ${{ matrix.install-scipy }}
OMP_NUM_THREADS: '2'
OPENBLAS_NUM_THREADS: '2'
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install build dependencies
run: |
sudo apt-get install libatlas-base-dev liblapack-dev gfortran libgmp-dev libmpfr-dev libsuitesparse-dev libmpc-dev
pip install --upgrade pip setuptools
- name: Setup cache
uses: hendrikmuhs/ccache-action@v1
with:
max-size: 1000M
- name: Install python dependencies
run: |
pip install pytest pytest-xdist cython wheel
- name: Install numpy
run: |
export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH"
if [[ $INSTALL_NUMPY == "wheel" ]]; then
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
else
git clone --recursive https://github.com/numpy/numpy.git ~/numpy
cd ~/numpy && pip install -e .
fi
- name: Install scipy
run: |
export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH"
if [[ $INSTALL_SCIPY == "wheel" ]]; then
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy
else
git clone --recursive https://github.com/scipy/scipy.git ~/scipy
cd ~/scipy && pip install -e .
fi
- name: Install scikit-learn
run: |
export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH"
pip install -v --no-build-isolation -e .
- name: Test scikit-learn
run: |
cd /tmp && python -m pytest -n 2 --pyargs sklearn
Copy link
Member

@ogrisel ogrisel Sep 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if our OPENMP-based Cython code should now avoid oversubscription problems in docker containers, numpy/scipy might still use too many threads and slow down the execution.

To be safe I think we should set OMP_NUM_THREADS=2 / OPENBLAS_NUM_THREADS=2 globally for this workflow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thomasjpfan Shall we set the environment variable in this file as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The environment variables are set globally in lines 42-45:

    env:
      INSTALL_NUMPY: ${{ matrix.install-numpy }}
      INSTALL_SCIPY: ${{ matrix.install-scipy }}
      OMP_NUM_THREADS: '2'
      OPENBLAS_NUM_THREADS: '2'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this work and be useful?

Suggested change
cd /tmp && python -m pytest -n 2 --pyargs sklearn
cd /tmp && python -m pytest -n $OMP_NUM_THREADS --pyargs sklearn