Skip to content

CI Create environment with conda rather than conda-lock when possible #29176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Jun 6, 2024
Merged

CI Create environment with conda rather than conda-lock when possible #29176

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
8e05b40
CI Create conda environment with conda rather than conda-lock
lesteve Jun 4, 2024
0f38ffb
[doc] [arm] Trigger CI
lesteve Jun 4, 2024
c6115ae
Can only install with conda when there is no pip package
lesteve Jun 4, 2024
aee9a53
[doc] [arm] Trigger CI
lesteve Jun 4, 2024
3178607
[doc] [arm] Fix
lesteve Jun 4, 2024
22521e4
[doc] [arm] Fix
lesteve Jun 4, 2024
18e3ce0
[doc] [arm] Fix
lesteve Jun 4, 2024
570829d
[doc] [arm] Fix
lesteve Jun 4, 2024
d41caa2
[doc] [arm] Fix
lesteve Jun 4, 2024
d5c1ec0
[doc] [arm] There is no difference between condaforge and mambaforge …
lesteve Jun 4, 2024
530e393
[doc build] [arm] There is no difference between condaforge and mamba…
lesteve Jun 4, 2024
11af8f2
[doc build] [cirrus arm] fix
lesteve Jun 4, 2024
a7e4ae8
Tweak comments
lesteve Jun 4, 2024
aea1218
[azure parallel] [doc build] Seriously PyPy?
lesteve Jun 4, 2024
4809b6b
[azure parallel] [doc build] Fix grep command
lesteve Jun 4, 2024
8da38e7
Merge branch 'main' of https://github.com/scikit-learn/scikit-learn i…
lesteve Jun 6, 2024
f8456eb
Use create_environment_from_lock_file in create_gpu_environment.sh
lesteve Jun 6, 2024
6e5b2d7
Use env var for more explicit argument purposes
lesteve Jun 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions build_tools/azure/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,7 @@ check_packages_dev_version() {

python_environment_install_and_activate() {
if [[ "$DISTRIB" == "conda"* ]]; then
# Install/update conda with the libmamba solver because the legacy
# solver can be slow at installing a specific version of conda-lock.
conda install -n base conda conda-libmamba-solver -y
conda config --set solver libmamba
conda install -c conda-forge "$(get_dep conda-lock min)" -y
conda-lock install --name $VIRTUALENV $LOCK_FILE
create_conda_environment_from_lock_file $VIRTUALENV $LOCK_FILE
source activate $VIRTUALENV

elif [[ "$DISTRIB" == "ubuntu" || "$DISTRIB" == "debian-32" ]]; then
Expand Down
23 changes: 11 additions & 12 deletions build_tools/circle/build_doc.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,21 +159,20 @@ if [[ `type -t deactivate` ]]; then
deactivate
fi

MAMBAFORGE_PATH=$HOME/mambaforge
# Install dependencies with mamba
wget -q https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh \
-O mambaforge.sh
chmod +x mambaforge.sh && ./mambaforge.sh -b -p $MAMBAFORGE_PATH
export PATH="/usr/lib/ccache:$MAMBAFORGE_PATH/bin:$PATH"

# Install Miniforge
MINIFORGE_URL="https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh"
curl -L --retry 10 $MINIFORGE_URL -o miniconda.sh
MINIFORGE_PATH=$HOME/miniforge3
bash ./miniconda.sh -b -p $MINIFORGE_PATH
source $MINIFORGE_PATH/etc/profile.d/conda.sh
conda activate
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for CI, micromamba is a lot smaller and quite fast. Would it be okay to use it or you rather not bother?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I seem to remember that you had issues with updating the lock-files though with micromamba, right, and that the fix for you was micromamba install mamba right? My wild guess is that there were issues with virtual packages that were fixed in conda (I did one of the PR by the way). Because micromamba reimplements these parts, they were issues, there is a small chance that someone reported it and that it was fixed.

I am not convinced the "lot smaller" this matters that much in practice for the, CI maybe you save a few seconds for the downloads and 2-3 lines for the installation code.

All in all, I would be in favour of being conservative here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The solution for me was to install both mamba and conda to be able to update lock files.

The smaller thing is significant since micromamba doesn't come with a base environment. So there's one whole environment saved by not using conda here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR is already a net improvement. Can we discuss micromamba in a follow up PR?


export PATH="/usr/lib/ccache:$PATH"
ccache -M 512M
export CCACHE_COMPRESS=1

# pin conda-lock to latest released version (needs manual update from time to time)
mamba install "$(get_dep conda-lock min)" -y

conda-lock install --log-level DEBUG --name $CONDA_ENV_NAME $LOCK_FILE
source activate $CONDA_ENV_NAME
create_conda_environment_from_lock_file $CONDA_ENV_NAME $LOCK_FILE
conda activate $CONDA_ENV_NAME

show_installed_libraries

Expand Down
2 changes: 1 addition & 1 deletion build_tools/cirrus/arm_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ linux_aarch64_test_task:
CONDA_ENV_NAME: testenv
LOCK_FILE: build_tools/cirrus/pymin_conda_forge_linux-aarch64_conda.lock
CONDA_PKGS_DIRS: /root/.conda/pkgs
HOME: / # $HOME is not defined in image and is required to install mambaforge
HOME: / # $HOME is not defined in image and is required to install Miniforge
# Upload tokens have been encrypted via the CirrusCI interface:
# https://cirrus-ci.org/guide/writing-tasks/#encrypted-variables
# See `maint_tools/update_tracking_issue.py` for details on the permissions the token requires.
Expand Down
23 changes: 10 additions & 13 deletions build_tools/cirrus/build_test_arm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,16 @@ setup_ccache() {
ccache -M 0
}

MAMBAFORGE_URL="https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-aarch64.sh"

# Install Mambaforge
curl -L --retry 10 $MAMBAFORGE_URL -o mambaforge.sh
MAMBAFORGE_PATH=$HOME/mambaforge
bash ./mambaforge.sh -b -p $MAMBAFORGE_PATH
export PATH=$MAMBAFORGE_PATH/bin:$PATH
mamba init --all --verbose
mamba update --yes mamba
mamba update --yes conda
mamba install "$(get_dep conda-lock min)" -y
conda-lock install --name $CONDA_ENV_NAME $LOCK_FILE
source activate $CONDA_ENV_NAME
# Install Miniforge
MINIFORGE_URL="https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh"
curl -L --retry 10 $MINIFORGE_URL -o miniconda.sh
MINIFORGE_PATH=$HOME/miniforge3
bash ./miniconda.sh -b -p $MINIFORGE_PATH
source $MINIFORGE_PATH/etc/profile.d/conda.sh
conda activate

create_conda_environment_from_lock_file $CONDA_ENV_NAME $LOCK_FILE
conda activate $CONDA_ENV_NAME

setup_ccache

Expand Down
6 changes: 3 additions & 3 deletions build_tools/github/create_gpu_environment.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,6 @@ source "${HOME}/conda/etc/profile.d/conda.sh"
source build_tools/shared.sh
conda activate base

# XXX switch once https://github.com/scikit-learn/scikit-learn/pull/29176 is merged
conda install -c conda-forge "$(get_dep conda-lock min)" -y
conda-lock install --name sklearn build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock
CONDA_ENV_NAME=sklearn
LOCK_FILE=build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock
create_conda_environment_from_lock_file $CONDA_ENV_NAME $LOCK_FILE
16 changes: 16 additions & 0 deletions build_tools/shared.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,19 @@ activate_environment() {
source $VIRTUALENV/bin/activate
fi
}

create_conda_environment_from_lock_file() {
ENV_NAME=$1
LOCK_FILE=$2
# Because we are using lock-files with the "explicit" format, conda can
# install them directly, provided the lock-file does not contain pip solved
# packages. For more details, see
# https://conda.github.io/conda-lock/output/#explicit-lockfile
lock_file_has_pip_packages=$(grep -q files.pythonhosted.org $LOCK_FILE && echo "true" || echo "false")
if [[ "$lock_file_has_pip_packages" == "false" ]]; then
conda create --name $ENV_NAME --file $LOCK_FILE
else
conda install "$(get_dep conda-lock min)" -y
conda-lock install --name $ENV_NAME $LOCK_FILE
fi
}
Loading