✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: c0b62a5. Link to the linter CI: here}

This is a draft to be able to experiment.

I checked all packages in pip_dependencies if the required version is available on the conda channel that is used in this CI setting.

I was unsure and have not moved meson, because it was not explicitly mentioned in this comment and I assume there is a reason for it?

Other dependencies have not defined a version in our pyproject.toml and I have moved them to conda_dependencies regardless if they were available: lightgbm, array-api-strict, pytest-xdist, ninja, jupyterlite-sphinx, and jupyterlite-pyodide-kernel. That means they will probably come in different versions as with pypi and we'll see where this leads, when the CIs have run.

Okay, seems that the version change in array-api-strict, which has no specified version our pyproject.toml, but was 2.4 (edit: no it was 2.3.1) with pip, and would be 2.2 with conda defaults as required by thepylatest_pip_openblas_pandas CI caused some tests to fail. So I think I should move it back to the pip_dependencies.

I have moved array-api-strict back to the pip_dependencies, and now it shows up in the diff of the lock file for the first time (before it didn't which I thought was strange) as:

# pip array-api-strict @ https://files.pythonhosted.org/packages/fe/c7/a97e26083985b49a7a54006364348cf1c26e5523850b8522a39b02b19715/array_api_strict-2.3.1-py3-none-any.whl#sha256=0ca6988be1c82d2f05b6cd44bc7e14cb390555d1455deb50f431d6d0cf468ded
# pip array-api-strict @ https://files.pythonhosted.org/packages/e5/33/cede42b7b866db4b77432889314fc652ecc5cb6988f831ef08881a767089/array_api_strict-2.4-py3-none-any.whl#sha256=1cb20acd008f171ad8cce49589cc59897d8a242d1acf8ce6a61c3d57b61ecd14

That shows that before version 2.3.1 was used and now it is the newest version on pypi 2.4, that was released 6 days ago.
I don't know why it is not version-locked since it is a fast developing library, but let's hope now the CI passes.

Edit: Nice they all passed. 🎉

Now it's only the Windows CI failing, and this is unrelated and coming from main.

This PR updates update_environments_and_lock_files.py to use conda for packages that were installed via pip but are now available on conda.

Do you know why we used pip for these and not conda? Was it because the version we wanted were not available on conda-forge? I don't know the answer but I assume that (like so many things in scikit-learn) it was done for a specific reason. If we know the reason then it is less likely to break something by accident

@betatim: I didn't know this before, but @adrinjalali told me that's a maintanance task we do from time to time. It is necessary because the versions we need were not available (before) on the respective conda channel that is used for this job (some use conda-forge and other use conda defaults channel) - but now they might be and we prefer this.

I also took notes while I checked everything. Let me share these, maybe it helps with reviewing:

We have specific minimal version requirements in our pyproject.toml:

• cython>=3.0.10, and 3.0.11 available on https://anaconda.org/main/cython
• threadpoolctl>=3.1.0, and 3.5.0 available on https://anaconda.org/main/threadpoolctl
• meson-python>=0.17.1, and 0.17.1 available on https://anaconda.org/main/meson-python
• "sphinx>=7.3.7", 8.2.3 available on https://anaconda.org/main/sphinx
• "numpydoc>=1.2.0", 1.7.0 available on https://anaconda.org/main/numpydoc
• "scikit-image>=0.19.0", 0.25.2 avlailable on https://anaconda.org/main/scikit-image
• "pytest>=7.1.2", 8.4.1 available on https://anaconda.org/conda-forge/pytest and 8.3.4 on https://anaconda.org/main/pytest
• "sphinxext-opengraph>=0.9.1", 0.10.0 available on https://anaconda.org/conda-forge/sphinxext-opengraph

We do not have a version specified in pyproject.toml and have thus used the newest version available on pypi:

• lightgbm, available with version 4.6.0 on https://anaconda.org/main/lightgbm
• array-api-strict, available with version 2.2 on https://anaconda.org/main/array-api-strict <-- this broke and I reverted (see above)
• pytest-xdist, available with version 3.6.1 on https://anaconda.org/main/pytest-xdist
• ninja, available with version 1.12.1 on https://anaconda.org/main/ninja
• jupyterlite-sphinx, available with version 0.20.2 on https://anaconda.org/conda-forge/jupyterlite-sphinx
• jupyterlite-pyodide-kernel, available with version 0.6.1 on https://anaconda.org/conda-forge/jupyterlite-pyodide-kernel

Whether I have checked the version from the main ("defaults") channel or the conda-forge channel depends on how the channel in the job was specified in build_tools/update_environments_and_lock_files.py. Did I do all that correctly 🤷 ?

Cool! Thanks for doing the "chopping wood and carrying water" on moving things from PyPI to conda.

As far as I can tell all the moves make sense. I think we should wait for the CI to be working again and run it again. Maybe even using the [cd build] marker in a commit message to run the builds?

Thanks for reviewing and your support @betatim and @lesteve! I have reverted the changes in the jobs that have "_pip_" in their name, resolved the conflicts with main and build the lock files again (with --select-build only those that I have changed in this PR: pylatest_conda_mkl_no_openmp, doc_min_dependencies and doc)

Oh, it seems that without re-building the lock files for pylatest_pip_openblas_pandas and pylatest_pip_scipy_dev too, the old lock files from my PR (that moved dependencies from conda to pip) where still on this branch. That doesn't fit at all with my current understanding of how resolving conflicts with lock files works (I have used the bash command that we have in our Resolve conflicts in lock files and to my understanding this should entirely prioritise the lock files from main over whatever lock files I had build before.

I have now rebuild the lock files for pylatest_pip_openblas_pandas and pylatest_pip_scipy_dev jobs too and these look better (only updates to some packages but no moving from pip to conda anymore).

Can you fix the conflicts? Each time we merged the automatic lock-files update on Monday, this may create new conflicts unfortunately ...

Based on the CI builds you are changing, I would suggest to only update the lock-files you want to change with the --select-build argument:

python build_tools/update_environments_and_lock_files.py --select-build <some-text-goes-here>

By default build_tools/update_environment_and_lock_files.py updates all the lock-files.

Thanks @lesteve! I have pulled main and re-generated the relevant lock files. Let's hope everything does fine with the CI. :)

LGTM, I enabled auto-merge and pushed a commit to make sure we run the full doc build with [doc build] marker just to be sure.

@StefanieSenger as a follow-up if you feel like it, you could tackle this TODO (noticed it while reviewing):

# scipy 1.12.x crashes on this platform (https://github.com/scipy/scipy/pull/20086)
# TODO: release scipy constraint when 1.13 is available in the "default"
# channel.
"scipy": "<1.12",

scipy >= 1.13 is available in the defaults channel and so removing the scipy constraint should "just work", but the only way to know is to try ...

Oh yes, nice, I will take care of this. Thanks for the hint, @lesteve! :)