MAINT bump minimum version for pytest #26184

glemaitre · 2023-04-14T19:04:30Z

Bump the minimum version of pytest from 5.3.1 to 5.4.3 (the latest of the 5.X release).

It seems that it removes the INVALIDERROR observed in #26091

I, therefore, have high hopes for solving a similar error observed here: #25639 (comment)

glemaitre · 2023-04-14T19:51:19Z

OK it seems that conda-lock does not like me:

❯ python build_tools/update_environments_and_lock_files.py
Writing conda environments
Writing conda lock files
pylatest_conda_forge_mkl_linux-64
pylatest_conda_forge_mkl_osx-64
pylatest_conda_mkl_no_openmp
pylatest_conda_forge_mkl_no_coverage
py38_conda_defaults_openblas
py38_conda_forge_openblas_ubuntu_2204
pylatest_pip_openblas_pandas
Traceback (most recent call last):
  File "/home/glemaitre/Documents/packages/scikit-learn/build_tools/update_environments_and_lock_files.py", line 596, in <module>
    main()
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/glemaitre/Documents/packages/scikit-learn/build_tools/update_environments_and_lock_files.py", line 582, in main
    write_all_conda_lock_files(filtered_conda_build_metadata_list)
  File "/home/glemaitre/Documents/packages/scikit-learn/build_tools/update_environments_and_lock_files.py", line 483, in write_all_conda_lock_files
    create_conda_lock_file(build_metadata)
  File "/home/glemaitre/Documents/packages/scikit-learn/build_tools/update_environments_and_lock_files.py", line 477, in create_conda_lock_file
    conda_lock(environment_path, lock_file_path, platform)
  File "/home/glemaitre/Documents/packages/scikit-learn/build_tools/update_environments_and_lock_files.py", line 464, in conda_lock
    execute_command(shlex.split(command))
  File "/home/glemaitre/Documents/packages/scikit-learn/build_tools/update_environments_and_lock_files.py", line 383, in execute_command
    raise RuntimeError(
RuntimeError: Command exited with non-zero exit code.
Exit code: 1
Command:
conda-lock lock --mamba --kind explicit --platform linux-64 --file build_tools/azure/pylatest_pip_openblas_pandas_environment.yml --filename-template build_tools/azure/pylatest_pip_openblas_pandas_linux-64_conda.lock
stdout:

stderr:
Locking dependencies for ['linux-64']...
INFO:conda_lock.conda_solver:linux-64 using specs ['python 3.9.*', 'ccache', 'sphinx', 'pip *']
Traceback (most recent call last):
  File "/home/glemaitre/miniconda3/envs/dev/bin/conda-lock", line 10, in <module>
    sys.exit(main())
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1353, in lock
    lock_func(
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1083, in run_lock
    make_lock_files(
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 408, in make_lock_files
    lock_content = lock_content | create_lockfile_from_spec(
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 801, in create_lockfile_from_spec
    deps = _solve_for_arch(
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 737, in _solve_for_arch
    pip_deps = solve_pypi(
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/pypi_solver.py", line 269, in solve_pypi
    result = s.solve(use_latest=to_update)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 65, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 233, in _solve
    result = resolve_version(
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/__init__.py", line 7, in resolve_version
    return solver.solve()
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 84, in solve
    next = self._choose_package_version()
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 400, in _choose_package_version
    version = self._provider.complete_package(version)
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/provider.py", line 433, in complete_package
    self._pool.package(
  File "/home/glemaitre/miniconda3/envs/dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/repositories/pool.py", line 149, in package
    raise PackageNotFound("Package {} ({}) not found.".format(name, version))
conda_lock._vendor.poetry.repositories.exceptions.PackageNotFound: Package tzdata (2023c) not found.

lorentzenchr

CI should become green, otherwise good to go no-brainer, i.e. no change of a runtime dependence.

glemaitre · 2023-04-17T07:42:09Z

@lesteve did you see in the past an error linked to a missing package?

lesteve · 2023-04-17T08:00:46Z

This one in particular does not ring a bell but it seems like a conda vs pip thing though

conda has tzdata (latest version 2023c) and python-tzdata (latest version 2023.3)
pip has tzdata (latest version 2023.3)

Generally weird thing can happen in conda-lock if you mix conda and pip for the same package. Sometimes you can find a work-around by moving some dependencies from the conda to the pip section or the other way around ... not sure if there is such a work-around in this case ...

glemaitre · 2023-04-17T08:06:43Z

OK I see. I will check the lock file. I assume tzdata is a dependence introduce by pandas here.

glemaitre · 2023-04-17T09:19:23Z

After investigating a bit, installing python via conda requires tzdata. pandas started to have tzdata as a mandatory dependency in 2.0 and it explains the current lock.

adrinjalali · 2023-04-17T09:58:28Z

build_tools/update_environments_and_lock_files.py

+            # TODO: remove the following pin when we have a solution since the current
+            # environment defeat the purpose of the test (e.g. `pylastest`).
+            # pandas 2.0 depends on tzdata as well as python installed via conda
+            # We currently have a mix of conda/pip and run into the following issue:
+            # https://github.com/conda/conda-lock/issues/179
+            "pandas": "1.5.3",


what happens if we get it from conda-forge instead of pypi?

Then we defeat the purpose of installing everything via pip as intended.
Here, I chose to keep what is currently happening on the CI.

hmm. so here's my issue. One can pip install pandas=2.0 in a conda-forge environment w/o an issue. Loading tzdata in a python process will then load the one installed by pip. So from the user's perspective, they can use the latest pandas, and scikit-learn, in a conda-forge env. But here in the CI now we're not testing that.

~ ⌚ 11:26:06 $ mmamba create -n delete-me python=3.11 __ __ ______ ___ ____ _____ ___ / /_ ____ _ / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/ / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ / / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/ /_/ conda-forge/noarch 11.9MB @ 3.2MB/s 3.8s conda-forge/linux-64 30.9MB @ 3.4MB/s 9.4s Transaction Prefix: /home/adrin/miniforge3/envs/delete-me Updating specs: - python=3.11 Package Version Build Channel Size ────────────────────────────────────────────────────────────────────────────────────── Install: ────────────────────────────────────────────────────────────────────────────────────── + _libgcc_mutex 0.1 conda_forge conda-forge/linux-64 Cached + _openmp_mutex 4.5 2_gnu conda-forge/linux-64 Cached + bzip2 1.0.8 h7f98852_4 conda-forge/linux-64 Cached + ca-certificates 2022.12.7 ha878542_0 conda-forge/linux-64 Cached + ld_impl_linux-64 2.40 h41732ed_0 conda-forge/linux-64 Cached + libexpat 2.5.0 hcb278e6_1 conda-forge/linux-64 78kB + libffi 3.4.2 h7f98852_5 conda-forge/linux-64 Cached + libgcc-ng 12.2.0 h65d4601_19 conda-forge/linux-64 Cached + libgomp 12.2.0 h65d4601_19 conda-forge/linux-64 Cached + libnsl 2.0.0 h7f98852_0 conda-forge/linux-64 Cached + libsqlite 3.40.0 h753d276_0 conda-forge/linux-64 Cached + libuuid 2.38.1 h0b41bf4_0 conda-forge/linux-64 Cached + libzlib 1.2.13 h166bdaf_4 conda-forge/linux-64 Cached + ncurses 6.3 h27087fc_1 conda-forge/linux-64 Cached + openssl 3.1.0 h0b41bf4_0 conda-forge/linux-64 Cached + pip 23.1 pyhd8ed1ab_0 conda-forge/noarch 1MB + python 3.11.3 h2755cc3_0_cpython conda-forge/linux-64 31MB + readline 8.2 h8228510_1 conda-forge/linux-64 Cached + setuptools 67.6.1 pyhd8ed1ab_0 conda-forge/noarch Cached + tk 8.6.12 h27826a3_0 conda-forge/linux-64 Cached + tzdata 2023c h71feb2d_0 conda-forge/noarch 118kB + wheel 0.40.0 pyhd8ed1ab_0 conda-forge/noarch Cached + xz 5.2.6 h166bdaf_0 conda-forge/linux-64 Cached Summary: Install: 23 packages Total download: 33MB ────────────────────────────────────────────────────────────────────────────────────── Confirm changes: [Y/n] Transaction starting tzdata 117.6kB @ 352.0kB/s 0.3s libexpat 78.0kB @ 220.3kB/s 0.4s pip 1.4MB @ 2.8MB/s 0.5s python 31.1MB @ 14.4MB/s 2.2s Linking _libgcc_mutex-0.1-conda_forge Linking ld_impl_linux-64-2.40-h41732ed_0 Linking ca-certificates-2022.12.7-ha878542_0 Linking libgomp-12.2.0-h65d4601_19 Linking _openmp_mutex-4.5-2_gnu Linking libgcc-ng-12.2.0-h65d4601_19 Linking libffi-3.4.2-h7f98852_5 Linking bzip2-1.0.8-h7f98852_4 Linking ncurses-6.3-h27087fc_1 Linking openssl-3.1.0-h0b41bf4_0 Linking libuuid-2.38.1-h0b41bf4_0 Linking libexpat-2.5.0-hcb278e6_1 Linking xz-5.2.6-h166bdaf_0 Linking libzlib-1.2.13-h166bdaf_4 Linking libnsl-2.0.0-h7f98852_0 Linking readline-8.2-h8228510_1 Linking tk-8.6.12-h27826a3_0 Linking libsqlite-3.40.0-h753d276_0 Linking tzdata-2023c-h71feb2d_0 Linking python-3.11.3-h2755cc3_0_cpython Linking wheel-0.40.0-pyhd8ed1ab_0 Linking setuptools-67.6.1-pyhd8ed1ab_0 Linking pip-23.1-pyhd8ed1ab_0 Transaction finished ~ ⌚ 12:22:24 $ mmamba activate delete-me (delete-me) ~ ⌚ 12:22:34 $ pip install pandas Collecting pandas Downloading pandas-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 16.0 MB/s eta 0:00:00 Collecting python-dateutil>=2.8.2 (from pandas) Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) Collecting pytz>=2020.1 (from pandas) Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 502.3/502.3 kB 16.0 MB/s eta 0:00:00 Collecting tzdata>=2022.1 (from pandas) Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.8/341.8 kB 18.3 MB/s eta 0:00:00 Collecting numpy>=1.21.0 (from pandas) Downloading numpy-1.24.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 16.3 MB/s eta 0:00:00 Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas) Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Installing collected packages: pytz, tzdata, six, numpy, python-dateutil, pandas Successfully installed numpy-1.24.2 pandas-2.0.0 python-dateutil-2.8.2 pytz-2023.3 six-1.16.0 tzdata-2023.3 (delete-me) ~ ⌚ 12:22:47 $ python Python 3.11.3 | packaged by conda-forge | (main, Apr 6 2023, 08:57:19) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import tzdata >>> tzdata.__version__ '2023.3' >>>

We are testing pandas development version in CI installed from pip is that good enough?

+1 for merging this as it is, and if the conda-lock problem does not get resolved in the next few months, open a follow-up PR to add the (unpinned) pandas optional dependency to one of the pylatest_conda_forge_* configurations.

ogrisel

Assuming CI goes green after conflict resolution.

build_tools/azure/test_script.sh

build_tools/update_environments_and_lock_files.py

ogrisel · 2023-04-25T16:47:58Z

build_tools/update_environments_and_lock_files.py

+            # TODO: remove the following pin when we have a solution since the current
+            # environment defeat the purpose of the test (e.g. `pylastest`).
+            # pandas 2.0 depends on tzdata as well as python installed via conda
+            # We currently have a mix of conda/pip and run into the following issue:
+            # https://github.com/conda/conda-lock/issues/179
+            "pandas": "1.5.3",


+1 for merging this as it is, and if the conda-lock problem does not get resolved in the next few months, open a follow-up PR to add the (unpinned) pandas optional dependency to one of the pylatest_conda_forge_* configurations.

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

adrinjalali · 2023-04-27T14:49:28Z

Trying to run python build_tools/update_environments_and_lock_files.py I get:

    STDERR:
PackageRecord(_hash=-5225189611440125633, name='__glibc', version='0', build='0', build_number=0, channel=Channel("@"), subdir='linux-64', fn='__glibc', md5='12345678901234567890123456789012', package_type='virtual_system')

Traceback (most recent call last):
  File "/home/adrin/miniforge3/envs/sklearn/bin/conda-lock", line 10, in <module>
    sys.exit(main())
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1353, in lock
    lock_func(
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1083, in run_lock
    make_lock_files(
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 408, in make_lock_files
    lock_content = lock_content | create_lockfile_from_spec(
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 801, in create_lockfile_from_spec
    deps = _solve_for_arch(
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 723, in _solve_for_arch
    conda_deps = solve_conda(
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/conda_lock/conda_solver.py", line 168, in solve_conda
    dry_run_install = solve_specs_for_arch(
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/conda_lock/conda_solver.py", line 355, in solve_specs_for_arch
    proc.check_returncode()
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py", line 168, in check_returncode
    raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['/home/adrin/miniforge3/condabin/mamba', 'create', '--prefix', '/tmp/tmpsiq55z9o/prefix', '--dry-run', '--json', '--override-channels', '--channel', 'conda-forge', '--channel', 'file:///tmp/tmp5xvoymnu', 'python', 'numpy', 'blas * mkl', 'scipy', 'cython', 'joblib', 'threadpoolctl', 'matplotlib', 'pandas', 'pyamg', 'pytest', 'pytest-xdist 2.5.0.*', 'pillow', 'pytest-cov', 'coverage', 'ccache', 'pytorch 1.13.*', 'pytorch-cpu', 'array-api-compat']' returned non-zero exit status 1.

lesteve · 2023-04-27T15:34:10Z

Hmmm weird I can try to take a closer look but probably not before tomorrow afternoon.

A possible work-around is to update only a selected build, something like:

python build_tools/update_environments_and_lock_files.py --select-build debian

adrinjalali · 2023-04-27T15:44:34Z

@lesteve that fails too:

$ python build_tools/update_environments_and_lock_files.py --select-build conda_forge_mkl
Writing conda environments
Writing conda lock files
pylatest_conda_forge_mkl_linux-64
Traceback (most recent call last):
  File "/home/adrin/Projects/sklearn/scikit-learn/build_tools/update_environments_and_lock_files.py", line 611, in <module>
    main()
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/adrin/miniforge3/envs/sklearn/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/adrin/Projects/sklearn/scikit-learn/build_tools/update_environments_and_lock_files.py", line 597, in main
    write_all_conda_lock_files(filtered_conda_build_metadata_list)
  File "/home/adrin/Projects/sklearn/scikit-learn/build_tools/update_environments_and_lock_files.py", line 498, in write_all_conda_lock_files
    create_conda_lock_file(build_metadata)
  File "/home/adrin/Projects/sklearn/scikit-learn/build_tools/update_environments_and_lock_files.py", line 492, in create_conda_lock_file
    conda_lock(environment_path, lock_file_path, platform)
  File "/home/adrin/Projects/sklearn/scikit-learn/build_tools/update_environments_and_lock_files.py", line 479, in conda_lock
    execute_command(shlex.split(command))
  File "/home/adrin/Projects/sklearn/scikit-learn/build_tools/update_environments_and_lock_files.py", line 398, in execute_command
    raise RuntimeError(
RuntimeError: Command exited with non-zero exit code.
Exit code: 1
Command:
conda-lock lock --mamba --kind explicit --platform linux-64 --file build_tools/azure/pylatest_conda_forge_mkl_linux-64_environment.yml --filename-template build_tools/azure/pylatest_conda_forge_mkl_linux-64_conda.lock

lesteve · 2023-04-27T16:04:42Z

What I was suggesting as a short term option to get this PR merged, is to only update the lock file that need updating due to pytest min dependency bump or and give up on the problematic ones 😉

But it looks like there are no conflicts anymore so this is good enough I think

adrinjalali · 2023-04-27T17:07:29Z

There are no conflicts here cause I've used the file from main, the file needs an update before merging this PR.

lesteve · 2023-04-28T04:33:12Z

All I am trying to advocate here is to update only the debian_32 lock file with --select-build in this PR since bumping minimum pytest dependency seems like a blocker in #26091 AFAIU.

Investigating conda-lock issues about updating the other lock files can be tackled in a separate issue.

build_tools/update_environments_and_lock_files.py

lesteve · 2023-04-28T05:55:31Z

FYI I opened conda/conda-lock#401 about the tzdata issue, since I believe this is a slightly different one than the other mixed conda/pip dependencies issue.

glemaitre · 2023-05-02T12:58:57Z

Wonderful, we need Rust to build our doc now ;). I am going to debug a bit to understand why when installing conda-lock some stuff needs to be rebuilt using rust.

thomasjpfan · 2023-05-02T15:06:06Z

From looking at the lock files, pywinpty is a new dependency for the doc building that is a source distribution, which requires a rust compiler to build. From looking at pywinpty's PyPI files, it has pre-build wheels only for Windows, so conda-lock is picking up the source distribution on Linux.

I suspect that pywinpty should be not installed on Linux anyways, but one of the doc building dependencies is depending on it.

glemaitre · 2023-05-02T15:10:08Z

Uhm but the issue happens when installing conda-lock so before installing any dependencies, isn't it?

glemaitre · 2023-05-02T15:23:27Z

Uhm you are right, this is the line just below when invoking conda-lock. So I assume that we need to find which of the dependency is the culprit.

glemaitre · 2023-05-02T18:44:43Z

So it seems that jupyter-server (that is a dependence of jupyterlite-pyiode-kernel) should have a link to pywinpty. I don't really understand why conda-lock pick up the package because it is declared with an exception depending on the OS:

https://github.com/jupyter-server/jupyter_server/blob/main/pyproject.toml#L40

Maybe there is a bug there.

glemaitre · 2023-05-03T09:13:48Z

Some partial info here. I am digging into conda-lock source code and the vendor version of poetry. It seems that the issue is linked to the PyPI solver from poetry that adds pywinpty while the environment created by conda-lock is a linux platform.

I think that we could merge the current PR since it is green. I deleted by ending the pywinpty package from the lock file.

My idea is to find the bug and it should be solved upstream such that we don't have anything to do in our repository.

If anyone is convinced about this approach, feel free to merge this PR.

glemaitre · 2023-05-03T16:31:37Z

Actually, the problem was already brought upstream by @lesteve: conda/conda-lock#336 (comment) and solve in main.

we would need only a release of conda-lock then.

adrinjalali · 2023-05-03T17:46:22Z

Confused, if we're to merge before conda-lock release, then we need rust, but it's commented out right now. What am I missing?

glemaitre · 2023-05-04T09:17:39Z

Confused, if we're to merge before conda-lock release, then we need rust, but it's commented out right now. What am I missing?

Before the release of conda-lock then we need to manually edit the lock file to remove pywinpty because it is not required (and bring trouble with the rust stuff).

So we can merge it now. But if someone makes an update of the lock file, we need to think again to remove the pywinpty.

adrinjalali · 2023-05-04T12:02:28Z

So we can merge it now. But if someone makes an update of the lock file, we need to think again to remove the pywinpty.

Then could you please make the update script to print out this information for whoever runs the script?

thomasjpfan · 2023-05-04T13:33:47Z

I prefer to have the script remove pywinpty from the lock file with a comment explaining the situation. This way we do not need to manually do it each time the lock files are updated.

glemaitre · 2023-05-04T14:14:08Z

I prefer to have the script remove pywinpty from the lock file with a comment explaining the situation. This way we do not need to manually do it each time the lock files are updated.

Will do.

adrinjalali

Otherwise I think we can merge.

adrinjalali · 2023-05-04T14:52:34Z

build_tools/circle/build_doc.sh

+# # Install Rust
+# # TODO: needs to find out which package requires this
+# curl https://sh.rustup.rs -sSf > rustup-init.sh
+# bash rustup-init.sh -y
+# source "$HOME/.cargo/env"`
+


so remove this?

build_tools/circle/build_doc.sh

MAINT bump minimum version for pytest

a143fd9

glemaitre mentioned this pull request Apr 14, 2023

FEA Implementation of "threshold-dependent metric per threshold value" curve #25639

Open

7 tasks

lorentzenchr approved these changes Apr 16, 2023

View reviewed changes

betatim approved these changes Apr 17, 2023

View reviewed changes

glemaitre mentioned this pull request Apr 17, 2023

MAINT Parameters validation for graph.single_source_shortest_path_length #26091

Merged

glemaitre added 2 commits April 17, 2023 11:44

iter

ceeb177

use defaults as previous file

05bc7cb

adrinjalali reviewed Apr 17, 2023

View reviewed changes

lesteve added 4 commits April 18, 2023 10:52

Ignore deprecation warnings due to pyamg using pkg_resources

e0c0828

fix

07a57d5

fix

8413a44

[azure parallel] trigger CI

f74116a

ogrisel approved these changes Apr 25, 2023

View reviewed changes

adrinjalali and others added 3 commits April 27, 2023 15:44

Update build_tools/update_environments_and_lock_files.py

2e4f935

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

Update build_tools/azure/test_script.sh

edf15b8

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

Merge remote-tracking branch 'upstream/main' into bump_pytest_version

e3e7ae2

lesteve reviewed Apr 28, 2023

View reviewed changes

build_tools/update_environments_and_lock_files.py Outdated Show resolved Hide resolved

lesteve mentioned this pull request Apr 28, 2023

Issue with mixed conda/pip dependencies when a name does not mean the same package in conda and pip conda/conda-lock#401

Closed

2 tasks

glemaitre added 5 commits May 2, 2023 15:06

why not

356a4d9

iter

d864831

iter

a1192d3

iter

23cee8b

debug

2436e51

iter

12ff705

iter

bdd8e01

manually remove pywinpty

221c70d

Merge remote-tracking branch 'origin/main' into bump_pytest_version

c9fbae8

remove a given line

cd87446

adrinjalali reviewed May 4, 2023

View reviewed changes

glemaitre commented May 5, 2023

View reviewed changes

build_tools/circle/build_doc.sh Outdated Show resolved Hide resolved

Update build_tools/circle/build_doc.sh

40feac6

adrinjalali merged commit fa5f43f into scikit-learn:main May 5, 2023

lesteve mentioned this pull request May 24, 2023

MNT Fix regression in lock file update script when using --select-build #26427

Merged

adrinjalali mentioned this pull request Jun 6, 2023

CI Update to conda-lock 2.0.0 #26510

Merged

MAINT bump minimum version for pytest #26184

MAINT bump minimum version for pytest #26184

Conversation

glemaitre commented Apr 14, 2023

glemaitre commented Apr 14, 2023

lorentzenchr left a comment

Choose a reason for hiding this comment

glemaitre commented Apr 17, 2023

lesteve commented Apr 17, 2023 • edited Loading

glemaitre commented Apr 17, 2023

glemaitre commented Apr 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogrisel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrinjalali commented Apr 27, 2023

lesteve commented Apr 27, 2023

adrinjalali commented Apr 27, 2023

lesteve commented Apr 27, 2023 • edited Loading

adrinjalali commented Apr 27, 2023

lesteve commented Apr 28, 2023

lesteve commented Apr 28, 2023

glemaitre commented May 2, 2023 • edited Loading

thomasjpfan commented May 2, 2023 • edited Loading

glemaitre commented May 2, 2023

glemaitre commented May 2, 2023 • edited Loading

glemaitre commented May 2, 2023

glemaitre commented May 3, 2023

glemaitre commented May 3, 2023

adrinjalali commented May 3, 2023

glemaitre commented May 4, 2023

adrinjalali commented May 4, 2023

thomasjpfan commented May 4, 2023

glemaitre commented May 4, 2023

adrinjalali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lesteve commented Apr 17, 2023 •

edited

Loading

lesteve commented Apr 27, 2023 •

edited

Loading

glemaitre commented May 2, 2023 •

edited

Loading

thomasjpfan commented May 2, 2023 •

edited

Loading

glemaitre commented May 2, 2023 •

edited

Loading