Skip to content

MNT update Cython 3.0.10 to 3.1.2 #31905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Aug 18, 2025
Merged

Conversation

lorentzenchr
Copy link
Member

@lorentzenchr lorentzenchr commented Aug 9, 2025

Reference Issues/PRs

This will help #31882 a bit. Similar to #28743.

What does this implement/fix? Explain your changes.

Update minimum Cython from 3.0.10 to 3.1.2.

Any other comments?

Some bugfixes (e.g. fused ctypes) and some nice features that we want to have sooner or later (e.g. Support for freethreading builds of CPython 3.13).

- fixes for fused ctuples
Copy link

github-actions bot commented Aug 9, 2025

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: 0a419be. Link to the linter CI: here

Copy link
Member

@thomasjpfan thomasjpfan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you push a commit with [cd build] to check that the wheels build correctly?

@lorentzenchr lorentzenchr added cython Quick Review For PRs that are quick to review dependencies Pull requests that update a dependency file labels Aug 11, 2025
@adrinjalali
Copy link
Member

lock file conflicts, could you please rerender them?

@adrinjalali adrinjalali enabled auto-merge (squash) August 11, 2025 09:48
@lesteve
Copy link
Member

lesteve commented Aug 11, 2025

I would wait for #31918 to be merged first and for now there seems to be a OpenML issue that prevents the doc build to be succesful.

Also can you use python build_tools/update_environment_and_lock_files.py --select-build '<some_regex>' to update only the lock-files you care about? I guess these ones from a quick git grep:

❯ git grep -P 'cython.+3\.0' build_tools/**/*lock*
build_tools/azure/pymin_conda_forge_openblas_min_dependencies_linux-64_conda.lock:https://conda.anaconda.org/conda-forge/linux-64/cython-3.0.10-py310hc6cd4ac_0.conda#bd1d71ee240be36f1d85c86177d6964f
build_tools/azure/ubuntu_atlas_lock.txt:cython==3.0.10
build_tools/circle/doc_min_dependencies_linux-64_conda.lock:https://conda.anaconda.org/conda-forge/linux-64/cython-3.0.10-py310hc6cd4ac_0.conda#bd1d71ee240be36f1d85c86177d6964f

When you update all the lock-files, there are more things to check like:

  • push a commit with [doc build] if doc lock-files are touched to make sure the full doc build passes
  • same thing with [scipy-dev] if scipy-dev lock-file is changed
  • label the PR with "CUDA CI" if CUDA CI lock-file is changed
  • maybe more things ...

@lesteve lesteve disabled auto-merge August 11, 2025 09:52
@adrinjalali
Copy link
Member

@StefanieSenger since you've been working on GH actions and CI, maybe you fancy implementing a secure @sklearn-bot please re-render kinda thing like what we had but disabled.

@lesteve
Copy link
Member

lesteve commented Aug 11, 2025

I am -0.5 on reviving the "update lock-file bot" attempt (too little functionality for too many security risks) but we can discuss this elsewhere ...

@lesteve
Copy link
Member

lesteve commented Aug 11, 2025

Just for completeness you would need a commit with [doc build] to merge this since you are updating the doc lock-files. The CI will likely be red because there are OpenML issues right now ...

@lorentzenchr
Copy link
Member Author

Just for completeness you would need a commit with [doc build] to merge this since you are updating the doc lock-files. The CI will likely be red because there are OpenML issues right now ...

Let's try.

@lesteve
Copy link
Member

lesteve commented Aug 18, 2025

I think you still need to update the doc-min-dependencies lock-file to have cython 3.1.2:

python build_tools/update_environments_and_lock_files.py --select-build doc_min_dependencies

and then push a commit with [doc build] to make sure that the full doc-min-dependencies actually work.

OpenML seems to work fine now at least it did this morning in #31918.

…-build doc_min_dependencies [cd build] [scipy-dev] [doc build]
@lorentzenchr
Copy link
Member Author

@lesteve Either the one test failure is unrelated to this PR, or, I'm about to give up.

Details on failing Wheel builder / Build wheel for cp313-win_amd64- (pull_request)

...
Build options:
  platform: windows
  allow_empty: False
  architectures: AMD64, ARM64, x86
  build_selector: 
    build_config: cp313-win_amd64
    skip_config: 
    requires_python: >=3.10
    enable: []
...
Building cp313-win_amd64 wheel
CPython 3.13 Windows 64bit
...

Installing Python cp313...
                                                             ✓ 12.46s
Setting up build environment...
                                                              ✓ 7.28s
Installing build tools...
                                                              ✓ 8.48s
Running before_build...
                                                              ✓ 0.05s
Building wheel...
                                                            ✓ 169.17s
Repairing wheel...
                                                              ✓ 3.75s
Testing wheel... [THIS ONE FAILS]

System:
      python: 3.13.7 (tags/v3.13.7:bcee1c3, Aug 14 2025, 14:15:11) [MSC v.1944 64 bit (AMD64)]
  executable: C:\Python\python.exe
     machine: Windows-2022Server-10.0.20348-SP0
  
  Python dependencies:
        sklearn: 1.8.dev0
            pip: 25.2
     setuptools: None
          numpy: 2.3.2
          scipy: 1.16.1
         Cython: None
         pandas: 2.3.1
     matplotlib: None
         joblib: 1.5.1
  threadpoolctl: 3.6.0
  
  Built with OpenMP: True
  
  threadpoolctl info:
         user_api: openmp
     internal_api: openmp
      num_threads: 4
           prefix: vcomp
         filepath: C:\Python\Lib\site-packages\sklearn\.libs\vcomp140.dll
          version: None
  
         user_api: blas
     internal_api: openblas
      num_threads: 4
           prefix: libscipy_openblas
         filepath: C:\Python\Lib\site-packages\numpy.libs\libscipy_openblas64_-860d95b1c38e637ce4509f5fa24fbf2a.dll
          version: 0.3.30
  threading_layer: pthreads
     architecture: Haswell
  
         user_api: blas
     internal_api: openblas
      num_threads: 4
           prefix: libscipy_openblas
         filepath: C:\Python\Lib\site-packages\scipy.libs\libscipy_openblas-6b2103f2ae4d8547998b5d188e9801fb.dll
          version: 0.3.28
  threading_layer: pthreads
     architecture: Haswell
  + docker container run -e SKLEARN_SKIP_NETWORK_TESTS=1 --rm scikit-learn/minimal-windows powershell -Command 'pytest --pyargs sklearn'
  ============================= test session starts =============================
...
================================== FAILURES ===================================
  __________________________ test_parallel[42-float64] __________________________
  
  global_dtype = <class 'numpy.float64'>, global_random_seed = 42
  
      def test_parallel(global_dtype, global_random_seed):
          centers = np.array([[1, 1], [-1, -1], [1, -1]]) + 10
          X, _ = make_blobs(
              n_samples=50,
              n_features=2,
              centers=centers,
              cluster_std=0.4,
              shuffle=True,
              random_state=global_random_seed,
          )
      
          X = X.astype(global_dtype, copy=False)
      
          ms1 = MeanShift(n_jobs=2)
  >       ms1.fit(X)
  
  Python\Lib\site-packages\sklearn\cluster\tests\test_mean_shift.py:95: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  Python\Lib\site-packages\sklearn\base.py:1366: in wrapper
      return fit_method(estimator, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Python\Lib\site-packages\sklearn\cluster\_mean_shift.py:506: in fit
      all_res = Parallel(n_jobs=self.n_jobs)(
  Python\Lib\site-packages\sklearn\utils\parallel.py:82: in __call__
      return super().__call__(iterable_with_config_and_warning_filters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Python\Lib\site-packages\joblib\parallel.py:1977: in __call__
      n_jobs = self._initialize_backend()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  Python\Lib\site-packages\joblib\parallel.py:1379: in _initialize_backend
      n_jobs = self._backend.configure(
  Python\Lib\site-packages\joblib\_parallel_backends.py:639: in configure
      self._workers = get_memmapping_executor(
  Python\Lib\site-packages\joblib\executor.py:18: in get_memmapping_executor
      return MemmappingExecutor.get_memmapping_executor(n_jobs, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Python\Lib\site-packages\joblib\executor.py:48: in get_memmapping_executor
      manager = TemporaryResourcesManager(temp_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Python\Lib\site-packages\joblib\_memmapping_reducer.py:601: in __init__
      self.set_current_context(context_id)
  Python\Lib\site-packages\joblib\_memmapping_reducer.py:605: in set_current_context
      self.register_new_context(context_id)
  Python\Lib\site-packages\joblib\_memmapping_reducer.py:627: in register_new_context
      self.register_folder_finalizer(new_folder_path, context_id)
  Python\Lib\site-packages\joblib\_memmapping_reducer.py:643: in register_folder_finalizer
      resource_tracker.register(pool_subfolder, "folder")
  Python\Lib\multiprocessing\resource_tracker.py:244: in register
      self._send('REGISTER', name, rtype)
  Python\Lib\multiprocessing\resource_tracker.py:261: in _send
      self._ensure_running_and_write(msg)
  Python\Lib\multiprocessing\resource_tracker.py:220: in _ensure_running_and_write
      self._launch()
  Python\Lib\multiprocessing\resource_tracker.py:185: in _launch
      pid = util.spawnv_passfds(exe, args, fds_to_pass)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  path = 'C:\\Python\\python.exe'
  args = ['C:\\Python\\python.exe', '-c', 'from multiprocessing.resource_tracker import main;main(12)']
  passfds = [9, 12]
  
      def spawnv_passfds(path, args, passfds):
  >       import _posixsubprocess
  E       ModuleNotFoundError: No module named '_posixsubprocess'
  
  Python\Lib\multiprocessing\util.py:515: ModuleNotFoundError
...
  =========================== short test summary info ===========================
  FAILED Python/Lib/site-packages/sklearn/cluster/tests/test_mean_shift.py::test_parallel[42-float64]
  FAILED Python/Lib/site-packages/sklearn/compose/tests/test_column_transformer.py::test_column_transformer_auto_memmap[42]
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_dict_learning.py::test_sparse_encode_shapes_omp
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_dict_learning.py::test_dict_learning_reconstruction_parallel
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_dict_learning.py::test_dict_learning_lassocd_readonly_data
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_dict_learning.py::test_sparse_coder_parallel_mmap
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_dict_learning.py::test_cd_work_on_joblib_memmapped_data
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_online_lda.py::test_lda_multi_jobs[online-csr_matrix]
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_online_lda.py::test_lda_multi_jobs[online-csr_array]
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_online_lda.py::test_lda_multi_jobs[batch-csr_matrix]
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_online_lda.py::test_lda_multi_jobs[batch-csr_array]
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_online_lda.py::test_lda_partial_fit_multi_jobs[csr_matrix]
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_online_lda.py::test_lda_partial_fit_multi_jobs[csr_array]
  FAILED Python/Lib/site-packages/sklearn/decomposition/tests/test_sparse_pca.py::test_fit_transform_parallel[42]
  FAILED Python/Lib/site-packages/sklearn/ensemble/tests/test_bagging.py::test_parallel_classification
  FAILED Python/Lib/site-packages/sklearn/ensemble/tests/test_bagging.py::test_parallel_regression
  FAILED Python/Lib/site-packages/sklearn/ensemble/tests/test_bagging.py::test_estimator
  FAILED Python/Lib/site-packages/sklearn/ensemble/tests/test_forest.py::test_backend_respected
  FAILED Python/Lib/site-packages/sklearn/ensemble/tests/test_voting.py::test_parallel_fit[42]
  FAILED Python/Lib/site-packages/sklearn/feature_selection/tests/test_mutual_info.py::test_mutual_info_n_jobs[42-mutual_info_regression-make_regression]
  FAILED Python/Lib/site-packages/sklearn/feature_selection/tests/test_mutual_info.py::test_mutual_info_n_jobs[42-mutual_info_classif-make_classification]
  FAILED Python/Lib/site-packages/sklearn/feature_selection/tests/test_rfe.py::test_rfe_cv_n_jobs[42]
  FAILED Python/Lib/site-packages/sklearn/feature_selection/tests/test_rfe.py::test_rfe_with_joblib_threading_backend[42]
  FAILED Python/Lib/site-packages/sklearn/feature_selection/tests/test_rfe.py::test_results_per_cv_in_rfecv[42]
  FAILED Python/Lib/site-packages/sklearn/gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel0]
  FAILED Python/Lib/site-packages/sklearn/gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel1]
  FAILED Python/Lib/site-packages/sklearn/gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel2]
  FAILED Python/Lib/site-packages/sklearn/gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel3]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression[None-0.5-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression[None-1.0-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression[ones-0.5-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression[ones-1.0-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression_pandas[0.5-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression_pandas[1.0-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_robustness_to_high_cardinality_noisy_feature[0.5-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_robustness_to_high_cardinality_noisy_feature[1.0-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_equivalence_sequential_parallel[500]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_equivalence_sequential_parallel[1.0]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_equivalence_array_dataframe[0.5-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_equivalence_array_dataframe[1.0-2]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_large_memmaped_data[array]
  FAILED Python/Lib/site-packages/sklearn/inspection/tests/test_permutation_importance.py::test_permutation_importance_large_memmaped_data[dataframe]
  FAILED Python/Lib/site-packages/sklearn/linear_model/tests/test_coordinate_descent.py::test_linear_models_cv_fit_with_loky[ElasticNetCV]
  FAILED Python/Lib/site-packages/sklearn/linear_model/tests/test_coordinate_descent.py::test_linear_models_cv_fit_with_loky[LassoCV]
  FAILED Python/Lib/site-packages/sklearn/linear_model/tests/test_sgd.py::test_multi_core_gridsearch_and_early_stopping
  FAILED Python/Lib/site-packages/sklearn/linear_model/tests/test_theil_sen.py::test_theil_sen_parallel
  FAILED Python/Lib/site-packages/sklearn/manifold/tests/test_mds.py::test_MDS
  FAILED Python/Lib/site-packages/sklearn/metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring0]
  FAILED Python/Lib/site-packages/sklearn/metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring1]
  FAILED Python/Lib/site-packages/sklearn/metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring2]
  FAILED Python/Lib/site-packages/sklearn/metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring3]
  FAILED Python/Lib/site-packages/sklearn/metrics/tests/test_pairwise.py::test_sparse_manhattan_readonly_dataset[csr_matrix]
  FAILED Python/Lib/site-packages/sklearn/metrics/tests/test_pairwise.py::test_sparse_manhattan_readonly_dataset[csr_array]
  FAILED Python/Lib/site-packages/sklearn/model_selection/tests/test_search.py::test_empty_cv_iterator_error
  FAILED Python/Lib/site-packages/sklearn/model_selection/tests/test_search.py::test_random_search_bad_cv
  FAILED Python/Lib/site-packages/sklearn/model_selection/tests/test_search.py::test_search_with_2d_array
  FAILED Python/Lib/site-packages/sklearn/model_selection/tests/test_validation.py::test_cross_validate_many_jobs
  FAILED Python/Lib/site-packages/sklearn/model_selection/tests/test_validation.py::test_cross_validate_return_indices[42]
  FAILED Python/Lib/site-packages/sklearn/neighbors/tests/test_kd_tree.py::test_kdtree_picklable_with_joblib[KDTree64]
  FAILED Python/Lib/site-packages/sklearn/neighbors/tests/test_kd_tree.py::test_kdtree_picklable_with_joblib[KDTree32]
  FAILED Python/Lib/site-packages/sklearn/neighbors/tests/test_neighbors.py::test_knn_forcing_backend[ball_tree-loky]
  FAILED Python/Lib/site-packages/sklearn/neighbors/tests/test_neighbors.py::test_knn_forcing_backend[kd_tree-loky]
  FAILED Python/Lib/site-packages/sklearn/neighbors/tests/test_neighbors.py::test_knn_forcing_backend[auto-loky]
  FAILED Python/Lib/site-packages/sklearn/tests/test_calibration.py::test_parallel_execution[True-sigmoid]
  FAILED Python/Lib/site-packages/sklearn/tests/test_calibration.py::test_parallel_execution[True-isotonic]
  FAILED Python/Lib/site-packages/sklearn/tests/test_calibration.py::test_parallel_execution[True-temperature]
  FAILED Python/Lib/site-packages/sklearn/tests/test_calibration.py::test_parallel_execution[False-sigmoid]
  FAILED Python/Lib/site-packages/sklearn/tests/test_calibration.py::test_parallel_execution[False-isotonic]
  FAILED Python/Lib/site-packages/sklearn/tests/test_calibration.py::test_parallel_execution[False-temperature]
  FAILED Python/Lib/site-packages/sklearn/tests/test_config.py::test_config_threadsafe_joblib[loky]
  FAILED Python/Lib/site-packages/sklearn/tests/test_config.py::test_config_threadsafe_joblib[multiprocessing]
  FAILED Python/Lib/site-packages/sklearn/tests/test_multioutput.py::test_multi_output_classification_partial_fit_parallelism
  FAILED Python/Lib/site-packages/sklearn/tests/test_pipeline.py::test_feature_union_parallel
  FAILED Python/Lib/site-packages/sklearn/utils/tests/test_parallel.py::test_configuration_passes_through_to_joblib[loky-2]
  FAILED Python/Lib/site-packages/sklearn/utils/tests/test_parallel.py::test_configuration_passes_through_to_joblib[multiprocessing-2]
  FAILED Python/Lib/site-packages/sklearn/utils/tests/test_parallel.py::test_dispatch_config_parallel[2]
  FAILED Python/Lib/site-packages/sklearn/utils/tests/test_parallel.py::test_filter_warning_propagates[loky-2]
  FAILED Python/Lib/site-packages/sklearn/utils/tests/test_parallel.py::test_filter_warning_propagates[multiprocessing-2]
  FAILED Python/Lib/site-packages/sklearn/utils/tests/test_parallel.py::test_filter_warning_propagates_no_side_effect_with_loky_backend
  = 79 failed, 34043 passed, 6152 skipped, 152 xfailed, 65 xpassed, 4316 warnings in 785.84s (0:13:05) =

@lesteve
Copy link
Member

lesteve commented Aug 18, 2025

The wheels failure is unrelated (it was noticed in main see #31955), so I am going to merge this one.

Full disclosure: I am not sure it was needed to test the wheels for this PR (I would think that the latest cython version is always used in the wheels build), but maybe I am missing something.

@lesteve lesteve merged commit 2831228 into scikit-learn:main Aug 18, 2025
60 of 61 checks passed
@lorentzenchr lorentzenchr deleted the update_cython branch August 18, 2025 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cython dependencies Pull requests that update a dependency file No Changelog Needed Quick Review For PRs that are quick to review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants