Add support for Python 3.13 free-threaded build #28978

ngoldbaum · 2024-05-08T16:51:30Z

I'm currently working on adding support for the Python 3.13 free-threaded build to projects in the scientific python ecosystem. We are tracking this work at https://github.com/Quansight-Labs/free-threaded-compatibility. Right now we're focusing on projects relatively low in the stack. scikit-learn isn't the lowest in the stack but it has a lot of tests that perform multithreaded workflows so running the scikit-learn tests is a productive way to elucidate threading bugs in scikit-learn and its dependencies.

Currently, scikit-learn builds fine and almost all the tests pass with the GIL disabled 🎉

The following tests have failures:

FAILED sklearn/ensemble/tests/test_voting.py::test_sample_weight[42] - AssertionError:
FAILED sklearn/model_selection/tests/test_search.py::test_random_search_cv_results_multimetric - AssertionError:
FAILED sklearn/semi_supervised/tests/test_self_training.py::test_classification[threshold-base_estimator1] - AssertionError:
FAILED sklearn/semi_supervised/tests/test_self_training.py::test_classification[k_best-base_estimator1] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-linear-X_train0-y_train0-X_test0] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-linear-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-linear-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-poly-X_train0-y_train0-X_test0] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-poly-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-poly-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-rbf-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-rbf-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-sigmoid-X_train0-y_train0-X_test0] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-sigmoid-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_matrix-sigmoid-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-linear-X_train0-y_train0-X_test0] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-linear-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-linear-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-poly-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-poly-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-rbf-X_train0-y_train0-X_test0] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-rbf-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-rbf-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-sigmoid-X_train0-y_train0-X_test0] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-sigmoid-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[csr_array-sigmoid-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-linear-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-linear-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-poly-X_train0-y_train0-X_test0] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-poly-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-poly-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-rbf-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-rbf-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-sigmoid-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_matrix-sigmoid-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-linear-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-linear-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-poly-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-poly-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-rbf-X_train0-y_train0-X_test0] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-rbf-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-rbf-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-sigmoid-X_train2-y_train2-X_test2] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_svc[lil_array-sigmoid-X_train3-y_train3-X_test3] - AssertionError:
FAILED sklearn/svm/tests/test_sparse.py::test_consistent_proba - AssertionError:
FAILED sklearn/svm/tests/test_svm.py::test_libsvm_iris - AssertionError:
FAILED sklearn/svm/tests/test_svm.py::test_svc_clone_with_callable_kernel - AssertionError:
FAILED sklearn/svm/tests/test_svm.py::test_consistent_proba - AssertionError:
FAILED sklearn/tests/test_build.py::test_openmp_parallelism_enabled - AssertionError:
FAILED sklearn/tests/test_pipeline.py::test_pipeline_memory - AssertionError:

Note that some of these failures are flaky and you may not get the same list if you run the scikit-learn tests. I also saw one deadlock in test_calibration.py::test_parallel_execution, so I'd also suggest running pytest with e.g. -o fauthandler_timeout=60 set so you will get python tracebacks for all threads if any test hangs and runs longer than 60 seconds. Your mileage may vary on how fast your computer is, so 60 seconds may not be sufficient if any tests take that long to run on your system.

My plan is to go through these failures and figure out what's going wrong. Some of these may not be specific to the free-threaded build.

To run the tests with the GIL disabled, you'll need to install 3.13b1 or the latest version of the CPython 3.13 branch, build cython, numpy, and scipy from the latest version of their respective git repositories. You'll need to e.g. pip install pybind11 and pythran after installing numpy and before building scipy. Then build scikit-learn as an editable install and run:

PYTHON_GIL=0 python -m pytest sklearn

The text was updated successfully, but these errors were encountered:

ogrisel · 2024-05-10T08:05:08Z

Thanks for your investigation.

My plan is to go through these failures and figure out what's going wrong. Some of these may not be specific to the free-threaded build.

Indeed we do not have a Python 3.13 build yet so maybe some of those failures are related to that.

To run the tests with the GIL disabled, you'll need to install 3.13b1 or the latest version of the CPython 3.13 branch, build cython, numpy, and scipy from the latest version of their respective git repositories.

That would probably be too long to build all of those dependencies from source on the CI. Maybe upstream packages such as numpy and scipy could publish nightly builds for Python 3.13? As far as I know this is not the case yet:

However, according to https://peps.python.org/pep-0703/#python-build-modes, there are two CPython ABIs for the foreseeable future and I am not sure how those two ABIs can be reflected in wheel filenames.

https://peps.python.org/pep-0703/#distribution mentions a dedicated conda channel but I am not sure about the status of this channel.

ogrisel · 2024-05-10T08:12:36Z

In any case, feel free to open PRs to address some of those failures you discovered even if we don't have a nogil CPython 3.13.X CI yet. We can merge them even if they can only be properly tested manually for now.

Note that we do have a 3.9 nogil CI running nightly on Azure Pipelines and it has been mostly green for a year or more. But I assume that 3.13 nogil has evolved a bit from 3.9 nogil hence the new failures.

ngoldbaum · 2024-05-10T16:55:09Z

Indeed we do not have a Python 3.13 build yet so maybe some of those failures are related to that.

Great, I might look at setting that up along with fixing the 3.13-specific issues.

Testing the free-threaded build will need to wait a little bit. I'm hoping that the setup-python action will gain support for installing it via a configuration flag (actions/setup-python#771). If that ends up not being quickly fixed now that the first beta is out, I'll look at setting up an Ubuntu-based CI job using the deadsnakes repo (once that gets updated with a 3.13 free-threaded build). See @hugovk's blog post.

However, according to https://peps.python.org/pep-0703/#python-build-modes, there are two CPython ABIs for the foreseeable future and I am not sure how those two ABIs can be reflected in wheel filenames.

There is a new ABI tag (3.13t), so you'd need to upload two python 3.13 wheels, one with the normal GIL-enabled ABI and one with the free-threaded ABI.

Also, one big issue is that pip doesn't know about the ABI tag yet (pypa/pip#12634). However that should be fixed in pip 24.1.

https://peps.python.org/pep-0703/#distribution mentions a dedicated conda channel but I am not sure about the status of this channel.

I don't think this has been set up yet. The last I heard the Anaconda Inc team was planning to start work on free-threaded support around the release of the first beta (so, nowish).

In any case, feel free to open PRs to address some of those failures you discovered even if we don't have a nogil CPython 3.13.X CI yet. We can merge them even if they can only be properly tested manually for now.

Great, thanks for this. Would creating a nogil or free-threaded label to tag issues and PRs with be OK?

ngoldbaum · 2024-05-10T17:21:04Z

Note that we do have a 3.9 nogil CI running nightly on Azure Pipelines and it has been mostly green for a year or more. But I assume that 3.13 nogil has evolved a bit from 3.9 nogil hence the new failures.

And it probably makes sense to retire this CI job since the 3.13 free-threaded build should replace it and we've been upstreaming support using the Py_GIL_DISABLED macro, which isn't used in the 3.9 nogil fork.

lesteve · 2024-05-13T00:49:52Z

I am Interested to help on this, don't hesitate to ping me on this kind of topics! Together with @ogrisel I was involved with setting up the scikit-learn CI for the nogil Python 3.9 and opening upstream bugs when we encountered them.

I have created the no-GIL label (following the same naming as scipy and numpy for consistency although I have always used nogil personally). Note that label search will match "nogil" and "free-threaded" since both terms appear in the label description.

Issues and PRs will need manual labeling by a scikit-learn maintainer for now, but I guess this is good enough for now, let me know if you have better suggestions!

hugovk · 2024-05-13T09:42:33Z

Naming things: when accepting PEP 703, the Steering Council said they wanted us to use a positive term like "free-threaded" rather than negative terms like "no-GIL":

We want to avoid negatives in terms and flags and such, so we won’t get into double-negative terrain (like we do when we talk about ‘non no-GIL’). We’d like a positive, clear term to talk about the no-GIL build, and we’re suggesting ‘free-threaded’.

And we use "topic-free-threading" for CPython:

https://github.com/python/cpython/labels/topic-free-threading

lesteve · 2024-05-13T10:27:16Z

Fair enough, I renamed the label to "free-threading", personally I don't care too much as long as I can find the label easily in the labels search for example like in the screenshot below (finding free-threaded label when searching nogil):

You may want to advocate for a similar change in numpy and scipy 😉

Edit: contrary to the screenshot, I ended up using free-threading (and not free-threaded) to match CPython.

lesteve · 2024-05-13T10:54:48Z

That would probably be too long to build all of those dependencies from source on the CI. Maybe upstream packages such as numpy and scipy could publish nightly builds for Python 3.13?

This seems definitely desirable. Alternatively in the shorter term, if there was a Docker image (even in an experimental state) with Python free-threaded build + latest numpy release and scipy release, this may make it more likely that curious but busy people (for example me) give it a go.

ogrisel · 2024-05-14T09:09:23Z

with Python free-threaded build + latest numpy release and scipy release

It could even be a nightly/weekly updated docker image with the dev versions of numpy and scipy. If a free docker hub account is enough we could use the auto-build feature of docker hub or similar:

https://docs.docker.com/docker-hub/builds/

Not sure about build time limit/usage quotas though.

EDIT: still I would prefer installing from a pip wheel house, e.g. https://pypi.anaconda.org/scientific-python-nightly-free-threaded/ to workaround any pip ABI tag conflicts that would prevent us to directly reuse https://pypi.anaconda.org/scientific-python-nightly-wheels/ .

ngoldbaum · 2024-05-14T16:48:42Z

to workaround any pip ABI tag conflicts that would prevent us to directly reuse https://pypi.anaconda.org/scientific-python-nightly-wheels/

There's a new ABI tag, so the new nightly wheels can live alongside the old ones on https://pypi.anaconda.org/scientific-python-nightly-wheels/. We just need to wait for a new pip release that can recognize and install the new ABI tag and for the the actual wheels to be produced. I'd like to look at that for NumPy at least later this week.

ngoldbaum · 2024-05-14T16:51:58Z

Ah but we probably will need to wait on cibuildwheel support first: pypa/cibuildwheel#1657

So maybe a docker image would be best until beta 2?

rgommers · 2024-05-14T17:05:05Z

if a free docker hub account is enough we could use the auto-build feature of docker hub or similar:

Just a note that we got rid of Docker images and weekly pushes to Docker Hub in NumPy and SciPy after they limited their free plan (and the machinery was a bit of a pain to begin with). Setting that up again isn't worth it imho. CPython 3.13 beta 2 is only two weeks away (May 28th), so I'd wait for that and if you want something in the meantime then building a one-off Docker image by hand to use for a few weeks until nightly wheels arrive should be okay I'd think. Investing time in auto-building Docker images is probably not the most productive way of spending time.

betatim · 2024-05-15T06:55:42Z

👍 for being patient for two weeks instead of building something "bespoke" that we should then un-build again.

ogrisel · 2024-05-15T07:56:01Z

We just need to wait for a new pip release that can recognize and install the new ABI tag and for the the actual wheels to be produced.

We would also need to make sure that older pip releases are not confused by the presence of wheels with the new ABI tag and do not fetch those wheels by mistake when trying to run nightly builds on the non-free-threaded version of CPython 3.13.

ogrisel · 2024-05-23T15:17:33Z

I built CPython main locally with the --disable-gil with:

  GDBM_CFLAGS="-I$(brew --prefix gdbm)/include" \                                                             
  GDBM_LIBS="-L$(brew --prefix gdbm)/lib -lgdbm" \
  ./configure --config-cache \
              --with-pydebug \
              --with-system-libmpdec \
              --with-openssl="$(brew --prefix openssl@3.0)" \
              --disable-gil && make -s -j10

some manual CFLAGS/LDFLAGS for zlib installed with brew.

Then I could successfully run ./python.exe -m ensurepip to install pip needed by venv.

Then I created a venv with ./python.exe -m ~/.venvs/free-threading without running make install

And then I tried to build numpy with pip install --no-build-isolation -e . but I got the following error that I don't know how to fix.

      + /Users/ogrisel/.venvs/free-threading/bin/python /Users/ogrisel/code/numpy/vendored-meson/meson/meson.py setup /Users/ogrisel/code/numpy /Users/ogrisel/code/numpy/build/cp314td -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/Users/ogrisel/code/numpy/build/cp314td/meson-python-native-file.ini
      The Meson build system
      Version: 1.2.99
      Source dir: /Users/ogrisel/code/numpy
      Build dir: /Users/ogrisel/code/numpy/build/cp314td
      Build type: native build
      Project name: NumPy
      Project version: 2.1.0.dev0+git20240519.2970735
      C compiler for the host machine: cc (clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.3.9.4)")
      C linker for the host machine: cc ld64 1053.12
      C++ compiler for the host machine: c++ (clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.3.9.4)")
      C++ linker for the host machine: c++ ld64 1053.12
      Cython compiler for the host machine: cython (cython 3.0.10)
      Host machine cpu family: aarch64
      Host machine cpu: aarch64
      Program python found: YES (/Users/ogrisel/.venvs/free-threading/bin/python)
      Found pkg-config: /opt/homebrew/bin/pkg-config (0.29.2)
      Run-time dependency python found: NO (tried pkgconfig, pkgconfig and sysconfig)
      
      ../../meson.build:41:12: ERROR: Python dependency not found

any suggestion @ngoldbaum?

ogrisel · 2024-05-23T15:18:05Z

I think @lesteve had a similar problem on Linux so it's probably not macOS specific?

rgommers · 2024-05-23T16:00:32Z

The Meson build system
Version: 1.2.99

Not sure if it matters, but that's not latest numpy main (or an outdated git submodule). In the free-threaded CI job in the numpy repo (log), it's at 1.4.99.

lesteve · 2024-05-23T16:29:26Z

I tried again, not sure what I did differently this time to be honest, but it went a bit further. This time, I get a build error in numpy error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’? ...

More details

  FAILED: numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/meson-generated_numpy_random__mt19937.pyx.c.o
  ccache cc -Inumpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p -Inumpy/random -I../numpy/random -I../numpy/random/src -Inumpy/_core -I../numpy/_core -Inumpy/_core/include -I../numpy/_core/include -I../numpy/_core/src/common -Inumpy -I/opt/python-free-threading/include/python3.13td -I/home/lesteve/dev/numpy/.mesonpy-2arbmkr6/meson_cpu -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -Wall -Winvalid-pch -std=c11 -O3 -fno-strict-aliasing -msse -msse2 -msse3 -DNPY_HAVE_SSE2 -DNPY_HAVE_SSE -DNPY_HAVE_SSE3 -fPIC -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DNPY_NO_DEPRECATED_API=0 -MD -MQ numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/meson-generated_numpy_random__mt19937.pyx.c.o -MF numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/meson-generated_numpy_random__mt19937.pyx.c.o.d -o numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/meson-generated_numpy_random__mt19937.pyx.c.o -c numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:2380:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
   2380 | static CYTHON_INLINE PyObject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw);
        |                                                                                ^~~~~~~~~~~~~~~~~~~~
        |                                                                                vectorcallfunc
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:11238:69: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
  11238 | static PyObject *__Pyx_PyVectorcall_FastCallDict_kw(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
        |                                                                     ^~~~~~~~~~~~~~~~~~~~
        |                                                                     vectorcallfunc
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:11283:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
  11283 | static CYTHON_INLINE PyObject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw)
        |                                                                                ^~~~~~~~~~~~~~~~~~~~
        |                                                                                vectorcallfunc
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c: In function ‘__Pyx_CyFunction_CallAsMethod’:
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:11972:6: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
  11972 |      __pyx_vectorcallfunc vc = __Pyx_CyFunction_func_vectorcall(cyfunc);
        |      ^~~~~~~~~~~~~~~~~~~~
        |      vectorcallfunc
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:2470:45: error: initialization of ‘int’ from ‘vectorcallfunc’ {aka ‘struct _object * (*)(struct _object *, struct _object * const*, long unsigned int,  struct _object *)’} makes integer from pointer without a cast [-Wint-conversion]
   2470 | #define __Pyx_CyFunction_func_vectorcall(f) (((PyCFunctionObject*)f)->vectorcall)
        |                                             ^
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:11972:32: note: in expansion of macro ‘__Pyx_CyFunction_func_vectorcall’
  11972 |      __pyx_vectorcallfunc vc = __Pyx_CyFunction_func_vectorcall(cyfunc);
        |                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:11975:16: error: implicit declaration of function ‘__Pyx_PyVectorcall_FastCallDict’; did you mean ‘__Pyx_PyObject_FastCallDict’? [-Wimplicit-function-declaration]
  11975 |         return __Pyx_PyVectorcall_FastCallDict(func, vc, &PyTuple_GET_ITEM(args, 0), (size_t)PyTuple_GET_SIZE(args), kw);
        |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        |                __Pyx_PyObject_FastCallDict
  numpy/random/_mt19937.cpython-313td-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:11975:16: error: returning ‘int’ from a function with return type ‘PyObject *’ {aka ‘struct _object *’} makes pointer from integer without a cast [-Wint-conversion]
  11975 |         return __Pyx_PyVectorcall_FastCallDict(func, vc, &PyTuple_GET_ITEM(args, 0), (size_t)PyTuple_GET_SIZE(args), kw);
        |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  [502/610] Compiling C++ object numpy/_core/_multiarray_umath.cpython-313td-x86_64-linux-gnu.so.p/src_umath_stringdtype_ufuncs.cpp.o
  In file included from ../numpy/_core/src/umath/stringdtype_ufuncs.cpp:21:
  ../numpy/_core/src/umath/string_buffer.h:265:23: warning: template-id not allowed for constructor in C++20 [-Wtemplate-id-cdtor]
    265 |     inline Buffer<enc>()
        |                       ^
  ../numpy/_core/src/umath/string_buffer.h:265:23: note: remove the ‘< >’
  ../numpy/_core/src/umath/string_buffer.h:270:23: warning: template-id not allowed for constructor in C++20 [-Wtemplate-id-cdtor]
    270 |     inline Buffer<enc>(char *buf_, npy_int64 elsize_)
        |                       ^
  ../numpy/_core/src/umath/string_buffer.h:270:23: note: remove the ‘< >’
  [503/610] Compiling C object numpy/_core/_multiarray_umath.cpython-313td-x86_64-linux-gnu.so.p/meson-generated_lowlevel_strided_loops.c.o
  [504/610] Compiling C object numpy/_core/_multiarray_umath.cpython-313td-x86_64-linux-gnu.so.p/src_umath_ufunc_object.c.o
  [505/610] Compiling C++ object numpy/_core/_multiarray_umath.cpython-313td-x86_64-linux-gnu.so.p/src_npysort_selection.cpp.o
  [506/610] Compiling C++ object numpy/_core/_multiarray_umath.cpython-313td-x86_64-linux-gnu.so.p/src_npysort_quicksort.cpp.o
  [507/610] Compiling C++ object numpy/_core/_multiarray_umath.cpython-313td-x86_64-linux-gnu.so.p/src_umath_string_ufuncs.cpp.o
  In file included from ../numpy/_core/src/umath/string_ufuncs.cpp:21:
  ../numpy/_core/src/umath/string_buffer.h:265:23: warning: template-id not allowed for constructor in C++20 [-Wtemplate-id-cdtor]
    265 |     inline Buffer<enc>()
        |                       ^
  ../numpy/_core/src/umath/string_buffer.h:265:23: note: remove the ‘< >’
  ../numpy/_core/src/umath/string_buffer.h:270:23: warning: template-id not allowed for constructor in C++20 [-Wtemplate-id-cdtor]
    270 |     inline Buffer<enc>(char *buf_, npy_int64 elsize_)
        |                       ^
  ../numpy/_core/src/umath/string_buffer.h:270:23: note: remove the ‘< >’


</details>

ngoldbaum · 2024-05-23T16:29:48Z

FWIW, I also see that using a virtualenv like @ogrisel is trying to do.

Using pip 24.0 from /Users/goldbaum/.venvs/free-threading/lib/python3.13/site-packages/pip (python 3.13)
/Users/goldbaum/.venvs/free-threading/lib/python3.13/site-packages/pip/_internal/metadata/importlib/_envs.py:111: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pip._vendor.pkg_resources import find_distributions
Processing /Users/goldbaum/Documents/numpy
  Running command Preparing metadata (pyproject.toml)
  + /Users/goldbaum/.venvs/free-threading/bin/python /Users/goldbaum/Documents/numpy/vendored-meson/meson/meson.py setup /Users/goldbaum/Documents/numpy /Users/goldbaum/Documents/numpy/.mesonpy-guksbdeb -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/Users/goldbaum/Documents/numpy/.mesonpy-guksbdeb/meson-python-native-file.ini
  The Meson build system
  Version: 1.4.99
  Source dir: /Users/goldbaum/Documents/numpy
  Build dir: /Users/goldbaum/Documents/numpy/.mesonpy-guksbdeb
  Build type: native build
  Project name: NumPy
  Project version: 2.1.0.dev0+git20240522.83b4a23
  C compiler for the host machine: ccache cc (clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.1.0.2.5)")
  C linker for the host machine: cc ld64 1022.1
  C++ compiler for the host machine: ccache c++ (clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.1.0.2.5)")
  C++ linker for the host machine: c++ ld64 1022.1
  Cython compiler for the host machine: cython (cython 3.0.10)
  Host machine cpu family: aarch64
  Host machine cpu: aarch64
  Program python found: YES (/Users/goldbaum/.venvs/free-threading/bin/python)
  Found pkg-config: YES (/opt/homebrew/bin/pkg-config) 0.29.2
  Run-time dependency python found: NO (tried pkgconfig, pkgconfig and sysconfig)

  ../meson.build:41:12: ERROR: Python dependency not found

I think maybe because python isn't visible in your $PATH? FWIW I've been using pyenv to manage python builds and dealing with $PATH for local builds. In numpy's meson configuration we detect python using find_package.

ngoldbaum · 2024-05-23T16:30:16Z

I tried again, not sure what I did differently this time to be honest, but it went a bit further. This time, I get a build error in numpy error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’? ...

This is fixed in cython main, you'll need to compile cython from the git repo before building numpy.

ngoldbaum · 2024-05-23T16:37:50Z

Run-time dependency python found: NO (tried pkgconfig, pkgconfig and sysconfig)

Ahh, sorry! This is because meson can't find python.h or libpython, so you need to set e.g. C_INCLUDE_PATH and LD_LIBRARY_PATH, I think.

lesteve · 2024-05-24T12:44:02Z

Thanks for your inputs, I managed to get numpy, scipy, scikit-learn to build. I was able to run the full scikit-learn test suite without failures. Maybe I was just lucky, since you said some of the failures are flaky? I did try to run multiple times some of the tests you mention in your original post but I did not manage to get them to fail.

What I am seeing though is segmentation fault at exit, i.e. after the test have run.

The gdb info looks like this:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
_Py_DECREF_DecRefTotal () at Objects/object.c:238
238	    reftotal_add(_PyThreadState_GET(), -1);
(gdb) bt
#0  _Py_DECREF_DecRefTotal () at Objects/object.c:238
#1  0x00007fff9129a0ef in (anonymous namespace)::._anon_193::~._anon_193() ()
   from /home/lesteve/venvs/free-threading/lib/python3.13/site-packages/scipy/_lib/_uarray/_uarray.cpython-313td-x86_64-linux-gnu.so
#2  0x00007ffff7cfefa1 in __run_exit_handlers (status=0, listp=0x7ffff7ea2680 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, 
    run_dtors=run_dtors@entry=true) at exit.c:108
#3  0x00007ffff7cff06e in __GI_exit (status=<optimized out>) at exit.c:138
#4  0x00007ffff7ce5c8f in __libc_start_call_main (main=main@entry=0x5555555d4839 <main>, argc=argc@entry=2, argv=argv@entry=0x7fffffffdac8)
    at ../sysdeps/nptl/libc_start_call_main.h:74
#5  0x00007ffff7ce5d4c in __libc_start_main_impl (main=0x5555555d4839 <main>, argc=2, argv=0x7fffffffdac8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffdab8) at ../csu/libc-start.c:360
#6  0x00005555555d4765 in _start ()

It seems like this happens when garbage-collecting uarray objects after a Python thread has been started, here is a reproducer with only scipy:

import scipy.fft

from concurrent.futures import ThreadPoolExecutor

def noop():
    return

tpe = ThreadPoolExecutor(max_workers=1)
tpe.submit(noop).result()

It does not seem to be one of the known issues in Scipy scipy/scipy#20669.

Edit: actually looking a bit further the problem still happens with PYTHON_GIL=1, so maybe this is more a Python 3.13 issue than a free-threading issue?

For further reference, here are the commits I used:
CPython: 6892b400dc
cython: 19f67cb1f
numpy: 1f73030
scipy: 8ce393f
scikit-learn: 5cbc419

ngoldbaum · 2024-05-24T13:40:56Z

Slightly dumb question: are you sure the GIL is disabled running the test suite? I still need to mark numpy's extension modules as safe without the GIL, so it needs to be explicitly disabled with e.g. the PYTHON_GIL environment variable.

Also cython has seen some updates, maybe cython changes fixed things? It's been a week or two since I tried the scikit-learn test suite.

lesteve · 2024-05-24T14:01:16Z

Slightly dumb question: are you sure the GIL is disabled running the test suite?

Yes, I did set PYTHON_GIL=0 when running the tests.

lesteve · 2024-06-05T15:56:32Z

Update: I have a PR with a CPython 3.13 free-threaded CI build in #29191 and the CI is green 🎉

Once this is merged, the next step would be to upload nightly wheels to scientific-python-nightly-wheels as Numpy and Scipy already do.

lesteve · 2024-06-19T14:14:34Z

Update: manylinux scikit-learn free-threaded wheel is available 🎉, see https://anaconda.org/scientific-python-nightly-wheels/scikit-learn/files

ogrisel · 2024-06-24T11:02:00Z

I think we should keep this issue open as long as:

we depend on nightly build versions or pre-release versions of our build dependencies (Python, NumPy, SciPy, Cython),
pypi.org does not accept cp3XYt wheels (to check if it is the case or not),
make sure that pip does not confuse pip free-threaded wheels for GIL-based wheels.

rgommers · 2024-06-24T11:09:12Z

make sure that pip does not confuse pip free-threaded wheels for GIL-based wheels.

pip 24.1 was released just a few days ago, that is the first version that does the right thing. Older versions happily install the wrong wheels.

betatim · 2024-06-26T15:29:00Z

Older versions happily install the wrong wheels.

In the sense that if we upload a cp3XYt wheel it will be downloaded by an old pip when it shouldn't download it?

ngoldbaum · 2024-06-26T15:33:55Z

In the sense that if we upload a cp3XYt wheel it will be downloaded by an old pip when it shouldn't download it?

I don't think that's possible. The failure mode pip fixed was that in Python 3.13, Pip didn't know about the existence of the cp313t ABI tag, so it installed the regular cp313 wheel instead, which it did know about. I don't see how the new ABI tag could inflence pip's decision to install a wheel or not on older python versions.

ogrisel · 2024-10-04T08:47:20Z

Note: the scikit-learn 1.5.2 release of scikit-learn now included regular (GIL enabled) wheels for CPython 3.13 but not the cp313t variant because we need scipy to ship cp313t wheels first. See the discussion in #29987.

ngoldbaum · 2024-10-30T21:38:33Z

Let me know if you need any support unblocking Windows wheel uploads.

lesteve · 2024-10-31T08:30:11Z

I guess we would need scipy to upload Windows free-threaded wheels first, which does not seem to be the case https://anaconda.org/scientific-python-nightly-wheels/scipy/files. I haven't found an ongoing Scipy PR adding Windows free-threaded wheels, but maybe I have missed it.

Also we are probably going to have something similar to pandas-dev/pandas#59057 (comment) that for Windows wheels we use a custom minimal Docker image to avoid relying on Windows system DLLs which make it a bit more cumbersome (see #18802 for more details about why we need this).

ngoldbaum · 2024-10-31T14:04:17Z

Ah sorry missed that SciPy was an issue. That is being worked on by @andfoy but is trickier than other libraries because SciPy uses a GNU toolchain to build on Windows.

lesteve · 2024-11-25T16:56:27Z

@ngoldbaum FYI, the Windows free-threaded wheel has been added in #30313, this should be available tomorrow in scientific-python-nightly-wheels 🤞.

ogrisel · 2024-11-28T18:34:51Z

I confirm they are now. We can close. Thanks to everybody involved in making this happen!

github-actions bot added the Needs Triage Issue requires triage label May 8, 2024

ngoldbaum mentioned this issue May 8, 2024

Status of scikit-learn compatibility Quansight-Labs/free-threaded-compatibility#11

Closed

ogrisel added Enhancement Build / CI and removed Needs Triage Issue requires triage labels May 10, 2024

lesteve added the free-threading PRs and issues related to support for free-threaded CPython (a.k.a. nogil or no-GIL, PEP 703) label May 13, 2024

This was referenced Jun 4, 2024

CI Add CPython 3.13 free-threaded test joblib/joblib#1589

Merged

⚠️ CI failed on Linux_nogil.pylatest_pip_nogil (last failure: Jun 10, 2024) ⚠️ #29189

Closed

CI Add CPython 3.13 free-threaded build and remove historic nogil Python 3.9 build #29191

Merged

lesteve mentioned this issue Jun 13, 2024

CI Add CPython 3.13 free-threaded manylinux wheel #29247

Merged

ogrisel mentioned this issue Jul 30, 2024

Truly parallel execution of pairwise_kernels and pairwise_distances #29587

Closed

tylerjereddy mentioned this issue Oct 8, 2024

BUG: Errors at compiling through pip for python 3.13 with option "disable-gil" scipy/scipy#21675

Closed

Zheaoli mentioned this issue Nov 1, 2024

BUG: Build failed in CPython main branch numpy/numpy#27688

Closed

glemaitre mentioned this issue Nov 5, 2024

DOC add priorities update 09/2024 glemaitre/probabl-blog-post#1

Merged

3 tasks

lesteve mentioned this issue Nov 20, 2024

CI Add Windows free-threaded wheel #30313

Merged

ogrisel closed this as completed Nov 28, 2024

tyralla mentioned this issue Jan 13, 2025

Free threading? hydpy-dev/hydpy#161

Open

Add support for Python 3.13 free-threaded build #28978

Add support for Python 3.13 free-threaded build #28978

Comments

ngoldbaum commented May 8, 2024

ogrisel commented May 10, 2024

ogrisel commented May 10, 2024

ngoldbaum commented May 10, 2024

ngoldbaum commented May 10, 2024

lesteve commented May 13, 2024 • edited Loading

hugovk commented May 13, 2024

lesteve commented May 13, 2024 • edited Loading

lesteve commented May 13, 2024

ogrisel commented May 14, 2024 • edited Loading

ngoldbaum commented May 14, 2024

ngoldbaum commented May 14, 2024

rgommers commented May 14, 2024

betatim commented May 15, 2024

ogrisel commented May 15, 2024

ogrisel commented May 23, 2024

ogrisel commented May 23, 2024

rgommers commented May 23, 2024

lesteve commented May 23, 2024

ngoldbaum commented May 23, 2024

ngoldbaum commented May 23, 2024

ngoldbaum commented May 23, 2024 • edited Loading

lesteve commented May 24, 2024 • edited Loading

ngoldbaum commented May 24, 2024

lesteve commented May 24, 2024

lesteve commented Jun 5, 2024

lesteve commented Jun 19, 2024 • edited Loading

ogrisel commented Jun 24, 2024

rgommers commented Jun 24, 2024

betatim commented Jun 26, 2024

ngoldbaum commented Jun 26, 2024

ogrisel commented Oct 4, 2024

ngoldbaum commented Oct 30, 2024

lesteve commented Oct 31, 2024 • edited Loading

ngoldbaum commented Oct 31, 2024

lesteve commented Nov 25, 2024

ogrisel commented Nov 28, 2024 • edited Loading

lesteve commented May 13, 2024 •

edited

Loading

lesteve commented May 13, 2024 •

edited

Loading

ogrisel commented May 14, 2024 •

edited

Loading

ngoldbaum commented May 23, 2024 •

edited

Loading

lesteve commented May 24, 2024 •

edited

Loading

lesteve commented Jun 19, 2024 •

edited

Loading

lesteve commented Oct 31, 2024 •

edited

Loading

ogrisel commented Nov 28, 2024 •

edited

Loading