Skip to content

ENH: Support for Python 3.13.2 free-threading version #60972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
3 tasks done
qfcy opened this issue Feb 20, 2025 · 9 comments
Open
3 tasks done

ENH: Support for Python 3.13.2 free-threading version #60972

qfcy opened this issue Feb 20, 2025 · 9 comments
Assignees
@rhshadrach
Labels
Build Library building on various platforms Enhancement

Comments

@qfcy
Copy link

qfcy commented Feb 20, 2025 •
edited
Loading

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Firstly, use python3.13t -m pip install pandas to build and install pandas on VS 16.11.43 and MSVC 19.29.30158.
Then try to import it:

E:\Git-repositories\project179>python3.13t
Python 3.13.2 experimental free-threading build (tags/v3.13.2:4f8bb39, Feb  4 2025, 15:33:40) [MSC v.1942 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
<Segmentation fault caused there>

Issue Description

The line import pandas failed in Python 3.13 free-threading version on x64 Windows.
As I tried to execute it, a segmentation fault caused because I've seen werfault.exe appeared in the task manager.

Expected Behavior

The line import pandas successfully be executed.

Installed Versions

pandas 2.2.3 (Note: cannot call pd.show_versions() as I couldn't import pandas at the first step)

Additionally, this is the output during compiling pandas that contains the MSVC version:

 + meson setup C:\Users\admin\AppData\Local\Temp\pip-install-yk54bvta\pandas_93cffcf8ab364c6d8ccc39766c5b148a C:\Users\admin\AppData\Local\Temp\pip-install-yk54bvta\pandas_93cffcf8ab364c6d8ccc39766c5b148a\.mesonpy-6btzqpgz\build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv --native-file=C:\Users\admin\AppData\Local\Temp\pip-install-yk54bvta\pandas_93cffcf8ab364c6d8ccc39766c5b148a\.mesonpy-6btzqpgz\build\meson-python-native-file.ini
      The Meson build system
      Version: 1.2.1
      Source dir: C:\Users\admin\AppData\Local\Temp\pip-install-yk54bvta\pandas_93cffcf8ab364c6d8ccc39766c5b148a
      Build dir: C:\Users\admin\AppData\Local\Temp\pip-install-yk54bvta\pandas_93cffcf8ab364c6d8ccc39766c5b148a\.mesonpy-6btzqpgz\build
      Build type: native build
      Project name: pandas
      Project version: 2.2.3
      Activating VS 16.11.43
      C compiler for the host machine: cl (msvc 19.29.30158 "用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30158 版")
      C linker for the host machine: link link 14.29.30158.0
      C++ compiler for the host machine: cl (msvc 19.29.30158 "用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30158 版")
      C++ linker for the host machine: link link 14.29.30158.0
      Cython compiler for the host machine: cython (cython 3.0.12)
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      Program python found: YES (D:\Python\Python313\python3.13t.exe)
      Run-time dependency python found: YES 3.13
      Build targets in project: 53

      pandas 2.2.3

        User defined options
          Native files: C:\Users\admin\AppData\Local\Temp\pip-install-yk54bvta\pandas_93cffcf8ab364c6d8ccc39766c5b148a\.mesonpy-6btzqpgz\build\meson-python-native-file.ini
          buildtype   : release
          vsenv       : True
          b_ndebug    : if-release
          b_vscrt     : md

      Found ninja.EXE-1.11.1.git.kitware.jobserver-1 at C:\Users\admin\AppData\Local\Temp\pip-build-env-3jrexmoi\normal\Scripts\ninja.EXE

      Visual Studio environment is needed to run Ninja. It is recommended to use Meson wrapper:
      C:\Users\admin\AppData\Local\Temp\pip-build-env-3jrexmoi\overlay\Scripts\meson compile -C .
      + meson compile
      [1/151] Generating pandas/_libs/algos_common_helper_pxi with a custom command
      [2/151] Generating pandas/_libs/khash_primitive_helper_pxi with a custom command
      [3/151] Generating pandas/_libs/hashtable_class_helper_pxi with a custom command
      [4/151] Generating pandas/_libs/algos_take_helper_pxi with a custom command
      [5/151] Generating pandas/_libs/hashtable_func_helper_pxi with a custom command
      [6/151] Generating pandas/_libs/intervaltree_helper_pxi with a custom command
      [7/151] Generating pandas/_libs/index_class_helper_pxi with a custom command
      [8/151] Generating pandas/_libs/sparse_op_helper_pxi with a custom command
      [9/151] Generating pandas/__init__.py with a custom command
      [10/151] Compiling C object pandas/_libs/tslibs/parsing.cp313t-win_amd64.pyd.p/.._src_parser_tokenizer.c.obj
      ..\..\pandas\_libs\include\pandas/vendored/klib/khash.h(729): warning C4090: “函数”: 不同的“const”限定符
      [11/151] Compiling Cython source C:/Users/admin/AppData/Local/Temp/pip-install-yk54bvta/pandas_93cffcf8ab364c6d8ccc39766c5b148a/pandas/_libs/tslibs/base.pyx
      [12/151] Compiling C object pandas/_libs/tslibs/base.cp313t-win_amd64.pyd.p/meson-generated_pandas__libs_tslibs_base.pyx.c.obj
@qfcy qfcy added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Feb 20, 2025
@qfcy qfcy changed the title BUG: import pandas failed in Python 3.13 free-threading version BUG: import pandas failed in Python 3.13 free-threading version Feb 20, 2025
@qfcy qfcy changed the title BUG: import pandas failed in Python 3.13 free-threading version BUG: import pandas failed in Python 3.13.2 free-threading version Feb 20, 2025
@rhshadrach
Copy link
Member

Thanks for the report. Cython is still working on support (but it seems 3.1 is getting close). pandas cannot support free-threading until after Cython is ready. Even then, it's not clear to me what the level of effort it will take to support free-threading. Work on this is certainly appreciated!

Reworking this issue as an enhancement.

@rhshadrach rhshadrach added Enhancement Build Library building on various platforms and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 5, 2025
@rhshadrach rhshadrach changed the title BUG: import pandas failed in Python 3.13.2 free-threading version ENH: Support for Python 3.13.2 free-threading version Mar 5, 2025
@rgommers
Copy link
Contributor

Cython 3.1.0 is available, so this should now work fine. Also, nightly Windows wheels for free-threaded CPython are available at https://anaconda.org/scientific-python-nightly-wheels/pandas, so there's no longer a need to build from source. They pass all tests, so I think this issue can be closed.

@rhshadrach
Copy link
Member

Thanks @rgommers - I plan to close this issue after updating our dev environment to use Cython 3.1. Want to run the full ASV suite first.

@rhshadrach rhshadrach self-assigned this May 19, 2025
@rgommers
Copy link
Contributor

Sounds good, thanks @rhshadrach

@qfcy
Copy link
Author

qfcy commented May 21, 2025 •
edited
Loading

Now pandas can be certainly built successfully, but the segmentation fault hasn't been fixed:

C:\Users\admin>py313t -m pip install pandas
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple/
Collecting pandas
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9c/d6/9f8431bacc2e19dca897724cd097b1bb224a6ad5433784a44b587c7c13af/pandas-2.2.3.tar.gz (4.4 MB)
     ---------------------------------------- 4.4/4.4 MB 36.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.26.0 in d:\python\python313\lib\site-packages (from pandas) (2.2.6)
Requirement already satisfied: python-dateutil>=2.8.2 in d:\python\python313\lib\site-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in d:\python\python313\lib\site-packages (from pandas) (2025.1)
Requirement already satisfied: tzdata>=2022.7 in d:\python\python313\lib\site-packages (from pandas) (2025.1)
Requirement already satisfied: six>=1.5 in d:\python\python313\lib\site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)
Building wheels for collected packages: pandas
  Building wheel for pandas (pyproject.toml) ... done
  Created wheel for pandas: filename=pandas-2.2.3-cp313-cp313t-win_amd64.whl size=39033716 sha256=b99517674c7dfaae923c9682bd970bb9db0c7cce192423355e394879c7347837
  Stored in directory: c:\users\admin\appdata\local\pip\cache\wheels\b3\3a\9c\4853eb9b64bf66e233d1ec029b2098492524487e3ba8370e28
Successfully built pandas
Installing collected packages: pandas
Successfully installed pandas-2.2.3

The same segmentation fault:

C:\Users\admin>py313t
Python 3.13.3 experimental free-threading build (tags/v3.13.3:6280bb5, Apr  8 2025, 14:53:12) [MSC v.1943 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
<Segmentation fault caused there>

@rhshadrach
Copy link
Member

@qfcy - what version of cython are you using to build?

@qfcy
Copy link
Author

qfcy commented May 22, 2025

@qfcy - what version of cython are you using to build?

The default Cython version 3.0.5 in pyproject.toml

@rhshadrach
Copy link
Member

Cython 3.0.5 will not work with freethreading, you need to compile with Cython >= 3.1

@rhshadrach
Copy link
Member

rhshadrach commented May 24, 2025 •
edited
Loading

Did two runs on main and with bumping Cython, benchmarks above a 5% cutoff are below. Looks like noise to me, will put up a PR to bump Cython in the next day or two.

                                                                                                                                           cy31      cy30        abs      pct
name                                                                params                                                                                                   
categoricals.Indexing.time_reindex_missing                                                                                              0.00774  0.006917  822.731us  11.894%
frame_methods.Dropna.time_dropna_axis_mixed_dtypes                  how='any', axis=0                                                  0.007919  0.007175  744.550us  10.378%
array.ArrowExtensionArray.time_to_numpy                             dtype='string[pyarrow]', hasna=False                               0.002367  0.002148  219.137us  10.202%
frame_methods.Dropna.time_dropna                                    how='any', axis=0                                                  0.007844  0.007124  720.241us  10.110%
                                                                    how='any', axis=1                                                  0.007324  0.006655  668.543us  10.045%
series_methods.NanOps.time_func                                     func='std', N=1000000, dtype='boolean'                             0.002304  0.002101  202.650us   9.644%
algorithms.Duplicated.time_duplicated                               unique=False, keep='last', dtype='datetime64[ns]'                  0.006026  0.005502  523.607us   9.517%
multiindex_object.Equals.time_equals_non_object_index                                                                                  0.000001  0.000001   58.995ns   9.422%
frame_methods.Dropna.time_dropna_axis_mixed_dtypes                  how='all', axis=0                                                  0.007583  0.006951  632.144us   9.095%
inference.ToTimedelta.time_convert_string_days                                                                                         0.008791  0.008171  620.322us   7.592%
algorithms.Duplicated.time_duplicated                               unique=False, keep=False, dtype='string'                           0.016058  0.015034    1.024ms   6.811%
frame_methods.ToDict.time_to_dict_ints                              orient='dict'                                                      0.003155  0.002961  194.213us   6.560%
timeseries.DatetimeIndex.time_to_date                               index_type='repeated'                                              0.008714  0.008183  530.847us   6.487%
plotting.BackendLoading.time_get_plot_backend_fallback                                                                                 0.002702  0.002539  163.185us   6.426%
algorithms.Duplicated.time_duplicated                               unique=False, keep='first', dtype='datetime64[ns, tz]'             0.005909  0.005555  353.359us   6.361%
                                                                    unique=False, keep='last', dtype='datetime64[ns, tz]'              0.005855  0.005507  347.920us   6.318%
join_merge.MergeMultiIndex.time_merge_sorted_multiindex             dtypes=('datetime64[ns]', 'int64'), how='right'                    0.005677   0.00535  327.196us   6.116%
join_merge.ConcatIndexDtype.time_concat_series                      dtype='string[python]', structure='monotonic', axis=1, sort=False  0.003117   0.00294  176.640us   6.007%
frame_methods.Count.time_count                                      axis=0                                                             0.009141  0.008638  503.169us   5.825%
join_merge.ConcatIndexDtype.time_concat_series                      dtype='string[python]', structure='monotonic', axis=1, sort=True    0.00311  0.002941  169.216us   5.753%
index_cached_properties.IndexCache.time_values                      index_type='PeriodIndex'                                           0.000001  0.000001   30.001ns   5.650%
frame_methods.Count.time_count                                      axis=1                                                             0.010255  0.009733  521.870us   5.362%
algorithms.Duplicated.time_duplicated                               unique=False, keep='first', dtype='datetime64[ns]'                 0.005857   0.00556  297.089us   5.344%
arithmetic.MixedFrameWithSeriesAxis.time_frame_op_with_series_axis1 opname='floordiv'                                                  0.011908  0.011332  576.210us   5.085%
timeseries.DatetimeAccessor.time_dt_accessor_date                   t=None                                                             0.008546  0.008138  408.001us   5.013%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rhshadrach rhshadrach

Labels
Build Library building on various platforms Enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rgommers @rhshadrach @qfcy