Skip to content

BUG: Build from source can fail on Windows for scikit-learn v1.6.1 with Ninja mkdir error #31123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Molier opened this issue Apr 1, 2025 · 4 comments
Closed

BUG: Build from source can fail on Windows for scikit-learn v1.6.1 with Ninja mkdir error #31123

Molier opened this issue Apr 1, 2025 · 4 comments
Labels
Build / CI Needs Investigation Issue requires investigation OS:Windows Problem specific to Windows

Comments

@Molier
Copy link

Molier commented Apr 1, 2025

Labels: Bug, Build / CI, Needs Triage (Suggested)

Describe the bug

Scikit-learn (v1.6.1) fails to build from source on a native Windows 11 ARM64 machine using the MSYS2 ClangARM64 toolchain. The build proceeds through the Meson setup phase correctly identifying the clang compiler, but fails during the ninja compilation phase with an error indicating it cannot create a specific, deeply nested intermediate build directory.

This occurs despite successfully building other complex dependencies like NumPy (v2.2.4) and SciPy (v1.15.2) from source in the exact same environment. Pandas (v2.2.3) also builds successfully after setting MESON_DISABLE_VSENV=1 (otherwise it incorrectly selects MSVC). This suggests the issue might be specific to how scikit-learn's build structure interacts with Meson/Ninja within this particular toolchain environment.

This is related to, but distinct from, #30567 which requests pre-built wheels. This issue focuses on a specific build-from-source failure.

Steps/Code to Reproduce

  1. Environment Setup:

    • OS: Windows 11 Pro ARM64 (via Parallels on Apple Silicon M2, or on native hardware like Windows Dev Kit 2023)
    • MSYS2: Latest version, updated via pacman -Syu.
    • MSYS2 Environment: CLANGARM64 shell launched.
    • Key MSYS2 Packages (installed via pacman -S mingw-w64-clang-aarch64-<package>):
      • python (3.12.x)
      • clang (20.1.1)
      • flang (20.1.1)
      • meson (1.7.0)
      • ninja (1.12.1)
      • pkgconf
      • openblas
      • lapack
      • openssl
      • hdf5
      • rust
      • zlib
    • Project Location: Tried both native MSYS2 path (/home/user/project) and WSL interop path (//wsl.localhost/Ubuntu/...) - error persists in both.

  2. Python Virtual Environment:

    # In CLANGARM64 shell, navigate to project directory
python -m venv .venv
source .venv/bin/activate # Or Scripts/activate
python -m pip install --upgrade pip setuptools wheel

  3. Environment Variables: Before attempting install, set the following in the activated CLANGARM64 shell:

    # Unset potential conflicts
unset CL _CL_ OPENSSL_LIB_DIR OPENSSL_INCLUDE_DIR OPENSSL_DIR

# Set compilers (though often picked up correctly by MSYS2 shell)
export CC=clang
export CXX=clang++

# Attempt to prevent Meson from activating MSVC (worked for Pandas)
export MESON_DISABLE_VSENV=1

# Ensure pkg-config uses MSYS2 paths
export PKG_CONFIG_PATH="/clangarm64/lib/pkgconfig:/clangarm64/share/pkgconfig"

# Attempt to use short temporary paths (tried /tmp and C:/pip_build_temp)
export TMPDIR="C:/pip_build_temp" # Or /tmp
mkdir -p $TMPDIR

  4. Attempt Installation:

    python -m pip install --verbose scikit-learn==1.6.1
# (Or use `pip install -r requirements.txt` containing scikit-learn==1.6.1)

Expected Results

Scikit-learn==1.6.1 should build and install successfully, similar to how NumPy and SciPy do in the same configured environment.

Actual Results

The build process starts, Meson correctly identifies clang, dependencies are resolved, Cython files are processed, but compilation fails during the Ninja execution phase.

The key error message is:

[...]
[48/251] Compiling Cython source C:/pip_build_temp/pip-install-.../scikit-learn.../utils/_fast_dict.pyx
ninja: error: mkdir(sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cp312-mingw_aarch64_ucrt_llvm.pyd.p/sklearn/metrics/_pairwise_distances_reduction): No such file or directory
ninja: build stopped: .
[...]
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
[...]
error: metadata-generation-failed

(Note: The exact file being compiled just before the error ([48/251] in the log provided previously) might vary slightly, but the ninja: error: mkdir(...) is consistent).

full log

full error trace

@energypc CLANGARM64 ~/projects_msys/func-ecopower-pwm-forecast-durable
$ python -m pip install --verbose scikit-learn==1.6.1
Using pip 25.0.1 from C:/msys64/home/oscar/projects_msys/func-ecopower-pwm-forecast-durable/.venv/lib/python3.12/site-packages/pip (python 3.12)
Collecting scikit-learn==1.6.1
  Using cached scikit_learn-1.6.1.tar.gz (7.1 MB)
  Running command pip subprocess to install build dependencies
  Using pip 25.0.1 from C:/msys64/home/oscar/projects_msys/func-ecopower-pwm-forecast-durable/.venv/lib/python3.12/site-packages/pip (python 3.12)
  Collecting meson-python>=0.16.0
    Obtaining dependency information for meson-python>=0.16.0 from https://files.pythonhosted.org/packages/7d/ec/40c0ddd29ef4daa6689a2b9c5ced47d5b58fa54ae149b19e9a97f4979c8c/meson_python-0.17.1-py3-none-any.whl.metadata
    Using cached meson_python-0.17.1-py3-none-any.whl.metadata (4.1 kB)
  Collecting Cython>=3.0.10
    Obtaining dependency information for Cython>=3.0.10 from https://files.pythonhosted.org/packages/27/6b/7c87867d255cbce8167ed99fc65635e9395d2af0f0c915428f5b17ec412d/Cython-3.0.12-py2.py3-none-any.whl.metadata
    Using cached Cython-3.0.12-py2.py3-none-any.whl.metadata (3.3 kB)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/3a/be/650f9c091ef71cb01d735775d554e068752d3ff63d7943b26316dc401749/numpy-1.21.2.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/5f/d6/ad58ded26556eaeaa8c971e08b6466f17c4ac4d786cd3d800e26ce59cc01/numpy-1.21.3.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/c2/a8/a924a09492bdfee8c2ec3094d0a13f2799800b4fdc9c890738aeeb12c72e/numpy-1.21.5.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/45/b7/de7b8e67f2232c26af57c205aaad29fe17754f793404f59c8a730c7a191a/numpy-1.21.6.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Collecting numpy>=2
    Using cached numpy-2.2.4-cp312-cp312-mingw_aarch64_ucrt_llvm.whl
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/99/f1/c00d6be56e1a718a3068079e3ec8ce044d7179345280f6a3f5066068af0d/scipy-1.6.2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/fe/fd/8704c7b7b34cdac850485e638346025ca57c5a859934b9aa1be5399b33b7/scipy-1.6.3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/bb/bb/944f559d554df6c9adf037aa9fc982a9706ee0e96c0d5beac701cb158900/scipy-1.7.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/47/33/a24aec22b7be7fdb10ec117a95e1e4099890d8bbc6646902f443fc7719d1/scipy-1.7.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/0e/23/58c4f995475a2a97cb5f4a032aedaf881ad87cd976a7180c55118d105a1d/scipy-1.7.2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/61/67/1a654b96309c991762ee9bc39c363fc618076b155fe52d295211cf2536c7/scipy-1.7.3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/c0/ad/e3c052ed4e0027a8abef0a5e8441a044427d252d17d9aee06d56e62fc698/scipy-1.8.0rc1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/29/d2/151a54944b333e465f98804dced31dab1284f3c37b752b9cefa710b64681/scipy-1.8.0rc2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/e4/26/83dd1c6378513a6241d984bda9f08c512b6e35fff13fba3acc1b3c195f02/scipy-1.8.0rc3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/22/78/056cc43e7737811b6f50886788a940f852773dd9804f5365952805db9648/scipy-1.8.0rc4.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/b4/a2/4faa34bf0cdbefd5c706625f1234987795f368eb4e97bde9d6f46860843e/scipy-1.8.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/26/b5/9330f004b9a3b2b6a31f59f46f1617ce9ca15c0e7fe64288c20385a05c9d/scipy-1.8.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/87/82/8b9bf8bb8030f1eef281ad2af87cc648e06e4a4974c2b9201a57d991b12c/scipy-1.9.0rc1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/5a/87/27f9ee75c32138d6eaabff2b91744c97a3fdbbbdd78948372dac93e0079f/scipy-1.9.0rc2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/57/72/484c38e5b814b48b41ac6432ef69077f2e7c8094925d3ce17bc3d61359b0/scipy-1.9.0rc3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/a8/e3/4ec401f609d34162b7023a09165da491630879e4cfa2336667fe2102cd06/scipy-1.9.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
    Link requires a different Python (3.12.9 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/db/af/16906139f52bc6866c43401869ce247662739ad71afa11c6f18505eb0546/scipy-1.9.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
    Link requires a different Python (3.12.9 not in: '<3.12,>=3.8'): https://files.pythonhosted.org/packages/a2/ce/2592c3b550cf8f68879d4ff2159f3c689ee6f032f8fc9059022074f7bd75/scipy-1.10.0rc1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:<3.12,>=3.8)
    Link requires a different Python (3.12.9 not in: '<3.12,>=3.8'): https://files.pythonhosted.org/packages/6b/a3/7de758a32569e06877a6b298bdf7b529ce97a33c55deb2b1637b0162fd9d/scipy-1.10.0rc2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:<3.12,>=3.8)
    Link requires a different Python (3.12.9 not in: '<3.12,>=3.8'): https://files.pythonhosted.org/packages/d6/bd/2d13a273d95f7b7d9903c906c486040b0aebb85e008f93a5dd0891f21f1f/scipy-1.10.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:<3.12,>=3.8)
    Link requires a different Python (3.12.9 not in: '<3.12,>=3.8'): https://files.pythonhosted.org/packages/84/a9/2bf119f3f9cff1f376f924e39cfae18dec92a1514784046d185731301281/scipy-1.10.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:<3.12,>=3.8)
  Collecting scipy>=1.6.0
    Using cached scipy-1.15.2-cp312-cp312-mingw_aarch64_ucrt_llvm.whl
  Collecting meson>=1.2.3 (from meson-python>=0.16.0)
    Obtaining dependency information for meson>=1.2.3 from https://files.pythonhosted.org/packages/ab/3b/63fdad828b4cbeb49cef3aad26f3edfbc72f37a0ab54917d445ec0b9d9ff/meson-1.7.0-py3-none-any.whl.metadata
    Using cached meson-1.7.0-py3-none-any.whl.metadata (1.8 kB)
  Collecting packaging>=19.0 (from meson-python>=0.16.0)
    Obtaining dependency information for packaging>=19.0 from https://files.pythonhosted.org/packages/88/ef/eb23f262cca3c0c4eb7ab1933c3b1f03d021f2c48f54763065b6f0e321be/packaging-24.2-py3-none-any.whl.metadata
    Using cached packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
  Collecting pyproject-metadata>=0.7.1 (from meson-python>=0.16.0)
    Obtaining dependency information for pyproject-metadata>=0.7.1 from https://files.pythonhosted.org/packages/7e/b1/8e63033b259e0a4e40dd1ec4a9fee17718016845048b43a36ec67d62e6fe/pyproject_metadata-0.9.1-py3-none-any.whl.metadata
    Using cached pyproject_metadata-0.9.1-py3-none-any.whl.metadata (7.0 kB)
  Using cached meson_python-0.17.1-py3-none-any.whl (27 kB)
  Using cached Cython-3.0.12-py2.py3-none-any.whl (1.2 MB)
  Using cached meson-1.7.0-py3-none-any.whl (990 kB)
  Using cached packaging-24.2-py3-none-any.whl (65 kB)
  Using cached pyproject_metadata-0.9.1-py3-none-any.whl (18 kB)
  Installing collected packages: packaging, numpy, meson, Cython, scipy, pyproject-metadata, meson-python
    Creating C:/pip_build_temp/pip-build-env-4z56ip4a/overlay/bin
  Successfully installed Cython-3.0.12 meson-1.7.0 meson-python-0.17.1 numpy-2.2.4 packaging-24.2 pyproject-metadata-0.9.1 scipy-1.15.2
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  + meson setup C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508 C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/.mesonpy-lhh9mbac -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/.mesonpy-lhh9mbac/meson-python-native-file.ini
  The Meson build system
  Version: 1.7.0
  Source dir: C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508
  Build dir: C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/.mesonpy-lhh9mbac
  Build type: native build
  Project name: scikit-learn
  Project version: 1.6.1
  C compiler for the host machine: clang (clang 20.1.1 "clang version 20.1.1")
  C linker for the host machine: clang ld.lld 20.1.1
  C++ compiler for the host machine: clang++ (clang 20.1.1 "clang version 20.1.1")
  C++ linker for the host machine: clang++ ld.lld 20.1.1
  Cython compiler for the host machine: cython (cython 3.0.12)
  Host machine cpu family: aarch64
  Host machine cpu: aarch64
  Compiler for C supports arguments -Wno-unused-but-set-variable: YES
  Compiler for C supports arguments -Wno-unused-function: YES
  Compiler for C supports arguments -Wno-conversion: YES
  Compiler for C supports arguments -Wno-misleading-indentation: YES
  Library m found: YES
  Program python found: YES (C:/msys64/home/oscar/projects_msys/func-ecopower-pwm-forecast-durable/.venv/bin/python.exe)
  Run-time dependency OpenMP for c found: YES 5.1
  Found pkg-config: YES (C:\msys64\clangarm64\bin/pkg-config.EXE) 2.3.0
  Run-time dependency python found: YES 3.12
  Build targets in project: 111

  scikit-learn 1.6.1

    User defined options
      Native files: C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/.mesonpy-lhh9mbac/meson-python-native-file.ini
      b_ndebug    : if-release
      b_vscrt     : md
      buildtype   : release

  Found ninja.EXE-1.12.1 at C:\msys64\clangarm64\bin/ninja.EXE
  + meson compile
  [1/251] Generating sklearn/metrics/_dist_metrics_pxd with a custom command
  [2/251] Copying file sklearn/utils/_cython_blas.pxd
  [3/251] Copying file sklearn/utils/__init__.py
  [4/251] Copying file sklearn/__init__.py
  [5/251] Copying file sklearn/utils/_openmp_helpers.pxd
  [6/251] Generating sklearn/utils/_seq_dataset_pxd with a custom command
  [7/251] Copying file sklearn/utils/_sorting.pxd
  [8/251] Copying file sklearn/utils/_random.pxd
  [9/251] Generating sklearn/utils/_weight_vector_pxd with a custom command
  [10/251] Copying file sklearn/utils/_heap.pxd
  [11/251] Copying file sklearn/_loss/_loss.pxd
  [12/251] Copying file sklearn/utils/_vector_sentinel.pxd
  [13/251] Generating sklearn/metrics/_pairwise_distances_reduction/_datasets_pair_pxd with a custom command
  [14/251] Generating sklearn/metrics/_pairwise_distances_reduction/_base_pxd with a custom command
  [15/251] Generating sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_pxd with a custom command
  [16/251] Copying file sklearn/metrics/__init__.py
  [17/251] Generating sklearn/neighbors/_binary_tree_pxi with a custom command
  [18/251] Copying file sklearn/utils/_typedefs.pxd
  [19/251] Generating sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer_pxd with a custom command
  [20/251] Generating sklearn/metrics/_pairwise_distances_reduction/_argkmin_pxd with a custom command
  [21/251] Generating sklearn/_loss/_loss_pyx with a custom command
  [22/251] Copying file sklearn/metrics/_pairwise_distances_reduction/__init__.py
  [23/251] Copying file sklearn/metrics/_pairwise_distances_reduction/_classmode.pxd
  [24/251] Copying file sklearn/linear_model/__init__.py
  [25/251] Copying file sklearn/neighbors/_partition_nodes.pxd
  [26/251] Copying file sklearn/neighbors/__init__.py
  [27/251] Generating sklearn/utils/_weight_vector_pyx with a custom command
  [28/251] Generating sklearn/utils/_seq_dataset_pyx with a custom command
  [29/251] Generating sklearn/metrics/_dist_metrics_pyx with a custom command
  [30/251] Generating sklearn/metrics/_pairwise_distances_reduction/_base_pyx with a custom command
  [31/251] Generating sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer_pyx with a custom command
  [32/251] Generating sklearn/metrics/_pairwise_distances_reduction/_datasets_pair_pyx with a custom command
  [33/251] Generating sklearn/metrics/_pairwise_distances_reduction/_argkmin_pyx with a custom command
  [34/251] Generating sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_pyx with a custom command
  [35/251] Generating sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode_pyx with a custom command
  [36/251] Generating sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode_pyx with a custom command
  [37/251] Generating sklearn/linear_model/_sgd_fast_pyx with a custom command
  [38/251] Generating sklearn/neighbors/_ball_tree_pyx with a custom command
  [39/251] Generating sklearn/linear_model/_sag_fast_pyx with a custom command
  [40/251] Generating sklearn/neighbors/_kd_tree_pyx with a custom command
  [41/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/__check_build/_check_build.pyx
  [42/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/utils/_openmp_helpers.pyx
  [43/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/metrics/cluster/_expected_mutual_info_fast.pyx
  [44/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/utils/_random.pyx
  [45/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/utils/_heap.pyx
  [46/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/_isotonic.pyx
  [47/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/utils/murmurhash.pyx
  [48/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/utils/_cython_blas.pyx
  [49/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/utils/_fast_dict.pyx
  [50/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/utils/_sorting.pyx
  [51/251] Compiling Cython source C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508/sklearn/utils/arrayfuncs.pyx
  ninja: error: mkdir(sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cp312-mingw_aarch64_ucrt_llvm.pyd.p/sklearn/metrics/_pairwise_distances_reduction): No such file or directory
  ninja: build stopped: .
  INFO: autodetecting backend as ninja
  INFO: calculating backend command to run: C:\msys64\clangarm64\bin/ninja.EXE
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: C:/msys64/home/oscar/projects_msys/func-ecopower-pwm-forecast-durable/.venv/bin/python.exe C:/msys64/home/oscar/projects_msys/func-ecopower-pwm-forecast-durable/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel C:/pip_build_temp/tmpx1jq7lm1
  cwd: C:/pip_build_temp/pip-install-yjv7z3j9/scikit-learn_75cabb5c67ca4de1b01dff74d7ec4508
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
(.venv)
oscar@pc CLANGARM64 ~/projects_msys/func-ecopower-pwm-forecast-durable
$

Collapsible until here.

**Versions**

(Cannot run sklearn.show_versions() as installation failed)

  • scikit-learn: 1.6.1 (attempted build from source)
  • Python: 3.12.x (mingw-w64-clang-aarch64-python)
  • OS: Windows 11 Pro 23H2 ARM64
  • Compiler: Clang 20.1.1 (mingw-w64-clang-aarch64-clang)
  • Meson: 1.7.0 (mingw-w64-clang-aarch64-meson)
  • Ninja: 1.12.1 (mingw-w64-clang-aarch64-ninja)
  • NumPy: 2.2.4 (built from source successfully in same venv)
  • SciPy: 1.15.2 (built from source successfully in same venv as build dependency)
  • MSYS2 Environment: CLANGARM64

Possible Cause:

The most likely cause is either hitting a path length limit within one of the build tools (Ninja?) despite OS support, or a bug in handling paths containing multiple dots or specific patterns like .pyd.p within the MSYS2 Clang toolchain environment on Windows ARM64.

Thanks for looking into this! Building natively on Win ARM64 is becoming increasingly relevant. I thought this was a good place to start to gather any insights on the issue.
@github-actions github-actions bot added the Needs Triage Issue requires triage label Apr 1, 2025
@lesteve lesteve added Needs Investigation Issue requires investigation and removed Needs Triage Issue requires triage labels Apr 2, 2025
@lesteve
Copy link
Member

lesteve commented Apr 2, 2025 •
edited
Loading

Thanks for the detailed issue 🙏! Unfortunately this will likely be a tricky one. I don't think any of the maintainers has access to a Windows ARM64 machine and even if we did, we are not super familiar with Windows compilation.

If there is a way to change our meson files without complicating it too much and to make it work for your use case, we would be willing to accept such a PR. In particular maybe there is a way to avoid such a nested folder structure sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cp312-mingw_aarch64_ucrt_llvm.pyd.p/sklearn/metrics/_pairwise_distances_reduction?

For the long path hypothesis, I am a bit surprised since MAXPATH is 260 characters according to this Windows doc and in the error message the path is 149 characters long but maybe this is the absolute path that counts 🤔:

ninja: error: mkdir(sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cp312-mingw_aarch64_ucrt_llvm.pyd.p/sklearn/metrics/_pairwise_distances_reduction): No such file or directory

I did find a ninja issue ninja-build/ninja#2442 and an ongoing PR ninja-build/ninja#2552 that you could maybe test to see if that fixes the issue.

@vitorpohlenz vitorpohlenz mentioned this issue Apr 4, 2025
Closed
@lesteve
Copy link
Member

lesteve commented Apr 14, 2025 •
edited
Loading

Actually looking a bit more, this is not ARM64 specific and I can reproduce on a amd64 Windows VM doing:

# This use --no-binary scikit-learn to make sure to compile from sdist rather than use the wheel
pip install scikit-learn --no-binary scikit-learn -v

@Molier as a work-around could you try if this command works for you (it does for me):

python -m pip install --verbose scikit-learn==1.6.1 -v -Cbuild-dir=c:\my-build

This passes an explicit build directory to meson-python following the doc and should avoid too long paths. By default, the build dir is auto-generated even when setting TMPDIR the meson build dir is shorter but still a bit too long for you apparently, e.g. something like this C:/tmp/pip-req-build-kc1tt4we/sklearn ...

There may be ways to reduce a bit the lengths of meson auto-generated files (maybe by 40 characters or so) in our meson.build but I would need to spend more time figuring how to do it and whether this is worth it or not.

@lesteve lesteve changed the title BUG: Build fails for scikit-learn v1.6.1 on Windows ARM64 (MSYS2 ClangARM64) with Ninja mkdir error BUG: Build from source can fail on Windows for scikit-learn v1.6.1 with Ninja mkdir error Apr 14, 2025
@lesteve lesteve added the OS:Windows Problem specific to Windows label Apr 16, 2025
@lesteve lesteve mentioned this issue Apr 16, 2025
Merged
@lesteve
Copy link
Member

lesteve commented Apr 16, 2025 •
edited
Loading

I opened #31212 that reduces the generated files maximum path length by ~50 characters and make it less likely that the Windows path limitation is hit.

@lesteve
Copy link
Member

lesteve commented May 5, 2025 •
edited
Loading

#31212 has been merged which should reduce the likelihood of this happening, let's close this one.

@lesteve lesteve closed this as completed May 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Build / CI Needs Investigation Issue requires investigation OS:Windows Problem specific to Windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lesteve @Molier