BUG: Build from source fails for scikit-learn v1.6.1 on Windows 11 with Visual Studio Build Tools 2022, Ninja subprocess error #31149
Assignees
Labels
Comments
|
Thanks for the report. For the conda-forge method, you still need to build scikit-learn from source. That is: After you do: you need to: and then only check that you can import "sklearn" with: |
|
The ninja error message is not very explicit but it's definitely different from #31123 which is about running mkdir in a non existing folder. Your issue does not seem related to non-existing folders. |
|
I discussed this IRL with @jeremiedbb and @lesteve and they both said that they could build scikit-learn from source by following the instructions (while replacing the 2019 SDK by the 2022 version as you did). So there must be something specific to you setup that we missed. Maybe they did not use the same Windows version? Anyway, a PR would be welcome to fix the instructions (and link) to tell the reader to use the 2022 version o Visual Studio Build Tools. |
|
Just a wild guess, maybe this is due to windows path length limitation, see https://scikit-learn.org/dev/install.html#error-caused-by-file-path-length-limit-on-windows for a possible work-around. Here is the part of your detailed log where the error seems to happen, notice the |
Thank you for replying @ogrisel ! Sorry, I didn't notice that I hadn't built it from source in the |
Yes, I also thought that |
|
Can you include the output of the build command (pip install in editable mode) when you build within the conda forge env? The error you get when importing sklearn implies that scikit-learn wasn't built in successfully. |
Thanks for the heads up @lesteve ! But my path length limit was disabled some time ago:
|
I forgot to save the editions before, sorry about that @ogrisel . Now I have updated the Steps to Reproduce and the Actual Results including a Gist of full conda log |
|
@lesteve, your "wild guess" was quite accurate! The build runs smoothly in those folders, an I can run the command sklearn.show_versions() in C:/ venv
But in conda env I still get an error when building: Conda env error in C:/
Using pip 25.0.1 from C:\Users\vitor.pohlenz\AppData\Local\anaconda3\envs\sklearn-env-new\Lib\site-packages\pip (python 3.13)
scikit-learn 1.7.dev0 Found ninja.EXE-1.12.1 at C:\Users\vitor.pohlenz\AppData\Local\anaconda3\envs\sklearn-env-new\Library\bin\ninja.EXE
× Preparing editable metadata (pyproject.toml) did not run successfully. note: This error originates from a subprocess, and is likely not a problem with pip. × Encountered error while generating package metadata. note: This is an issue with the package mentioned above, not pip. From my side, I do not need to use conda env, I can work completely fine just with the |
|
On the conda env, for some unknown reason you are using free-threaded in your conda env, note cp313t (rather than cp313) in your build for python: Can you try in a new conda env where you explicitly mention Footnotes
|
|
Sorry I didn't see your newer messages before posting my comment about your conda env which you can forget about. Great if you managed to fix your issue! The error message is not helpful at all, not sure if there is a way to improve it under our control ...
That would be great if you can open a PR indeed! No need to create a separate issue, you can link to this issue in your PR. |
|
For later reference, I have found reports that seem to indicate that cl.exe doesn't support long paths even when allowed in the registry, not sure how relevant they still are ... There is also this comment that seems to say that you need to enable long paths with a manifest https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107974#c3. The manifest thingy is also mentioned in the documentation: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#application-manifest-updates-to-declare-long-path-capability. |
@lesteve great tip! If I downgrade Python to 3.12.9 inside the conda env, I can build it from source. The problem seems to be associated with Python 3.13, as you mentioned. Here are the versions that work in conda env: sklearn.show_versions() in C:/ conda env
System: Python dependencies: Built with OpenMP: True threadpoolctl info: internal_api: openmp |
|
Thanks for the help with this issue, everyone! Quite strange that using an "older" and newer version of Python in two different kinds of virtual environments, I faced different problems. It is almost like Murphy's Law. |
|
/take I will open a PR with these adjustments in the Installing the development version of scikit-learn tutorial. |
|
@vitorpohlenz another thing you could do if you want: it is very surprising than you got a free-threading python when installing your conda environment i.e. #31149 (comment). If you manage to reproduce with the last version of |
1 task
|
I actually managed to reproduce on my Windows VM and opened conda-forge/python-feedstock#792 about the free-threading Python by default on Windows when install scipy. Thanks @vitorpohlenz for the report! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
First of all, thank you guys for the fantastic job with Sklearn.
I'm trying to build from source to start contributing to the project, but it ended with me bringing more issues to you.
After struggling for some days with this problem, I'm seeking help. Maybe if you have some clue or workaround, I could open a Pull Request with the solution for this.
I am following the guidelines for Contributing with Scikit-learn, and for that, it is necessary to Build from source on Windows, which recomends install Build Tools for Visual Studio 2019, but nowadays is not possible to download the 2019 version just the Build Tools for Visual Studio 2022 installer.
The installation of Build Tools for Visual Studio 2022 runs smoothly(and also the initialization of its Environment), as well as the creation of the Python virtual environment and the installation of the packages
wheel, numpy, scipy, cython, meson-python, ninja.But in the step of building from source using the
pip install --editable, the build breaks when Compiling C objects after some C4090 warnings, throwing anmetadata-generation-failed errorfrom a subprocess ofninja build.This seems related/similar to issue #31123. Despite not being the same problem, if we find a solution, it may work for both issues.
Steps/Code to Reproduce
I have tried the steps using
pip installand alsoconda-forgein different versions of Python: pip :{3.10.11, 3.12.7} conda:{ 3.13.2} to check if it was a problem with Python/pip itself.C:\Users\vitor.pohlenz\Documents\Python\vpz\scikit-learnInstalled Microsoft build Tools 2022
Initialized the enviroment:
Which gives the output:
Navigated to the project directory:
C:\Users\vitor.pohlenz\Documents\Python\vpz\scikit-learnCreated the Python virtual environment, upgraded pip, and installed the dependencies
Until here, everything works as expected. But when building from source, in the command below, the error occurs:
To check if it wasn't a problem with pip I have also used
conda-forge, as described in the Building from Source tutorial:conda create -n sklearn-env -c conda-forge python numpy scipy cython meson-python ninjaconda activate sklearn-envpip install --editable . --verbose --no-build-isolation --config-settings editable-verbose=trueExpected Results
The build of scikit-learn should finish and install without error, or at least with a few warnings.
Actual Results
Using pip install
It seems that the Meson build system finds the compiler for C, C+++, and also the
ninja.exein the environment. But ninja subprocess breaks when compilingC object sklearn/metrics/_dist_metrics.cp310-win_amd64.pyd.p/meson-generated_sklearn_metrics__dist_metrics.pyx.c.objError message:
Full Log
Using Conda
For
conda env, the problem seems to be associated withundeclared identifiersandsyntax errorsGist of Full conda log
Versions
The text was updated successfully, but these errors were encountered: