Skip to content

Random Windows wheel build error: failed with exit status 1158 #22310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ogrisel opened this issue Jan 27, 2022 · 3 comments
Open

Random Windows wheel build error: failed with exit status 1158 #22310

ogrisel opened this issue Jan 27, 2022 · 3 comments
Labels
Build / CI

Comments

@ogrisel
Copy link
Member

ogrisel commented Jan 27, 2022 •
edited by thomasjpfan
Loading

The nightly windows builds randomly fail with the following error:

error: Command "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\cibw\python\python.3.10.1\tools\libs /LIBPATH:C:\cibw\python\python.3.10.1\tools\PCbuild\amd64 /LIBPATH:build\temp.win-amd64-3.10 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\um\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\um\x64 /EXPORT:PyInit__hierarchical_fast build\temp.win-amd64-3.10\Release\sklearn\cluster\_hierarchical_fast.obj /OUT:build\lib.win-amd64-3.10\sklearn\cluster\_hierarchical_fast.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.10\Release\sklearn\cluster\_hierarchical_fast.cp310-win_amd64.lib /openmp" failed with exit status 1158

https://github.com/scikit-learn/scikit-learn/runs/4961291550?check_suite_focus=true

The above happened with cp310-win_amd64 but apparently this can happen with other Python versions. Generally restarting the build is enough to get it to work again.

Let's try to collect below which config and file fail, to see if there is a pattern:

  • cp310-win_amd64 / sklearn\cluster\_hierarchical_fast.cp310-win_amd64.lib
  • cp38-win_amd64 / sklearn\ensemble\_gradient_boosting.cp38-win_amd64.lib
  • cp38-win32 / sklearn\ensemble\_hist_gradient_boosting\_binning.cp38-win32.lib
  • cp38-win32 / sklearn\utils\_openmp_helpers.cp38-win32.lib
  • cp39-win32/ sklearn\decomposition\_cdnmf_fast.cp39-win32.lib

Link to all wheel building runs that were scheduled
@thomasjpfan thomasjpfan mentioned this issue Jan 27, 2022
Closed
@thomasjpfan
Copy link
Member

Could be related to pypa/setuptools#2212

A race condition would explain why it happens randomly on different files and on different instances.

@thomasjpfan
Copy link
Member

thomasjpfan commented Jan 28, 2022 •
edited
Loading

So pypa/distutils#6 was opened to fix pypa/setuptools#2212 which did not work with numpy.distutils. NumPy was updated with a fix here: numpy/numpy#20640

The fix in numpy/numpy#20640 was backported to 1.22, but since we build with older versions of NumPy the windows build will sometimes error out because of the original linking issue.

@ogrisel ogrisel mentioned this issue Feb 18, 2022
Closed
@ogrisel
Copy link
Member Author

ogrisel commented Feb 18, 2022

So we just have to wait and restart the failing builds during the release process and hope for the best.

It it really becomes too frequent we can maintain our own backport of the fix as a monkey patch but since we plan to switch the build system to meson, it's probably not worth investing time in a monkeypatch maintenance.

@ogrisel ogrisel mentioned this issue Feb 18, 2022
Merged
This was referenced Feb 26, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Build / CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ogrisel @thomasjpfan