CI Enable parallel build on CI via an env variable #18826
Conversation
Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>
|
Actually the azure builds now last 30 min instead of 25 min on master. I suspect that this is because it's worth using So maybe I will remove the auto mode and just allow to pass an integer as an environment variable. This will be simpler to maintain. |
You did not finish your sentence :) What's the issue with this ? |
Ooops. I edited the description. |
|
Setting parallelism to 3 instead of 2 on a 2 core machine is not necessarily that beneficial in retrospect. But it does not seem to hurt either. Anyways, I don't plan to tweak this PR any further. |
|
Thanks! |
On the wheel build, it's not easy to do the usual
python setup.py build_ext -j 3because cibuildwheel does not allow to customize the build command and it uses pip wheel that does the build in an isolated environment (which is otherwise nice to make sure we respect the pinned versions of the build dependencies from pyproject.toml).Furthermore, I expect that on some future ARM builds we will get many more cores than on x86 hosts but we need to be able to automatically defect the number of usable cores for optimal performance and avoiding over-subscription caused by docker CPU quotas for instance.
So here is a tentative update of our custom
build_extcommandto automatically use as many core as possible by settingto use 3 parallel tasks by settingSKLEARN_BUILD_PARALLEL=autoSKLEARN_BUILD_PARALLEL=3in the various CI environments.Edit: use a static level of parallel build task to allow for overlapping IO and CPU bound tasks.