Skip to content

BLD Remove build_utils/ from the built wheels, and associated Meson cleanups #29821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

agriyakhetarpal
Copy link
Contributor

Reference Issues/PRs

Stemmed off from #29791

What does this implement/fix? Explain your changes.

This PR aims to remove the sklearn/_build_utils/ folder from the wheel (while keeping it in the sdist to allow building wheels). The utility file sklearn/_build_utils/tempita.py seems to have created a runtime dependency on Cython in the PR linked above that aims to improve the Pyodide builds, because Cython is being imported at the time of running the test suite. I'm unsure how it works for the current builds with the [pyodide] commit marker, but either way: since the _build_utils/ files are needed at just build time, it would make sense to remove them from the wheel artifacts.

This behaviour matches that of NumPy, where the contents of this folder have likely been inspired from back in #28040. The NumPy source distribution on PyPI includes this folder since it's needed at build-time to build wheels off of the sdist, but removes it from the wheel. There are a few more potential and possible Meson-related cleanups to do here but I am not too eager to push them in this PR to keep the diff small :)

In particular, here's a short summary of the changes:

  • Replaced files() with the more standard find_program()
  • Used the exclude_directories: keyword argument for install_subdir() to remove the build utilities from the wheels from the installation
  • Added a shebang at the top of the tempita.py CLI utility to invoke the tempita executable directly

Any other comments?

N/A

Copy link

github-actions bot commented Sep 9, 2024

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: acc3976. Link to the linter CI: here

@agriyakhetarpal
Copy link
Contributor Author

Hi @lesteve, this PR is ready for review – I pushed 84d2cc6 to trigger the wheel builder and the Azure Pyodide builds, just to confirm that they work as intended. Once this is reviewed and gets merged, I can rebase #29791. Thanks!

Comment on lines +1 to +2
#!/usr/bin/env python3

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the reason why this shebang has been added, please refer to numpy/numpy#24971.

@agriyakhetarpal
Copy link
Contributor Author

The wheels build fine, but the _build_utils/ folder still exists there :/ I'll take another look at this later in the day.

@agriyakhetarpal agriyakhetarpal marked this pull request as draft September 9, 2024 22:25
@lesteve
Copy link
Member

lesteve commented Sep 10, 2024

The wheels build fine, but the _build_utils/ folder still exists there :/ I'll take another look at this later in the day.

I pushed a commit that I think fixes it c5a71bf

@agriyakhetarpal
Copy link
Contributor Author

Oops, yes – thanks!

@lesteve
Copy link
Member

lesteve commented Sep 10, 2024

Oh well not so simple, looks like this breaks the test discovery see build log ...

__________________ ERROR collecting _build_utils/__init__.py ___________________
ImportError while importing test module '/home/vsts/work/1/s/sklearn/_build_utils/__init__.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/share/miniconda/envs/testvenv/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'sklearn._build_utils'
___________________ ERROR collecting _build_utils/tempita.py ___________________
ImportError while importing test module '/home/vsts/work/1/s/sklearn/_build_utils/tempita.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/share/miniconda/envs/testvenv/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'sklearn._build_utils'
___________________ ERROR collecting _build_utils/version.py ___________________
ImportError while importing test module '/home/vsts/work/1/s/sklearn/_build_utils/version.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/share/miniconda/envs/testvenv/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'sklearn._build_utils'

@agriyakhetarpal
Copy link
Contributor Author

I think there should be a way to manoeuvre around it for the tests. I guess it's because of the doctest_modules suggestion you mentioned in the other PR.

@czgdp1807
Copy link

The solution to it is from my perspective will be to use install_tags. Add meson.build file inside sklearn/_build_utils and use install_tags. See my PRs on how we selectively install tests. You can use similiar logic to avoid installing sklearn/_build_utils maybe.

PRs - scipy/scipy#20712

@agriyakhetarpal
Copy link
Contributor Author

Thanks for chiming in, @czgdp1807. I've experimented with adding install tags locally, and doing that seems to be a larger change which would need more consideration and reviewing effort in another PR (it's certainly useful to do, though); is there a more—for the lack of a better word—"organic" method to exclude files from sdists (and subsequently, wheels) in the meantime? In any case, I plan to revisit this PR next week.

@agriyakhetarpal
Copy link
Contributor Author

Closing this PR as #31058 is now merged. Thank you!

@agriyakhetarpal agriyakhetarpal deleted the build/refactor-tempita branch March 24, 2025 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants