This is an experimental branch to explore a way to adopt PEP 770 for our vendored runtime dependencies. The goal is to make software composition analysis (SCA) tools such as syft able to properly report metadata about vendored components for vulnerability detection or licensing compliance concerns.

This involves using the development branch of vendoring as documented in the sklearn/_vendored/README.md file instead of custom bash scripts. vendoring allows ignoring a list of files in the destination folder but not folders. Hence, I decided to move array-api-extra and array-api-compat to sklearn/_vendored instead of sklearn/externals to avoid deleting our sklearn/externals/_scipy compat folder structure. Note that sklearn/externals should better be renamed to sklearn/_compat because the code there cannot be automatically managed by a vendoring tool.

Also note that the current state of this PR is not enough to implement PEP 770 for our vendored runtime dependencies, as we would also need to move the generated SBOM file under .dist-info/sboms when building the wheel: psf/sboms-for-python-packages#5 (comment)

Note that a SBOM file for vendored build-time injected native runtime dependencies (such as libgomp on Linux) will be generated by auditwheel repair once pypa/auditwheel#577 is merged and released. Similar efforts in wheel post-processing tools will be necessary for macOS and Windows.

This PR is loosely related to #28151, in the sense that #28151 might also involve adopting SBOM files, although for another purpose: explicitly recording metadata of build-time dependencies (including compilers) for the sake of making the build bitwise reproducible.

TODO:

• Find a way to move or copy the BOM file into the .dist-info/sboms folder when building the wheel.
• Wait for a stable release of vendoring? Instruct maintainainers to use install a specific commit hash of vendoring if no planned release?