ENH: meson: implement BLAS/LAPACK auto-detection and many CI jobs #24899
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@charris you may want to squash-merge the two commits and only keep the commit message of the first one - including |
|
Oddly, none of the That was because the second commit disabled them. |
f22d7fa to
ad4f480
Compare
…eel build]
This reimplements the auto-detection and many of the switches that
`numpy.distutils` offered. Beyond that, it implements several new
features:
- Auto-detect the symbol suffix for ILP64 OpenBLAS (can be none or
`64_`)
- MKL ILP64 support, threading control, and use of the layered library
model for MKL >=2023.0
- FlexiBLAS support (LP64 and ILP64)
- Support for the upcoming standard in Reference LAPACK for `_64` ILP64
symbol suffix convention.
- A test suite for BLAS/LAPACK libraries, covering:
- OpenBLAS: LP64, ILP64 detected via pkg-config and with a "system
dependency" (i.e., custom code inside Meson)
- MKL: LP64, ILP64 (layered) and LP64 (SDL)
- Accelerate: LP64, ILP64 on macOS >=13.3
- FlexiBLAS: LP64, ILP64 on Fedora
- ATLAS (LP64, via pkg-config only)
- BLIS (LP64, via pkg-config only)
- plain libblas/liblapack (Netlib, LP64 only)
The list of libraries that is tried with the default 'auto' setting
excludes a couple of libraries, because they're either no longer
developed (ATLAS), not mature (libflame), or can't be tested and
may be re-added later (ArmPL, ssl2). Those libraries can still be
quite easily used via pkg-config.
The new CI jobs are running by default right now. Once things settle
down, the plan is to disable them by default and allow triggering
them via a `[blas ci]` command in the commit message (just like for
wheel builds).
Docs will be included in a separate PR with the pending rewrite of
all the build/install docs. For now, the CI jobs and the
`meson_options.txt` file serve as guidance for how to use this.
Note that the test suite contains a few hacks, because of packaging
bugs for MKL on PyPI (broken .pc files) and BLIS (missing .pc file in
Debian).
ad4f480 to
cbdc785
Compare
|
I think both of these fail because they use ILP64
We could probably delete them, but it would be nice to know where things go wrong. Not clear why travis is failing, I don't think it should be affected. It would be good to keep travis until we figure out why #24814 is failing. |
|
Okay, all the wheel builds are happy, and the failures are happening in jobs that aren't present on |
This is because the change to |
|
I don't know what's up with the That does not seem related to the last two commits. EDIT: oh wait, it is: |
|
The Musl thing would be much easier to diagnose with a decent traceback. @stefanv here is another example of If I reproduce the issue locally and |
This should fix the MUSL CI job failure after the Meson auto-BLAS/LAPACK update.
[skip circle] [skip cirrus]
40cdb4e to
891fb22
Compare
|
All green now and the commits are all backports that look fine, so in it goes. Thanks Chuck! I'll propose content for the release notes as well. EDIT: see gh-24902 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #24893.
This reimplements the auto-detection and many of the switches that
numpy.distutilsoffered. Beyond that, it implements several new features:Auto-detect the symbol suffix for ILP64 OpenBLAS (can be none or
64_)MKL ILP64 support, threading control, and use of the layered library model for MKL >=2023.0
FlexiBLAS support (LP64 and ILP64)
Support for the upcoming standard in Reference LAPACK for
_64ILP64 symbol suffix convention.A test suite for BLAS/LAPACK libraries, covering:
The list of libraries that is tried with the default 'auto' setting excludes a couple of libraries, because they're either no longer developed (ATLAS), not mature (libflame), or can't easily be tested and may be re-added later (ArmPL, ssl2). Those libraries can still be quite easily used via pkg-config.
The new CI jobs are running by default right now. Once things settle down, the plan is to disable them by default and allow triggering them via a
[blas ci]command in the commit message (just like for wheel builds).Docs will be included in a separate PR with the pending rewrite of all the build/install docs. For now, the CI jobs and the
meson_options.txtfile serve as guidance for how to use this.Note that the test suite contains a few hacks, because of packaging bugs for MKL on PyPI (broken .pc files) and BLIS (missing .pc file in Debian).
Closes gh-24808
Closes gh-24846
Addresses the most important remaining task from the introduction of Meson (see gh-23981).