Skip to content

CI: Bump array-api-tests to v2025.05.23 #29149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 11, 2025

Conversation

jorenham
Copy link
Member

@jorenham jorenham commented Jun 8, 2025

@jorenham jorenham added component: CI 40 - array API standard PRs and issues related to support for the array API standard labels Jun 8, 2025
@jorenham
Copy link
Member Author

jorenham commented Jun 8, 2025

cc @ev-br

@ev-br
Copy link
Contributor

ev-br commented Jun 9, 2025

This update picks up several improvements in the test suite for the 2024.12 array api revision: the updated suite has a full functional coverage of the 2024.12 spec. Thus this PR LGTM.
The failing test is a known issue where numpy deviates from the Array API spec, and so do basically all other libraries:
data-apis/array-api-tests#317
Thus it seems reasonable to just add an xfail similar to https://github.com/data-apis/array-api-compat/blob/main/numpy-xfails.txt#L3 and move on.


One potential follow-up improvement could be to increase the number of hypothesis examples the CI job runs. It currently uses --max-examples=100 which is rather small. This is a kind of important because hypothesis is essentially fuzzing, and a small number of examples means that it does not even probe "interesting" cases.

In array-api-compat, we recently bumped the number of examples to 1000 while keeping the runtime under 5 minutes in data-apis/array-api-compat#321

@jorenham
Copy link
Member Author

jorenham commented Jun 9, 2025

I'm a bit out of my comfort zone here, but I think I xfailed the failing tests now, and I increased the --max-examples to a modest 500.

Co-authored-by: Evgeni Burovski <evgeny.burovskiy@gmail.com>
@jorenham jorenham requested a review from ev-br June 11, 2025 18:39
@ev-br
Copy link
Contributor

ev-br commented Jun 11, 2025

LGTM.

@charris charris merged commit a3ec9af into numpy:main Jun 11, 2025
74 checks passed
@charris
Copy link
Member

charris commented Jun 11, 2025

Thanks Joren.

@jorenham jorenham deleted the array-api-tests/2025.05.23 branch June 11, 2025 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
05 - Testing 40 - array API standard PRs and issues related to support for the array API standard component: CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants