I don't really like the module rename. The XP abbreviation is pretty opaque to me.

Everything else here looks good.

I know the dependencies thing is still a TODO, but a very simple thing we can recommend now is to always start with test_signatures.py, since this will tell you which functions are implemented, and of those which have the expected signature.

We should also mention that the suite is designed so that it can be vendored in a library, if desired. I thought the README already mentioned that, but I guess not. Probably if someone wants to actually do that, though, they should tell us, so that we can start doing actual release tags.

Just to be clear, "stacking" in the context of an elementwise function means testing that a function called on an array of values gives the same result as each individual value (an array is a "stack" of its 0-D elements. It might also be called "vectorizing". But it's not actually something I would expect would ever fail for elementwise functions, as opposed to linear algebra where a library might not implement stacking at all.

It'd be good to use "stacking (also called batching)" or some such explanation. Batching is the much more commonly used word.

I don't really like the module rename. The XP abbreviation is pretty opaque to me.

XP is opaque as an abbreviation, but it's possibly the best option and seems to be the convention the spec runs with. Namely I was interested in making running pytest a lot on specific test cases easier, because even with smart shells the verbosity and underscores of "array_api_tests/" got a bit annoying. Just these last two days I've appreciated the name change a lot. Happy if an end-user was to tie-break on reverting the change heh.

Just to be clear, "stacking" in the context of an elementwise function means testing that a function called on an array of values gives the same result as each individual value (an array is a "stack" of its 0-D elements. It might also be called "vectorizing". But it's not actually something I would expect would ever fail for elementwise functions, as opposed to linear algebra where a library might not implement stacking at all.

This is what confused me, because from my understanding of what you're saying, the elementwise tests don't do this explicitly. Hypothesis will generate related examples such as [[0, 1], [2, 3]]/[2, 3]/2, and so test this implicitly, but for floating-point outputs with rough assertions, we currently (and I'd say appropriately) allow for results not to stack.

XP is opaque as an abbreviation, but it's possibly the best option and seems to be the convention the spec runs with.

Where is xp used in the spec? I only see it used once in the "purpose and scope" section as an example variable name for the array namespace. "array_api_tests" on the other hand matches the name of this repo, the array-api repo, and many implementations, and also is pretty self explanatory.

Namely I was interested in making running pytest a lot on specific test cases easier, because even with smart shells the verbosity and underscores of "array_api_tests/" got a bit annoying. Just these last two days I've appreciated the name change a lot. Happy if an end-user was to tie-break on reverting the change heh.

You don't use tab completion? It's the only file in the directory that starts with a. Also unless you are running just a single file you can just use pytest with no argument and it will run the tests, because those are the only ones it will find.

This is what confused me, because from my understanding of what you're saying, the elementwise tests don't do this explicitly. Hypothesis will generate related examples such as [[0, 1], [2, 3]]/[2, 3]/2, and so test this implicitly, but for floating-point outputs with rough assertions, we currently (and I'd say appropriately) allow for results not to stack.

Yes I don't think it's actually implemented yet. That's why I wanted to make a table/checklist for what's missing in the coverage, because I know there's a lot of cases where a test already exists but there's something or other that it doesn't test that it could.

You don't use tab completion? It's the only file in the directory that starts with a.

I do. It's still annoying as it's faster to write it on remotes, and even locally as my computer is slow. I think it's nice for first-time users generally as well.

Also unless you are running just a single file you can just use pytest with no argument and it will run the tests, because those are the only ones it will find.

Right, but 99% of the time I'm specifying tests. Sure I can stay inside the tests folder, but I prefer working in the root.