IMO I think such a granular table isn't worth it. I'm going to spend time filling it out, and then 100% forget to update it in lockstep with test suite changes—at the very least it will be a finicky thing to worry about. We're already 80%+ Yes with #39 anywho. Maybe something better to do after we think the suite is complete, to verify that it indeed is.

I think it's better to just summarise the goals of the test suite i.e. the column categories you've outlined, and state that it's WIP for now. Maybe giving broad allusions to what it lacks (testing some smaller function groups, usability things like nice error messages, etc.).

Well my understanding from the meeting is that we do want it. What do you think @rgommers @kgryte? If we're already close, then we should think about whether we are missing any categories. Otherwise, I think this will be useful to help us understand the remaining 20% that is missing.

Outlining other big TODOs here like better reporting is a good idea. Right now most of those ideas only live in the issue tracker, but that isn't very visible to people using the test suite, and I've often found myself re-explaining it to people.

@honno's point is about the level of granularity I think. This will be a huge table, perhaps it will be easier to parse if it had a single entry/row for "statistical functions" rather than one for each function. That would make sense to me. You'd implement support for a particular type of test (say promotion behavior) for a set of functions at once normally. So little info will be lost, and you may make the table ~5x shorter.

Well now I'm writing tests that that covers all relevant bases at once, and all the existing tests cover all relevant bases, so we can just say now "we need to implement set + utility + searching functions" in a sentence and not have any table or list at all.

Let's clarify what we want. My impression was we want a resource that could give Array API implementers a high-level overview of what this test suite does and doesn't. The current page does a great job doing that, so I'm thinking we just need to tweak it (remove table, readability changes, etc.) and we'll be done.

As discussed in our meeting, let's prioritise expressing the broad strokes for now, and sit with a "completed" test suite sit for a while before documenting the granular details. Good news that's mostly done, so I'll:

• Review the page as-is and fix any minor errors
• See about readability changes i.e. just restructuring the content a bit so skim-readers will still get a good impression
• Go half-way with granularity for now, expressing what our tests generally do per function group

I think users of the test suite have gotten the jist of what the test suite is actually testing with the README as-is, so closing (feel free to re-open).