Skip to content

Commit 65182a5

Browse files
honnohonno
authored
Merge pull request #53 from honno/readme
Usability improvements
2 parents 2260ae0 + 5a4bc96 commit 65182a5

File tree

91 files changed

+184
-233
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+184
-233
lines changed

‎.github/workflows/numpy.yml

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -23,30 +23,30 @@ jobs:
2323
python -m pip install -r requirements.txt
2424
- name: Run the test suite
2525
env:
26-
ARRAY_API_TESTS_MODULE: numpy.array_api
26+
XPTESTS_MODULE: numpy.array_api
2727
run: |
2828
# Mark some known issues as XFAIL
2929
cat << EOF >> xfails.txt
3030
3131
# https://github.com/numpy/numpy/issues/18881
32-
array_api_tests/test_creation_functions.py::test_linspace
32+
xptests/test_creation_functions.py::test_linspace
3333
# einsum is not yet completed in the spec
34-
array_api_tests/test_signatures.py::test_has_names[einsum]
34+
xptests/test_signatures.py::test_has_names[einsum]
3535
# dlpack support is not yet implemented in NumPy
3636
# See https://github.com/numpy/numpy/pull/19083
37-
array_api_tests/test_signatures.py::test_function_positional_args[__dlpack__]
38-
array_api_tests/test_signatures.py::test_function_positional_args[__dlpack_device__]
39-
array_api_tests/test_signatures.py::test_function_positional_args[from_dlpack]
40-
array_api_tests/test_signatures.py::test_function_positional_args[to_device]
41-
array_api_tests/test_signatures.py::test_function_keyword_only_args[__dlpack__]
37+
xptests/test_signatures.py::test_function_positional_args[__dlpack__]
38+
xptests/test_signatures.py::test_function_positional_args[__dlpack_device__]
39+
xptests/test_signatures.py::test_function_positional_args[from_dlpack]
40+
xptests/test_signatures.py::test_function_positional_args[to_device]
41+
xptests/test_signatures.py::test_function_keyword_only_args[__dlpack__]
4242
# floor_divide has an issue related to https://github.com/data-apis/array-api/issues/264
43-
array_api_tests/test_elementwise_functions.py::test_floor_divide
43+
xptests/test_elementwise_functions.py::test_floor_divide
4444
# mesgrid doesn't return all arrays as the promoted dtype
45-
array_api_tests/test_type_promotion.py::test_meshgrid
45+
xptests/test_type_promotion.py::test_meshgrid
4646
# https://github.com/numpy/numpy/pull/20066#issuecomment-947056094
47-
array_api_tests/test_type_promotion.py::test_where
47+
xptests/test_type_promotion.py::test_where
4848
# shape mismatches are not handled
49-
array_api_tests/test_type_promotion.py::test_tensordot
49+
xptests/test_type_promotion.py::test_tensordot
5050
5151
EOF
5252

‎README.md

Lines changed: 158 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -1,166 +1,212 @@
1-
# Array API Standard Test Suite
1+
# Test Suite for Array API Compliance
22

3-
This is the test suite for the Python array API standard.
3+
This is the test suite for array libraries adopting the [Python Array API
4+
standard](https://data-apis.org/array-api/).
45

5-
**NOTE: This test suite is still a work in progress.**
6+
Note the suite is still a **work in progress**. Feedback and contributions are
7+
welcome!
68

7-
Feedback and contributions are welcome, but be aware that this suite is not
8-
yet completed. In particular, there are still many parts of the array API
9-
specification that are not yet tested here.
10-
11-
## Running the tests
9+
## Quickstart
1210

1311
### Setup
1412

15-
To run the tests, first install the testing dependencies
13+
To run the tests, install the testing dependencies.
14+
15+
```bash
16+
$ pip install -r requirements
17+
```
1618

17-
pip install pytest hypothesis
19+
Ensure you have the array library that you want to test installed.
1820

19-
or
21+
### Specifying the array module
2022

21-
conda install pytest hypothesis
23+
You need to specify the array library to test. It can be specified via the
24+
`XPTESTS_MODULE` environment variable, e.g.
2225

23-
as well as the array libraries that you want to test.
26+
```bash
27+
$ export XPTESTS_MODULE=numpy.array_api
28+
```
2429

25-
### Specifying the array module
30+
Alternately, change the `array_module` variable in `xptests/_array_module.py`
31+
line, e.g.
2632

27-
To run the tests, you need to set the array library that is to be tested. There
28-
are two ways to do this. One way is to set the `ARRAY_API_TESTS_MODULE`
29-
environment variable. For example you can set it when running `pytest`
33+
```diff
34+
- array_module = None
35+
+ import numpy.array_api as array_module
36+
```
3037

31-
ARRAY_API_TESTS_MODULE=numpy pytest
38+
### Run the suite
3239

33-
Alternately, edit the `array_api_tests/_array_module.py` file and change the
34-
line
40+
Simply run `pytest` against the `xptests/` folder to run the full suite.
3541

36-
```py
37-
array_module = None
42+
```bash
43+
$ pytest xptests/
3844
```
3945

40-
to
46+
The suite tries to logically organise its tests. `pytest` allows you to only run
47+
a specific test case, which is useful when developing functions.
4148

42-
```py
43-
import numpy as array_module
49+
```bash
50+
$ pytest xptests/test_creation_functions.py::test_zeros
4451
```
4552

46-
(replacing `numpy` with the array module namespace to be tested).
53+
## What the test suite covers
54+
55+
We are interested in array libraries conforming to the
56+
[spec](https://data-apis.org/array-api/latest/API_specification/index.html).
57+
Ideally this means that if a library has fully adopted the Array API, the test
58+
suite passes. We take great care to _not_ test things which are out-of-scope, so
59+
as to not unexpectedly fail the suite.
60+
61+
### Primary tests
62+
63+
Every function—including array object methods—has a respective test method. We
64+
use [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) to generate a
65+
diverse set of valid inputs. This means array inputs will cover different dtypes
66+
and shapes, as well as contain interesting elements. These examples generate
67+
with interesting arrangements of non-array positional arguments and keyword
68+
arguments.
69+
70+
Each test case will cover the following areas if relevant:
71+
72+
* **Smoking**: We pass our generated examples to all functions. As these
73+
examples solely consist of *valid* inputs, we are testing that functions can
74+
be called using their documented inputs without raising errors.
4775

48-
### Specifying test cases
76+
* **Data type**: For functions returning/modifying arrays, we assert that output
77+
arrays have the correct data types. Most functions
78+
[type-promote](https://data-apis.org/array-api/latest/API_specification/type_promotion.html)
79+
input arrays and some functions have bespoke rules—in both cases we simulate
80+
the correct behaviour to find the expected data types.
4981

50-
The test suite tries to logically organise its tests so you can find specific
51-
test cases whilst developing something in particular. So to avoid running the
52-
rather slow complete suite, you can specify particular test cases like any other
53-
test suite.
82+
* **Shape**: For functions returning/modifying arrays, we assert that output
83+
arrays have the correct shape. Most functions
84+
[broadcast](https://data-apis.org/array-api/latest/API_specification/broadcasting.html)
85+
input arrays and some functions have bespoke rules—in both cases we simulate
86+
the correct behaviour to find the expected shapes.
5487

55-
pytest array_api_tests/test_creation_functions.py::test_zeros
88+
* **Values**: We assert output values (including the elements of
89+
returned/modified arrays) are as expected. Except for manipulation functions
90+
or special cases, the spec allows floating-point inputs to have inexact
91+
outputs, so with such examples we only assert values are roughly as expected.
5692

57-
## Notes on Interpreting Errors
93+
### Additional tests
5894

59-
- Some tests cannot be run unless other tests pass first. This is because very
60-
basic APIs such as certain array creation APIs are required for a large
61-
fraction of the tests to run. TODO: Write which tests are required to pass
62-
first here.
95+
In addition to having one test case for each function, we test other properties
96+
of the functions and some miscellaneous things.
6397

64-
- If an error message involves `_UndefinedStub`, it means some name that is
65-
required for the test to run is not defined in the array library.
98+
* **Special cases**: For functions with special case behaviour, we assert that
99+
these functions return the correct values.
66100

67-
- Due to the nature of the array api spec, virtually every array library will
68-
produce a large number of errors from nonconformance. It is still a work in
69-
progress to enable reporting the errors in a way that makes them easy to
70-
understand, even if there are a large number of them.
101+
* **Signatures**: We assert functions have the correct signatures.
71102

72-
- The spec documents are the ground source of truth. If the test suite appears
73-
to be testing something that is different from the spec, or something that
74-
isn't actually mentioned in the spec, this is a bug. [Please report
75-
it](https://github.com/data-apis/array-api-tests/issues/new). Furthermore,
76-
be aware that some aspects of the spec are either impossible or extremely
77-
difficult to actually test, so they are not covered in the test suite (TODO:
78-
list what these are).
103+
* **Constants**: We assert that
104+
[constants](https://data-apis.org/array-api/latest/API_specification/constants.html)
105+
behave expectedly, are roughly the expected value, and that any related
106+
functions interact with them correctly.
107+
108+
Be aware that some aspects of the spec are impractical or impossible to actually
109+
test, so they are not covered in the suite <!-- TODO: note what these are -->
79110

80-
## Configuring Tests
111+
## Interpreting errors
112+
113+
First and foremost, note that most tests have to assume that certain aspects of
114+
the Array API have been correctly adopted, as fundamental APIs such as array
115+
creation and equalities are hard requirements for many assertions. This means a
116+
test case for one function might fail because another function has bugs or even
117+
no implementation.
118+
119+
This means adopting libraries at first will result in a vast number of errors
120+
due to cascading errors. Generally the nature of the spec means many granular
121+
details such as type promotion is likely going to also fail nearly-conforming
122+
functions.
123+
124+
We hope to improve user experience in regards to "noisy" errors in
125+
[#51](https://github.com/data-apis/array-api-tests/issues/51). For now, if an
126+
error message involves `_UndefinedStub`, it means an attribute of the array
127+
library (including functions) and it's objects (e.g. the array) is missing.
128+
129+
The spec is the suite's source of truth. If the suite appears to assume
130+
behaviour different from the spec, or test something that is not documented,
131+
this is a bug—please [report such
132+
issues](https://github.com/data-apis/array-api-tests/issues/) to us.
133+
134+
## Configuration
81135

82136
By default, tests for the optional Array API extensions such as
83137
[`linalg`](https://data-apis.org/array-api/latest/extensions/linear_algebra_functions.html)
84138
will be skipped if not present in the specified array module. You can purposely
85139
skip testing extension(s) via the `--disable-extension` option, and likewise
86140
purposely test them via the `--enable-extension` option.
87141

88-
The tests make heavy use of the
89-
[Hypothesis](https://hypothesis.readthedocs.io/en/latest/) testing library.
90-
Hypothesis generates random input values for the tests. You can configure how
91-
many values are generated and run using the `--max-examples` flag. The default
92-
`--max-examples` is 100. For example, `--max-examples 50` will only generate
93-
half as many examples and as a result, the test suite will run in about half
94-
the time. Setting `--max-examples` to a lower value can be useful when you
95-
want to have a faster test run. It can also be useful to set `--max-examples`
96-
to a large value to do a longer, more rigorous run of the tests. For example,
97-
`--max-examples 10000` will do a very rigorous check of the tests, but may
98-
take a few hours to run.
99-
100-
## Contributing
101-
102-
### Adding Tests
142+
The tests make heavy use
143+
[Hypothesis](https://hypothesis.readthedocs.io/en/latest/). You can configure
144+
how many examples are generated using the `--max-examples` flag, which defaults
145+
to 100. Lower values can be useful for quick checks, and larger values should
146+
result in more rigorous runs. For example, `--max-examples 10000` may find bugs
147+
where default runs don't, but will take a much longer time.
103148

104-
It is important that every test in the test suite only uses APIs that are part
105-
of the standard. This means that, for instance, when creating test arrays, you
106-
should only use array creation functions that are part of the spec, such as
107-
`ones` or `full`. It also means that many array testing functions that are
108-
built-in to libraries like numpy are reimplemented in the test suite (see
109-
`array_api_tests/pytest_helpers.py`, `array_api_tests/array_helpers.py`, and
110-
`array_api_tests/hypothesis_helpers.py`).
149+
<!-- TODO: howto on CI -->
111150

112-
In order to enforce this, the `array_api_tests._array_module` should be used
113-
everywhere in place of the actual array module that is being tested.
151+
## Contributing
114152

115-
### Hypothesis
153+
### Remain in-scope
116154

117-
The test suite uses [Hypothesis](https://hypothesis.readthedocs.io/en/latest/)
118-
to generate random input data. Any test that should be applied over all
119-
possible array inputs should use hypothesis tests. Custom Hypothesis
120-
strategies are in the `array_api_tests/hypothesis_helpers.py` file.
155+
It is important that every test only uses APIs that are part of the standard.
156+
For instance, when creating input arrays you should only use the [array creation
157+
functions](https://data-apis.org/array-api/latest/API_specification/creation_functions.html)
158+
that are documented in the spec. The same goes for testing arrays—you'll find
159+
many utilities that parralel NumPy's own test utils in the `*_helpers.py` files.
121160

122-
### Parameterization
161+
### Tools
123162

124-
Any test that applies over all functions in a module should use
125-
`pytest.mark.parametrize` to parameterize over them. For example,
163+
Hypothesis should always be used for the primary tests, and can be useful
164+
elsewhere. Effort should be made so drawn arguments are labeled with their
165+
respective names. For
166+
[`st.data()`](https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.data),
167+
draws should be accompanied with the `label` kwarg i.e. `data.draw(<strategy>,
168+
label=<label>)`.
126169

127-
```py
128-
from . import function_stubs
170+
[`pytest.mark.parametrize`](https://docs.pytest.org/en/latest/how-to/parametrize.html)
171+
should be used to run tests over multiple arguments. Parameterization should be
172+
preferred over using Hypothesis when there are a small number of possible
173+
inputs, as this allows better failure reporting. Note using both parametrize and
174+
Hypothesis for a single test method is possible and can be quite useful.
129175

130-
@pytest.mark.parametrize('name', function_stubs.__all__)
131-
def test_whatever(name):
132-
...
133-
```
176+
### Error messages
134177

135-
will parameterize `test_whatever` over all the functions stubs generated from
136-
the spec. Parameterization should be preferred over using Hypothesis whenever
137-
there are a finite number of input possibilities, as this will cause pytest to
138-
report failures for all input values separately, as opposed to Hypothesis
139-
which will only report one failure.
178+
Any assertion should be accompanied with a descriptive error message, including
179+
the relevant values. Error messages should be self-explanatory as to why a given
180+
test fails, as one should not need prior knowledge of how the test is
181+
implemented.
140182

141-
### Error Strings
183+
### Generated files
142184

143-
Any assertion or exception should be accompanied with a useful error message.
144-
The test suite is designed to be ran by people who are not familiar with the
145-
test suite code, so the error messages should be self explanatory as to why
146-
the module fails a given test.
185+
Some files in the suite are automatically generated from the spec, and should
186+
not be edited directly. To regenerate these files, run the script
147187

148-
### Meta-errors
188+
./generate_stubs.py path/to/array-api
149189

150-
Any error that indicates a bug in the test suite itself, rather than in the
151-
array module not following the spec, should use `RuntimeError` whenever
152-
possible.
190+
where `path/to/array-api` is the path to a local clone of the [`array-api`
191+
repo](https://github.com/data-apis/array-api/). Edit `generate_stubs.py` to make
192+
changes to the generated files.
153193

154-
(TODO: Update this policy to something better. See [#5](https://github.com/data-apis/array-api-tests/issues/5).)
194+
## Future plans
155195

156-
### Automatically Generated Files
196+
Keeping full coverage of the spec is an on-going priority as the Array API
197+
evolves.
157198

158-
Some files in the test suite are automatically generated from the API spec
159-
files. These files should not be edited directly. To regenerate these files,
160-
run the script
199+
Additionally, we have features and general improvements planned. Work on such
200+
functionality is guided primarily by the concerete needs of developers
201+
implementing and using the Array API—be sure to [let us
202+
know](https://github.com/data-apis/array-api-tests/issues) any limitations you
203+
come across.
161204

162-
./generate_stubs.py path/to/array-api
205+
* A dependency graph for every test case, which could be used to modify pytest's
206+
collection so that low-dependency tests are run first, and tests with faulty
207+
dependencies would skip/xfail.
163208

164-
where `path/to/array-api` is the path to the local clone of the `array-api`
165-
repo. To modify the automatically generated files, edit the code that
166-
generates them in the `generate_stubs.py` script.
209+
* In some tests we've found it difficult to find appropaite assertion parameters
210+
for output values (particularly epsilons for floating-point outputs), so we
211+
need to review these and either implement assertions or properly note the lack
212+
thereof.

‎conftest.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@
44
from pytest import mark
55
from hypothesis import settings
66

7-
from array_api_tests import _array_module as xp
8-
from array_api_tests._array_module import _UndefinedStub
7+
from xptests import _array_module as xp
8+
from xptests._array_module import _UndefinedStub
99

1010

1111
settings.register_profile('xp_default', deadline=800)
@@ -77,7 +77,7 @@ def xp_has_ext(ext: str) -> bool:
7777
if xfails_path.exists():
7878
with open(xfails_path) as f:
7979
for line in f:
80-
if line.startswith('array_api_tests'):
80+
if line.startswith('xptests'):
8181
id_ = line.strip('\n')
8282
xfail_ids.append(id_)
8383

0 commit comments

Comments
 (0)