Skip to content

BUG: can't run tests after JSON reporting addition #138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tylerjereddy opened this issue Aug 25, 2022 · 2 comments · Fixed by #139
Closed

BUG: can't run tests after JSON reporting addition #138

tylerjereddy opened this issue Aug 25, 2022 · 2 comments · Fixed by #139

Comments

@tylerjereddy
Copy link
Contributor

Following the merge of gh-131 a few days ago, our testing with the suite here started to fail as described at kokkos/pykokkos#63

I can reproduce in act and locally as well:

platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/tyler/github_projects/array-api-tests
plugins: forked-1.4.0, hypothesis-6.54.3, metadata-2.0.2, json-report-1.5.0, xdist-2.5.0
collected 9 items                                                                                                                                                                                                                                                                    
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/_pytest/main.py", line 321, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/_pytest/main.py", line 332, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/_pytest/main.py", line 659, in perform_collect
INTERNALERROR>     self.config.pluginmanager.check_pending()
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_manager.py", line 262, in check_pending
INTERNALERROR>     raise PluginValidationError(
INTERNALERROR> pluggy._manager.PluginValidationError: unknown hook 'pytest_json_modifyreport' in plugin <module 'conftest' from '/home/tyler/github_projects/array-api-tests/conftest.py'>

From a quick check, this diff gets rid of the problem, but I'm not sufficiently familiar with the hook side of pytest to say if this is what you'd want (and how you'd adjust your CI to catch the original issue).

diff --git a/reporting.py b/reporting.py
index b15b336..f7c7d6b 100644
--- a/reporting.py
+++ b/reporting.py
@@ -91,6 +91,7 @@ def add_extra_json_metadata(request, json_metadata):
 
     request.addfinalizer(finalizer)
 
+@mark.optionalhook
 def pytest_json_modifyreport(json_report):
     # Deduplicate warnings. These duplicate warnings can cause the file size
     # to become huge. For instance, a warning from np.bool which is emitted

In fact, looking more closely at the merged PR, your CI did fail with the same error we are seeing in pykokkos! Should I just pin to a commit for a while or was that accidental?
@asmeurer
Copy link
Member

The pytest-json-report plugin should be required. Are you getting this error with it installed? It's in the requirements.txt

With that being said, if this diff makes it optional, I'm fine with that.

Also you should only test against tagged releases. I wonder if there's an easy for people to test the "latest" tag without having to manually update it, but also not just always testing master. Maybe we should make a latest branch which always points to the latest tag, and make that the default branch. CC @honno

@tylerjereddy
Copy link
Contributor Author

tylerjereddy commented Aug 25, 2022 via email

asmeurer added a commit to asmeurer/array-api-tests that referenced this issue Aug 26, 2022
@asmeurer
Fix the tests when --json-report is not used
4afa895 
Fixes data-apis#138.
@asmeurer asmeurer mentioned this issue Aug 26, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix the tests when --json-report is not used asmeurer/array-api-tests
2 participants
@asmeurer @tylerjereddy