Add option to use strict xfail mode in parametrize_with_checks
#31951
+96
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In strict mode unexpectedly passing tests lead to a test failure. This helps keep the xfail list up to date.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Fixes #31931
What does this implement/fix? Explain your changes.
This adds an option to
parametrize_with_checks(and the functions it uses) that allow you to specify "strict mode" for xfailed tests. In strict mode unexpectedly passing tests lead to a test failure. This helps keep the xfail list up to date.You can configure this globally via
pytest.inifor the whole test suite or for individual tests via thexfailmarker.Any other comments?
In cuml we have a lot of common checks that fail and hence are marked as xfail. As we go through fixing things it would be good to have strict mode enabled to help keep the xfail list up to date. We don't really want to turn on strict mode for the whole test suite though (this would be a bigger discussion).
I am not super happy with the testing of this testing infrastructure. For example I only noticed that passing
strict=Noneto the mark overrides the config frompytest.initby trying it out for real, I couldn't figure out a test that would catch it (except the explicit one I added). Ideas welcome, also comments that say "this looks super good enough" :D