Add option to use strict xfail mode in `parametrize_with_checks` #31951

betatim · 2025-08-15T09:56:38Z

Reference Issues/PRs

Fixes #31931

What does this implement/fix? Explain your changes.

This adds an option to parametrize_with_checks (and the functions it uses) that allow you to specify "strict mode" for xfailed tests. In strict mode unexpectedly passing tests lead to a test failure. This helps keep the xfail list up to date.

You can configure this globally via pytest.ini for the whole test suite or for individual tests via the xfail marker.

Any other comments?

In cuml we have a lot of common checks that fail and hence are marked as xfail. As we go through fixing things it would be good to have strict mode enabled to help keep the xfail list up to date. We don't really want to turn on strict mode for the whole test suite though (this would be a bigger discussion).

I am not super happy with the testing of this testing infrastructure. For example I only noticed that passing strict=None to the mark overrides the config from pytest.init by trying it out for real, I couldn't figure out a test that would catch it (except the explicit one I added). Ideas welcome, also comments that say "this looks super good enough" :D

In strict mode unexpectedly passing tests lead to a test failure. This helps keep the xfail list up to date.

github-actions · 2025-08-15T09:57:34Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: fa808d2. Link to the linter CI: here}

…ecks-strict-mode

doc/whats_new/upcoming_changes/sklearn.utils/31951.enhancement.rst

sklearn/utils/estimator_checks.py

sklearn/utils/tests/test_estimator_checks.py

lesteve · 2025-08-28T15:59:27Z

I am not super happy with the testing of this testing infrastructure. For example I only noticed that passing strict=None to the mark overrides the config from pytest.init by trying it out for real, I couldn't figure out a test that would catch it (except the explicit one I added). Ideas welcome, also comments that say "this looks super good enough" :D

To make sure I follow, you are saying that doing pytest.mark.xfail(strict=None) is overriding the pytest config? If that's the case that doesn't surprise me too much the function-level value overrides the config one and strict=None is the same as strict=False. If the worry is more about "oh but what happens if someone changes the pytest config and this makes some test fail, could we avoid this", I would say YAGNI (or in a longer way "let's not try to tackle this for now and revisit if this happens often enough for us to care") 🤷.

Co-authored-by: Loïc Estève <loic.esteve@ymail.com>

More consistent with the name of the pytest config option

betatim · 2025-08-29T13:49:04Z

To make sure I follow, you are saying that doing pytest.mark.xfail(strict=None) is overriding the pytest config? If that's the case that doesn't surprise me too much the function-level value overrides the config one and strict=None is the same as strict=False.

This is what I meant. The thing I didn't love was that the tests won't tell you this, you have to know it (or try it out). Basically this whole section

https://github.com/scikit-learn/scikit-learn/pull/31951/files#diff-49f814863816a0e8d199ba647472450ecc59a4375171db23cb1ca36063d6dc25R462-R468

doc/whats_new/upcoming_changes/sklearn.utils/31951.enhancement.rst

….rst

lesteve

I pushed a small tweak in the test, LGTM.

It would be nice if someone more familiar with the estimator checks could have a look, maybe @adrinjalali?

adrinjalali

we could potentially spawn a process and run pytest, but that seems a bit excessive I guess.

sklearn/utils/estimator_checks.py

sklearn/utils/tests/test_estimator_checks.py

…ode' into parametrize-with-checks-strict-mode

Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com>

betatim · 2025-09-01T09:31:53Z

Thanks for the comments.

adrinjalali · 2025-09-01T10:15:06Z

We might even try to have this in sklearn and see how many tests are passing now. Nice idea!

betatim · 2025-09-01T12:03:00Z

I'll make a PR enabling this for scikit-learn just to see what happens.

Add option to use strict xfail mode in parametrize_with_checks

076c65e

In strict mode unexpectedly passing tests lead to a test failure. This helps keep the xfail list up to date.

github-actions bot added the module:utils label Aug 15, 2025

what's new

53d24b8

This comment was marked as off-topic.

Sign in to view

Merge remote-tracking branch 'upstream/main' into parametrize-with-ch…

010f428

…ecks-strict-mode