Describe the workflow you want to enable

I'd like to be able to use parametrize_with_checks and use "strict mode" to notice when checks that are marked as xfail start passing. But I don't want to turn on strict mode for my whole test suite (xfail_strict = true in pytest.ini)

Describe your proposed solution

We use pytest.mark.xfail internally when generating all the estimator + check combinations. I think we could pass strict=True there to make it a failure for a test, that is marked as xfail, to pass.

I think we want to make this behaviour configurable, so we need a new parameter for parametrize_with_checks, something like strict=None with the option to set it to True/False.

I'd set the default to None so that not setting it does not override the setting in pytest.ini (to be checked if this actually works). If you are using pytest.ini to control strict mode then not passing strict to parametrize_with_checks should not change anything.

Describe alternatives you've considered, if relevant

I tried layering @pytest.mark.xfail(strict=True) on top of @parametrize_with_checks but that doesn't seem to work.

@pytest.mark.xfail(strict=True)
@parametrize_with_checks(...)
def test_sklearn_compat(estimator, check):
   ...

Additional context

No response