It's true that having a way to differentiate common tests used internally for scikit-learn and those intented to run on 3rd-party packages would be very useful. Related to #6715 . Maybe using something like,

if estimator.__module__.startswith('sklearn.'):
    # run some scikit-learn specific check

in common tests would be better? For the examples you provide they should never apply outside of scikit-learn and this would avoid passing this parameter everywhere..

I would rather give the user the control rather than being magic about it. It would add the parameter in a lot of places.
On the other hand, we would have to add your snippet in a lot of places as well (though not in the signatures).
Maybe someone wants to have error messages consistent with sklearn?
I'm not entirely decided but right now tending towards a parameter.

I think not having the exact error message is good. Now...are we okay with not even checking the types when strict=False?

Or strict could be 'message', 'type' or False? Not sure if that's necessary though.

Since this seems to be the place, do we want to have a deprecation cycle for certain API changes in check_estomator? Seems reasonable to me. Certain checks can then move from non-strict to strict with a deprecation cycle (related: #14241 (comment))

I'm working on this. NYC WiMLDS

@hirenmayani I would recommend you start with something smaller if you haven't worked on sklearn before, you can also pick up something more tricky later.

I think I can fix it. Will ask you if I need any help.

@hirenmayani ok go for it then.

Re-opening since we still need to make another PR to use the mechanism added in #17361 in more common tests. Also add a what's new entry.

Not validating error messages is done in #18415.

What else do we want to relax when strict mode is off?

From slack conversation, it would be nice for strict=False to relax this check:

To just:

assert y_pred.dtype.kind == "f"  # any precision floating-point

Similar to #16241, moving to 1.1

very unlikely to be part of 1.1, moving to 1.2

I think now that we have #29699 merged and released.