Skip to content

ENH XFAIL in common tests with estimator tags (v3) #16502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Feb 20, 2020
Merged

ENH XFAIL in common tests with estimator tags (v3) #16502

merged 18 commits into from
Feb 20, 2020

Conversation

rth
Copy link
Member

@rth rth commented Feb 20, 2020

A third version of #16306 and #16328

Puts the common checks to mark as xfail into the _xfail_test estimator tag.

Hopefully addresses concerns raised in the two previous implementations. In particular, this would make the workflow identical between scikit-learn and contrib projects.

cc @thomasjpfan @jnothman @glemaitre

rth
rth commented Feb 20, 2020
View reviewed changes
sklearn/tests/test_common.py
if xfail_checks:
if check_name in xfail_checks:
msg = xfail_checks[check_name]
request.applymarker(pytest.mark.xfail(reason=msg))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is probably a way to make this work for contrib projects automatically with parametrize_with_checks, but for now I just want something that we can use in scikit-learn as the ability to XFAIL is blocking multiple PRs.

jnothman
jnothman approved these changes Feb 20, 2020
View reviewed changes
Copy link
Member

@jnothman jnothman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm. much happier with this..

@rth @jnothman
Update doc/developers/develop.rst
490228a 
Co-Authored-By: Joel Nothman <joel.nothman@gmail.com>
@glemaitre glemaitre merged commit 1c1f273 into scikit-learn:master Feb 20, 2020
@glemaitre
Copy link
Member

Thanks @rth and @thomasjpfan.

I think that we should look at the check_invariance to see if we can make it more granular such that we can try to test more methods for the skipped estimators.

@rth
Copy link
Member Author

rth commented Feb 20, 2020

Thanks for the reviews! Looks like somebody broke master,

Could not get the latest source version for repository scikit-learn/scikit-learn hosted on https://github.com/ using ref 1c1f273. GitHub reported the error, "Internal Server Error"

hope it's intermittent.

If you have other comments @thomasjpfan let me know, we can address them in follow up PRs.

@rth rth deleted the xfail-with-estimator-tags branch February 20, 2020 21:37
@rth rth mentioned this pull request Feb 21, 2020
Merged
@thomasjpfan thomasjpfan mentioned this pull request Feb 21, 2020
Merged
thomasjpfan pushed a commit to thomasjpfan/scikit-learn that referenced this pull request Feb 22, 2020
@rth @thomasjpfan
ENH XFAIL in common tests with estimator tags (scikit-learn#16502)
13af4d8
panpiort8 pushed a commit to panpiort8/scikit-learn that referenced this pull request Mar 3, 2020
@rth
ENH XFAIL in common tests with estimator tags (scikit-learn#16502)
c3a1c55
gio8tisu pushed a commit to gio8tisu/scikit-learn that referenced this pull request May 15, 2020
@rth @gio8tisu
ENH XFAIL in common tests with estimator tags (scikit-learn#16502)
1fac914
@lesteve lesteve mentioned this pull request Jul 19, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jnothman jnothman jnothman approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rth @glemaitre @jnothman