Skip to content

CI Remove pytest maxfail argument #28584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 7, 2024
Merged

CI Remove pytest maxfail argument #28584

merged 1 commit into from
Mar 7, 2024

Conversation

lesteve
Copy link
Member

@lesteve lesteve commented Mar 6, 2024 •
edited
Loading

--maxfail was added in #27910 (comment). The main argument for it was:

We might want to add --max-fail=10 by default in all our CIs as most of the time, when we get a very large number of failures at once, they are caused by a few common causes and trying to exhaustively analyze a gigantic error log is counter productive. We tend to fix problems incrementally when it happens.

Personally in most cases I find it convenient to get all the failures from the CI, so I can then work on them locally without having to run the full test suite.

In scipy-dev, for example, it can happen quite quickly that you go over 10 failures and I want the tracking issue to list all of them to get a faithful status of the kind of failures and a faithful history of the failures.

I am not too bothered by the gigantic error log, since typically I go to the very end of the log to look at the summary get a quick feeling of the the different kind of errors and then can investigate failures one by one:

=========================== short test summary info ============================
FAILED datasets/tests/test_arff_parser.py::test_pandas_arff_parser_strip_single_quotes[_liac_arff_parser] - DeprecationWarning: The copy keyword is deprecated and will be removed in a...
FAILED datasets/tests/test_arff_parser.py::test_pandas_arff_parser_strip_double_quotes[_liac_arff_parser] - DeprecationWarning: The copy keyword is deprecated and will be removed in a...
FAILED datasets/tests/test_openml.py::test_fetch_openml_as_frame_true[True-liac-arff-61-dataset_params0-150-4-1] - DeprecationWarning: The copy keyword is deprecated and will be removed in a...
FAILED datasets/tests/test_openml.py::test_fetch_openml_as_frame_true[True-liac-arff-61-dataset_params1-150-4-1] - DeprecationWarning: The copy keyword is deprecated and will be removed in a...
FAILED datasets/tests/test_openml.py::test_fetch_openml_as_frame_true[True-liac-arff-2-dataset_params2-11-38-1] - DeprecationWarning: The copy keyword is deprecated and will be removed in a...
FAILED datasets/tests/test_openml.py::test_fetch_openml_as_frame_true[True-liac-arff-2-dataset_params3-11-38-1] - DeprecationWarning: The copy keyword is deprecated and will be removed in a...
FAILED datasets/tests/test_openml.py::test_fetch_openml_as_frame_true[True-liac-arff-561-dataset_params4-209-7-1] - DeprecationWarning: The copy keyword is deprecated and will be removed in a...
.
.
.
.

One recent example where I was annoyed by the --maxfail is #28583, I wanted to get an idea of the remaining scipy-dev issues and I got only a partial status. This has happened to me a few times (maybe 2-3 times?) since December where --maxfail was added.

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 6, 2024

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: 22ed7bd. Link to the linter CI: here

@lesteve lesteve added the Quick Review For PRs that are quick to review label Mar 6, 2024
@jeremiedbb
Copy link
Member

jeremiedbb commented Mar 6, 2024 •
edited
Loading

Maybe we can remove maxfail for scipy-dev and keep it for other jobs ?
(personally I don't mind removing it in all jobs)

@lesteve
Copy link
Member Author

lesteve commented Mar 6, 2024 •
edited
Loading

We could but really I don't much point. IMO, --maxfail was useful in #27910 for some weird reason that happens very rarely : there was a timeout and that you would not get any information about failures because of the timeout.

You could imagine a high number of errors for other builds as well, for example when we update lock-files. Even in a more "standard" PR, it can happen to have more than 10 failures. I'd rather have all the failures to get the full picture, rather than fixing some issues based on the CI (forgetting that this is a partial result), push and wait for 30-45 more minutes to look at the CI status, expecting it to be green and realise there are more failures that I need to fix.

adrinjalali
adrinjalali approved these changes Mar 7, 2024
View reviewed changes
Copy link
Member

@adrinjalali adrinjalali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then let's remove and see how it goes, we can always put it back.

@adrinjalali adrinjalali merged commit 15ecc0e into scikit-learn:main Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adrinjalali adrinjalali adrinjalali approved these changes

Assignees
No one assigned
Labels
Build / CI Quick Review For PRs that are quick to review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lesteve @jeremiedbb @adrinjalali