[MNT]: Flaky Windows_py31x tests on Azure Pipelines #29797
Comments
|
This does also occur, albeit less frequently, on A commonality between the failure cases seems to be that one or more |
|
Question: do all of the test cases that exhibit this use Superficially the failures appear similar to this bugreport (note: the PR mentioned in the description is where the bug was encountered -- not a subsequent PR to address it): python/cpython#85481 |
|
Sionce these are all on Python 3.10 (I think?), and according to NEP29 we can drop support for Python 3.10 on 4th April (https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table), it's probably not worth spending much time looking into this? |
|
Not quite, it also happens on newer python versions, e.g. #29797 (comment). But apparently less frequently. |
|
I think it's unlikely to be due to a bug in Even so: if it's a sample case for a problem that could be figured out and resolved elsewhere, then that could be helpful (and eventually provide reduced test flakiness, meaning fewer distractions/delays, here as a benefit). |
|
I have no statistics, but the issue still happens on 3.11, and I have the feeling that it's more common now on 3.11 since we have removed 3.10. 🤔 |
The odd thing to me is that the test that fails is which makes me think we are somehow crossing state what launching the subprocesses. |
This is a somewhat wild guess based on https://github .com/matplotlib/issues/29797#issuecomment-2833795708 > which makes me think we are somehow crossing state what launching the subprocesses. Using the `xdist_group` with `--dist=loadgroup` should put all tests of that group to the same worker according to https://pytest-xdist .readthedocs .io/en/stable/distribution.html. I've only added `--dist=loadgroup` to the windows pipelines, so tests on other systems are not affected at all . The first test is to see whether this works in the PR. - But to be sure, it think we would need to put this on master and monitor whether the timeouts disappear.
This is a somewhat wild guess based on matplotlib#29797 (comment) > which makes me think we are somehow crossing state what launching the subprocesses. Using the `xdist_group` with `--dist=loadgroup` should put all tests of that group to the same worker according to https://pytest-xdist.readthedocs.io/en/stable/distribution.html. I've only added `--dist=loadgroup` to the windows pipelines, so tests on other systems are not affected at all. The first test is to see whether this works in the PR. - But to be sure, it think we would need to put this on master and monitor whether the timeouts disappear.
This is a somewhat wild guess based on matplotlib#29797 (comment) > which makes me think we are somehow crossing state what launching the subprocesses. Using the `xdist_group` with `--dist=loadgroup` should put all tests of that group to the same worker according to https://pytest-xdist.readthedocs.io/en/stable/distribution.html. I've only added `--dist=loadgroup` to the windows pipelines, so tests on other systems are not affected at all. The first test is to see whether this works in the PR. - But to be sure, it think we would need to put this on master and monitor whether the timeouts disappear.
This is a somewhat wild guess based on matplotlib#29797 (comment) > which makes me think we are somehow crossing state what launching the subprocesses. Using the `xdist_group` with `--dist=loadgroup` should put all tests of that group to the same worker according to https://pytest-xdist.readthedocs.io/en/stable/distribution.html. I've only added `--dist=loadgroup` to the windows pipelines, so tests on other systems are not affected at all. The first test is to see whether this works in the PR. - But to be sure, it think we would need to put this on master and monitor whether the timeouts disappear.
This is a somewhat wild guess based on matplotlib#29797 (comment) > which makes me think we are somehow crossing state what launching the subprocesses. Using the `xdist_group` with `--dist=loadgroup` should put all tests of that group to the same worker according to https://pytest-xdist.readthedocs.io/en/stable/distribution.html. I've only added `--dist=loadgroup` to the windows pipelines, so tests on other systems are not affected at all. The first test is to see whether this works in the PR. - But to be sure, it think we would need to put this on master and monitor whether the timeouts disappear.
|
Confirmation from #29986
|
|
This may be a bad interaction between pytest-xdist and pytest-timeout. Related topics (I have not yet dug into them in detail): |
When we dropped py310 we switched py311 to use the windows-2019 image. Could the image be the problem? I went searching for info about it and (if I’m looking at the right thing?) it is soon to be deprecated |
|
Good observation. Certainly worth a try to see whether switching to windows-2022 or windows-latest fixes the issue. |
The windows-2019 image will be deprecated on 1.6.2025. Also, this is a test whether the timeouts matplotlib#29797 are related to the windows-2019 image.
The windows-2019 image will be deprecated on 1.6.2025. Also, this is a test whether the timeouts matplotlib#29797 are related to the windows-2019 image.
The windows-2019 image will be deprecated on 1.6.2025. Also, this is a test whether the timeouts matplotlib#29797 are related to the windows-2019 image.
The windows-2019 image will be deprecated on 1.6.2025. Also, this is a test whether the timeouts matplotlib#29797 are related to the windows-2019 image.
Summary
The
Windows_py310Azure Pipelines test jobs run by GitHub Actions for this repo appear to be unreliable at the moment.Test timeouts and failures have occurred on both pull requests and mainline branch checks, and the failures do not appear related to the commits-under-test:
It's possible that this is a temporary/ephemeral issue.
Proposed fix
N/A (yet) - I'm not sure what the cause of these failures is yet
The text was updated successfully, but these errors were encountered: