Skip to content

Generalize reusable Windows CI jobs #121766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 14, 2024

Conversation

webknjaz
Copy link
Contributor

@webknjaz webknjaz commented Jul 14, 2024

Previously, both the reusable and calling workflows had a lot of unnecessary duplication in the windows job definitions. This patch simplifies that into a single parametrized job that is included though a matrix in different modes.

webknjaz added 3 commits July 14, 2024 14:28
Sometimes, GitHub Actions turn things into strings when passing the
data around. So it's usually safer to turn them into proper booleans
in expressions explicitly.
Previously, it didn't have a human-readable explanation which is a
bad tone as it's less obvious how it's supposed to be used.
Previously, it was duplicated in each job which is not DRY.
@webknjaz webknjaz changed the title 🧪💅Evaluate free-threading var as JSON @ CI 🧪💅 Generalize reusable windows jobs @ CI Jul 14, 2024
@webknjaz webknjaz force-pushed the maintenance/gha-win-collapse branch from d97631c to 2a92204 Compare July 14, 2024 13:01
webknjaz added 2 commits July 14, 2024 15:04
Previously, the reusable windows workflow defined a number of per-arch
jobs with a lot of copy-paste. Now, there is a single job and a matrix
definition on the calling side.
Previously, these were defined through two separate matrices but there
is no technical reason to keep them like that.
@webknjaz webknjaz force-pushed the maintenance/gha-win-collapse branch from 2a92204 to 21a36a2 Compare July 14, 2024 13:04
@webknjaz webknjaz marked this pull request as ready for review July 14, 2024 14:33
@hugovk
Copy link
Member

hugovk commented Jul 14, 2024

Thanks for the PR. A quick note for myself, will come back to review properly later:

So before, the Windows jobs showed up in the big misc box at the bottom (marked in yellow):

image

https://github.com/python/cpython/actions/runs/9929374848

With this PR, they're in their own Windows box:

image

@hugovk hugovk changed the title 🧪💅 Generalize reusable windows jobs @ CI Generalize reusable Windows CI jobs Jul 14, 2024
Copy link
Member

@hugovk hugovk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the config feels a bit cleaner this way.

@hugovk hugovk added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Jul 14, 2024
@hugovk hugovk merged commit 7982363 into python:main Jul 14, 2024
41 checks passed
@miss-islington-app
Copy link

Thanks @webknjaz for the PR, and @hugovk for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 14, 2024
(cherry picked from commit 7982363)

Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <wk@sydorenko.org.ua>
@miss-islington-app
Copy link

Sorry, @webknjaz and @hugovk, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7982363b479e22fffc72481e54c9f40ace8a0021 3.12

@bedevere-app
Copy link

bedevere-app bot commented Jul 14, 2024

GH-121775 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jul 14, 2024
@webknjaz
Copy link
Contributor Author

Extra note: one job name changed from x86 to Win32 in the UI. I forgot to mention this but I think, this is more consistent even.

hugovk pushed a commit that referenced this pull request Jul 14, 2024
Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <wk@sydorenko.org.ua>
@bedevere-app
Copy link

bedevere-app bot commented Jul 14, 2024

GH-121776 is a backport of this pull request to the 3.12 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.12 only security fixes label Jul 14, 2024
@webknjaz
Copy link
Contributor Author

Sorry, @webknjaz and @hugovk, I could not cleanly backport this to 3.12 due to a conflict. Please backport using cherry_picker on command line.

cherry_picker 7982363b479e22fffc72481e54c9f40ace8a0021 3.12

Manual backport: #121776

webknjaz added a commit to webknjaz/cpython that referenced this pull request Jul 14, 2024
@webknjaz
Copy link
Contributor Author

So before, the Windows jobs showed up in the big misc box at the bottom

FTR, this happens because of a weird corner case in GHA. Basically, it renders some of the job group representations differently depending on whether there's interpolation syntax in name:. And it's especially odd if you want to have a static group name. For this, I invented a hack of adding ${{ '' }} to the end of the value just to trigger the better grouping behavior in the UI.

webknjaz added a commit to webknjaz/cpython that referenced this pull request Jul 14, 2024
@hugovk
Copy link
Member

hugovk commented Jul 16, 2024

@webknjaz This PR can't be merged:

https://github.com/python/cpython/actions/runs/9950331379/job/27488343862?pr=121831

build_windows → ⬜ None [required to succeed or be skipped]

Do you know what is needed?

webknjaz added a commit to webknjaz/cpython that referenced this pull request Jul 16, 2024
Previously, those flags would sometimes end up having empty string
values which tends to break evaluating them as JSON. This patch adds
`false` fallbacks to all such outputs.

This allows feeding them to `fromJSON()` without a fear of them
causing surprising internal behaviors in the GitHub Actions CI/CD
workflows platform itself [[1]]. The behavior observed was that
some skipped jobs wouldn't show up in the workflow sidebar view at
all, would display in the graph view as `Waiting for pending jobs`
and in the `${{ needs }}` context, they would have a
`result: failure` entry.

This should help make PRs like python#121831 mergeable again.

[1]: python#121766 (comment)
webknjaz added a commit to webknjaz/cpython that referenced this pull request Jul 16, 2024
Previously, those flags would sometimes end up having empty string
values which tends to break evaluating them as JSON. This patch adds
`false` fallbacks to all such outputs.

This allows feeding them to `fromJSON()` without a fear of them
causing surprising internal behaviors in the GitHub Actions CI/CD
workflows platform itself [[1]]. The behavior observed was that
some skipped jobs wouldn't show up in the workflow sidebar view at
all, would display in the graph view as `Waiting for pending jobs`
and in the `${{ needs }}` context, they would have a
`result: failure` entry [[2]].

This should help make PRs like python#121831 mergeable again.

[1]: python#121766 (comment)
[2]: https://github.com/python/cpython/actions/runs/9950331379/job/27501637459?pr=121831#step:2:244
webknjaz added a commit to webknjaz/cpython that referenced this pull request Jul 16, 2024
Previously, those flags would sometimes end up having empty string
values, which tends to break evaluating them as JSON. This patch adds
`false` fallbacks to all such outputs.

This allows feeding them to `fromJSON()` without a fear of them
causing surprising internal behaviors in the GitHub Actions CI/CD
workflows platform itself [[1]]. The behavior observed was that
some skipped jobs wouldn't show up in the workflow sidebar view at
all, would display in the graph view as `Waiting for pending jobs`
and in the `${{ needs }}` context, they would have a
`result: failure` entry [[2]].

This should help make PRs like python#121831 mergeable again.

[1]: python#121766 (comment)
[2]: https://github.com/python/cpython/actions/runs/9950331379/job/27501637459?pr=121831#step:2:244
webknjaz added a commit to webknjaz/cpython that referenced this pull request Jul 16, 2024
Previously, those flags would sometimes end up having empty string
values, which tends to break evaluating them as JSON. This patch adds
`false` fallbacks to all such outputs.

This allows feeding them to `fromJSON()` without a fear of them
causing surprising internal behaviors in the GitHub Actions CI/CD
workflows platform itself [[1]]. The behavior observed was that
some skipped jobs wouldn't show up in the workflow sidebar view at
all, would display in the graph view as `Waiting for pending jobs`
and in the `${{ needs }}` context, they would have a
`result: failure` entry [[2]].

This should help make PRs like python#121831 mergeable again.

[1]: python#121766 (comment)
[2]: https://github.com/python/cpython/actions/runs/9950331379/job/27501637459?pr=121831#step:2:244
webknjaz added a commit to webknjaz/cpython that referenced this pull request Jul 16, 2024
Previously, those flags would sometimes end up having empty string
values, which tends to break evaluating them as JSON. This patch adds
`false` fallbacks to all such outputs.

This allows feeding them to `fromJSON()` without a fear of them
causing surprising internal behaviors in the GitHub Actions CI/CD
workflows platform itself [[1]]. The behavior observed was that
some skipped jobs wouldn't show up in the workflow sidebar view at
all, would display in the graph view as `Waiting for pending jobs`
and in the `${{ needs }}` context, they would have a
`result: failure` entry [[2]].

This should help make PRs like python#121831 mergeable again.

[1]: python#121766 (comment)
[2]: https://github.com/python/cpython/actions/runs/9950331379/job/27501637459?pr=121831#step:2:244
@webknjaz
Copy link
Contributor Author

@webknjaz This PR can't be merged:

python/cpython/actions/runs/9950331379/job/27488343862?pr=121831

build_windows → ⬜ None [required to succeed or be skipped]

Do you know what is needed?

Back-referencing the possible fix: #121848

webknjaz added a commit to webknjaz/cpython that referenced this pull request Jul 16, 2024
estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants