Skip to content

Add Coverage workflow #623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 41 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
7078342
Add coverage workflow
Daraan Jul 7, 2025
56faf0f
Add coverage summary
Daraan Jul 7, 2025
045b6f7
add PR write permission
Daraan Jul 7, 2025
21b240c
report line misses in logs
Daraan Jul 7, 2025
71daf3d
Merge branch 'main' into add-coverage
Daraan Jul 7, 2025
3b6319d
Remove install check again
Daraan Jul 7, 2025
e69ace4
Merge remote-tracking branch 'refs/remotes/origin/add-coverage' into …
Daraan Jul 7, 2025
4384206
coverage not on mypy
Daraan Jul 7, 2025
b852953
Feedback; include temp files in .coverage
Daraan Jul 7, 2025
022d37b
Merge remote-tracking branch 'upstream' into add-coverage
Daraan Jul 7, 2025
9f1b3e1
check files present
Daraan Jul 7, 2025
4bd0aa1
Add omit to final output
Daraan Jul 7, 2025
7a7d6b3
Read env
Daraan Jul 7, 2025
ba46124
Use pyproject.toml for coverage
Daraan Jul 7, 2025
b9613f7
clean comment
Daraan Jul 7, 2025
525e198
Merge branch 'main' into add-coverage
Daraan Jul 7, 2025
034ec43
Zizmor feedback: Pin actions
Daraan Jul 7, 2025
88ba8f6
Merge branch 'main' into add-coverage
Daraan Aug 13, 2025
6c43f31
Test without permissions override
Daraan Aug 18, 2025
943aba7
Merge branch 'main' into add-coverage
Daraan Aug 18, 2025
5bbafa0
Revert "Test without permissions override"
Daraan Aug 18, 2025
ee8f319
rework pr comment with workflow_run
Daraan Aug 18, 2025
2757de5
overwrite pr number
Daraan Aug 18, 2025
084e39d
na
Daraan Aug 18, 2025
6ed9889
continue on error as v4 has a bug
Daraan Aug 18, 2025
f48d961
slim workflow_run
Daraan Aug 18, 2025
8d7600b
WIP: checkout files
Daraan Aug 18, 2025
8677db5
adjust filename
Daraan Aug 18, 2025
f77717e
fix dir
Daraan Aug 18, 2025
4cc732b
recheck
Daraan Aug 18, 2025
30a30d1
recheck
Daraan Aug 18, 2025
b03f51f
recheck
Daraan Aug 18, 2025
1a7c6e5
Fix file/path
Daraan Aug 18, 2025
355ccd7
cleanup
Daraan Aug 18, 2025
90c0e0c
wrong event name
Daraan Aug 18, 2025
b16e770
add pr write
Daraan Aug 18, 2025
26ace2b
Merge branch 'main' into add-coverage
Daraan Aug 18, 2025
b41735a
Upload xml and create markdown downstream, get PR number via API
Daraan Aug 19, 2025
50eaad7
Rework pipeline and PR number acquisition
Daraan Aug 19, 2025
4547b9c
No need to format json if processing instantly
Daraan Aug 19, 2025
7f67cf3
Split steps for minimal permissions
Daraan Aug 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 70 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,14 +66,41 @@ jobs:
python-version: ${{ matrix.python-version }}
allow-prereleases: true

- name: Test typing_extensions
- name: Install coverage
if: ${{ !startsWith(matrix.python-version, 'pypy') }}
run: |
# Be wary that this does not install typing_extensions in the future
pip install coverage

- name: Test typing_extensions with coverage
if: ${{ !startsWith(matrix.python-version, 'pypy') }}
run: |
# Be wary of running `pip install` here, since it becomes easy for us to
# accidentally pick up typing_extensions as installed by a dependency
cd src
python --version # just to make sure we're running the right one
# Run tests under coverage
export COVERAGE_FILE=.coverage_${{ matrix.python-version }}
python -m coverage run -m unittest test_typing_extensions.py
- name: Test typing_extensions no coverage on pypy
if: ${{ startsWith(matrix.python-version, 'pypy') }}
run: |
# Be wary of running `pip install` here, since it becomes easy for us to
# accidentally pick up typing_extensions as installed by a dependency
cd src
python --version # just to make sure we're running the right one
python -m unittest test_typing_extensions.py

- name: Archive code coverage results
if: ${{ !startsWith(matrix.python-version, 'pypy') }}
uses: actions/upload-artifact@v4
with:
name: .coverage_${{ matrix.python-version }}
path: ./src/.coverage*
include-hidden-files: true
compression-level: 0 # no compression


- name: Test CPython typing test suite
# Test suite fails on PyPy even without typing_extensions
if: ${{ !startsWith(matrix.python-version, 'pypy') }}
Expand Down Expand Up @@ -111,3 +138,45 @@ jobs:
title: `Daily tests failed on ${new Date().toDateString()}`,
body: "Runs listed here: https://github.com/python/typing_extensions/actions/workflows/ci.yml",
})
prepare-report-coverage:
name: Report coverage

runs-on: ubuntu-latest

needs: [tests]

if: ${{ always() }}

steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3"
- name: Download coverage artifacts
uses: actions/download-artifact@v4
with:
pattern: .coverage_*
path: .
# merge only when files are named differently
merge-multiple: true
- name: Install dependencies
run: pip install coverage
- name: Combine coverage results
run: |
# List the files to see what we have
echo "Combining coverage files..."
ls -aR .coverage*
coverage combine .coverage*
echo "Creating coverage report..."
coverage report
coverage xml

- name: Archive code coverage report
uses: actions/upload-artifact@v4
with:
name: coverage
path: coverage.xml
compression-level: 0 # no compression
85 changes: 85 additions & 0 deletions .github/workflows/coverage_report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
name: Add coverage comment on PR

on:
workflow_run:
workflows: ["Test and lint"]
types:
- completed

permissions: {}

jobs:
get-pr-number:
runs-on: ubuntu-latest
permissions: {}
if: >
github.event.workflow_run.event == 'pull_request'
&& github.repository == github.event.workflow_run.repository.full_name

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I don't think this works -- workflow_run always runs in the context of the "parent" repository, so the second hand of this && expression always evaluate to true (I think).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(But also, if the intention is to limit this to first-party PRs, then I think this workflow_run can be removed entirely and you can move these steps with their first-party check to ci.yml instead 🙂)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For what it's worth, if this works only for first-party PRs that would be a dealbreaker to me; we get lots of third-party PRs and even maintainer PRs often come from forks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to follow zizmors advise below, in no way it should inhibit any PRs that are made onto here.

If you have to use a dangerous trigger, consider adding a github.repository == ... check to only run for your repository but not in forks of your repository (in case the user has enabled Actions there). This avoids exposing forks to danger in case you fix a vulnerability in the workflow but the fork still contains an old vulnerable version.

I ported their suggestion on github.actor == 'dependabot[bot]' && github.repository == github.event.pull_request.head.repo.full_name and ported it.
envoyproxy implements such a guard in two of its repositories.
https://github.com/envoyproxy/toolshed/blob/f70658cd0445b7d763107eaebde543be927c4f7a/gh-actions/github/env/load/action.yml#L45
and in envoy. Beside that it does not appear to be a common guard.

However, possibly they meant a hardcoded repository string, i.e. python/typing_extensions.
On third thought I now also think it might not make sense but leaving it here will have no negative consequences for this repo and all PRs against it.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to follow zizmors advise below, in no way it should inhibit any PRs that are made onto here.

If you have to use a dangerous trigger, consider adding a github.repository == ... check to only run for your repository but not in forks of your repository (in case the user has enabled Actions there). This avoids exposing forks to danger in case you fix a vulnerability in the workflow but the fork still contains an old vulnerable version.

This is my fault -- that advice is only correct for pull_request_target, not for workflow_run 😅. I'll file a bug to fix those docs later today.

(This is another datapoint for workflow_run being quite a footgun -- I think GitHub at one point also suggested this as a mitigation, which is where I got it from, without realizing that it doesn't actually protect anything on that trigger.)

# The github.repository == ... check will prevent this workflow from running on forks.
steps:
- name: Get PR number
id: get_pr_number
env:
SHA: "${{ github.event.workflow_run.head_sha }}"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any docs for this field? I couldn't find anything that clearly documents that it would point to the HEAD to the PR's branch, rather than the HEAD of the main branch.

(It's also not clear to me how this works when PRs contain some shared history or have identical commits due to cherry picking, so this feels pretty risky.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.github.com/de/webhooks/webhook-events-and-payloads#workflow_run does not exactly specify it.
From my tests it always points to the latest commit in a PR which triggered the ci workflow.

For example: me as 3rd party no fork without access:
dsp-unima#9

A reference I found is here:
https://github.com/marketplace/actions/commit-hash

workflow_run events are handled by Workflows within the context of the main branch, therefore the github.sha context value does not represent the commit that triggered the Workflow and we must use the head_sha value on the event instead.

REPO: "${{ github.repository }}"
run: |
# Query GitHub API to find the PR number associated with the head commit SHA
PR_DATA=$(curl -s "https://api.github.com/search/issues?q=repo:$REPO+type:pr+sha:$SHA")
# Check if head commit is on a single PR else ambiguous which PR triggered the workflow
ITEMS_COUNT=$(jq '.items | length' <<< "$PR_DATA")
if [ "$ITEMS_COUNT" -ne 1 ]; then
echo "Error: Expected exactly one PR, but found $ITEMS_COUNT." >&2
exit 1
fi
PR_NUMBER=$(jq '.items[0].number' <<< "$PR_DATA")
echo "pr_number=$PR_NUMBER" >> $GITHUB_OUTPUT
outputs:
pr_number: ${{ steps.get_pr_number.outputs.pr_number }}

create-md-comment:
runs-on: ubuntu-latest
permissions: {}
if: >
github.event.workflow_run.event == 'pull_request'
&& github.repository == github.event.workflow_run.repository.full_name
steps:
- name: Download coverage report
uses: actions/download-artifact@v4
with:
name: coverage
github-token: ${{ secrets.GITHUB_TOKEN }}
run-id: ${{ github.event.workflow_run.id }}
- name: Code Coverage Report
uses: irongut/CodeCoverageSummary@51cc3a756ddcd398d447c044c02cb6aa83fdae95 # v1.3.0
with:
filename: coverage.xml
badge: true
fail_below_min: false
format: markdown
hide_branch_rate: false
hide_complexity: true
indicators: true
output: both
thresholds: '80 90'
- name: Upload coverage summary
uses: actions/upload-artifact@v4
with:
name: code-coverage-results
path: code-coverage-results.md
post-comment:
runs-on: ubuntu-latest
needs: [get-pr-number, create-md-comment]
permissions:
pull-requests: write
steps:
- name: Download coverage report
uses: actions/download-artifact@v4
with:
name: code-coverage-results
- name: Add Coverage PR Comment
uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2.9.3
with:
recreate: true
path: code-coverage-results.md
number: ${{ needs.get-pr-number.outputs.pr_number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,6 @@ venv*/
*.swp
*.pyc
*.egg-info/

.coverage*
coverage.xml
7 changes: 7 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -120,3 +120,10 @@ ignore = [
[tool.ruff.lint.isort]
extra-standard-library = ["tomllib"]
known-first-party = ["typing_extensions", "_typed_dict_test_helper"]

[tool.coverage.report]
show_missing = true
# Omit files that are created in temporary directories during tests.
# If not explicitly omitted they will result in warnings in the report.
omit = ["inspect*", "ann*"]
ignore_errors = true
Loading