Skip to content

CFNv2: Standardise on test reporting #12984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

CFNv2: Standardise on test reporting #12984

wants to merge 5 commits into from

Conversation

simonrw
Copy link
Contributor

@simonrw simonrw commented Aug 9, 2025 •
edited
Loading

Motivation

It would be good going forward to be able to automatically parse the number of tests executed and skipped so that we can easily report progress.

Changes

  • Standardise the skip decorator format and structure so we can automatically parse the values in a plugin
  • Add plugin in conftest.py which adds a reporting line for number of skips if PROVIDER_OVERRIDE_CLOUDFORMATION=engine-v2, for example (at time of creating):
CFNV2: skipped 99 v1 skips and 63 v2 skips
- CFNV2(Transform): 26
- CFNV2(LanguageExtensions): 6
- CFNV2(Resolve): 5
- CFNV2(Engine): 4
- CFNV2(DescribeStacks): 3
- CFNV2(Validation): 2
- CFNV2(Other): 2
- CFNV2(Imports): 2
- CFNV2(GetTemplate): 2
- CFNV2(Metadata,Other): 1
- CFNV2(Test): 1
- CFNV2(Deletion): 1
- CFNV2(DescribeStack): 1
- CFNV2(AWS::Include): 1
- CFNV2(UpdateStack): 1
- CFNV2(ResourceProvider): 1
- CFNV2(Provider): 1
- CFNV2(DescribeStackResource): 1
- CFNV2(Macros): 1
- CFNV2(Fn::Transform): 1

See an example from this PR's CI here

We can then read this value from the latest CI run to see progress

Note

See how we technically support more of the test suite with the v2 provider than the v1 provider!

@simonrw simonrw added the semver: patch Non-breaking changes which can be included in patch releases label Aug 9, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results - Preflight, Unit

22 063 tests  ±0   20 329 ✅ ±0   6m 20s ⏱️ -4s
     1 suites ±0    1 734 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit 21aabb6. ± Comparison against base commit c1d19e2.

♻️ This comment has been updated with latest results.

@simonrw simonrw marked this pull request as ready for review August 9, 2025 14:27
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 17s ⏱️ +5s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit 21aabb6. ± Comparison against base commit c1d19e2.

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

LocalStack Community integration with Pro

  2 files  ±    0    2 suites  ±0   21m 20s ⏱️ - 1h 23m 8s
553 tests  - 4 072  329 ✅  - 3 857  224 💤  - 215  0 ❌ ±0 
555 runs   - 4 072  329 ✅  - 3 857  226 💤  - 215  0 ❌ ±0 

Results for commit 21aabb6. ± Comparison against base commit c1d19e2.

This pull request removes 4072 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results (amd64) - Integration, Bootstrap

  5 files    5 suites   33m 38s ⏱️
577 tests 354 ✅ 223 💤 0 ❌
583 runs  354 ✅ 229 💤 0 ❌

Results for commit 21aabb6.

♻️ This comment has been updated with latest results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pinzon pinzon Awaiting requested review from pinzon pinzon is a code owner

@dominikschubert dominikschubert Awaiting requested review from dominikschubert dominikschubert is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@simonrw