Skip to content

CFNv2: Standardise on test reporting #12984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

CFNv2: Standardise on test reporting #12984

wants to merge 6 commits into from
+100 −68

Conversation

simonrw
Copy link
Contributor

@simonrw simonrw commented Aug 9, 2025 •
edited
Loading

Motivation

It would be good going forward to be able to automatically parse the number of tests executed and skipped so that we can easily report progress.

Changes

  • Standardise the skip decorator format and structure so we can automatically parse the values in a plugin
  • Add plugin in conftest.py which adds a reporting line for number of skips if PROVIDER_OVERRIDE_CLOUDFORMATION=engine-v2, for example (at time of creating):
CFNV2: skipped 99 v1 skips and 63 v2 skips
- CFNV2(Transform): 26
- CFNV2(LanguageExtensions): 6
- CFNV2(Resolve): 5
- CFNV2(Engine): 4
- CFNV2(DescribeStacks): 3
- CFNV2(Validation): 2
- CFNV2(Other): 2
- CFNV2(Imports): 2
- CFNV2(GetTemplate): 2
- CFNV2(Metadata,Other): 1
- CFNV2(Test): 1
- CFNV2(Deletion): 1
- CFNV2(DescribeStack): 1
- CFNV2(AWS::Include): 1
- CFNV2(UpdateStack): 1
- CFNV2(ResourceProvider): 1
- CFNV2(Provider): 1
- CFNV2(DescribeStackResource): 1
- CFNV2(Macros): 1
- CFNV2(Fn::Transform): 1

See an example from this PR's CI here

We can then read this value from the latest CI run to see progress

Note

See how we technically support more of the test suite with the v2 provider than the v1 provider!

@simonrw simonrw added the semver: patch Non-breaking changes which can be included in patch releases label Aug 9, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results - Preflight, Unit

22 063 tests  ±0   20 329 ✅ ±0   6m 42s ⏱️ -3s
     1 suites ±0    1 734 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit df5eea9. ± Comparison against base commit 4258f76.

♻️ This comment has been updated with latest results.

@simonrw simonrw marked this pull request as ready for review August 9, 2025 14:27
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 8s ⏱️ -14s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit df5eea9. ± Comparison against base commit 4258f76.

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

LocalStack Community integration with Pro

  2 files  ±    0    2 suites  ±0   21m 45s ⏱️ - 1h 21m 8s
555 tests  - 4 073  330 ✅  - 3 858  225 💤  - 215  0 ❌ ±0 
557 runs   - 4 073  330 ✅  - 3 858  227 💤  - 215  0 ❌ ±0 

Results for commit df5eea9. ± Comparison against base commit 4258f76.

This pull request removes 4073 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results (amd64) - Integration, Bootstrap

  5 files    5 suites   33m 10s ⏱️
579 tests 355 ✅ 224 💤 0 ❌
585 runs  355 ✅ 230 💤 0 ❌

Results for commit df5eea9.

♻️ This comment has been updated with latest results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pinzon pinzon Awaiting requested review from pinzon pinzon is a code owner

@dominikschubert dominikschubert Awaiting requested review from dominikschubert dominikschubert is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@simonrw