Migrate full-run integration tests from CircleCI to GH Actions except docker push and push of all artifacts #12545

k-a-il · 2025-04-22T13:24:41Z

Motivation

This PR is part of initiative to migrate from CircleCI to GithubActions. It migrates part of full-run tests to GithubActions. This PR does not include merged artifact uploads or Docker push to registry jobs; they will be added in a separate PR.

Changes

Removed parallel coverage runs from service tests that execute on a single runner
re-uses Github Action to build community image
introduces Github Action to prepare test environment and load docker image from artifacts
caching of pip dependencies is done using actions/setup-python@v5

Testing

Pipeline testing acceptance tests run: https://github.com/k-a-il/localstack/actions/runs/14730738450
Full integration tests run (tests in TestDockerClient were skipped for now): https://github.com/k-a-il/localstack/actions/runs/14795627190

github-actions · 2025-04-25T15:46:05Z

S3 Image Test Results (AMD64 / ARM64)

2 files ±0 2 suites ±0 8m 16s ⏱️ +11s
488 tests ±0 438 ✅ ±0 50 💤 ±0 0 ❌ ±0
976 runs ±0 876 ✅ ±0 100 💤 ±0 0 ❌ ±0

Results for commit c39dbbd. ± Comparison against base commit 4ea4102.

♻️ This comment has been updated with latest results.

github-actions · 2025-04-25T16:49:32Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 1h 43m 5s ⏱️ - 11m 15s
4 399 tests +26 4 037 ✅ +21 362 💤 +5 0 ❌ ±0
4 401 runs +26 4 037 ✅ +21 364 💤 +5 0 ❌ ±0

Results for commit 20d7e05. ± Comparison against base commit 4ea4102.

This pull request removes 16 and adds 42 tests. Note that renamed tests count towards both.

tests.aws.services.apigateway.test_apigateway_api.TestApiGatewayApiRestApi ‑ test_create_rest_api_with_custom_id_tag
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_dynamic]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_parameter_for_condition_create_resource]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_unrelated_property]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_unrelated_property_not_create_only]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_mapping_scenarios[update_string_referencing_resource]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_conditions
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_direct_update
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_dynamic_update
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_execute_with_ref
…

tests.aws.services.apigateway.test_apigateway_common.TestApigatewayRouting ‑ test_routing_with_custom_api_id
tests.aws.services.cloudformation.v2.test_change_set_parameters.TestChangeSetParameters ‑ test_update_parameter_default_value
tests.aws.services.cloudformation.v2.test_change_set_parameters.TestChangeSetParameters ‑ test_update_parameter_default_value_with_dynamic_overrides
tests.aws.services.cloudformation.v2.test_change_set_parameters.TestChangeSetParameters ‑ test_update_parameter_with_added_default_value
tests.aws.services.cloudformation.v2.test_change_set_parameters.TestChangeSetParameters ‑ test_update_parameter_with_removed_default_value
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_dynamic]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_parameter_for_condition_create_resource]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_unrelated_property]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_unrelated_property_not_create_only]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_mapping_scenarios[update_string_referencing_resource]
…

This pull request removes 14 skipped tests and adds 18 skipped tests. Note that renamed tests count towards both.

tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_dynamic]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_parameter_for_condition_create_resource]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_unrelated_property]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_unrelated_property_not_create_only]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_base_mapping_scenarios[update_string_referencing_resource]
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_conditions
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_direct_update
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_dynamic_update
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_execute_with_ref
tests.aws.services.cloudformation.api.test_changesets.TestCaptureUpdateProcess ‑ test_mappings_with_parameter_lookup
…

tests.aws.services.cloudformation.v2.test_change_set_parameters.TestChangeSetParameters ‑ test_update_parameter_default_value
tests.aws.services.cloudformation.v2.test_change_set_parameters.TestChangeSetParameters ‑ test_update_parameter_default_value_with_dynamic_overrides
tests.aws.services.cloudformation.v2.test_change_set_parameters.TestChangeSetParameters ‑ test_update_parameter_with_added_default_value
tests.aws.services.cloudformation.v2.test_change_set_parameters.TestChangeSetParameters ‑ test_update_parameter_with_removed_default_value
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_dynamic]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_parameter_for_condition_create_resource]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_unrelated_property]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_dynamic_parameter_scenarios[change_unrelated_property_not_create_only]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_base_mapping_scenarios[update_string_referencing_resource]
tests.aws.services.cloudformation.v2.test_change_sets.TestCaptureUpdateProcess ‑ test_conditions
…

This pull request skips 1 test.

tests.aws.services.cloudformation.v2.test_change_sets ‑ test_single_resource_static_update

♻️ This comment has been updated with latest results.

k-a-il · 2025-04-29T07:18:38Z

I will add a link to a successful run shortly, and then add other codeowners to this PR.

silv-io

LGTM in principle! Some awesome ideas in there with the composite actions and more 🥳

I will give the final approval after having looked at the full test runs.

.github/workflows/tests-integration.yml

.github/actions/load-localstack-docker-from-artifacts/action.yml

.github/workflows/tests-integration.yml

Makefile

… step

silv-io

LGTM! It's very easily readable which I think is very important for pipelines like these 🚀

Only the publishing of the results to the GHA summary would be left now. (Sorry that I came up with that just now :P) However, we can also add that in a follow up.

silv-io · 2025-05-02T14:01:31Z

.github/workflows/aws-tests.yml

+          path: |
+            target/pytest-junit-integration-${{ env.PLATFORM }}-${{ matrix.group }}.xml
+            target/.coverage.integration-${{ env.PLATFORM }}-${{ matrix.group }}
+          retention-days: 30


nit: Just noticed that we will also need to add this here, like we have in ext: https://github.com/EnricoMi/publish-unit-test-result-action so that we can quickly browse the results in the Summary.

We can either add this now or in a follow-up PR. And we'll need that for all test jobs

I will add this in the follow-up PR

… docker push and push of all artifacts (#12545)

Install and build jobs for integration tests from CircleCI

3847bab

k-a-il changed the title ~~Install and build jobs for integration tests from CircleCI~~ Full-run integration tests from CircleCI Apr 22, 2025

k-a-il requested a review from silv-io April 22, 2025 13:25

k-a-il added the semver: patch Non-breaking changes which can be included in patch releases label Apr 22, 2025

k-a-il self-assigned this Apr 22, 2025

k-a-il added 4 commits April 22, 2025 18:51

Added test selection and pip cache in python installation

4c348eb

Added unit-tests to preflight, fixed docker image load

e430097

Added test-bootstrap job

9e62c32

Added tests for services, moved reusable components to actions

c9e43d5

k-a-il added this to the 4.4 milestone Apr 25, 2025

Renamed actions, added parity test, fixed test selection

502b862

k-a-il added 6 commits April 28, 2025 18:37

Added new CloudFront test, fixed artifacts for int test, added triggers

7e3224a

Changed path to actions to current branch

dc34b8b

Uses actions from repo

8a64877

Removed comments

4d0697d

Changed path to actions

f86ca3b

Renamed artifact names for integration tests and acceptance tests

dcf003f

k-a-il marked this pull request as ready for review April 29, 2025 07:13

k-a-il requested review from alexrashed, dfangl and dominikschubert as code owners April 29, 2025 07:13

k-a-il removed request for dominikschubert, alexrashed and dfangl April 29, 2025 07:17

k-a-il changed the title ~~Full-run integration tests from CircleCI~~ Migrate full-run integration tests from CircleCI to GH Actions except docker push and push of all artifacts Apr 29, 2025

silv-io reviewed Apr 29, 2025

View reviewed changes

Added platform to action which loads docker image, removed list files…

c46dac0

… step

k-a-il added 3 commits April 29, 2025 12:31

Checkout only latest commit for service tests, fixed comments from pr

60ea2bc

Removed matrix group from acceptance test

7b8e48f

Changed trigger events

aff7cff

k-a-il mentioned this pull request Apr 30, 2025

Added main workflow to trigger full-run workflow and push results to docker registry and pypi #12570

Merged

k-a-il added 2 commits April 30, 2025 18:23

Fixed path for not implemented results

20d7e05

Added test selection to service tests and lambda caching to build job

c39dbbd

k-a-il requested review from alexrashed and silv-io and removed request for alexrashed May 2, 2025 13:46

silv-io approved these changes May 2, 2025

View reviewed changes

k-a-il merged commit 940aa0a into master May 2, 2025
35 of 38 checks passed

k-a-il deleted the circleci-to-gh-actions-full-run branch May 2, 2025 14:11

sannya-singal mentioned this pull request May 5, 2025

Migrate MA/MR pipeline from CircleCI to GH Actions #12579

Merged

simonrw pushed a commit that referenced this pull request May 8, 2025

Migrate full-run integration tests from CircleCI to GH Actions except…

909e125

… docker push and push of all artifacts (#12545)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Migrate full-run integration tests from CircleCI to GH Actions except docker push and push of all artifacts #12545

Migrate full-run integration tests from CircleCI to GH Actions except docker push and push of all artifacts #12545

Uh oh!

k-a-il commented Apr 22, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 25, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 25, 2025 •

edited

Loading

Uh oh!

k-a-il commented Apr 29, 2025

Uh oh!

silv-io left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

silv-io left a comment

Uh oh!

silv-io May 2, 2025

Uh oh!

k-a-il May 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Migrate full-run integration tests from CircleCI to GH Actions except docker push and push of all artifacts #12545

Migrate full-run integration tests from CircleCI to GH Actions except docker push and push of all artifacts #12545

Uh oh!

Conversation

k-a-il commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

k-a-il commented Apr 29, 2025

Uh oh!

silv-io left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

silv-io left a comment

Choose a reason for hiding this comment

Uh oh!

silv-io May 2, 2025

Choose a reason for hiding this comment

Uh oh!

k-a-il May 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

k-a-il commented Apr 22, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading