Investigate CI analytics pollution #12671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft

joe4dev wants to merge 18 commits into master from investigate-ci-analytics-pollution

+36 −7

Member

joe4dev commented May 28, 2025

DO NOT MERGE, only for internal testing!

Motivation

Investigate why we receive analytic events in CI.

Changes

Introduce new test counter and raise intentional exception upon trying to publish CI metrics

joe4dev added this to the Playground milestone

github-actions bot commented May 28, 2025 •

edited

Loading

Test Results - Preflight, Unit

21 613 tests ±0 19 955 ✅ - 3 6m 10s ⏱️ -14s
1 suites ±0 1 658 💤 +3
1 files ±0 0 ❌ ±0

Results for commit fca8e3e. ± Comparison against base commit 02ad74e.

This pull request skips 3 tests.

tests.unit.cli.test_cli ‑ test_publish_analytics_event_on_command_invocation[--debug config show --format plain-localstack config show-expected_params2]
tests.unit.cli.test_cli ‑ test_publish_analytics_event_on_command_invocation[config show-localstack config show-expected_params1]
tests.unit.cli.test_cli ‑ test_publish_analytics_event_on_command_invocation[stop-localstack stop-expected_params0]

♻️ This comment has been updated with latest results.

github-actions bot commented May 28, 2025 •

edited

Loading

Test Results - Alternative Providers

987 tests 589 ✅ 30m 15s ⏱️
4 suites 398 💤
4 files 0 ❌

Results for commit fca8e3e.

♻️ This comment has been updated with latest results.

github-actions bot commented May 28, 2025 •

edited

Loading

LocalStack Community integration with Pro

2 files 2 suites 1h 44m 10s ⏱️
4 873 tests 4 096 ✅ 777 💤 0 ❌
4 875 runs 4 096 ✅ 779 💤 0 ❌

Results for commit fca8e3e.

♻️ This comment has been updated with latest results.

github-actions bot commented May 28, 2025 •

edited

Loading

S3 Image Test Results (AMD64 / ARM64)

2 files 2 suites 8m 11s ⏱️
495 tests 445 ✅ 50 💤 0 ❌
990 runs 890 ✅ 100 💤 0 ❌

Results for commit fca8e3e.

♻️ This comment has been updated with latest results.

joe4dev force-pushed the investigate-ci-analytics-pollution branch from 673729a to 78df7ee Compare

June 2, 2025 14:41

github-actions bot commented Jun 2, 2025 •

edited

Loading

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 7s ⏱️ ±0s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit fca8e3e. ± Comparison against base commit 02ad74e.

♻️ This comment has been updated with latest results.

github-actions bot commented Jun 2, 2025 •

edited

Loading

Test Results (amd64) - Integration, Bootstrap

5 files 5 suites 2h 20m 43s ⏱️
5 230 tests 4 300 ✅ 929 💤 1 ❌
5 236 runs 4 300 ✅ 935 💤 1 ❌

For more details on these failures, see this check.

Results for commit fca8e3e.

♻️ This comment has been updated with latest results.

joe4dev force-pushed the investigate-ci-analytics-pollution branch from cf5ba72 to 38f72f5 Compare

June 3, 2025 15:04

github-actions bot commented Jun 3, 2025 •

edited

Loading

Test Results (MA/MR) - Preflight, Unit

21 613 tests 19 955 ✅ 6m 1s ⏱️
1 suites 1 658 💤
1 files 0 ❌

Results for commit fca8e3e.

♻️ This comment has been updated with latest results.

github-actions bot commented Jun 3, 2025 •

edited

Loading

Test Results (amd64, MA/MR) - Acceptance

7 tests 5 ✅ 3m 8s ⏱️
1 suites 2 💤
1 files 0 ❌

Results for commit fca8e3e.

♻️ This comment has been updated with latest results.

github-actions bot commented Jun 3, 2025 •

edited

Loading

Test Results (MA/MR) - Alternative Providers

987 tests 589 ✅ 30m 22s ⏱️
4 suites 398 💤
4 files 0 ❌

Results for commit fca8e3e.

♻️ This comment has been updated with latest results.

github-actions bot commented Jun 3, 2025 •

edited

Loading

Test Results (amd64, MA/MR) - Integration, Bootstrap

5 files 5 suites 2h 23m 3s ⏱️
5 230 tests 4 301 ✅ 929 💤 0 ❌
5 236 runs 4 301 ✅ 935 💤 0 ❌

Results for commit fca8e3e.

♻️ This comment has been updated with latest results.

joe4dev added 15 commits

June 13, 2025 14:53


          Add CI analytics pollution testing

d99089e

# Conflicts:
#	localstack-core/localstack/services/sns/analytics.py
#	localstack-core/localstack/utils/analytics/metrics.py


          Bump log levels for CI filtering

38f9d0a


          Add DISABLE_EVENTS to Makefile to mitigate CI analytics pollution

df23ae0

# Conflicts:
#	Makefile


          Use LOCALSTACK_ prefix to work around failing bootstrap tests

8da960b


          Fix prefixed reference of DISABLE_EVENTS

19afbd0


          Switch back to non-prefixed variable because it is not recognized

a205a65


          Fix typo in env var prefix warning

d9ee78e


          Add workaround for failing unit tests due to missing env var prefix

f2213fe


          Revert "Add workaround for failing unit tests due to missing env var …

fb21a6f

…prefix"

This reverts commit 42509c7.


          Switch to prefixed env var to fix bootstrap tests

3e3dc98

The following bootstrap tests fail due to the missing `LOCALSTACK_` prefix required for CLI tests:
https://app.circleci.com/pipelines/github/localstack/localstack/32859/workflows/4f30e6c8-ed63-4ef5-8705-fd9aa2bb7777/jobs/297211

* tests/bootstrap/test_container_configurators.py::test_container_configurator_no_deprecation_warning_on_prefix
* tests/bootstrap/test_container_configurators.py::test_container_configurator_no_deprecation_warning_for_ci_env_var


          Revert "Switch to prefixed env var to fix bootstrap tests"

e5bbfd9

This reverts commit 78df7ee.


          Add workaround for failing unit tests due to missing env var prefix

6ea4959


          Fix monkeypatching on correct tests

c8c7f2b


          Explicitly disable analytic events in CI pipelines


          Temporarily skip failing tests due to experimentation to trigger othe…

ed883c2

…r pipelines

joe4dev force-pushed the investigate-ci-analytics-pollution branch from 10cd525 to ed883c2 Compare

June 13, 2025 13:04


          Remove metrics.py due to accidential rebase error

3ecd235

joe4dev added 2 commits

June 13, 2025 16:13


          Remove legacy CircleCi config

ae62753


          Introduce intentional exception

fca8e3e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

bentsku Awaiting requested review from bentsku bentsku will be requested when the pull request is marked ready for review bentsku is a code owner

baermat Awaiting requested review from baermat baermat will be requested when the pull request is marked ready for review baermat is a code owner

simonrw Awaiting requested review from simonrw simonrw will be requested when the pull request is marked ready for review simonrw is a code owner

thrau Awaiting requested review from thrau thrau will be requested when the pull request is marked ready for review thrau is a code owner

alexrashed Awaiting requested review from alexrashed alexrashed will be requested when the pull request is marked ready for review alexrashed is a code owner

dfangl Awaiting requested review from dfangl dfangl will be requested when the pull request is marked ready for review dfangl is a code owner

dominikschubert Awaiting requested review from dominikschubert dominikschubert will be requested when the pull request is marked ready for review dominikschubert is a code owner

silv-io Awaiting requested review from silv-io silv-io will be requested when the pull request is marked ready for review silv-io is a code owner

k-a-il Awaiting requested review from k-a-il k-a-il will be requested when the pull request is marked ready for review k-a-il is a code owner

Labels

None yet