Skip to content

Make event invoke function update test more predicatable #12444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Make event invoke function update test more predicatable #12444

wants to merge 3 commits into from

Conversation

joe4dev
Copy link
Member

@joe4dev joe4dev commented Mar 26, 2025 •
edited
Loading

Depends on: #12441

Motivation

The test test_async_invoke_queue_upon_function_update has been flaky in CircleCI (e.g., old example) and locally (assertion error 14 != 15).

Changes

  • Migrate test to new SQS-based lambda notifier infrastructure introduced in Fix flaky lambda test event retry reserved concurrency #12441
    • This allows us to wait until the first invoke is actually being processed rather than using a flaky sleep.
    • This should speeds up the test because the SQS message collection uses long polling
  • Migrate to label-based snapshots for easier snapshot skip

TODO

  • Fix buggy LocalStack behavior not respecting the SQS message deletion and therefore causing a snapshot key issue (guarded via assert). It seems that the SQS message does not get deleted properly although the delete_message call was successful 🤔 . The same test passes against AWS.
    • Increasing the visibility timeout didn't help sqs_create_queue(Attributes={"VisibilityTimeout": "90"}) (the default 20s should be ok as long as the cold start doesn't take more than 10s)
    • The sleep-01 key appears multiple times because the initial invoke is aborted with the update instead of continuted and then the retry triggers another invoke. To fix this, Lambda should not abort pending invokes upon function update.

@joe4dev joe4dev added the semver: patch Non-breaking changes which can be included in patch releases label Mar 26, 2025
@joe4dev joe4dev added this to the 4.4 milestone Mar 26, 2025
@joe4dev joe4dev self-assigned this Mar 26, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 26, 2025 •
edited
Loading

LocalStack Community integration with Pro

    2 files  ±    0      2 suites  ±0   1h 30m 14s ⏱️ - 21m 13s
3 144 tests  - 1 162  2 925 ✅  - 1 061  218 💤  - 102  1 ❌ +1 
3 146 runs   - 1 162  2 925 ✅  - 1 061  220 💤  - 102  1 ❌ +1 

For more details on these failures, see this check.

Results for commit 00d944c. ± Comparison against base commit ccddefd.

This pull request removes 1162 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

Base automatically changed from fix-flaky-lambda-test-event-retry-reserved-concurrency to master March 28, 2025 15:48
@joe4dev joe4dev force-pushed the fix-flaky-lambda-test-event-function-update branch from c0ced8a to 00d944c Compare March 28, 2025 16:06
@joe4dev joe4dev modified the milestones: 4.4, Playground May 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dominikschubert dominikschubert Awaiting requested review from dominikschubert dominikschubert will be requested when the pull request is marked ready for review dominikschubert is a code owner

@dfangl dfangl Awaiting requested review from dfangl dfangl will be requested when the pull request is marked ready for review dfangl is a code owner

@gregfurman gregfurman Awaiting requested review from gregfurman gregfurman will be requested when the pull request is marked ready for review gregfurman is a code owner

Assignees

@joe4dev joe4dev

Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Milestone
Playground
Development

Successfully merging this pull request may close these issues.
1 participant
@joe4dev