Added pagination and filtering for s3 list buckets operation #12609

bryansan-local · 2025-05-12T18:00:41Z

Motivation

Right now, the list buckets operation is ignoring the Prefix, MaxBuckets, BucketRegion and ContinuationToken which results in the operation always returning all buckets.

Closes #12585

Changes

Process correctly Prefix, MaxBuckets, BucketRegion and ContinuationToken to filter and paginate accordingly.
Added integration tests + parity validations for these scenarios

…d integration and parity tests for this.

localstack-bot · 2025-05-12T18:00:52Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

bryansan-local · 2025-05-12T18:02:23Z

I have read the CLA Document and I hereby sign the CLA

github-actions · 2025-05-12T18:09:54Z

S3 Image Test Results (AMD64 / ARM64)

2 files 2 suites 8m 41s ⏱️
493 tests 443 ✅ 50 💤 0 ❌
986 runs 886 ✅ 100 💤 0 ❌

Results for commit 3c33f88.

♻️ This comment has been updated with latest results.

github-actions · 2025-05-12T18:46:04Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 1h 2m 13s ⏱️ - 40m 16s
1 800 tests - 2 625 1 575 ✅ - 2 467 225 💤 - 158 0 ❌ ±0
1 802 runs - 2 625 1 575 ✅ - 2 467 227 💤 - 158 0 ❌ ±0

Results for commit 3c33f88. ± Comparison against base commit b961fee.

This pull request removes 2630 and adds 5 tests. Note that renamed tests count towards both.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

tests.aws.services.s3.test_s3_list_operations.TestS3ListBuckets ‑ test_list_buckets_by_bucket_region
tests.aws.services.s3.test_s3_list_operations.TestS3ListBuckets ‑ test_list_buckets_by_prefix_with_case_sensitivity
tests.aws.services.s3.test_s3_list_operations.TestS3ListBuckets ‑ test_list_buckets_when_continuation_token_is_empty
tests.aws.services.s3.test_s3_list_operations.TestS3ListBuckets ‑ test_list_buckets_with_continuation_token
tests.aws.services.s3.test_s3_list_operations.TestS3ListBuckets ‑ test_list_buckets_with_max_buckets

♻️ This comment has been updated with latest results.

…f string comparisons when listing a bucket

bentsku

Awesome, great work, this is a nice addition!

I have a few comments, one blocking regarding the snapshot transformations of fields that are changing between every runs.

There's also a case that seems to be missing: when the continuation token provided is an empty string.

Once these are addressed, we're good to go 🚀 nice work!

tests/aws/services/s3/test_s3_list_operations.py

tests/aws/services/s3/test_s3_list_operations.snapshot.json

bentsku · 2025-05-13T12:17:46Z

localstack-core/localstack/services/s3/provider.py

+        if continuation_token == "":
+            raise InvalidArgument(
+                "The continuation token provided is incorrect",
+                ArgumentName="continuation-token",
+            )


it seems we do not have a snapshot test covering this, it would be great to add a snapshot validated test for it

I have checked what AWS does with this empty continuation token scenario and it turns out that it allows it instead of throwing an error. Thus, I have removed this raise error code block and created a test to validate that this implementation accepts and ignores an empty continuation token. Let me know what you think about it

@bentsku is this fine for you? Do you agree?

This is good, thanks for checking this! 👍

tests/aws/services/s3/test_s3_list_operations.py

bentsku

Thanks for addressing the comments! This looks good 👌 I've answered the BucketRegion negative case question, and after that we should be good to go!
Seems like the test failures are unrelated, and linked to the Transcribe service?

tests/aws/services/s3/test_s3_list_operations.py

bryansan-local · 2025-05-13T16:13:20Z

Thanks for addressing the comments! This looks good 👌 I've answered the BucketRegion negative case question, and after that we should be good to go! Seems like the test failures are unrelated, and linked to the Transcribe service?

Yea, no idea at all why those unrelated tests are failing :(

… buckets

localstack-core/localstack/services/s3/provider.py

bentsku

LGTM! This is great, thanks for taking the time to address the comments, this looks awesome! 💯

Also, thank you for taking the time to properly transform the region field, making sure the tests are always passing even with non-default regions.

Let's ship it!

bentsku · 2025-05-14T13:33:09Z

tests/aws/services/s3/test_s3_list_operations.py

+        snapshot.match("list-objects-with-empty-continuation-token", response)
+
+    @markers.aws.validated
+    # In some regions AWS returns the Owner display name (us-west-2) but in some it doesnt (eu-central-1)


nice comment, thanks for adding it! 👌

Added support for pagination and filtering in s3 bucket listing. Adde…

8bef29f

…d integration and parity tests for this.

bryansan-local self-assigned this May 12, 2025

bryansan-local changed the title ~~Added support for pagination and filtering in s3 bucket listing. Adde…~~ Added pagination and filtering for s3 list buckets operation May 12, 2025

bryansan-local added the semver: patch Non-breaking changes which can be included in patch releases label May 12, 2025

localstack-bot added a commit that referenced this pull request May 12, 2025

@bryansan-local has signed the CLA in #12609

c51ae46

bryansan-local mentioned this pull request May 12, 2025

Added pagination and filtering to s3 list buckets #12370

Closed

bryansan-local requested a review from alexrashed May 12, 2025 19:07

Added a check and comment to take into account the case sensitivity o…

040e9c2

…f string comparisons when listing a bucket

bryansan-local marked this pull request as ready for review May 13, 2025 11:40

bryansan-local requested a review from bentsku as a code owner May 13, 2025 11:40

bentsku reviewed May 13, 2025

View reviewed changes

bryansan-local added 5 commits May 13, 2025 16:14

Tackle some PR comments

8a1447b

Regenerated snapshot files

8ce9d81

Fixed tests to take into account a parallel s3 bucket being created

2f23b92

Fixed tests to take into account a parallel s3 bucket being created

1844772

Fixed tests to take into account a parallel s3 bucket being created

accde56

bentsku reviewed May 13, 2025

View reviewed changes

tests/aws/services/s3/test_s3_list_operations.py Outdated Show resolved Hide resolved

tests/aws/services/s3/test_s3_list_operations.py Show resolved Hide resolved

Added tests for checking empty responses and test for regions without…

ca21e13

… buckets

bentsku reviewed May 14, 2025

View reviewed changes

localstack-core/localstack/services/s3/provider.py Outdated Show resolved Hide resolved

Skip owner prefix snapshot verification since it is region dependant

3c33f88

bryansan-local marked this pull request as draft May 14, 2025 09:18

bryansan-local marked this pull request as ready for review May 14, 2025 09:54

bryansan-local requested a review from bentsku May 14, 2025 09:54

bentsku approved these changes May 14, 2025

View reviewed changes

bryansan-local merged commit 2f84418 into master May 14, 2025
39 checks passed

bryansan-local deleted the feature/pagination_and_filtering_in_s3_bucket_listing branch May 14, 2025 15:08

Uh oh!

Added pagination and filtering for s3 list buckets operation #12609

Added pagination and filtering for s3 list buckets operation #12609

Uh oh!

Conversation

bryansan-local commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

localstack-bot commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bryansan-local commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

github-actions bot commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

bentsku left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bentsku May 13, 2025

Choose a reason for hiding this comment

Uh oh!

bryansan-local May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryansan-local May 14, 2025

Choose a reason for hiding this comment

Uh oh!

bentsku May 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bentsku left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bryansan-local commented May 13, 2025

Uh oh!

Uh oh!

bentsku left a comment

Choose a reason for hiding this comment

Uh oh!

bentsku May 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bryansan-local commented May 12, 2025 •

edited

Loading

localstack-bot commented May 12, 2025 •

edited

Loading

github-actions bot commented May 12, 2025 •

edited

Loading

github-actions bot commented May 12, 2025 •

edited

Loading

bryansan-local May 13, 2025 •

edited

Loading