Skip to content

DDB Global table: add logic for streams #12641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
May 27, 2025
Merged

DDB Global table: add logic for streams #12641

merged 17 commits into from
May 27, 2025

Conversation

giograno
Copy link
Member

@giograno giograno commented May 19, 2025

Motivation

In LocalStack, we support global DynamoDB tables by keeping track of the region of the original table and its replicas. Whenever we receive a request for a replica table, we redirect the request to the original region. Therefore, we do not really replicate any data, unlike AWS.

Global tables also come with associated streams. According to the docs, "each global table produces an independent stream based on all its writes, regardless of the origination point for those writes".
However, we did not implement any redirect logic for DynamoDB Streams, as the one I just described for DDB.

Changes

  • Implementing the redirect logic for the DDBStreams of global tables.
  • Adding a dictionary to the provider class, mapping the emitted shard iterators to the original regions (in case of a global table). This allows us to get the records that are kept in the original region.
  • Adding a validated test with snapshots.

Changes have been implemented to both v1 and v2 providers.

@giograno giograno self-assigned this May 19, 2025
@giograno giograno added aws:dynamodb Amazon DynamoDB aws:dynamodbstreams AWS DynamoDB Streams semver: patch Non-breaking changes which can be included in patch releases labels May 19, 2025
Copy link

github-actions bot commented May 19, 2025

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 44m 20s ⏱️ +58s
4 465 tests ±0  4 077 ✅ +1  388 💤  - 1  0 ❌ ±0 
4 467 runs  ±0  4 077 ✅ +1  390 💤  - 1  0 ❌ ±0 

Results for commit b1c0818. ± Comparison against base commit bae7a0e.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented May 20, 2025

Test Results - Preflight, Unit

21 579 tests  ±0   19 927 ✅ ±0   6m 11s ⏱️ -1s
     1 suites ±0    1 652 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit b1c0818. ± Comparison against base commit bae7a0e.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented May 20, 2025

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 9s ⏱️ -2s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit b1c0818. ± Comparison against base commit bae7a0e.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented May 20, 2025

Test Results - Alternative Providers

597 tests  ±0   420 ✅ +1   14m 56s ⏱️ -6s
  4 suites ±0   177 💤  - 1 
  4 files   ±0     0 ❌ ±0 

Results for commit b1c0818. ± Comparison against base commit bae7a0e.

♻️ This comment has been updated with latest results.

@giograno giograno force-pushed the global-tables-streams branch from 5c1fe2d to c45289c Compare May 20, 2025 17:16
Copy link

github-actions bot commented May 20, 2025

Test Results (amd64) - Integration, Bootstrap

    5 files  ±0      5 suites  ±0   2h 22m 9s ⏱️ + 1m 25s
4 820 tests ±0  4 279 ✅ +1  541 💤  - 1  0 ❌ ±0 
4 826 runs  ±0  4 279 ✅ +1  547 💤  - 1  0 ❌ ±0 

Results for commit b1c0818. ± Comparison against base commit bae7a0e.

♻️ This comment has been updated with latest results.

@giograno giograno force-pushed the global-tables-streams branch 2 times, most recently from b5e0cd5 to db15e4c Compare May 22, 2025 13:53
@giograno giograno marked this pull request as ready for review May 22, 2025 15:28
@giograno giograno force-pushed the global-tables-streams branch from 13b2900 to f22abbb Compare May 22, 2025 16:52
@viren-nadkarni
Copy link
Member

@giograno Would this fix #7405? Could you run a quick check?

@giograno
Copy link
Member Author

@giograno Would this fix #7405? Could you run a quick check?

Actually, it won't 🤔 We use the table name to detect the original region, and this won't work if we just invoke ListStreams without a TableName parameter (this parameter is optional). I did not consider this case.

Copy link
Member

@viren-nadkarni viren-nadkarni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this 👍

@@ -33,6 +39,14 @@ def on_after_init(self):
def on_before_start(self):
self.server.start_dynamodb()

def _forward_request(
self, context: RequestContext, region: str | None, service_request: ServiceRequest
) -> ServiceResponse:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have a docstring explaining how this is different from self.forward_request()

@giograno giograno force-pushed the global-tables-streams branch from 3655b1b to 85f2231 Compare May 27, 2025 06:04
giograno and others added 2 commits May 27, 2025 08:04
Co-authored-by: Viren Nadkarni <viren.nadkarni@localstack.cloud>
@giograno giograno merged commit cbd8ac1 into master May 27, 2025
57 checks passed
@giograno giograno deleted the global-tables-streams branch May 27, 2025 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws:dynamodb Amazon DynamoDB aws:dynamodbstreams AWS DynamoDB Streams semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants