Skip to content

[ESM] Handle DynamoDB-local Invalid ShardID exception #12036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Dec 13, 2024
Merged

[ESM] Handle DynamoDB-local Invalid ShardID exception #12036

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
81174f6
[ESM] Handle DynamoDB-local Invalid ShardID exception
gregfurman Dec 13, 2024
125d5d9
[ESM] Catch TrimmedDataAccessException
gregfurman Dec 13, 2024
d5ef268
Handle TrimmedDataAccessException in DynamoDBStreams Poller
gregfurman Dec 13, 2024
b379cfd
Handle TrimmedDataAccessException in in StreamPoller
gregfurman Dec 13, 2024
ef48cb4
Fix...
gregfurman Dec 13, 2024
e02a1af
Remove blank line
gregfurman Dec 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions localstack-core/localstack/services/lambda_/event_source_mapping/pollers/stream_poller.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,12 +283,27 @@ def get_records(self, shard_iterator: str) -> dict:
)
raise CustomerInvocationError from e
elif "ResourceNotFoundException" in str(e):
LOG.warning(
"Source stream %s does not exist: %s",
# FIXME: The 'Invalid ShardId in ShardIterator' error is returned by DynamoDB-local. Unsure when/why this is returned.
if "Invalid ShardId in ShardIterator" in str(e):
LOG.warning(
"Invalid ShardId in ShardIterator for %s. Re-initializing shards.",
self.source_arn,
)
self.initialize_shards()
else:
LOG.warning(
"Source stream %s does not exist: %s",
self.source_arn,
e,
)
raise CustomerInvocationError from e
elif "TrimmedDataAccessException" in str(e):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you so much for adding this! 🙏 I'll tell the users next week about it for them to confirm to us it solves the issue and they don't need to run the command manually anymore! Thank you very much for going the extra mile here 🙇

LOG.debug(
"Attempted to iterate over trimmed record or expired shard iterator %s for stream %s, re-initializing shards",
shard_iterator,
self.source_arn,
e,
)
raise CustomerInvocationError from e
self.initialize_shards()
else:
LOG.debug("ClientError during get_records for stream %s: %s", self.source_arn, e)
raise PipeInternalError from e
Expand Down
Loading