Feature: Firehose/Kinesis: Add resharding (split merge) kinesis event stream validated tests #21
Conversation
…nt failures better
…#10143) Co-authored-by: Benjamin Simon <benjh.simon@gmail.com>
…f Message Arguments (localstack#10165)
There was a problem hiding this comment.
PR Summary
This pull request introduces new tests and enhancements for Kinesis Firehose functionality in LocalStack, focusing on resharding operations and multiple delivery streams.
- Added tests for Firehose with Kinesis as source, validating behavior during resharding (split and merge) operations
- Implemented new test for multiple Firehose delivery streams connected to a single Kinesis stream
- Enhanced Firehose provider to support unique DynamoDB lease tables for each delivery stream
- Improved Kinesis mock server log level handling for better consistency with LocalStack logging
- Updated fixtures in conftest.py to support more flexible Firehose and Kinesis testing scenarios
30 file(s) reviewed, 5 comment(s)
Edit PR Review Bot Settings
| # publish metric only if CloudWatch service is available | ||
| if not is_api_enabled("cloudwatch"): | ||
| return | ||
| cw_client = connect_to(region_name=region_name).cloudwatch | ||
| cw_client = connect_to(aws_access_key_id=account_id, region_name=region_name).cloudwatch |
There was a problem hiding this comment.
style: Consider using a consistent parameter order across functions (e.g., account_id before region_name)
| @@ -3,16 +3,14 @@ | |||
| "use strict"; | |||
|
|
|||
| const AWS = require("aws-sdk"); | |||
| let dynamoDb; | |||
|
|
|||
| var config = {}; | |||
There was a problem hiding this comment.
style: Consider using 'const' instead of 'var' for better scoping
| @@ -3,16 +3,14 @@ | |||
| "use strict"; | |||
|
|
|||
| const AWS = require("aws-sdk"); | |||
| let dynamoDb; | |||
|
|
|||
| var config = {}; | |||
| if (process.env.AWS_ENDPOINT_URL) { | |||
There was a problem hiding this comment.
style: Consider using const instead of var for config object
| shard_id = shard["ShardId"][-2:] | ||
| starting_hash_keys[shard_id] = shard["HashKeyRange"]["StartingHashKey"] |
There was a problem hiding this comment.
logic: Extracting only the last 2 characters of ShardId may not be reliable for all shard naming conventions
| json_array_string = "[" + input_string.replace("}{", "},{") + "]" | ||
| message = json.loads(json_array_string) |
There was a problem hiding this comment.
style: This JSON parsing method assumes a specific format. Consider using a more robust parsing approach
Motivation
Changes