LocalStack Community integration with Pro

    2 files  ±    0      2 suites  ±0   1h 30m 8s ⏱️ - 25m 23s
3 143 tests  - 1 162  2 924 ✅  - 1 060  219 💤  - 102  0 ❌ ±0 
3 145 runs   - 1 162  2 924 ✅  - 1 060  221 💤  - 102  0 ❌ ±0 

Results for commit e5bb66c. ± Comparison against base commit 68df8dc.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

a) hotreload>operation: Should we use the operations label (e.g., create) if we envison that at some point we might want to track other operations (e.g., reload). Unfortunately, that's a bit tricky in this case because we would need to do that in our custom Lambda RIE Golang codebase.

I’d suggest yes if create and reload, or other operations in general will share common characteristics and, in the future you want to track them in a more fine-grained way with additional labels/dimensions.

b) function>operation: Should we add an operations label (e.g., invoke) if we envison to track create as well? If so, how to treat the invocation type label, which only applies for the invoke operation?

That’s a tough one, and it ties back to question (a) 🤣

If invocation_type is the only label that isn’t shared across the different operations you want to track, we could compromise by leaving invocation_type empty for create events.

However, if you anticipate that different operations will diverge and require operation-specific labels, then I’d suggest creating two separate metrics instead of using an operation label, even if the two metrics share some common characteristics.

c) function>status: What do you think about using typed definitions for such a status label? Are we too fine-grained here?

What’s the cardinality here, this? If this is the cardinality, it seems reasonable to track it, as long as it’s valuable for you. Additionally, if everything != 'success' can be considered an error, filtering for error statuses should be straightforward. However, grouping them might be less convenient, in that case, we would need to introduce a general error attribute to group all error events != 'success'.

Additional Note:
If it helps with your decision, I totally understand that it’s often difficult to anticipate how the measurement of a feature/service might evolve over time. That’s why I’m considering introducing a schema versioning mechanism to help mitigate this issue—allowing us to apply corrections to label usage (e.g., deprecations, additions, modifications, etc.).

That said, I wouldn’t want this to be over-relied upon or used as a fallback for every case. Ideally, we should still aim to anticipate future needs as much as possible.

If you’d like, we can have a quick chat and run some queries together to test different combinations or possibilities. That might give me a better understanding of your intentions and help refine the approach 💪🏼

However, grouping them might be less convenient, in that case, we would need to introduce a general error attribute to group all error events != 'success'.

@vittoriopolverino to confirm, do you suggest here to have 2 separate labels - one for general operation result, i.e. success/error, and another one for specific reason for error?

@joe4dev @vittoriopolverino I added a draft PR for EventBridge Pipes: https://github.com/localstack/localstack-ext/pull/4224
Since Pipes and ESM share a part of infrastructure, namely pollers, we need to coordinate both changes and maybe unify enums like status, wdyt?

to confirm, do you suggest here to have 2 separate labels - one for general operation result, i.e. success/error, and another one for specific reason for error?

Yes 👍🏼

• Advantage: Having both successful and failed invocations in a single metric, since they share more common characteristics than differences.
• Disadvantage: For successful invocations, the error reason label will be empty, but that’s an acceptable trade-off in this case.

I added a draft PR for EventBridge Pipes: https://github.com/localstack/localstack-ext/pull/4224
Since Pipes and ESM share a part of infrastructure, namely pollers, we need to coordinate both changes and maybe unify enums like status, wdyt?

From a data modeling perspective, if there are enums having the same meaning in both EventBridge Pipes and ESM, it would make sense to ensure consistency across them.

That way, you can use the same filter to reference a status across both Pipes and ESM, instead of needing two separate filters to represent the same status. (Not sure if this kind of cross-service analysis is something you’d ever do, but in any case, if we can ensure consistency, it’s better)

to confirm, do you suggest here to have 2 separate labels - one for general operation result, i.e. success/error, and another one for specific reason for error?

Yes 👍🏼

* Advantage: Having both successful and failed invocations in a single metric, since they share more common characteristics than differences.

* Disadvantage: For successful invocations, the error reason label will be empty, but that’s an acceptable trade-off in this case.

One of the main question for me is what do we categorize as "error":

• AWS API error responses can be expected errors and part of the intended application behavior (e.g., partial batch failure handling in Lambda ESM because an application entry fails and then gets retried). These errors are expected AWS behavior and from a LocalStack perspective not actionable errors because we correctly implement AWS error parity.
• Unhandled errors are actionable for LocalStack because they indicate bugs in LocalStack and can guide maintenance effort to certain code paths (e.g., unhandled_state_error).
• Some errors can be user-side errors (e.g., pending_state_error: invoke a Lambda function that is not yet active)
• For some errors, we have insufficient information to categorize them and they could be user-side or LocalStack bugs (e.g., invocation_error or failed_state_error)

@vittoriopolverino @tiurin What is your understanding/suggestion regarding what should be considered an "error"?

What is your understanding/suggestion regarding what should be considered an "error"?

🤔 If an error is expected (part of intended application behavior), doesn’t require intervention, and isn’t something we need to act on, then maybe we need a way to differentiate errors.

We have two options:
1️⃣ Use a single status label with three or more categories: success, actionable_error, and non_actionable_error, etc. The downside is that this makes it harder to filter all errors under a single error status.
2️⃣ Introduce an additional label, like is_error_actionable (true/false), while keeping status=error. This keeps all errors grouped but adds a second error-specific label alongside error_details.

Which approach makes the most sense really depends on the query patterns you/we anticipate using the most in the future. I don’t think there’s a strictly right or wrong choice between the two, it’s more about what will make querying and analysis more efficient for your use cases

From a data modeling perspective, if there are enums having the same meaning in both EventBridge Pipes and ESM, it would make sense to ensure consistency across them.

Pipes is a more broad case of ESM, moreover they have some shared code, so if we can align both well we will avoid "if pipes" conditions for counters.

Background on trade-off discussions:

1. When to introduce operation-specific counters given the tradeoff n/a fields vs duplicated fields? For example, Lambda function creation and invocation has some shared (e.g., runtime) and some operation-specific (e.g., invocation_type) fields.

Most dimensions seem relevant for both create and invoke operations, which allows us to compare usage of the control vs. data plane. We can live with the invocation_type field using n/a upon the create operation.

2. How to decide whether an additional top-level is_error field (i.e., status field) is helpful. Especially when things might not add up to 100% in ESM/Pipes.

In ESM, Pipes, and Lambda, it's not clearly possible to distinguish actionable and user-side errors (e.g., invocation_error can have multiple root causes). Therefore, we couldn't clearly identify high-level error categories and a boolean flag is_error could be misleading because in ESM and Pipes, poller errors happen in a different part of the system and don't naturally complement the success case (which happens only if records are available and a pipe execution or ESM execution is triggered).

Don't we need a similar or "n/a" for the invokes in the function counters as well, not just the creation?

absolutely spot on @dfangl 💯 Thank you!

I added the or "n/a" for invokes in the lambda_service (sync) and event_manager (async) 👍