EventBridge Connections tests plus implementation #11836

zaingz · 2024-11-12T17:08:54Z

Motivation

Implemented EventBridge Connections and API Destinations in LocalStack to support integration with external APIs and services.

Changes

Connections API:
- Added support for CreateConnection, UpdateConnection, ListConnections, and DeleteConnection.
- Implemented authentication types: BASIC, API_KEY, and OAUTH_CLIENT_CREDENTIALS.
- Handled sensitive auth parameters securely using SecretArn.
- Managed connection states (AUTHORIZED, AUTHORIZING, DELETING) with accurate timestamps.
- Matched AWS behavior for validation errors, ARN formats, and state transitions.
API Destinations API:
- Added support for CreateApiDestination, UpdateApiDestination, DescribeApiDestination, ListApiDestinations, and DeleteApiDestination.
- Integrated API Destinations with Connections, ensuring proper validation and state synchronization.
- Aligned error handling and parameter validation with AWS standards.
Testing and Fixtures:
- Created common fixtures (create_connection, create_api_destination) for reusable setup in tests.
- Updated tests to cover all operations, authentication types, and edge cases.
- Used snapshot testing to ensure responses and errors match AWS behavior.
- Fixed issues with parameter validation and key errors in tests.

Testing

Connections:
- Tested creation, updating, listing, and deletion with different auth types.
- Verified validation of auth parameters and connection names.
- Ensured correct state transitions and error handling.
API Destinations:
- Tested creation, updating, describing, listing, and deletion.
- Verified integration with Connections and proper state management.
- Checked validation errors and exception handling.
Snapshot Testing:
- Ensured API responses and errors match AWS responses exactly.
- Applied transformers to handle dynamic values in snapshots.

github-actions · 2024-11-12T17:50:22Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 1h 42m 8s ⏱️ - 1m 46s
3 540 tests +14 3 133 ✅ ±0 407 💤 +14 0 ❌ ±0
3 542 runs +14 3 133 ✅ ±0 409 💤 +14 0 ❌ ±0

Results for commit c87049c. ± Comparison against base commit 75436ef.

♻️ This comment has been updated with latest results.

joe4dev

Great work @zaingz with thorough validations and parametrized test coverage 👏 👏 👏

Most comments are improvement suggestions, but I see two things we need to fix:

Test fixtures miss cleanups when run against AWS. Each fixture should ensure that all created resources are automatically deleted upon test teardown. This is typically achieved by yielding and then cleaning up the created resource and potential dependency. Example: aws.services.events.conftest.events_put_rule
Persistence isn't implemented for connections and API destinations because they are not stored in the LS store in localstack-core/localstack/services/events/models.py. EventBridge tries to do persistence quite differently than most other services, which requires keeping track of instances at multiple places.

These shouldn't block API functionality, so we can consider merging this PR and tackling these fixes in an immediate follow-up. What do you think?

joe4dev · 2024-11-18T13:59:56Z

localstack-core/localstack/services/events/provider.py


    def on_before_start(self):
        JobScheduler.start()

    def on_before_stop(self):
        JobScheduler.shutdown()

+    ##########
+    # Helper Methods for connections and api destinations


Organization: We tend to externalize such helper methods into utils.py as standalone methods (PyCharm also suggests these helpers may be static)

joe4dev · 2024-11-18T14:10:51Z

localstack-core/localstack/services/events/provider.py

+            return ListConnectionsResponse(Connections=connections)
+
+        except Exception as e:
+            raise ValidationException(f"Error listing connections: {str(e)}")


Would it make sense to log in to LS logs here?
It would make it easier to debug having a stack trace in the logs.

joe4dev · 2024-11-18T14:13:31Z

localstack-core/localstack/services/events/provider.py

+            if limit:
+                connections = connections[:limit]
+
+            return ListConnectionsResponse(Connections=connections)


It seems that the listing supports pagination: https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_ListConnections.html. We could consider adding it in a follow up or upon request.

There should be plenty of examples in LS: https://github.com/localstack/localstack/pulls?q=is%3Apr+is%3Aclosed+paginate
Greg also knows about it :)

joe4dev · 2024-11-18T14:16:23Z

localstack-core/localstack/services/events/provider.py

+
+            return ListApiDestinationsResponse(
+                ApiDestinations=api_destination_summaries,
+                NextToken=None,  # Pagination token handling can be added if needed


👍 ok in a follow up or upon request

joe4dev · 2024-11-18T14:19:17Z

localstack-core/localstack/testing/snapshots/transformer_utility.py

@@ -740,6 +740,41 @@ def stepfunctions_api():
    # def custom(fn: Callable[[dict], dict]) -> Transformer:
    #     return GenericTransformer(fn)

+    @staticmethod
+    def eventbridge_api_destination(snapshot, connection_name: str):


nit: the other pytest fixture use the botocore name events as prefix rather than eventbridge, might be a little more discoverable and consistent to standardize

joe4dev · 2024-11-18T14:27:15Z

tests/aws/services/events/conftest.py

+                AuthParameters=auth_parameters,
+            )
+
+    return _create_connection


Aren't we missing some cleanup here or does that happen somewhere else?
That's mainly relevant when testing against AWS

joe4dev · 2024-11-18T14:28:02Z

tests/aws/services/events/conftest.py

+
+
+@pytest.fixture
+def create_connection(aws_client, connection_name):


nice usage and extensions of our snapshot utilities 👏

joe4dev · 2024-11-18T14:43:01Z

tests/aws/services/events/test_events.py

+
+class TestEventBridgeConnections:
+    @pytest.fixture
+    def connection_snapshots(self, snapshot, connection_name):


hint: if the snapshot fixture is used everywhere, it's possible to use @pytest.fixture(autouse=True)

Example from aws.services.lambda_.test_lambda.fixture_snapshot

@pytest.fixture(autouse=True) def fixture_snapshot(snapshot): snapshot.add_transformer(snapshot.transform.lambda_api()) snapshot.add_transformer( snapshot.transform.key_value("CodeSha256", reference_replacement=False) )

joe4dev · 2024-11-18T14:43:46Z

tests/aws/services/events/test_events.py

+        is_old_provider(),
+        reason="V1 provider does not support this feature",
+    )
+    @pytest.mark.parametrize("auth_params", API_DESTINATION_AUTH_PARAMS)


nicely parameterized 👏 👏👏

joe4dev · 2024-11-18T14:44:23Z

tests/aws/services/events/test_events.py

+            },
+        )
+
+        response = aws_client.events.list_connections(NamePrefix=connection_name)


well-handled scoping 💯

joe4dev · 2024-11-18T15:38:34Z

If I understand correctly, the current implementation covers the CRUD part, which is fine for reaching v1 parity. Nevertheless, we should be explicit about this in the docs that emulation is not (yet) implemented as it requires integrations with the secrets manager, HTTPs APIs, and incoming requests (e.g., matched rules):

joe4dev · 2024-11-19T10:39:05Z

Merging this after a tiny rebase (just an import error) because the pipeline was green and we need this missing feature for testing the default switch in #11834

zaingz requested review from maxhoheiser and joe4dev as code owners November 12, 2024 17:08

zaingz added this to the 4.0 milestone Nov 13, 2024

zaingz added the aws:events Amazon EventBridge label Nov 13, 2024

joe4dev added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Nov 13, 2024

zaingz marked this pull request as draft November 13, 2024 11:41

zaingz force-pushed the connections_implementation branch from c528d7f to 4e2faab Compare November 14, 2024 15:01

zaingz changed the title ~~[WIP] EventBridge Connections tests plus implementation~~ EventBridge Connections tests plus implementation Nov 14, 2024

zaingz marked this pull request as ready for review November 15, 2024 09:23

zaingz requested review from dominikschubert and steffyP as code owners November 15, 2024 09:23

zaingz force-pushed the connections_implementation branch from 65789a7 to c87049c Compare November 16, 2024 19:27

joe4dev approved these changes Nov 18, 2024

View reviewed changes

zaingz added 4 commits November 19, 2024 11:28

[WIP] Tests plus implementation

d4098d6

refactoring

cc7e06f

fixed failing test; method signature missmatch

a45b16a

implemented api destination

1f8b756

joe4dev force-pushed the connections_implementation branch from c87049c to 1f8b756 Compare November 19, 2024 10:30

joe4dev assigned zaingz Nov 19, 2024

joe4dev merged commit c072be8 into master Nov 19, 2024
24 of 27 checks passed

joe4dev deleted the connections_implementation branch November 19, 2024 10:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

EventBridge Connections tests plus implementation #11836

EventBridge Connections tests plus implementation #11836

Uh oh!

zaingz commented Nov 12, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Nov 12, 2024 •

edited

Loading

Uh oh!

joe4dev left a comment

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev Nov 18, 2024

Uh oh!

joe4dev commented Nov 18, 2024

Uh oh!

joe4dev commented Nov 19, 2024

Uh oh!

Uh oh!

Uh oh!



		@pytest.fixture
		def create_connection(aws_client, connection_name):

Uh oh!

EventBridge Connections tests plus implementation #11836

EventBridge Connections tests plus implementation #11836

Uh oh!

Conversation

zaingz commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

Uh oh!

github-actions bot commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

joe4dev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joe4dev commented Nov 18, 2024

Uh oh!

joe4dev commented Nov 19, 2024

Uh oh!

Uh oh!

Uh oh!

zaingz commented Nov 12, 2024 •

edited

Loading

github-actions bot commented Nov 12, 2024 •

edited

Loading