Resolve non-subdomain host prefixes to LocalStack #12653
Conversation
Test Results - Alternative Providers597 tests 419 ✅ 13m 57s ⏱️ Results for commit e5ee939. ♻️ This comment has been updated with latest results. |
Test Results (amd64) - Integration, Bootstrap 5 files 5 suites 2h 22m 15s ⏱️ Results for commit e5ee939. ♻️ This comment has been updated with latest results. |
cb5aed9 to
d8751a2
Compare
# Conflicts: # tests/aws/services/lambda_/test_lambda.snapshot.json
649d180 to
27cc631
Compare
Likely updated related to this change (just a guess): https://github.com/localstack/localstack/pull/12645/files#diff-1d20d60454c412d95e42d6c9d2626a4389b249eb1cc22a8c1a815cf81b1893f5L62 Couldn't find any operations that use the `discovery-` prefix anymore. ```py discovery_prefixes = df[df['hostPrefix'].str.startswith('discovery-', na=False)] ```
| @@ -460,3 +466,19 @@ def test_no_resolv_conf_overwriting_on_host(self, tmp_path: Path, monkeypatch): | |||
|
|
|||
| assert "nameserver 127.0.0.1" not in new_contents.splitlines() | |||
| assert "nameserver 127.0.0.11" in new_contents.splitlines() | |||
|
|
|||
| def test_host_prefix_no_subdomain( | |||
There was a problem hiding this comment.
Thank you for adding this test step, this will be great for preventing regressions and making sure we handle new services!
| unique_prefixes.add(operation.endpoint["hostPrefix"]) | ||
|
|
||
| non_dot_unique_prefixes = [prefix for prefix in unique_prefixes if not prefix.endswith(".")] | ||
| assert set(HOST_PREFIXES_NO_SUBDOMAIN) == set(non_dot_unique_prefixes) |
There was a problem hiding this comment.
suggestion(minor): I think the real test that would give me confidence is that if everything in non_dot_unique_prefixes is also included in NAME_PATTERNS_POINTING_TO_LOCALSTACK as this is the complete list of hostnames that target LocalStack. There may be some subsequent manipulation of HOST_PREFIXES_NO_SUBDOMAIN that affect the DNS behaviour.
Depends: on public DNS entries for the dash-prefixed domains deployed (e.g.,
data-localhost.localstack.cloud)Motivation
The botocore specs include 562 operations with prefixed endpoints such as
api.ordata-and non-subdomains (i.e.,-dash-prefixed operations) require special consideration.These operations are defined in the botocore specs with a
hostPrefix. Example for DiscoverInstances in CloudMap:"endpoint":{"hostPrefix":"data-"}Example of a failing AWS CLI command due to missing DNS resolution:
aws --profile localstack servicediscovery discover-instances --namespace-name my-namespace --service-name my-service Could not connect to the endpoint URL: "https://data-localhost.localstack.cloud:4566/"To fix this problem, we need to:
a) deploy public DNS entries for these currently 11 prefixes
b) add them to
NAME_PATTERNS_POINTING_TO_LOCALSTACKto ensure they get resolved to LocalStack by our DNS serverChanges
NAME_PATTERNS_POINTING_TO_LOCALSTACKsuch that these are resolved to LocalStack.test_resolve_localstack_hostto cover thehostPrefixAPI operation scenariohostPrefixoperation for Lambda intest_lambda_host_prefix_api_operationDiscussion
Why a static list?
I think it's clearer (and faster) to maintain a static list such that we have full control and easier visibility into which name patterns get resolved to LocalStack. It also helps detecting changes, which require manual action by deploying a new DNS entry into our public DNS.