Configure cosmetic and functional variables with LOCALSTACK_HOST and GATEWAY_LISTEN #7893

simonrw · 2023-03-17T12:00:09Z

We are switching to using LOCALSTACK_HOST and GATEWAY_LISTEN to configure LocalStack.

Variable types

These variables are split into two types: cosmetic and functional. Cosmetic are purely used as hostnames in returned URLs or addresses by LocalStack. LocalStack does not use these variables to configure services.

functional variables are used to configure the networking stack of LocalStack.

Roles

LOCALSTACK_HOST: (cosmetic) this variable is interpolated into URLs and addresses. It has the form <hostname>:<port> and defaults to localhost.localstack.cloud:4566.
GATEWAY_LISTEN: (functional) this configures the bind addresses of LocalStack. It has the form <ip address>:<port>(,<ip address>:<port>)* and defaults to 127.0.0.1:4566 on the host and 0.0.0.0:4566 in docker.

`GATEWAY_LISTEN` advantages

Since GATEWAY_LISTEN supports multiple addresses (separated by ,) we can provide configuration for pro users who wish to use SSL e.g. GATEWAY_LISTEN=0.0.0.0:4566,0.0.0.0:443.

Legacy configuration

Existing configuration of the system is maintained with the existing legacy variables (EDGE_PORT, EDGE_PORT_HTTP and EDGE_BIND_HOST) as these are now derived from GATEWAY_LISTEN if absent. Some examples:

GATEWAY_LISTEN: "" -> EDGE_PORT=4566,EDGE_PORT_HTTP=4566,EDGE_BIND_HOST=127.0.0.1/0.0.0.0
GATEWAY_LISTEN=0.0.0.0:9999 -> EDGE_PORT=9999,EDGE_PORT_HTTP=9999,EDGE_BIND_HOST=0.0.0.0

If GATEWAY_LISTEN is not given but legacy variables are, they pick up the supplied values rather than inheriting the default GATEWAY_LISTEN.

If GATEWAY_LISTEN includes multiple values, the first will be used to configure our hypercorn listener, and the rest will be transparent proxies (implementation in future PR).

Legacy variables have been marked as deprecated:

LOCALSTACK_HOSTNAME
HOSTNAME_EXTERNAL
EDGE_PORT
EDGE_PORT_HTTP
EDGE_BIND_HOST

Design document

coveralls · 2023-03-17T12:47:58Z

Coverage: 84.943% (+0.005%) from 84.938% when pulling df661df on deprecate-envars into 5f13258 on master.

github-actions · 2023-03-17T13:44:06Z

LocalStack integration with Pro

1 827 tests ±0 1 481 ✔️ ±0 1h 5m 35s ⏱️ +56s
      1 suites ±0   345 💤 ±0
      1 files ±0       1 ❌ ±0

For more details on these failures, see this check.

Results for commit f4d8a4b. ± Comparison against base commit fad6714.

♻️ This comment has been updated with latest results.

localstack/config.py

thrau

this looks great! really nice deprecation path. just a couple of nits regarding naming, but nothing blocking

thrau · 2023-03-23T13:19:41Z

localstack/config.py

+        port = int(localstack_host.split(":")[-1])
+        edge_bind = f"{default_ip}:{port}"
+    else:
+        components = edge_bind.split(",")


nit: should we maybe output a warning here if there are >1 components in here for now, since they are ignored? we could however pretty soon update the logic to start the gateway and proxies.

#7893 (comment)

This means we can test the configuration without mutating global state, which affects downstream tests.

If the user has not migrated yet, we don't want the new variables' defaults overriding their configuration.

localstack/localstack-python-client#45

#7893 (comment)

simonrw · 2023-03-23T15:26:16Z

The test_presigned_url_v4_signed_headers_in_qs S3 integration test is failing, but will be fixed with an upcoming merge of the new lambda provider. Thanks @alexrashed for the suggestion. I will merge this even though the pipeline is failing.

…GATEWAY_LISTEN (#7893) Switch to using `LOCALSTACK_HOST` and `GATEWAY_LISTEN` for configuring the cosmetic and functional properties of LocalStack. * `LOCALSTACK_HOST`: cosmetic - interpolated into URLs and hostnames/ports returned * `GATEWAY_LISTEN`: functional - replaces `EDGE_PORT`, `EDGE_PORT_HTTP`, `EDGE_BIND_HOST`.

simonrw self-assigned this Mar 17, 2023

simonrw temporarily deployed to localstack-ext-tests March 17, 2023 12:00 — with GitHub Actions Inactive

simonrw force-pushed the deprecate-envars branch from 5f465ff to cea3aa6 Compare March 20, 2023 09:45

simonrw temporarily deployed to localstack-ext-tests March 20, 2023 09:46 — with GitHub Actions Inactive

simonrw marked this pull request as ready for review March 20, 2023 10:13

simonrw requested a review from thrau as a code owner March 20, 2023 10:13

simonrw marked this pull request as draft March 20, 2023 10:42

simonrw force-pushed the deprecate-envars branch from cea3aa6 to 2a317cd Compare March 22, 2023 13:19

simonrw temporarily deployed to localstack-ext-tests March 22, 2023 13:19 — with GitHub Actions Inactive

simonrw temporarily deployed to localstack-ext-tests March 22, 2023 15:28 — with GitHub Actions Inactive

simonrw temporarily deployed to localstack-ext-tests March 22, 2023 16:31 — with GitHub Actions Inactive

simonrw changed the title ~~Add deprecation notices for HOSTNAME_EXTERNAL/LOCALSTACK_HOST~~ Configure cosmetic and networking variables with LOCALSTACK_HOST and EDGE_BIND Mar 22, 2023

simonrw changed the title ~~Configure cosmetic and networking variables with LOCALSTACK_HOST and EDGE_BIND~~ Configure cosmetic and functional variables with LOCALSTACK_HOST and EDGE_BIND Mar 22, 2023

simonrw temporarily deployed to localstack-ext-tests March 22, 2023 16:50 — with GitHub Actions Inactive

simonrw marked this pull request as ready for review March 22, 2023 16:51

simonrw temporarily deployed to localstack-ext-tests March 22, 2023 16:51 — with GitHub Actions Inactive

This was referenced Mar 22, 2023

New localstack_host format localstack/localstack-python-client#45

Merged

Update configuration document with new networking description localstack/docs#526

Closed

simonrw temporarily deployed to localstack-ext-tests March 23, 2023 12:13 — with GitHub Actions Inactive

simonrw changed the base branch from master to v2 March 23, 2023 12:52

simonrw force-pushed the deprecate-envars branch from a744854 to 1343947 Compare March 23, 2023 12:53

simonrw temporarily deployed to localstack-ext-tests March 23, 2023 12:55 — with GitHub Actions Inactive

thrau reviewed Mar 23, 2023

View reviewed changes

localstack/config.py Outdated Show resolved Hide resolved

thrau reviewed Mar 23, 2023

View reviewed changes

localstack/config.py Outdated Show resolved Hide resolved

thrau approved these changes Mar 23, 2023

View reviewed changes

simonrw temporarily deployed to localstack-ext-tests March 23, 2023 13:37 — with GitHub Actions Inactive

simonrw added a commit that referenced this pull request Mar 23, 2023

Output log entry when multiple GATEWAY_LISTEN addresses are provided

f4d8a4b

#7893 (comment)

simonrw temporarily deployed to localstack-ext-tests March 23, 2023 13:41 — with GitHub Actions Inactive

simonrw changed the title ~~Configure cosmetic and functional variables with LOCALSTACK_HOST and EDGE_BIND~~ Configure cosmetic and functional variables with LOCALSTACK_HOST and GATEWAY_LISTEN Mar 23, 2023

simonrw added 13 commits March 23, 2023 15:19

Add deprecation notices for HOSTNAME_EXTERNAL/LOCALSTACK_HOST

c2e0fdb

Deprecate EGDE_ variables

19dc3f9

Implement graceful fallback for EDGE_VARIABLES

86a4075

Include (accidentally deleted) EDGE_FORWARD_URL

59567ed

Extract pure functions to derive config variables

0ed9dc3

This means we can test the configuration without mutating global state, which affects downstream tests.

Support overriding new variables with legacy if not given

b5645de

If the user has not migrated yet, we don't want the new variables' defaults overriding their configuration.

Refactor splitting edge bind by commas

d05c243

Remove unused fixture

d0a0736

Remove redundant line

a10bf32

Rename EDGE_BIND -> GATEWAY_LISTEN

f3e141d

Bump localstack client requirement

3468485

localstack/localstack-python-client#45

fix missing EDGE_BIND -> GATEWAY_LISTEN instances

adb9037

Output log entry when multiple GATEWAY_LISTEN addresses are provided

d7aa1e2

#7893 (comment)

simonrw force-pushed the deprecate-envars branch from f4d8a4b to d7aa1e2 Compare March 23, 2023 15:19

simonrw temporarily deployed to localstack-ext-tests March 23, 2023 15:19 — with GitHub Actions Inactive

simonrw merged commit 9f67f51 into v2 Mar 23, 2023

simonrw deleted the deprecate-envars branch March 23, 2023 15:28

joe4dev mentioned this pull request Apr 11, 2023

Disable HTTPS(S) proxies for internal calls #8066

Merged

viren-nadkarni mentioned this pull request Jun 27, 2023

Update deprecated config options localstack/docs#676

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Configure cosmetic and functional variables with LOCALSTACK_HOST and GATEWAY_LISTEN #7893

Configure cosmetic and functional variables with LOCALSTACK_HOST and GATEWAY_LISTEN #7893

Uh oh!

simonrw commented Mar 17, 2023 •

edited

Loading

Uh oh!

coveralls commented Mar 17, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Mar 17, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

thrau left a comment

Uh oh!

thrau Mar 23, 2023

Uh oh!

simonrw commented Mar 23, 2023

Uh oh!

Uh oh!

Uh oh!

Configure cosmetic and functional variables with LOCALSTACK_HOST and GATEWAY_LISTEN #7893

Configure cosmetic and functional variables with LOCALSTACK_HOST and GATEWAY_LISTEN #7893

Uh oh!

Conversation

simonrw commented Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Variable types

Roles

GATEWAY_LISTEN advantages

Legacy configuration

Uh oh!

coveralls commented Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack integration with Pro

Uh oh!

Uh oh!

Uh oh!

thrau left a comment

Choose a reason for hiding this comment

Uh oh!

thrau Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

simonrw commented Mar 23, 2023

Uh oh!

Uh oh!

simonrw commented Mar 17, 2023 •

edited

Loading

`GATEWAY_LISTEN` advantages

coveralls commented Mar 17, 2023 •

edited

Loading

github-actions bot commented Mar 17, 2023 •

edited

Loading