LDM as provisioner of debug-enabled containers #12851

MEPalma · 2025-07-10T12:44:21Z

Motivation

The current LDM implementation depends heavily on the Lambda service’s container provisioning logic, injecting LDM-specific parameter overrides throughout various parts of the service to support debug-enabled containers. This fragmented integration makes LDM difficult to maintain and limits control over the lifecycle of debug-enabled containers. This refactor centralizes LDM responsibilities by transforming it into a dedicated provisioner for debug-enabled containers. As a result, the Lambda service is relieved of most LDM-related logic. Additionally, these changes ensure that debug-enabled containers persist across multiple invocations, allowing the debug client to remain connected or reconnect after disconnections.

Changes

LDM as a debug-enabled container provisioner
removed LDM fragments in the lambda provider
introduce new ExecutionEnvironment subtype for container-enabled execution environments
avoid container restarts between debug invokes, for compatibility python lambda functions must include a revised debug server logic:

def handler(event, context):
    print(event)
    return event

def wait_for_debug_client(port: int=19891, timeout: int=3600):
    import time, threading
    import sys, glob
    sys.path.append(glob.glob(".venv/lib/python*/site-packages")[0])
    import debugpy

    if not hasattr(wait_for_debug_client, "_debugpy_listening"):
        wait_for_debug_client._debugpy_listening = False

    if not wait_for_debug_client._debugpy_listening:
        try:
            debugpy.listen(("0.0.0.0", port))
            wait_for_debug_client._debugpy_listening = True
            print(f"debugpy is now listening on port {port}")
        except RuntimeError as e:
            print(f"debugpy.listen() failed or already active: {e}")

    if not debugpy.is_client_connected():
        print("Waiting for client to attach debugger...")

        def cancel_wait():
            time.sleep(timeout)
            print("Canceling debug wait task after timeout...")
            debugpy.wait_for_client.cancel()

        threading.Thread(target=cancel_wait, daemon=True).start()
        debugpy.wait_for_client()
    else:
        print("Debugger already attached.")

wait_for_debug_client()

several other related changes

github-actions · 2025-07-10T12:54:23Z

Test Results - Preflight, Unit

21 862 tests ±0 20 205 ✅ ±0 6m 27s ⏱️ +14s
1 suites ±0 1 657 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 0ce1ab4. ± Comparison against base commit 8057b19.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-10T13:03:08Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 7s ⏱️ ±0s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 0ce1ab4. ± Comparison against base commit 8057b19.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-10T13:36:09Z

Test Results (amd64) - Integration, Bootstrap

5 files ±0 5 suites ±0 2h 20m 19s ⏱️ - 1m 55s
5 288 tests ±0 4 358 ✅ ±0 930 💤 ±0 0 ❌ ±0
5 294 runs ±0 4 358 ✅ ±0 936 💤 ±0 0 ❌ ±0

Results for commit 0ce1ab4. ± Comparison against base commit 8057b19.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-10T13:48:25Z

LocalStack Community integration with Pro

2 files ±0 2 suites ±0 1h 46m 36s ⏱️ +6s
4 929 tests ±0 4 152 ✅ ±0 777 💤 ±0 0 ❌ ±0
4 931 runs ±0 4 152 ✅ ±0 779 💤 ±0 0 ❌ ±0

Results for commit 0ce1ab4. ± Comparison against base commit 8057b19.

♻️ This comment has been updated with latest results.

joe4dev

Nice refactoring to centralize the scattered LDM changes and enable multi-invoke debugging with the same debug connection 👏👏 .

I tested these changes with the base scenario of the adjusted pro sample in and they work as expected localstack-samples/localstack-pro-samples#258

I raised a couple of clarification questions, most seem already addressed/answered.

joe4dev · 2025-07-16T13:29:02Z

localstack-core/localstack/services/lambda_/invocation/assignment.py

@@ -79,10 +75,7 @@ def get_environment(

        try:
            yield execution_environment
-            if is_lambda_debug_timeout_enabled_for(lambda_arn=function_version.qualified_arn):
-                self.stop_environment(execution_environment)


praise: Nice simplification and quality of life improvement to not stop the LDM-enabled container anymore 👍

joe4dev · 2025-07-16T13:37:46Z

localstack-core/localstack/services/lambda_/invocation/execution_environment.py

@@ -139,7 +135,7 @@ def get_environment_variables(self) -> Dict[str, str]:
            # AWS_LAMBDA_DOTNET_PREJIT
            "TZ": ":UTC",
            # 2) Public AWS RIE interface: https://github.com/aws/aws-lambda-runtime-interface-emulator
-            "AWS_LAMBDA_FUNCTION_TIMEOUT": self._get_execution_timeout_seconds(),


question: Is a debug-enabled Lambda function killed pre-maturely by the Golang code if we don't extend this timeout anymore?

Background: AWS_LAMBDA_FUNCTION_TIMEOUT is used by the Golang Lambda RIE here.

I see that LDM overwrites this later here.

Exactly, like you pointed out, this value continues to be refined by the LDM through the debug-enabled execution environment

joe4dev · 2025-07-16T13:43:44Z

localstack-core/localstack/services/lambda_/invocation/version_manager.py

@@ -191,6 +192,28 @@ def invoke(self, *, invocation: Invocation) -> InvocationResult:
            LOG.warning(message)
            raise ServiceException(message)

+        if debug_execution_environment := LDM.get_execution_environment(


docs(nit): It might be worth adding some docstring here, clarifying that debug-enabled execution environments don't consider Lambda quotas

joe4dev · 2025-07-16T13:44:14Z

localstack-core/localstack/services/lambda_/lambda_debug_mode/ldm.py

+
+LOG = logging.getLogger(__name__)
+
+# Specifies the fault timeout value in seconds to be used by time restricted workflows when


typo (nit): fault -> default

joe4dev · 2025-07-16T14:02:17Z

localstack-core/localstack/services/lambda_/lambda_debug_mode/ldm.py

+        with self._mutex:
+            if self._debug_execution_environment is not None:
+                return
+            self.stop_debug_enabled_execution_environment()


question: Why are we stopping the environment here? Can this happen given the return above if there is an existing environment?

thank you, yes this is indeed redundant, I removed this statement

joe4dev · 2025-07-16T14:05:58Z

localstack-core/localstack/services/lambda_/lambda_debug_mode/ldm.py

+                lambda_function_debug_config=self.lambda_function_debug_config,
+                on_timeout=self._on_execution_environment_timeout,
+            )
+            LOG.info(


question: Shouldn't that info log happen after starting the environment? Like status == RuntimeStatus.READY?

In principle yes, I would like this Log to occur after the container is READY. However, due to the debug layer, the docker container will not notify LS until the user has connected the debug client. I think this is something we should refine soon: I added a todo.

joe4dev · 2025-07-16T14:10:45Z

localstack-core/localstack/services/lambda_/invocation/lambda_models.py

@@ -66,6 +66,7 @@ class Invocation:
    # = invocation_id
    request_id: str
    trace_context: dict
+    user_agent: Optional[str] = None


FYI: The Invocation is used in the event manager for async invokes. The optional new field should be ok, given it initializes with a default

joe4dev · 2025-07-16T14:15:33Z

localstack-core/localstack/services/lambda_/lambda_debug_mode/ldm_config_file.py

@@ -0,0 +1,178 @@
+from __future__ import annotations


FYI: refactoring where code is moved from https://github.com/localstack/localstack/pull/12851/files#diff-2266754b538bc83024fa9e051519928adf58f50124f471c186512e25731e8736L1

joe4dev · 2025-07-16T14:17:20Z

localstack-core/localstack/services/lambda_/lambda_debug_mode/ldm.py

+    def get_execution_environment(self) -> DebugEnabledExecutionEnvironment:
+        # TODO: add support for concurrent invokes, such as invoke object queuing, new container spinup
+        with self._mutex:
+            # TODO: move this start-up logic to lambda function creation.


question (regression): Does this limitation introduce a regression such that LDM is not applied to Lambda functions created or updated AFTER adding an LDM configuration?

Testing showed that this is not the case.

Brainstorming the list of change points potentially to consider (in the future):

CreateFunction: Marco pointed out that function creation is handled here

UpdateFunction: doesn't change ARN, should be fine

UpdateFunctionConfiguration: doesn't change ARN, should be fine

PublishVersion: creates a new function version, which might require spawning a debug-enabled container if the new function version ARN matches a debug config

CreateAlias: creates a new function ARN via the alias Name, which could match a debug config

Thank you for this list, it's very helpful. I'll refer to this to refine the revision of this logic we already implemented in follow-up changes to these.

joe4dev · 2025-07-16T14:22:18Z

localstack-core/localstack/services/lambda_/lambda_debug_mode/ldm.py

+    def _on_execution_environment_timeout(
+        self, version_manager_id: str, environment_id: str
+    ) -> None:
+        # TODO: add support


question: What does "add support" include (e.g., raising aws-validated timeout error)?
Does this happen if DEFAULT_LAMBDA_DEBUG_MODE_TIMEOUT_SECONDS (i.e., 1h) is exceeded or enforce-timeout is disabled and then running into a short function timeout?

This functionality should not be applicable for debug-enabled containers. It is used in the release of on-demand containers, however these are provisioned-concurrency. I revised the comment to be more informative on the situation.

MEPalma added this to the Playground milestone Jul 10, 2025

MEPalma self-assigned this Jul 10, 2025

MEPalma added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Jul 10, 2025

MEPalma marked this pull request as ready for review July 14, 2025 10:34

MEPalma requested review from joe4dev, dominikschubert, dfangl and gregfurman as code owners July 14, 2025 10:34

MEPalma removed request for dominikschubert, dfangl and gregfurman July 14, 2025 10:34

MEPalma added 12 commits July 16, 2025 09:25

LDM as provisioner

a63440f

minor

85fb5cc

fixes

7bbb5c7

LDM as provisioner

f25fbe2

minor

ad1471f

fixes

3e53491

remove configs on function delete

bcbd098

fix

a9c737a

minor

5de4922

minor

4111022

fix

827e83a

minor

6bc6c40

MEPalma force-pushed the MEP-LDM-provisioner branch from 90030a5 to 6bc6c40 Compare July 16, 2025 07:25

joe4dev approved these changes Jul 16, 2025

View reviewed changes

PR items

0ce1ab4

MEPalma merged commit 04aa2bc into main Jul 17, 2025
39 checks passed

MEPalma deleted the MEP-LDM-provisioner branch July 17, 2025 11:59


		LOG = logging.getLogger(__name__)

		# Specifies the fault timeout value in seconds to be used by time restricted workflows when

Uh oh!

LDM as provisioner of debug-enabled containers #12851

LDM as provisioner of debug-enabled containers #12851

Uh oh!

Conversation

MEPalma commented Jul 10, 2025

Motivation

Changes

Uh oh!

github-actions bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

github-actions bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

joe4dev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 10, 2025 •

edited

Loading

github-actions bot commented Jul 10, 2025 •

edited

Loading

github-actions bot commented Jul 10, 2025 •

edited

Loading

github-actions bot commented Jul 10, 2025 •

edited

Loading