Motivation

Lambda Debug Mode configurations allow users to specify a debug port for each qualified Lambda ARN. This means that as long as lease limits are not exceeded, the system can concurrently evaluate multiple invocations of the same Lambda function version. However, this also means that additional containers might attempt to establish new debugging sessions on the same port. A similar issue can arise when updating provisioned concurrency configurations, which can result in multiple containers being spun up for the same Lambda function version. To enhance the developer experience in debugging Lambda functions, these changes introduces bypass logics for leasing and provisioning multiple instances of the same Lambda function version when it is being debugged.

Changes

• utility function is_lambda_debug_enabled_for which computed whether the given lambda arn is being debugged
• check for non concurrent leases in localstack.services.lambda_.invocation.counting_service.CountingService.get_invocation_lease
• limit true provisioning to 1 environment in localstack.services.lambda_.invocation.assignment.AssignmentService.scale_provisioned_concurrency

Testing

As a testing setup for Lambda Debug Mode is in the works and this feature is in preview, testing is conducted manually. Here follow scenarios relevant for these changes.

Test for Concurrent Invocations

1. Start Lambda Debug Mode with config:

   functions:
     arn:aws:lambda:eu-central-1:000000000000:function:handler_1-handler_1:
       debug-port: 19891

2. deploy lambda functions handler_1-handler_1 :

      DEBUGGER_ENDPOINT = "0.0.0.0"
      DEBUGGER_PORT = 19891
      
      def handler_1(event, context):
          """Lambda handler that will get invoked by the LocalStack runtime"""
      
          # wait for the debugger to get attached
          import debugpy
          debugpy.listen((DEBUGGER_ENDPOINT, DEBUGGER_PORT))
          debugpy.wait_for_client()  # blocks execution until client is attached
      
          # print the incoming invocation event
          print(event)
          return {"body": "Hello from handler_1"}

3. Send a first invoke for handler_1-handler_1

4. Send a second invoke (on another thread) for handler_1-handler_1

5. Assert that in the console the warning message Concurrent lambda invocations disabled for 'arn:aws:lambda:eu-central-1:000000000000:function:handler_1-handler_1:$LATEST' by Lambda Debug Mode for the second invocation, whilst the first one can continue to be debugged as usual by connecting to 19891

6. Assert only one container for the lambda version was created

7. If the tool used to forward the invocation requests implements a periodic retry (such as boto) then the above error message would show up every time the request is received

Test for Provisioning

1. Start Lambda Debug Mode with config (note we are defining the qualifier for version 1 in this configuration, that means that arn:aws:lambda:eu-central-1:000000000000:function:handler_1-handler_1, which is equal to arn:aws:lambda:eu-central-1:000000000000:function:handler_1-handler_1$LATEST will not be affected by Lambda Debug Mode in any way):

   functions:
     arn:aws:lambda:eu-central-1:000000000000:function:handler_1-handler_1:1:
       debug-port: 19891

2. deploy lambda functions handler_1-handler_1 :

      DEBUGGER_ENDPOINT = "0.0.0.0"
      DEBUGGER_PORT = 19891
      
      def handler_1(event, context):
          """Lambda handler that will get invoked by the LocalStack runtime"""
      
          # wait for the debugger to get attached
          import debugpy
          debugpy.listen((DEBUGGER_ENDPOINT, DEBUGGER_PORT))
          debugpy.wait_for_client()  # blocks execution until client is attached
      
          # print the incoming invocation event
          print(event)
          return {"body": "Hello from handler_1"}

3. create a version for the lambda function: awslocal lambda publish-version --function-name handler_1-handler_1

4. request provisioning for more than one instance:

    awslocal lambda put-provisioned-concurrency-config \
      --function-name handler_1-handler_1 \
      --qualifier 1\
      --provisioned-concurrent-executions 30

5. assert in the console the following warning is logged: Environments for 'arn:aws:lambda:eu-central-1:000000000000:function:handler_1-handler_1:1' enforced to '1' by Lambda Debug Mode, configurations will continue to report the set value '30'

6. assert only one container for the lambda version is created

7. Send a first invoke for handler_1-handler_1:1

8. Send a second invoke (on another thread) for handler_1-handler_1:1

9. Assert only one container for the lambda version was created

10. Assert that in the console the warning message Concurrent lambda invocations disabled for 'arn:aws:lambda:eu-central-1:000000000000:function:handler_1-handler_1:1' by Lambda Debug Mode for the second invocation, whilst the first one can continue to be debugged as usual by connecting to 19891