Skip to content

bug: Errors connecting to the LocalStack instance using Lambda hot reloading #12452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
1 task done
GowsikChandran opened this issue Mar 30, 2025 · 7 comments
Closed
1 task done
Labels
area: docker Use Docker with LocalStack aws:lambda AWS Lambda status: resolved/stale Closed due to staleness status: response required Waiting for a response from the reporter type: bug Bug report

Comments

@GowsikChandran
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I'm experiencing an issue with lambda LocalStack setup. Hot-reload is enabled and works fine when I invoke my Lambda function normally (i.e. without using boto3). However, when I add a boto3 client call within my Lambda code, the function container automatically restarts four times and then errors out.
Here's a simplified version of the code in my Lambda function:
client = boto3.client(
"secretsmanager",
endpoint_url=os.getenv("AWS_ENDPOINT_URL")
)
secret = client.get_secret_value(SecretId="postgres-secret")
And my Docker Compose configuration is as follows:
localstack:
image: localstack/localstack:latest
container_name: container-localstack
restart: unless-stopped
ports:
- "4566:4566"
environment:
- SERVICES=iam,lambda,apigateway,cloudformation,secretsmanager,dynamodb
- DEFAULT_REGION=us-east-1
- DEBUG=1
- PERSISTENCE=1
- LAMBDA_KEEPALIVE_MS=0
- LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=120
- LAMBDA_DOCKER_NETWORK=backend_hub-net
volumes:
- localstack_data:/var/lib/localstack
- /var/run/docker.sock:/var/run/docker.sock
privileged: true
networks:
nexhub-net:
networks:
backend_hub-net:
I've increased LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT to 120 seconds and AWS_LAMBDA_FUNCTION_TIMEOUT to 600 seconds, but the issue persists. Without the boto3 client call, the function runs normally without restarting.
Does anyone have experience with this behavior when using hot-reload and boto3 in a Lambda function? Any insights or suggestions to resolve this would be greatly appreciated!
Thanks in advance.
Error trace
container-localstack | 2025-03-26T23:07:29.728 DEBUG --- [et.reactor-1] l.s.l.i.assignment : Starting new environment container-localstack | 2025-03-26T23:07:29.729 DEBUG --- [et.reactor-1] l.s.l.i.docker_runtime_exe : Assigning container name of container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534 to executor 465feaad8c1ec01178dd2c16eeac2534 container-localstack | 2025-03-26T23:07:29.758 DEBUG --- [et.reactor-1] l.u.c.docker_sdk_client : Creating container with attributes: {'self': <localstack.utils.container_utils.docker_sdk_client.SdkDockerClient object at 0x7fa43915cb50>, 'image_name': 'public.ecr.aws/lambda/python:3.12', 'name': 'container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534', 'entrypoint': '/var/rapid/init', 'remove': False, 'interactive': False, 'tty': False, 'detach': False, 'command': None, 'volumes': [VolumeBind(host_dir='D:/vscode-projects/backend/serverless', container_dir='/var/task', read_only=True)], 'ports': <PortMappings: {}>, 'exposed_ports': [], 'env_vars': {'AWS_DEFAULT_REGION': 'us-east-1', 'AWS_REGION': 'us-east-1', 'AWS_LAMBDA_FUNCTION_NAME': 'proxy-function', 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': 512, 'AWS_LAMBDA_FUNCTION_VERSION': '$LATEST', 'AWS_LAMBDA_INITIALIZATION_TYPE': 'on-demand', 'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/proxy-function', 'AWS_LAMBDA_LOG_STREAM_NAME': '2025/03/26/[$LATEST]465feaad8c1ec01178dd2c16eeac2534', 'AWS_ACCESS_KEY_ID': 'LSIAQAAAAAAAJ3TY2G2R', 'AWS_SECRET_ACCESS_KEY': 'aMX4xZnX44cAvcSf9Jnnx8EkNCmNiXdx5D+iSJ6+', 'AWS_SESSION_TOKEN': 'FQoGZXIvYXdzEBYaDfTA7ESZbCSkt4ofL2pRUdVaAG4paibWCovm70dCb/BI7zjkbM5SHfngvfCHMpFMLFQrPZppOytUoJmjyKQ8j/bZCO8SdpJrVRaqzZRD9NkWn6x2Hf5BJXPvCqsOyEP8v1ptYIZpGDKkFK+WsojOzOEKQHwe3AsI396UDsMB+3GNNLZzcm2uGnAxa2cSRw8pts6tWpEtNho3gwIoeVXFGxUc4ftiNeS8oCzHLpn/2DjkHrxGeqHYLK6T07UXOEjZJD6ttiAY/7IuAM0tTkU6VEUCBhJWfFAGXuzNMcxQHKtsjo9L3EWbJlF2Zk2voYIpxpPQmT5v6zsFsE5Qf6c=', 'LAMBDA_TASK_ROOT': '/var/task', 'LAMBDA_RUNTIME_DIR': '/var/runtime', 'AWS_XRAY_CONTEXT_MISSING': 'LOG_ERROR', 'AWS_XRAY_DAEMON_ADDRESS': '127.0.0.1:2000', '_AWS_XRAY_DAEMON_PORT': '2000', '_AWS_XRAY_DAEMON_ADDRESS': '127.0.0.1', 'TZ': ':UTC', 'AWS_LAMBDA_FUNCTION_TIMEOUT': 600, 'LOCALSTACK_HOSTNAME': '172.18.0.2', 'EDGE_PORT': '4566', 'LOCALSTACK_RUNTIME_ID': '465feaad8c1ec01178dd2c16eeac2534', 'LOCALSTACK_RUNTIME_ENDPOINT': 'http://172.18.0.2:4566/_localstack_lambda/465feaad8c1ec01178dd2c16eeac2534', 'LOCALSTACK_FUNCTION_ACCOUNT_ID': '000000000000', 'AWS_ENDPOINT_URL': 'http://172.18.0.2:4566/', '_HANDLER': 'main.lambda_handler', 'AWS_EXECUTION_ENV': 'AWS_Lambda_rapid', 'PYTHONPATH': '/var/task/deps:$PYTHONPATH', 'LOCALSTACK_MAX_PAYLOAD_SIZE': 6291556, 'LOCALSTACK_HOT_RELOADING_PATHS': '/var/task', 'LOCALSTACK_CHMOD_PATHS': '[{"path": "/tmp", "mode": "0700"}]'}, 'user': None, 'cap_add': None, 'cap_drop': None, 'security_opt': None, 'network': 'nexiam-backend_nexhub-net', 'dns': '172.18.0.2', 'additional_flags': '', 'workdir': None, 'privileged': False, 'labels': None, 'platform': 'linux/amd64', 'ulimits': None, 'init': None, 'log_config': None} container-localstack | 2025-03-26T23:07:29.918 DEBUG --- [et.reactor-1] l.u.c.docker_sdk_client : Copying file /usr/lib/localstack/lambda-runtime/v0.1.32-pre/x86_64/. into container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534:/ container-localstack | 2025-03-26T23:07:31.507 DEBUG --- [et.reactor-1] l.u.c.docker_sdk_client : Starting container container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534 container-localstack | 2025-03-26T23:07:32.222 DEBUG --- [et.reactor-1] l.u.c.container_client : Getting ipv4 address for container container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534 in network backend_hub-net. container-localstack | 2025-03-26T23:08:23.080 WARN --- [ Thread-126] l.s.l.i.execution_environm : Execution environment cbe420b4c58cddb555de27f6c30ae7cf for function arn:aws:lambda:us-east-1:000000000000:function:proxy-function:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT. container-localstack | 2025-03-26T23:08:23.160 DEBUG --- [ Thread-126] l.s.l.i.execution_environm : Logs from the execution environment cbe420b4c58cddb555de27f6c30ae7cf after startup timeout: container-localstack | [lambda cbe420b4c58cddb555de27f6c30ae7cf] container-localstack | 2025-03-26T23:08:23.160 DEBUG --- [ Thread-126] l.u.c.docker_sdk_client : Stopping container: container-localstack-lambda-proxy-function-cbe420b4c58cddb555de27f6c30ae7cf container-localstack | 2025-03-26T23:08:24.054 DEBUG --- [ Thread-126] l.u.c.docker_sdk_client : Removing container: container-localstack-lambda-proxy-function-cbe420b4c58cddb555de27f6c30ae7cf container-localstack | 2025-03-26T23:08:27.945 WARN --- [et.reactor-0] l.s.a.n.e.integrations.aws : Unexpected exception during integration invocation: 'Read timeout on endpoint URL: "http://localhost:4566/2015-03-31/functions/arn%3Aaws%3Alambda%3Aus-east-1%3A000000000000%3Afunction%3Aproxy-function/invocations"' container-localstack | 2025-03-26T23:08:27.946 INFO --- [et.reactor-0] l.s.a.n.e.h.gateway_except : Error raised during invocation: INTEGRATION_FAILURE container-localstack | 2025-03-26T23:08:27.948 INFO --- [et.reactor-0] localstack.request.http : GET /local/test => 502 container-localstack | 2025-03-26T23:09:26.086 WARN --- [ Thread-140] l.s.l.i.execution_environm : Execution environment 465feaad8c1ec01178dd2c16eeac2534 for function arn:aws:lambda:us-east-1:000000000000:function:proxy-function:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT. container-localstack | 2025-03-26T23:09:26.125 DEBUG --- [ Thread-140] l.s.l.i.execution_environm : Logs from the execution environment 465feaad8c1ec01178dd2c16eeac2534 after startup timeout: container-localstack | [lambda 465feaad8c1ec01178dd2c16eeac2534] container-localstack | 2025-03-26T23:09:26.125 DEBUG --- [ Thread-140] l.u.c.docker_sdk_client : Stopping container: container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534

Expected Behavior

No response

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

docker run localstack/localstack

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

awslocal s3 mb s3://mybucket

Environment

- OS:
- LocalStack:
  LocalStack version:
  LocalStack Docker image sha:
  LocalStack build date:
  LocalStack build git hash:

Anything else?

No response

@GowsikChandran GowsikChandran added status: triage needed Requires evaluation by maintainers type: bug Bug report labels Mar 30, 2025
@localstack-bot
Copy link
Collaborator

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

@dfangl
Copy link
Member

dfangl commented Mar 31, 2025

Hi @GowsikChandran !
Please remove the - LAMBDA_DOCKER_NETWORK=backend_hub-net section of your docker compose - LocalStack will automatically detect the right network, and the name has to include the docker compose project name itself as well - docker compose will usually prefix the network name with it.
In general, please take a look at our sample docker compose: https://github.com/localstack/localstack/blob/master/docker-compose.yml and try this file to see if this resolves your issues, before adding back some of the modifications you made.

@dfangl dfangl added area: docker Use Docker with LocalStack status: response required Waiting for a response from the reporter aws:lambda AWS Lambda and removed status: triage needed Requires evaluation by maintainers labels Mar 31, 2025
@dfangl dfangl changed the title bug: <title> bug: Errors connecting to the LocalStack instance using Lambda hot reloading Mar 31, 2025
@GowsikChandran
Copy link
Author

GowsikChandran commented Mar 31, 2025

Hi @dfangl
thank you for the response :)

I updated the docker compose

services:
localstack:
image: localstack/localstack:stable
container_name: container-localstack
restart: unless-stopped
ports:
- "127.0.0.1:4566:4566" # LocalStack Gateway
- "127.0.0.1:4510-4559:4510-4559" # external services port range
environment:
- SERVICES=iam,lambda,apigateway,cloudformation,secretsmanager,dynamodb
- DEFAULT_REGION=us-east-1
- DEBUG=1
- PERSISTENCE=1
- LAMBDA_KEEPALIVE_MS=0
- LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=120
volumes:
- localstack_data:/var/lib/localstack
- /var/run/docker.sock:/var/run/docker.sock
networks:
nexhub-net:

I am invoking the function via postman
http://kxfdkcnexl.execute-api.localhost.localstack.cloud:4566/local/test

the function and the hot-reload normally works but it not working if I add boto3 to my code
client = boto3.client(
"lambda",
endpoint_url=os.getenv("AWS_ENDPOINT_URL"),
aws_access_key_id="test",
aws_secret_access_key="test",
region_name=os.getenv("AWS_REGION")
)
result = client.list_functions()

the container restarts 4 times and stops. below is the log of the last restart.

container-localstack | 2025-03-31T10:36:24.320 DEBUG --- [et.reactor-4] l.s.l.i.version_manager : Got an invocation for function arn:aws:lambda:us-east-1:000000000000:function:proxy-function:$LATEST with request_id 4cfb498e-628a-41de-85f9-63006ba3adff
container-localstack | 2025-03-31T10:36:24.321 DEBUG --- [et.reactor-4] l.s.l.i.assignment : Starting new environment
container-localstack | 2025-03-31T10:36:24.327 DEBUG --- [et.reactor-4] l.s.l.i.docker_runtime_exe : Assigning container name of container-localstack-lambda-proxy-function-3a6023de14a29d118adedfd6ddc11532 to executor 3a6023de14a29d118adedfd6ddc11532
container-localstack | 2025-03-31T10:36:24.344 DEBUG --- [et.reactor-4] l.u.c.docker_sdk_client : Creating container with attributes: {'self': <localstack.utils.container_utils.docker_sdk_client.SdkDockerClient object at 0x7fe23efed810>, 'image_name': 'public.ecr.aws/lambda/python:3.12', 'name': 'container-localstack-lambda-proxy-function-3a6023de14a29d118adedfd6ddc11532', 'entrypoint': '/var/rapid/init', 'remove': False, 'interactive': False, 'tty': False, 'detach': False, 'command': None, 'volumes': [VolumeBind(host_dir='D:/vscode-projects/nexiam/nexiam-backend/serverless/nexhub', container_dir='/var/task', read_only=True)], 'ports': <PortMappings: {}>, 'exposed_ports': [], 'env_vars': {'AWS_DEFAULT_REGION': 'us-east-1', 'AWS_REGION': 'us-east-1', 'AWS_LAMBDA_FUNCTION_NAME': 'proxy-function', 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': 512, 'AWS_LAMBDA_FUNCTION_VERSION': '$LATEST', 'AWS_LAMBDA_INITIALIZATION_TYPE': 'on-demand', 'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/proxy-function', 'AWS_LAMBDA_LOG_STREAM_NAME': '2025/03/31/[$LATEST]3a6023de14a29d118adedfd6ddc11532', 'AWS_ACCESS_KEY_ID': 'LSIAQAAAAAAAOUTP6XD6', 'AWS_SECRET_ACCESS_KEY': 'UIHtXxkgUkkncxlNrqXksZJJ+nSxJYgKVPcTkfwy', 'AWS_SESSION_TOKEN': 'FQoGZXIvYXdzEBYaDCMs0L2RzVTwrh+jxz6ElQbwrOI/B0u7nbNNxM4M6MWmXK14ANfZTOh7vuwEQ78ctgneZbrbBLYuW0mym2qyAf17cBeFV27kbBgjbv+Aj9/MZHwHMeZZ/xwp/jjj0iTjfW3AYqFCnscFJ0abwFPv0gqKQqWT5Q+SVJTw9muTVRxWcIzJAU98Kxs/IlwIYLddcKAzHPuu+U9cdhXRPxOcI0fcT+TPNXGeyXHBSw6h0qNOkqoKtdASu1c7ltfOTtyoBWkL7bBRuQe8IKQNr5oe/cqw1zs3/ymEjHng07dDrLwn8O9XMrliO0wQGjvl7pr6aFrF+5YiXvQhgY2U588=', 'LAMBDA_TASK_ROOT': '/var/task', 'LAMBDA_RUNTIME_DIR': '/var/runtime', 'AWS_XRAY_CONTEXT_MISSING': 'LOG_ERROR', 'AWS_XRAY_DAEMON_ADDRESS': '127.0.0.1:2000', '_AWS_XRAY_DAEMON_PORT': '2000', '_AWS_XRAY_DAEMON_ADDRESS': '127.0.0.1', 'TZ': ':UTC', 'AWS_LAMBDA_FUNCTION_TIMEOUT': 600, 'LOCALSTACK_HOSTNAME': '172.18.0.3', 'EDGE_PORT': '4566', 'LOCALSTACK_RUNTIME_ID': '3a6023de14a29d118adedfd6ddc11532', 'LOCALSTACK_RUNTIME_ENDPOINT': 'http://172.18.0.3:4566/_localstack_lambda/3a6023de14a29d118adedfd6ddc11532', 'LOCALSTACK_FUNCTION_ACCOUNT_ID': '000000000000', 'AWS_ENDPOINT_URL': 'http://172.18.0.3:4566', '_HANDLER': 'main.lambda_handler', 'AWS_EXECUTION_ENV': 'AWS_Lambda_rapid', 'PYTHONPATH': '/var/task/deps:$PYTHONPATH', 'LOCALSTACK_MAX_PAYLOAD_SIZE': 6291556, 'LOCALSTACK_HOT_RELOADING_PATHS': '/var/task', 'LOCALSTACK_CHMOD_PATHS': '[{"path": "/tmp", "mode": "0700"}]'}, 'user': None, 'cap_add': None, 'cap_drop': None, 'security_opt': None, 'network': 'nexiam-backend_nexhub-net', 'dns': '172.18.0.3', 'additional_flags': '', 'workdir': None, 'privileged': False, 'labels': None, 'platform': 'linux/amd64', 'ulimits': None, 'init': None, 'log_config': None}
container-localstack | 2025-03-31T10:36:24.509 DEBUG --- [et.reactor-4] l.u.c.docker_sdk_client : Copying file /usr/lib/localstack/lambda-runtime/v0.1.32-pre/x86_64/. into container-localstack-lambda-proxy-function-3a6023de14a29d118adedfd6ddc11532:/
container-localstack | 2025-03-31T10:36:27.026 DEBUG --- [et.reactor-4] l.u.c.docker_sdk_client : Starting container container-localstack-lambda-proxy-function-3a6023de14a29d118adedfd6ddc11532
container-localstack | 2025-03-31T10:36:29.352 DEBUG --- [et.reactor-4] l.u.c.container_client : Getting ipv4 address for container container-localstack-lambda-proxy-function-3a6023de14a29d118adedfd6ddc11532 in network nexiam-backend_nexhub-net.
container-localstack | 2025-03-31T10:37:18.954 WARN --- [ Thread-85] l.s.l.i.execution_environm : Execution environment 92c9eb604159c0c4cfdfb660a25a683c for function arn:aws:lambda:us-east-1:000000000000:function:proxy-function:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT.
container-localstack | 2025-03-31T10:37:19.029 DEBUG --- [ Thread-85] l.s.l.i.execution_environm : Logs from the execution environment 92c9eb604159c0c4cfdfb660a25a683c after startup timeout:
container-localstack | [lambda 92c9eb604159c0c4cfdfb660a25a683c]
container-localstack | 2025-03-31T10:37:19.029 DEBUG --- [ Thread-85] l.u.c.docker_sdk_client : Stopping container: container-localstack-lambda-proxy-function-92c9eb604159c0c4cfdfb660a25a683c
container-localstack | 2025-03-31T10:37:20.432 DEBUG --- [ Thread-85] l.u.c.docker_sdk_client : Removing container: container-localstack-lambda-proxy-function-92c9eb604159c0c4cfdfb660a25a683c
container-localstack | 2025-03-31T10:37:22.087 WARN --- [et.reactor-1] l.s.a.n.e.integrations.aws : Unexpected exception during integration invocation: 'Read timeout on endpoint URL: "http://localhost:4566/2015-03-31/functions/arn%3Aaws%3Alambda%3Aus-east-1%3A000000000000%3Afunction%3Aproxy-function/invocations"'
container-localstack | 2025-03-31T10:37:22.088 INFO --- [et.reactor-1] l.s.a.n.e.h.gateway_except : Error raised during invocation: INTEGRATION_FAILURE
container-localstack | 2025-03-31T10:37:22.091 INFO --- [et.reactor-1] localstack.request.http : GET /local/test => 502

this is my aws config file

[default]
region=us-east-1
output=json
endpoint_url = http://localhost.localstack.cloud:4566

this is my aws credentials file

[default]
aws_access_key_id=test
aws_secret_access_key=test

@localstack-bot localstack-bot removed the status: response required Waiting for a response from the reporter label Mar 31, 2025
@dfangl
Copy link
Member

dfangl commented Mar 31, 2025

This new code, do you add it on the module level to your python function, or within the handler itself? If the former, can you try it within the handler, this should make the error more obvious.

@Anze1508 Anze1508 added the status: in progress Currently being worked on label Mar 31, 2025
@GowsikChandran
Copy link
Author

GowsikChandran commented Mar 31, 2025

Hi @dfangl

I tried creating boto3 client outside and also tried inside the handler but I get the same error.

LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT if I add the boto3 code, (i also increased the timeout of LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT and lambda function)

if I remove the boto3 code everything is fine

@dfangl
Copy link
Member

dfangl commented Apr 15, 2025

Hi @GowsikChandran , sorry for the delayed response.
Does the same thing happen without hot reloading, but including the boto3 code? Could you share a (stripped, but confirmed not working) complete code example of the lambda deployed?
Please also share the LocalStack version printed at startup!

@joe4dev joe4dev added status: response required Waiting for a response from the reporter and removed status: in progress Currently being worked on labels May 19, 2025
@localstack-bot
Copy link
Collaborator

Hello 👋! It looks like this issue hasn’t been active in longer than two weeks. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

@localstack-bot localstack-bot added the status: stale To be closed soon due to staleness label May 19, 2025
@localstack-bot localstack-bot added status: resolved/stale Closed due to staleness and removed status: stale To be closed soon due to staleness labels May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: docker Use Docker with LocalStack aws:lambda AWS Lambda status: resolved/stale Closed due to staleness status: response required Waiting for a response from the reporter type: bug Bug report
Projects
None yet
Development

No branches or pull requests

5 participants