-
-
Notifications
You must be signed in to change notification settings - Fork 4.2k
bug: Errors connecting to the LocalStack instance using Lambda hot reloading #12452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide. |
Hi @GowsikChandran ! |
Hi @dfangl I updated the docker compose services: I am invoking the function via postman the function and the hot-reload normally works but it not working if I add boto3 to my code the container restarts 4 times and stops. below is the log of the last restart. container-localstack | 2025-03-31T10:36:24.320 DEBUG --- [et.reactor-4] l.s.l.i.version_manager : Got an invocation for function arn:aws:lambda:us-east-1:000000000000:function:proxy-function:$LATEST with request_id 4cfb498e-628a-41de-85f9-63006ba3adff this is my aws config file [default] this is my aws credentials file [default] |
This new code, do you add it on the module level to your python function, or within the handler itself? If the former, can you try it within the handler, this should make the error more obvious. |
Hi @dfangl I tried creating boto3 client outside and also tried inside the handler but I get the same error. LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT if I add the boto3 code, (i also increased the timeout of LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT and lambda function) if I remove the boto3 code everything is fine |
Hi @GowsikChandran , sorry for the delayed response. |
Hello 👋! It looks like this issue hasn’t been active in longer than two weeks. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one. |
Is there an existing issue for this?
Current Behavior
I'm experiencing an issue with lambda LocalStack setup. Hot-reload is enabled and works fine when I invoke my Lambda function normally (i.e. without using boto3). However, when I add a boto3 client call within my Lambda code, the function container automatically restarts four times and then errors out.
Here's a simplified version of the code in my Lambda function:
client = boto3.client(
"secretsmanager",
endpoint_url=os.getenv("AWS_ENDPOINT_URL")
)
secret = client.get_secret_value(SecretId="postgres-secret")
And my Docker Compose configuration is as follows:
localstack:
image: localstack/localstack:latest
container_name: container-localstack
restart: unless-stopped
ports:
- "4566:4566"
environment:
- SERVICES=iam,lambda,apigateway,cloudformation,secretsmanager,dynamodb
- DEFAULT_REGION=us-east-1
- DEBUG=1
- PERSISTENCE=1
- LAMBDA_KEEPALIVE_MS=0
- LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=120
- LAMBDA_DOCKER_NETWORK=backend_hub-net
volumes:
- localstack_data:/var/lib/localstack
- /var/run/docker.sock:/var/run/docker.sock
privileged: true
networks:
nexhub-net:
networks:
backend_hub-net:
I've increased LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT to 120 seconds and AWS_LAMBDA_FUNCTION_TIMEOUT to 600 seconds, but the issue persists. Without the boto3 client call, the function runs normally without restarting.
Does anyone have experience with this behavior when using hot-reload and boto3 in a Lambda function? Any insights or suggestions to resolve this would be greatly appreciated!
Thanks in advance.
Error trace
container-localstack | 2025-03-26T23:07:29.728 DEBUG --- [et.reactor-1] l.s.l.i.assignment : Starting new environment container-localstack | 2025-03-26T23:07:29.729 DEBUG --- [et.reactor-1] l.s.l.i.docker_runtime_exe : Assigning container name of container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534 to executor 465feaad8c1ec01178dd2c16eeac2534 container-localstack | 2025-03-26T23:07:29.758 DEBUG --- [et.reactor-1] l.u.c.docker_sdk_client : Creating container with attributes: {'self': <localstack.utils.container_utils.docker_sdk_client.SdkDockerClient object at 0x7fa43915cb50>, 'image_name': 'public.ecr.aws/lambda/python:3.12', 'name': 'container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534', 'entrypoint': '/var/rapid/init', 'remove': False, 'interactive': False, 'tty': False, 'detach': False, 'command': None, 'volumes': [VolumeBind(host_dir='D:/vscode-projects/backend/serverless', container_dir='/var/task', read_only=True)], 'ports': <PortMappings: {}>, 'exposed_ports': [], 'env_vars': {'AWS_DEFAULT_REGION': 'us-east-1', 'AWS_REGION': 'us-east-1', 'AWS_LAMBDA_FUNCTION_NAME': 'proxy-function', 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': 512, 'AWS_LAMBDA_FUNCTION_VERSION': '$LATEST', 'AWS_LAMBDA_INITIALIZATION_TYPE': 'on-demand', 'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/proxy-function', 'AWS_LAMBDA_LOG_STREAM_NAME': '2025/03/26/[$LATEST]465feaad8c1ec01178dd2c16eeac2534', 'AWS_ACCESS_KEY_ID': 'LSIAQAAAAAAAJ3TY2G2R', 'AWS_SECRET_ACCESS_KEY': 'aMX4xZnX44cAvcSf9Jnnx8EkNCmNiXdx5D+iSJ6+', 'AWS_SESSION_TOKEN': 'FQoGZXIvYXdzEBYaDfTA7ESZbCSkt4ofL2pRUdVaAG4paibWCovm70dCb/BI7zjkbM5SHfngvfCHMpFMLFQrPZppOytUoJmjyKQ8j/bZCO8SdpJrVRaqzZRD9NkWn6x2Hf5BJXPvCqsOyEP8v1ptYIZpGDKkFK+WsojOzOEKQHwe3AsI396UDsMB+3GNNLZzcm2uGnAxa2cSRw8pts6tWpEtNho3gwIoeVXFGxUc4ftiNeS8oCzHLpn/2DjkHrxGeqHYLK6T07UXOEjZJD6ttiAY/7IuAM0tTkU6VEUCBhJWfFAGXuzNMcxQHKtsjo9L3EWbJlF2Zk2voYIpxpPQmT5v6zsFsE5Qf6c=', 'LAMBDA_TASK_ROOT': '/var/task', 'LAMBDA_RUNTIME_DIR': '/var/runtime', 'AWS_XRAY_CONTEXT_MISSING': 'LOG_ERROR', 'AWS_XRAY_DAEMON_ADDRESS': '127.0.0.1:2000', '_AWS_XRAY_DAEMON_PORT': '2000', '_AWS_XRAY_DAEMON_ADDRESS': '127.0.0.1', 'TZ': ':UTC', 'AWS_LAMBDA_FUNCTION_TIMEOUT': 600, 'LOCALSTACK_HOSTNAME': '172.18.0.2', 'EDGE_PORT': '4566', 'LOCALSTACK_RUNTIME_ID': '465feaad8c1ec01178dd2c16eeac2534', 'LOCALSTACK_RUNTIME_ENDPOINT': 'http://172.18.0.2:4566/_localstack_lambda/465feaad8c1ec01178dd2c16eeac2534', 'LOCALSTACK_FUNCTION_ACCOUNT_ID': '000000000000', 'AWS_ENDPOINT_URL': 'http://172.18.0.2:4566/', '_HANDLER': 'main.lambda_handler', 'AWS_EXECUTION_ENV': 'AWS_Lambda_rapid', 'PYTHONPATH': '/var/task/deps:$PYTHONPATH', 'LOCALSTACK_MAX_PAYLOAD_SIZE': 6291556, 'LOCALSTACK_HOT_RELOADING_PATHS': '/var/task', 'LOCALSTACK_CHMOD_PATHS': '[{"path": "/tmp", "mode": "0700"}]'}, 'user': None, 'cap_add': None, 'cap_drop': None, 'security_opt': None, 'network': 'nexiam-backend_nexhub-net', 'dns': '172.18.0.2', 'additional_flags': '', 'workdir': None, 'privileged': False, 'labels': None, 'platform': 'linux/amd64', 'ulimits': None, 'init': None, 'log_config': None} container-localstack | 2025-03-26T23:07:29.918 DEBUG --- [et.reactor-1] l.u.c.docker_sdk_client : Copying file /usr/lib/localstack/lambda-runtime/v0.1.32-pre/x86_64/. into container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534:/ container-localstack | 2025-03-26T23:07:31.507 DEBUG --- [et.reactor-1] l.u.c.docker_sdk_client : Starting container container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534 container-localstack | 2025-03-26T23:07:32.222 DEBUG --- [et.reactor-1] l.u.c.container_client : Getting ipv4 address for container container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534 in network backend_hub-net. container-localstack | 2025-03-26T23:08:23.080 WARN --- [ Thread-126] l.s.l.i.execution_environm : Execution environment cbe420b4c58cddb555de27f6c30ae7cf for function arn:aws:lambda:us-east-1:000000000000:function:proxy-function:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT. container-localstack | 2025-03-26T23:08:23.160 DEBUG --- [ Thread-126] l.s.l.i.execution_environm : Logs from the execution environment cbe420b4c58cddb555de27f6c30ae7cf after startup timeout: container-localstack | [lambda cbe420b4c58cddb555de27f6c30ae7cf] container-localstack | 2025-03-26T23:08:23.160 DEBUG --- [ Thread-126] l.u.c.docker_sdk_client : Stopping container: container-localstack-lambda-proxy-function-cbe420b4c58cddb555de27f6c30ae7cf container-localstack | 2025-03-26T23:08:24.054 DEBUG --- [ Thread-126] l.u.c.docker_sdk_client : Removing container: container-localstack-lambda-proxy-function-cbe420b4c58cddb555de27f6c30ae7cf container-localstack | 2025-03-26T23:08:27.945 WARN --- [et.reactor-0] l.s.a.n.e.integrations.aws : Unexpected exception during integration invocation: 'Read timeout on endpoint URL: "http://localhost:4566/2015-03-31/functions/arn%3Aaws%3Alambda%3Aus-east-1%3A000000000000%3Afunction%3Aproxy-function/invocations"' container-localstack | 2025-03-26T23:08:27.946 INFO --- [et.reactor-0] l.s.a.n.e.h.gateway_except : Error raised during invocation: INTEGRATION_FAILURE container-localstack | 2025-03-26T23:08:27.948 INFO --- [et.reactor-0] localstack.request.http : GET /local/test => 502 container-localstack | 2025-03-26T23:09:26.086 WARN --- [ Thread-140] l.s.l.i.execution_environm : Execution environment 465feaad8c1ec01178dd2c16eeac2534 for function arn:aws:lambda:us-east-1:000000000000:function:proxy-function:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT. container-localstack | 2025-03-26T23:09:26.125 DEBUG --- [ Thread-140] l.s.l.i.execution_environm : Logs from the execution environment 465feaad8c1ec01178dd2c16eeac2534 after startup timeout: container-localstack | [lambda 465feaad8c1ec01178dd2c16eeac2534] container-localstack | 2025-03-26T23:09:26.125 DEBUG --- [ Thread-140] l.u.c.docker_sdk_client : Stopping container: container-localstack-lambda-proxy-function-465feaad8c1ec01178dd2c16eeac2534
Expected Behavior
No response
How are you starting LocalStack?
With a docker-compose file
Steps To Reproduce
How are you starting localstack (e.g.,
bin/localstack
command, arguments, ordocker-compose.yml
)Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
Environment
Anything else?
No response
The text was updated successfully, but these errors were encountered: