Open
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
When a dependent docker container fails to start/returns unavailable, Localstack will not produce meaningful errors.
For example launching RDS Cluster instance via Terraform:
- Localstack starts mysql docker container
- mysql docker container fails to start (this was due to local setup, incorrect colima config causing initial startup to fail)
- Failure to contact container is logged
- No errors logged in localstack UI
- No meaningful error returned to Terraform
Due to the lack of errors, this was only discovered by running localstack not in daemon mode.
Localstack logs after mysql container fails to start:
2025-05-06T13:15:48.213 WARN --- [-functhread5] l.p.c.s.rds.db_utils : Unable to startup DB instance: Traceback (most recent call last):
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 275, in _raise_for_status
response.raise_for_status()
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.47/containers/87e90b733418a1806a0a29c1573681c3966fbb10b0e3b15010f769a1d0c0d548/exec
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 899, in exec_in_container
result = container.exec_run(
^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/models/containers.py", line 208, in exec_run
resp = self.client.api.exec_create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/exec_api.py", line 78, in exec_create
return self._result(res, True)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 281, in _result
self._raise_for_status(response)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 277, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.47/containers/87e90b733418a1806a0a29c1573681c3966fbb10b0e3b15010f769a1d0c0d548/exec: Conflict ("container 87e90b733418a1806a0a29c1573681c3966fbb10b0e3b15010f769a1d0c0d548 is not running")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/db_utils.py.enc", line 63, in A
try:B and B(status='creating');G=F.start_db_instance(*E,**D);B and B(status=_D);LOG.debug('DB started successfully.');return G
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 62, in start_db_instance
retry(T,sleep=3,retries=30,sleep_before=2)
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 63, in retry
raise raise_error
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/sync.py", line 59, in retry
return function(**kwargs)
^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 61, in T
def T():A='mysqladmin ping -p$MYSQL_ROOT_PASSWORD --protocol tcp';A=[P,'-c',A];assert D.container.exec(command=A)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
user 87262 0.0 0.7 1867179424 138832 ?? S 5:18p.m. 0:00.12 /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/135.0.7049.96/Helpers/Google Chrome Helper (Renderer).app/Contents/MacOS/Go
root 87209 0.0 0.0 410337536 2896 ?? Ss 5:18p.m. 0:00.06 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/com.apple.audio.SandboxHelper.xpc/Contents/MacOS/com.apple.audio.SandboxHelper
root 87208 0.0 0.0 426949376 3888 ?? Ss 5:18p.m. 0:00.07 /System/Library/Frameworks/AudioToolbox.framework/AudioComponentRegistrar -daemon
root 87207 0.0 0.1 426996672 9920 ?? Ss 5:18p.m. 0:00.09 /usr/sbin/systemsoundserverd
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/container.py.enc", line 20, in B
return A(self,*B,**C)
^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/utils/container/docker_container.py.enc", line 22, in exec
def exec(self,command):return DOCKER_CLIENT.exec_in_container(container_name_or_id=self.id,command=command)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 940, in exec_in_container
raise ContainerException() from e
localstack.utils.container_utils.container_client.ContainerException
stdout: 'None' stderr: 'None'
Terraform error:
│ Error: waiting for RDS Cluster (test-eu-west-1-tfmod-aurora) create: unexpected state 'error', wanted target 'available'. last error: %!s(<nil>)
│
│ with module.aurora.aws_rds_cluster.cluster,
│ on ../../../main.tf line 16, in resource "aws_rds_cluster" "cluster":
│ 16: resource "aws_rds_cluster" "cluster" {
│
╵
Expected Behavior
Meaningful logging or errors returned e.g. Unable to reach container xyz
How are you starting LocalStack?
With the localstack script
Steps To Reproduce
How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)
`localstack start`
Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
`tflocal apply --auto-approve`
Environment
- OS: Sequoia 15.4
- LocalStack:
LocalStack version: 4.3.1.dev101
LocalStack Docker image sha: `sha256:f4677d60c015c564d6c89f9bcff8d6258dafc97a7033fa709a9eb623f45afce0`
LocalStack build date: 2025-05-06
LocalStack build git hash: 90a008fe9Anything else?
No response