Localstack SSE-C Issue

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Current Behavior

As per https://github.com/localstack/localstack/issues/11428#issuecomment-2589753432, decided to raise issue. The `LS_LOG=trace` is useful as it gives actual AWS information too.

One of the apps, I work on uses presigned urls, SSE-C to upload/download objects to/from S3. It's been around >5yrs and currently works in Prod with AWS.

We were using the stable version of localstack in docker for our docker-compose and had no issues until 3.7.0 came out with it's new SSE-C support.

Using `stable`, our existing code (sequenced shortened/obfuscated a bit):

```
  localstack:
    image: localstack/localstack:stable
    hostname: localhost.localstack.cloud
    environment:
      - AWS_DEFAULT_REGION=us-east-1
      - SERVICES=sqs,sns,s3,kms
      - DEBUG=1
      - DISABLE_CORS_CHECKS=1
      - LS_LOG=trace
    ports:
      - "4566:4566"
      - "4510-4559:4510-4559"  # external services port range
    volumes:
      - 'localstack:/var/lib/localstack"'
      - '/var/run/docker.sock:/var/run/docker.sock'
      - './docker/aws/init-sns-sqs.sh:/etc/localstack/init/ready.d/init-sns-sqs.sh'
      - './docker/aws/kms.sh:/etc/localstack/init/ready.d/kms.sh'
      - './docker/aws/s3.sh:/etc/localstack/init/ready.d/s3.sh'

  sast-code-artifacts:
    image: ******/sast-code-artifacts:latest
...
    environment:
...
      JAVA_TOOL_OPTIONS: -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8085,server=y,suspend=n -Djava.security.egd=file:/dev/./urandom
      AWS_S3_ENDPOINT_URL: https://localhost.localstack.cloud:4566
      AWS_S3_BUCKET: sast-code-artifacts
      AWS_ACCESS_KEY_ID: test
      AWS_SECRET_ACCESS_KEY: local-development
      AWS_KMS_ENDPOINT_URL: http://localhost.localstack.cloud:4566
      AWS_KMS_MASTER_KEY_ID: alias/sast-code-artifacts-key
```

From the microservice log, it's able to upload the artifact correctly but when it comes to downloading (including a check for existence) it falls over:

```
2025-01-14 13:43:49 2025-01-14 13:43:49.798  INFO 1 --- [nio-9991-exec-3] c.c.s.a.interceptor.LoggingInterceptor   : Received HTTP request [httpMethod=POST, requestUri=/organizations/7e4dfc6d-88f9-4daf-b51a-77d51cbd1409/projects/de807db3-f19e-4076-b49a-3d8ebfc1ba50/code-artifacts/43e0b006-363f-440d-9554-7ac3a300a065/content, requestHandler=downloadCodeArtifactContent] [traceId=67866a153af577d991a21beafd7e6a56, spanId=91a21beafd7e6a56, organizationId=7e4dfc6d-88f9-4daf-b51a-77d51cbd1409, projectId=de807db3-f19e-4076-b49a-3d8ebfc1ba50, codeArtifactId=43e0b006-363f-440d-9554-7ac3a300a065] 
2025-01-14 13:43:50 2025-01-14 13:43:50.011  WARN 1 --- [nio-9991-exec-3] c.c.storage.s3.AwsS3StorageService       : Error checking if file exists in S3 [traceId=67866a153af577d991a21beafd7e6a56, spanId=91a21beafd7e6a56, organizationId=7e4dfc6d-88f9-4daf-b51a-77d51cbd1409, projectId=de807db3-f19e-4076-b49a-3d8ebfc1ba50, codeArtifactId=43e0b006-363f-440d-9554-7ac3a300a065] 
2025-01-14 13:43:50 
2025-01-14 13:43:50 software.amazon.awssdk.services.s3.model.S3Exception: BAD REQUEST (Service: S3, Status Code: 400, Request ID: 9dd17bad-f9ba-4eb7-97e4-276f628c6791, Extended Request ID: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=)
2025-01-14 13:43:50     at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104)
2025-01-14 13:43:50     at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58)
2025-01-14 13:43:50     at software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor.modifyException(ExceptionTranslationInterceptor.java:88)
2025-01-14 13:43:50     at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:181)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:39)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
2025-01-14 13:43:50     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
2025-01-14 13:43:50     at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
2025-01-14 13:43:50     at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
2025-01-14 13:43:50     at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:6661)
2025-01-14 13:43:50     at ********.storage.s3.AwsS3StorageService.fileExists(AwsS3StorageService.java:83)
2025-01-14 13:43:50     at ********.storage.StorageService.fileExists(StorageService.java:48)
2025-01-14 13:43:50     at ********Service.fileExists(CodeArtifactService.java:241)
```

the LocalStack trace has: 

```
2025-01-14 13:43:49 2025-01-14T13:43:49.936 DEBUG --- [et.reactor-1] rolo.gateway.wsgi          : HEAD localhost.localstack.cloud:4566/sast-code-artifacts/organizations/7e4dfc6d-88f9-4daf-b51a-77d51cbd1409/projects/de807db3-f19e-4076-b49a-3d8ebfc1ba50/code-artifacts/43e0b006-363f-440d-9554-7ac3a300a065/webgoat-server-8.2.3-SNAPSHOT.jar
2025-01-14 13:43:49 2025-01-14T13:43:49.941 DEBUG --- [et.reactor-1] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml
2025-01-14 13:43:49 2025-01-14T13:43:49.948  INFO --- [et.reactor-1] localstack.request.aws     : AWS s3.HeadObject => 400 (InvalidRequest); 000000000000/us-east-1; HeadObjectRequest({'Bucket': 'sast-code-artifacts', 'IfMatch': None, 'IfModifiedSince': None, 'IfNoneMatch': None, 'IfUnmodifiedSince': None, 'Key': 'organizations/7e4dfc6d-88f9-4daf-b51a-77d51cbd1409/projects/de807db3-f19e-4076-b49a-3d8ebfc1ba50/code-artifacts/43e0b006-363f-440d-9554-7ac3a300a065/webgoat-server-8.2.3-SNAPSHOT.jar', 'Range': None, 'ResponseCacheControl': None, 'ResponseContentDisposition': None, 'ResponseContentEncoding': None, 'ResponseContentLanguage': None, 'ResponseContentType': None, 'ResponseExpires': None, 'VersionId': None, 'SSECustomerAlgorithm': 'AES256', 'SSECustomerKey': '5O9Ux3oD/vUFNdxo5ltQeF6u+x2ZOfwhyYkg+auzIiU=', 'SSECustomerKeyMD5': None, 'RequestPayer': None, 'PartNumber': None, 'ExpectedBucketOwner': None, 'ChecksumMode': None}, headers={'Host': 'localhost.localstack.cloud:4566', 'amz-sdk-invocation-id': 'e05d19a4-9179-9b7f-0f92-2ec15b3d1c4a', 'amz-sdk-request': 'attempt=1; max=4', 'Authorization': 'AWS4-HMAC-SHA256 Credential=cratus/20250114/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption-customer-algorithm;x-amz-server-side-encryption-customer-key, Signature=4106302bcb5423546b847844444ee6f6495c68f2e4b2e87c15e2058bbe434a1d', 'User-Agent': 'aws-sdk-java/2.29.21 md/io#sync md/http#Apache ua/2.1 os/Linux#6.10.14-linuxkit lang/java#21.0.5 md/OpenJDK_64-Bit_Server_VM#21.0.5+11-LTS md/vendor#Eclipse_Adoptium cfg/auth-source#env m/D,N,N', 'x-amz-content-sha256': 'UNSIGNED-PAYLOAD', 'X-Amz-Date': '20250114T134349Z', 'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': '5O9Ux3oD/vUFNdxo5ltQeF6u+x2ZOfwhyYkg+auzIiU=', 'Connection': 'Keep-Alive', 'x-moto-account-id': '000000000000'}); InvalidRequest(The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object., headers={'Content-Type': 'application/xml', 'Content-Length': '284', 'x-amz-request-id': '9dd17bad-f9ba-4eb7-97e4-276f628c6791', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234='})
```

including the helpful `The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object` error

### Expected Behavior

With 3.6.0 ie prior to the added LS SSE-C, the HEAD request works:

```
2025-01-14 14:10:07 2025-01-14T14:10:07.159 DEBUG --- [et.reactor-0] rolo.gateway.wsgi          : HEAD localhost.localstack.cloud:4566/sast-code-artifacts/organizations/7e4dfc6d-88f9-4daf-b51a-77d51cbd1409/projects/de807db3-f19e-4076-b49a-3d8ebfc1ba50/code-artifacts/2775547f-936a-40af-aa56-5ebd1b4e0a29/webgoat-server-8.2.3-SNAPSHOT.jar
2025-01-14 14:10:07 2025-01-14T14:10:07.167 DEBUG --- [et.reactor-0] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml
2025-01-14 14:10:07 2025-01-14T14:10:07.175  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.HeadObject => 200; 000000000000/us-east-1; HeadObjectRequest({'Bucket': 'sast-code-artifacts', 'IfMatch': None, 'IfModifiedSince': None, 'IfNoneMatch': None, 'IfUnmodifiedSince': None, 'Key': 'organizations/7e4dfc6d-88f9-4daf-b51a-77d51cbd1409/projects/de807db3-f19e-4076-b49a-3d8ebfc1ba50/code-artifacts/2775547f-936a-40af-aa56-5ebd1b4e0a29/webgoat-server-8.2.3-SNAPSHOT.jar', 'Range': None, 'ResponseCacheControl': None, 'ResponseContentDisposition': None, 'ResponseContentEncoding': None, 'ResponseContentLanguage': None, 'ResponseContentType': None, 'ResponseExpires': None, 'VersionId': None, 'SSECustomerAlgorithm': 'AES256', 'SSECustomerKey': '5ztim7NU4CXyMj6rDIjgtKdQiuY8+JsbdPchzLO+wGs=', 'SSECustomerKeyMD5': None, 'RequestPayer': None, 'PartNumber': None, 'ExpectedBucketOwner': None, 'ChecksumMode': None}, headers={'Host': 'localhost.localstack.cloud:4566', 'amz-sdk-invocation-id': 'bf2b063c-a802-c883-a820-c6aa92bc181c', 'amz-sdk-request': 'attempt=1; max=4', 'Authorization': 'AWS4-HMAC-SHA256 Credential=cratus/20250114/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption-customer-algorithm;x-amz-server-side-encryption-customer-key, Signature=4e154c01c5d6931a9206e708d4fb42d93b9cd4448b945c24b38a4789cbc136f8', 'User-Agent': 'aws-sdk-java/2.29.21 md/io#sync md/http#Apache ua/2.1 os/Linux#6.10.14-linuxkit lang/java#21.0.5 md/OpenJDK_64-Bit_Server_VM#21.0.5+11-LTS md/vendor#Eclipse_Adoptium cfg/auth-source#env m/D,N,N', 'x-amz-content-sha256': 'UNSIGNED-PAYLOAD', 'X-Amz-Date': '20250114T141007Z', 'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': '5ztim7NU4CXyMj6rDIjgtKdQiuY8+JsbdPchzLO+wGs=', 'Connection': 'Keep-Alive', 'x-moto-account-id': '000000000000'}); HeadObjectOutput({'AcceptRanges': 'bytes', 'LastModified': 'Tue, 14 Jan 2025 14:09:40 GMT', 'ContentLength': '96639625', 'ETag': '"5bbe9d40c7360981e6c5ce571d0b3923-19"', 'ContentType': 'application/java-archive', 'ServerSideEncryption': <ServerSideEncryption.AES256: 'AES256'>}, headers={'Content-Type': 'application/java-archive', 'accept-ranges': 'bytes', 'Last-Modified': 'Tue, 14 Jan 2025 14:09:40 GMT', 'Content-Length': '96639625', 'ETag': '"5bbe9d40c7360981e6c5ce571d0b3923-19"', 'x-amz-server-side-encryption': <ServerSideEncryption.AES256: 'AES256'>, 'x-amz-request-id': '1cbae7c0-ad2b-4c84-a1e4-af4e36d39d23', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234='})
```

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

    docker-compose

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

    awslocal s3 mb s3://mybucket


### Environment

```markdown
- OS: Docker M1 Macbook
- LocalStack:
  LocalStack version: 3.6.0/3.7.0/stable

LocalStack version: 4.0.3
LocalStack build date: 2024-11-29
LocalStack build git hash: aa795ed1c
```

### Anything else?

_No response_

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Localstack SSE-C Issue #12136

Is there an existing issue for this?

Current Behavior

Expected Behavior

How are you starting LocalStack?

Steps To Reproduce

How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Environment

Anything else?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Localstack SSE-C Issue #12136

Description

Is there an existing issue for this?

Current Behavior

Expected Behavior

How are you starting LocalStack?

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Environment

Anything else?

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)