Skip to content

APIGW: binary support for non-proxy integration #12215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Feb 7, 2025

Conversation

bentsku
Copy link
Contributor

@bentsku bentsku commented Jan 31, 2025

Motivation

Follow up from #12199, is reviewable but will come out of draft once the base is merged to avoid the rebase review madness.

This PR implements the full Binary media support for REST APIs. The AWS documentation will do a much better job explaining what this means exactly, but in big line, it allows you to define some behavior in how your payload in being pass through to your integration: it can be pass as is, converted to a safe UTF-8 string, or base64 decoded or encoded.

There are precise tables on the behavior, but I tend to not trust the documentation so much, so I took it upon myself to write AWS validated tests, which was to be honest, quite a pain again 😅

The behavior in itself is not too complicated, it is mapping some table to if conditions (which might be simplified, if you see it during the review, please say so, I didn't optimize the conditions).

Some changes in behavior is how the responseTemplates VTL rendering actually needs the value to be a string, and so if it comes out of the "converter", it needs to be of the right type. We might want to add better log statements to help the user understand why it would fail for them.

I've also sneaked some TODO removal, and a small change in update_integration_response, this API op really needs to be validated and fixed.

Resources:

Changes

  • validate and remove snapshot skips for S3 AWS integration tests, now passing 🥳
  • add a good amount of binary handling tests in the S3 integration tests, as it's very easy to verify binary behavior with S3, and manipulate returned Content-Type headers
  • add unit tests for the Convert functions
  • implement the logic for converting the request and response bodies
  • modify the method (ported back to base PR) to check if a request if of "binary" type (AWS terms), meaning it matches the binaryMediaTypes
  • update UpdateIntegrationResponse to support our test case
  • remove some todos and update some of them

@bentsku bentsku added aws:apigateway Amazon API Gateway semver: patch Non-breaking changes which can be included in patch releases labels Jan 31, 2025
@bentsku bentsku added this to the 4.2 milestone Jan 31, 2025
@bentsku bentsku self-assigned this Jan 31, 2025
@bentsku bentsku linked an issue Feb 1, 2025 that may be closed by this pull request
1 task
@bentsku bentsku marked this pull request as ready for review February 1, 2025 17:53
@bentsku bentsku requested a review from cloutierMat as a code owner February 1, 2025 17:53
@bentsku bentsku marked this pull request as draft February 1, 2025 17:54
@bentsku bentsku force-pushed the apigw-binary-support-non-proxy branch from 2052964 to 768d6df Compare February 3, 2025 09:27
@bentsku bentsku force-pushed the apigw-binary-support-aws-proxy branch from 349b657 to 294e374 Compare February 3, 2025 09:31
@bentsku bentsku force-pushed the apigw-binary-support-non-proxy branch from 768d6df to 4d05456 Compare February 3, 2025 09:32
Base automatically changed from apigw-binary-support-aws-proxy to master February 3, 2025 13:05
@bentsku bentsku force-pushed the apigw-binary-support-non-proxy branch from 4d05456 to 3afb9e3 Compare February 3, 2025 13:06
Copy link

github-actions bot commented Feb 3, 2025

LocalStack Community integration with Pro

    2 files    2 suites   29m 22s ⏱️
1 061 tests 998 ✅ 63 💤 0 ❌
1 063 runs  998 ✅ 65 💤 0 ❌

Results for commit 3afb9e3.

@bentsku bentsku marked this pull request as ready for review February 3, 2025 13:43
@bentsku bentsku removed the semver: patch Non-breaking changes which can be included in patch releases label Feb 3, 2025
@bentsku bentsku added semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases semver: patch Non-breaking changes which can be included in patch releases and removed semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases semver: patch Non-breaking changes which can be included in patch releases labels Feb 3, 2025
@localstack-bot
Copy link
Collaborator

Currently, only patch changes are allowed on master. Your PR labels (aws:apigateway, semver: minor) indicate that it cannot be merged into the master at this time.

@bentsku bentsku added semver: patch Non-breaking changes which can be included in patch releases semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases and removed semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases semver: patch Non-breaking changes which can be included in patch releases labels Feb 5, 2025
Copy link
Contributor

@cloutierMat cloutierMat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! 🚀 this adds great parity with aws!

The s3 tests just shows how far we have gone now! The logic of the convert_body methods is super well covered in unit tests as well 🙏

Comment on lines +262 to +277
if is_binary_request:
if content_handling and content_handling == ContentHandlingStrategy.CONVERT_TO_TEXT:
body = base64.b64encode(body)
# if the content handling is not defined, or CONVERT_TO_BINARY, we do not touch the body and leave it as
# proper binary
else:
if not content_handling or content_handling == ContentHandlingStrategy.CONVERT_TO_TEXT:
body = body.decode(encoding="UTF-8", errors="replace")
else:
# it means we have CONVERT_TO_BINARY, so we need to try to decode the base64 string
try:
body = base64.b64decode(body)
except ValueError:
raise InternalServerError("Internal server error")

return body
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice digging! There are a few hoops to jump through here as well 🚀

@bentsku bentsku merged commit ccf99d7 into master Feb 7, 2025
57 of 61 checks passed
@bentsku bentsku deleted the apigw-binary-support-non-proxy branch February 7, 2025 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws:apigateway Amazon API Gateway semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feature request: Enabling binary support using the API Gateway REST API
3 participants