fix parse_response to handle streaming responses #6415
Conversation
| if response_dict["status_code"] >= 300: | ||
| response_dict["body"] = response.data |
There was a problem hiding this comment.
Just an FYI: This explicitly contains any redirections (3xx). There are no 3xx http response status codes in the specifications, but these responses might have a response body. But imho it's safe to assume that there are not streaming responses in 300+ status codes.
There was a problem hiding this comment.
yep, just doing what the client does 🤷
localstack/aws/client.py
Outdated
| # TODO | ||
| LOG.warning( | ||
| "don't know how to parse an event stream output for %s.%s", | ||
| operation.service_model.service_name, | ||
| operation.name, | ||
| ) | ||
| response_dict["body"] = response.data |
There was a problem hiding this comment.
I just dig a bit into this:
- The parsers create an instance of the
EventStreamwithresponse['body']in this case: parsers.py#L348 - The
EventStreamclass in botocore expects this param to be a raw bytes stream:
eventstream.py#L595
So I think we should just use the _ResponseStream here as well?
There was a problem hiding this comment.
cool, yea i added an additional _RawStream wrapper that exposes the interface botocore's EventStream object seems to expect. i'm too lazy to test this right now, so leaving a TODO for the next person :P
8fb5eed to
e4bd11d
Compare
e4bd11d to
26eb419
Compare
26eb419 to
cdd7552
Compare
cdd7552 to
81dada4
Compare
This PR fixes an issue/limitation of the
parse_responseoperation in our aws client module.The problem was that we were not really implementing the parsing logic of botocore correctly, in particular when streaming binary responses are expected, like in
s3.GetObject, which caused a problem here: https://github.com/boto/botocore/blob/ae44d1de241c13626d05a9301cf62e460833c950/botocore/parsers.py#L910-L914In boto, when you call
s3_client.get_object(...), theBodyattribute of the response (i.e., the payload), will not actually be bytes, but a readableioobject. This PR introduces the same behavior for theparse_responsemethod.I modeled the implementation after the version in boto core https://github.com/boto/botocore/blob/ae44d1de241c13626d05a9301cf62e460833c950/botocore/endpoint.py#L65-L74
Future work is
has_event_stream_output, which I think covers things like the kinesis shard iterator.