Motivation

This PR addresses #8793, which describes an issue with the handling of the Accept-Encoding header when communicating with OpenSearch clusters with enabled http compression (as enabled statically with #8628).
This bug is actually caused by our HTTP client used in the Proxy infrastructure used to proxy requests from the client via LocalStack to the started OpenSearch server. This HTTP client always adds Accept-Encoding: gzip, deflate to the proxied HTTP request if no Accept-Encoding is set in the originating request.

With enabling the compression support for OpenSearch (i.e. OpenSearch will compress the responses using GZIP Content-Encoding in responses if the client sends gzip in the Accept-Encoding request header), this issue surfaced since every request to OpenSearch via LocalStack (which is the default) without the Accept-Encoding header (which is the default f.e. for a plain curl request).

It turns out that sending HTTP requests without any Accept-Encoding header is not possible with requests because it uses urllib3 under the hood, which in turn uses Python's http.client standard lib:

• urllib3 automatically adds `` if the Accept-Encoding header is not set at all. See `urllib3.util.request.ACCEPT_ENCODING`.
  • This could be circumvented by setting Accept-Encoding explicitly to None
• BUT: One layer below - Python's http.client - sets the Accept-Encoding: identity in case it's not set or None.
  • However, as discussed in when setting the Accept-Encoding header to None, it is still sent as 'identity' psf/requests#2234, this behavior should be compliant to the specs / RFCs.

This means it's not really possible to identically relay a request which does not contain the Accept-Encoding header at all.

Changes

• This PR explicitly sets the Accept-Encoding header to identity in case it's not set or None.
  • An alternative would have been just to circumvent the behavior of urllib3 by setting the Accept-Encoding header to None, but I wanted to make this behavior as explicit as possible (because it was quite a pain to get to the root of this weird behavior).
• This PR also merges the SimpleHttpClient with the SimpleStreamingHttpClient. The latter was introduced with fix auto decoding of gzip for s3 vhost proxied requests #8148 in addition to the SimpleHttpClient where it should actually just have been an extension of the default (but that was understandable at the time in order to prevent too wide-ranging changes).
  • This is why I classified this change as "minor".

Testing

Testing this in Python is actually quite hard (since the test explicitly has to send a request without any Accept-Encoding header whatsoever). I actually had to directly use the HTTPConnection class of Python's std lib.

Fixes #8793.