@FoamyGuy I was going to tackle this tomorrow. The Python requests library doesn't do this based on status code, and think we should try to match their logic, thoughts?

200 without content-length and not chunked I thought was fixed by: #123

My understanding (which is shallow) is that CPython can detect when the server closes the socket, using socket.fileno() (or .select() or similar. In ESP32SPI I think we have .available or can access he specific states of socket setup and shutdown. But nothing I know of that's comparable in native wifi sockets. Not sure about WIZnet.

204 by definition has no response body:
https://www.rfc-editor.org/rfc/rfc7231.html#section-6.3.5
Also, 204 is not supposed to have content-length, and since there is no body it can't be chunked.

I may be missing a third way, but it seems we either need to condition on status code, or add capability to sense that the server has closed the socket (supporting ESP32SPI, WIZnet, and two native implementations). Otherwise, that leaves timeout.

In an environement where I can control the server to respond with a 204, the headers are:

HTTP/1.1 204 No Content
Date: Sun, 11 May 2025 18:14:13 GMT
Server: Apache
Upgrade: h2
Connection: Upgrade
Cache-Control: max-age=172800
Expires: Tue, 13 May 2025 18:14:13 GMT
Vary: User-Agent

No response body (and not chunked), no content-length header, (and also no keep-alive), but the client (curl, which didn't send Connection: close, and verbose mode reports: "* Connection #0 to host {redacted} left intact") exits immediately. keep-alive is default behavior in HTTP/1.1 unless there's a Connection: close in the request or response. That doesn't seem the case here, so not sure what options there are besides checking status code or timeout.

A quick look, and that PR uses _remaining, which is based on content length, which for a 204 is specifically not returned.

So the correct fix would see that _remaining is None and probably something else to make sure there wasn't any data...

I'm just wondering if that something needs to be a timeout, or if there's some other mechanism in the absence of content-length, chunked, or socket close. It doesn't seem like curl or CPython Requests have any discernable timeout.

If not conditioning on status code, a 200 is like a 204 except it has no body (a 200 could also have no body, but then it should have a content-length header of 0), but the client may not know there are no body bytes incoming without waiting to see?

My plan was to step through a normal 204 in CPython requests and see what it gets. And then see if I can reproduce it in a 3 in circuitpython...

I'm glad you all are on it. I'm baffled how it works given all of the above :-)

I would be happy to match the logic from CPython requests to the best that we can. Thanks for looking into it @justmobilize

@FoamyGuy would like you thoughts. I went down the rabbit-hole and got:

in Cpython

1. requests: calls self.raw.read(chunk_size), where raw is urllib3.response.HTTPResponse
2. urllib3 goes read, _raw_read, _fp_read to return self._fp.read(amt) if amt is not None else self._fp.read(), where _fp is a http.client.HTTPResponse
3. http.client.HTTPResponse read hits here and self.fpis a _io.BufferedReader

All this to say that the socket takes care of this. So I see 2 options:

1. Update the core, ESP32SPI and WIZNET5K to handle this correctly
2. Do a hack in requests, like this

I would prefer option 1, because I think we would keep fixing things where the socket code isn't handling things right.

Looking a little bit more, I see when using CPython and this library, it hangs on _recv_into, since we have this in _readinto:

            elif self._remaining is None:
                # the Content-Length is not provided in the HTTP header
                # so try parsing as long as their is data in the socket
                pass

which then does:

    read = self._recv_into(buf, nbytes)

which just sits there...

Everything I read and know, you should always have Content-Length unless Transfer-Encoding is chunked.

I did check a Hue (since #123 is what added that) and it indeed doesn't return a Content-Length.

I'm going to dig a little more, to try to find the right fix

okay, @FoamyGuy finally found it in http/client.py

        # does the body have a fixed length? (of zero)
        if (status == NO_CONTENT or status == NOT_MODIFIED or
            100 <= status < 200 or      # 1xx codes
            self._method == "HEAD"):
            self.length = 0

I think we can move this logic into Response._parse_headers to set the length to 0 matching this. So similar to what you have, just a little different and in a different spot

@justmobilize that works for me, do you want to PR that change? we can close this one. I'll submit it in another day or two if you are unable.