Skip to content

bpo-1346874: Add 'Expect: 100-Continue' support to httplib #14880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

bpo-1346874: Add 'Expect: 100-Continue' support to httplib #14880

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3d8b7b1
bpo-1346874: Add 'Expect: 100-Continue' support to httplib
tbartlett0 Jul 21, 2019
7390814
Fix rstlint errors
tbartlett0 Jul 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 80 additions & 6 deletions Doc/library/http.client.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ The module provides the following classes:


.. class:: HTTPConnection(host, port=None[, timeout], source_address=None, \
blocksize=8192)
blocksize=8192, continue_timeout=2.5)

An :class:`HTTPConnection` instance represents one transaction with an HTTP
server. It should be instantiated passing it a host and optional port
Expand All @@ -44,7 +44,10 @@ The module provides the following classes:
The optional *source_address* parameter may be a tuple of a (host, port)
to use as the source address the HTTP connection is made from.
The optional *blocksize* parameter sets the buffer size in bytes for
sending a file-like message body.
sending a file-like message body. Finally, the optional *continue_timeout*
parameter controls how long the connection will wait for a ``100 Continue``
response from the server before sending the body, if the request included
an ``Expect: 100-Continue`` header.

For example, the following calls all create instances that connect to the server
at the same host and port::
Expand All @@ -59,11 +62,14 @@ The module provides the following classes:

.. versionchanged:: 3.4
The *strict* parameter was removed. HTTP 0.9-style "Simple Responses" are
not longer supported.
no longer supported.

.. versionchanged:: 3.7
*blocksize* parameter was added.

.. versionchanged:: 3.9
*continue_timeout* parameter was added.


.. class:: HTTPSConnection(host, port=None, key_file=None, \
cert_file=None[, timeout], \
Expand Down Expand Up @@ -306,11 +312,24 @@ HTTPConnection Objects
No attempt is made to determine the Content-Length for file
objects.

.. method:: HTTPConnection.getresponse()
.. versionchanged:: 3.9
If the headers include ``Expect: 100-Continue`` and *body* is set,
the body will not be sent until either a ``100 Continue`` response is
received from the server or the :class:`HTTPConnection`'s *continue_timeout*
is reached; if a response code other than 100 is received, the body
will not be sent at all.

.. method:: HTTPConnection.getresponse(ignore_100_continue=True)

Should be called after a request is sent to get the response from the server.
Returns an :class:`HTTPResponse` instance.

By default, a server response of ``100 Continue`` will be ignored and the
call will not return until a response other than code 100 is received.
Setting *ignore_100_continue* to ``False`` will allow a 100 response to be
returned; you will then need to call :meth:`getresponse` a second time
(after transmitting the body) to get the final response.

.. note::

Note that you must have read the whole response before you can send a new
Expand All @@ -321,6 +340,10 @@ HTTPConnection Objects
:class:`HTTPConnection` object will be ready to reconnect when
a new request is sent.

.. versionchanged:: 3.9
Added the *ignore_100_continue* parameter. (In prior versions
a ``Continue`` response was always ignored.)


.. method:: HTTPConnection.set_debuglevel(level)

Expand Down Expand Up @@ -399,11 +422,17 @@ also send your request step by step, by using the four functions below.
an argument.


.. method:: HTTPConnection.endheaders(message_body=None, *, encode_chunked=False)
.. method:: HTTPConnection.endheaders(message_body=None, *, encode_chunked=False, \
expect_continue=False)

Send a blank line to the server, signalling the end of the headers. The
optional *message_body* argument can be used to pass a message body
associated with the request.
associated with the request. If a body is provided, setting
*expect_continue* to ``True`` will wait for a ``100 Continue`` response
from the server before sending the body. (This should generally be
used only when an ``Expect: 100-Continue`` header has been sent.) If no
response is received within the :class:`HTTPConnection`'s *continue_timeout*,
the body will be sent regardless.

If *encode_chunked* is ``True``, the result of each iteration of
*message_body* will be chunk-encoded as specified in :rfc:`7230`,
Expand All @@ -425,6 +454,9 @@ also send your request step by step, by using the four functions below.
Chunked encoding support. The *encode_chunked* parameter was
added.

.. versionadded:: 3.9
The *expect_continue* parameter was added.


.. method:: HTTPConnection.send(data)

Expand Down Expand Up @@ -579,6 +611,48 @@ request using http.client::
>>> print(response.status, response.reason)
200, OK

Since version 3.9, conditional transmission of the body is supported when an
``Expect: 100-Continue`` header is set. To use this in a simple case, just
set the header, and optionally the time for which the client should wait for
a ``100 Continue`` response before sending the body regardless::

>>> import http.client
>>> BODY = "***filecontents***"
>>> conn = http.client.HTTPConnection("localhost", 8080, continue_timeout=1.0)
>>> conn.request("PUT", "/file", BODY, headers={'Expect': '100-Continue'})
>>> response = conn.getresponse()
>>> # You will not see the '100' response, as it is handled internally
>>> print(response.status, response.reason)
200, OK

Here is a more complex example in which we manually check the response and
decide whether to send the body. This may be useful if the body must be
generated by some resource-intensive process which should be skipped if the
server will not accept it. ::

>>> import http.client
>>> conn = http.client.HTTPConnection("localhost", 8080)
>>> conn.putrequest("PUT", "/file")
>>> conn.putheader('Expect', '100-Continue')
>>> # Assuming you know in advance what the length will be
>>> # If not, you will need to encode it as chunked
>>> conn.putheader('Content-Length', '42')
>>> conn.endheaders()
>>> response = conn.getresponse(ignore_100_continue=False)
>>> print(response.status, response.reason)
100, Continue
>>> BODY = resource_intensive_calculation()
>>> conn.send(BODY)
>>> response = conn.getresponse()
>>> print(response.status, response.reason)
200, OK

.. note::

The *continue_timeout* setting does not apply when directly using
:meth:`getresponse`, so use the above example only if you are confident
that the server respects the ``Expect: 100-Continue`` header.

.. _httpmessage-objects:

HTTPMessage Objects
Expand Down
89 changes: 68 additions & 21 deletions Lib/http/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@
import http
import io
import re
import select
import socket
import collections.abc
from urllib.parse import urlsplit
Expand Down Expand Up @@ -293,15 +294,16 @@ def _read_status(self):
raise BadStatusLine(line)
return version, status, reason

def begin(self):
def begin(self, ignore_100_continue=True):
if self.headers is not None:
# we've already started reading the response
return

# read until we get a non-100 response
# (unless caller has requested return of 100 responses)
while True:
version, status, reason = self._read_status()
if status != CONTINUE:
if not (ignore_100_continue and status == CONTINUE):
break
# skip the header from the 100 response
while True:
Expand Down Expand Up @@ -813,13 +815,15 @@ def _get_content_length(body, method):
return None

def __init__(self, host, port=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
source_address=None, blocksize=8192):
source_address=None, blocksize=8192, continue_timeout=2.5):
self.timeout = timeout
self.continue_timeout = continue_timeout
self.source_address = source_address
self.blocksize = blocksize
self.sock = None
self._buffer = []
self.__response = None
self.__pending_response = None
self.__state = _CS_IDLE
self._method = None
self._tunnel_host = None
Expand All @@ -828,9 +832,10 @@ def __init__(self, host, port=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,

(self.host, self.port) = self._get_hostport(host, port)

# This is stored as an instance variable to allow unit
# tests to replace it with a suitable mockup
# These are stored as instance variables to allow unit
# tests to replace them with a suitable mockup
self._create_connection = socket.create_connection
self._select = select.select

def set_tunnel(self, host, port=None, headers=None):
"""Set up host and port for HTTP CONNECT tunnelling.
Expand Down Expand Up @@ -928,6 +933,10 @@ def close(self):
self.sock = None
sock.close() # close it manually... there may be other refs
finally:
pending_response = self.__pending_response
if pending_response:
self.__pending_response = None
pending_response.close()
response = self.__response
if response:
self.__response = None
Expand Down Expand Up @@ -992,7 +1001,8 @@ def _read_readable(self, readable):
datablock = datablock.encode("iso-8859-1")
yield datablock

def _send_output(self, message_body=None, encode_chunked=False):
def _send_output(self, message_body=None, encode_chunked=False,
expect_continue=False):
"""Send the currently buffered request and clear the buffer.

Appends an extra \\r\\n to the buffer.
Expand All @@ -1004,6 +1014,23 @@ def _send_output(self, message_body=None, encode_chunked=False):
self.send(msg)

if message_body is not None:
if expect_continue and not self.__response:
read_ready, _, _ = self._select([self.sock], [], [],
self.continue_timeout)
if read_ready:
if self.debuglevel > 0:
response = self.response_class(self.sock,
self.debuglevel,
method=self._method)
else:
response = self.response_class(self.sock,
method=self._method)
response.begin(ignore_100_continue=False)
if response.code != CONTINUE:
# Break without sending the body
self.__pending_response = response
return
response.close()

# create a consistent interface to message_body
if hasattr(message_body, 'read'):
Expand Down Expand Up @@ -1202,7 +1229,8 @@ def putheader(self, header, *values):
header = header + b': ' + value
self._output(header)

def endheaders(self, message_body=None, *, encode_chunked=False):
def endheaders(self, message_body=None, *, encode_chunked=False,
expect_continue=False):
"""Indicate that the last header line has been sent to the server.

This method sends the request to the server. The optional message_body
Expand All @@ -1213,7 +1241,8 @@ def endheaders(self, message_body=None, *, encode_chunked=False):
self.__state = _CS_REQ_SENT
else:
raise CannotSendHeader()
self._send_output(message_body, encode_chunked=encode_chunked)
self._send_output(message_body, encode_chunked=encode_chunked,
expect_continue=expect_continue)

def request(self, method, url, body=None, headers={}, *,
encode_chunked=False):
Expand Down Expand Up @@ -1258,20 +1287,33 @@ def _send_request(self, method, url, body, headers, encode_chunked):
else:
encode_chunked = False

# If the Expect: 100-continue header is set, we will try to honor it
# (if possible). We can only do so if 1) the request has a body, and
# 2) there is no current incomplete response (since we need to read
# the response stream to check if the code is 100 or not)
expect_continue = (
body and not self.__response
and 'expect' in header_names
and '100-continue' in {v.lower() for k, v in headers.items()
if k.lower() == 'expect'}
)

for hdr, value in headers.items():
self.putheader(hdr, value)
if isinstance(body, str):
# RFC 2616 Section 3.7.1 says that text default has a
# default charset of iso-8859-1.
body = _encode(body, 'body')
self.endheaders(body, encode_chunked=encode_chunked)
self.endheaders(body, encode_chunked=encode_chunked,
expect_continue=expect_continue)

def getresponse(self):
def getresponse(self, ignore_100_continue=True):
"""Get the response from the server.

If the HTTPConnection is in the correct state, returns an
instance of HTTPResponse or of whatever object is returned by
the response_class variable.
instance of HTTPResponse or of whatever object is returned by the
response_class variable. The connection will wait for a response other
than code 100 ('Continue') unless ignore_100_continue is set to False.

If a request has not been sent or if a previous response has
not be handled, ResponseNotReady is raised. If the HTTP
Expand Down Expand Up @@ -1302,27 +1344,32 @@ def getresponse(self):
if self.__state != _CS_REQ_SENT or self.__response:
raise ResponseNotReady(self.__state)

if self.debuglevel > 0:
if self.__pending_response:
response = self.__pending_response
self.__pending_response = None
elif self.debuglevel > 0:
response = self.response_class(self.sock, self.debuglevel,
method=self._method)
else:
response = self.response_class(self.sock, method=self._method)

try:
try:
response.begin()
response.begin(ignore_100_continue=ignore_100_continue)
except ConnectionError:
self.close()
raise
assert response.will_close != _UNKNOWN
self.__state = _CS_IDLE
if response.code != 100:
# Code 100 is effectively 'not a response' for this purpose
self.__state = _CS_IDLE

if response.will_close:
# this effectively passes the connection to the response
self.close()
else:
# remember this, so we can tell when it is complete
self.__response = response
if response.will_close:
# this effectively passes the connection to the response
self.close()
else:
# remember this, so we can tell when it is complete
self.__response = response

return response
except:
Expand Down
Loading