Closed
Description
Spotted while reviewing #1230. The code for sending bytes into the network currently looks like this:
# In the future we might manage an internal write buffer
# and send bytes asynchronously. For now, just block
# sending each request payload
self._sock.setblocking(True)
total_sent = 0
while total_sent < len(data):
sent_bytes = self._sock.send(data[total_sent:])
total_sent += sent_bytes
assert total_sent == len(data)
if self._sensors:
self._sensors.bytes_sent.record(total_sent)
self._sock.setblocking(False)The problem here is, that if 1 of the nodes goes down only on write (will stop reading bytes on the other end) this will lead to a stop in application code, as the sending Thread will suspend indefinitely. This is not too uncommon in Kubernetics or other virtual environments, that do not perfectly handle socket proxying.
To avoid the issue we have to make the call asynchronous or at least set the timeout based on request_timeout_ms configuration. For example:
# In the future we might manage an internal write buffer
# and send bytes asynchronously. For now, just block
# sending each request payload
self._sock.settimeout(self.config['request_timeout_ms'] / 1000.0)
total_sent = 0
while total_sent < len(data):
sent_bytes = self._sock.send(data[total_sent:])
total_sent += sent_bytes
assert total_sent == len(data)
if self._sensors:
self._sensors.bytes_sent.record(total_sent)
self._sock.setblocking(False)