Skip to content

Set a timeout on send in connection #1246

Closed
@tvoinarovskyi

Description

@tvoinarovskyi

Spotted while reviewing #1230. The code for sending bytes into the network currently looks like this:

            # In the future we might manage an internal write buffer
            # and send bytes asynchronously. For now, just block
            # sending each request payload
            self._sock.setblocking(True)
            total_sent = 0
            while total_sent < len(data):
                sent_bytes = self._sock.send(data[total_sent:])
                total_sent += sent_bytes
            assert total_sent == len(data)
            if self._sensors:
                self._sensors.bytes_sent.record(total_sent)
            self._sock.setblocking(False)

The problem here is, that if 1 of the nodes goes down only on write (will stop reading bytes on the other end) this will lead to a stop in application code, as the sending Thread will suspend indefinitely. This is not too uncommon in Kubernetics or other virtual environments, that do not perfectly handle socket proxying.
To avoid the issue we have to make the call asynchronous or at least set the timeout based on request_timeout_ms configuration. For example:

            # In the future we might manage an internal write buffer
            # and send bytes asynchronously. For now, just block
            # sending each request payload
            self._sock.settimeout(self.config['request_timeout_ms'] / 1000.0)
            total_sent = 0
            while total_sent < len(data):
                sent_bytes = self._sock.send(data[total_sent:])
                total_sent += sent_bytes
            assert total_sent == len(data)
            if self._sensors:
                self._sensors.bytes_sent.record(total_sent)
            self._sock.setblocking(False)

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions