Maybe the root cause could be an incomplete #1733 after all. I realized that the recurring timeouts will only occur if there is some in-flight-request, but there was none present before the connection was closing. Therefore I suspect a race condition between sending and storing to in-flight-request.

Ideally, addition to in-flight_requests in _send could be wrapped like so:

        with self._lock:
            correlation_id = self._protocol.send_request(request)
            log.debug('%s Request %d: %s', self, correlation_id, request)
            expect_response = request.expect_response()

            if expect_response:
                sent_time = time.time()
                assert correlation_id not in self.in_flight_requests, 'Correlation ID already in-flight!'
                self.in_flight_requests[correlation_id] = (future, sent_time)

        if not expect_response:
            future.success(None)

Afterwards, the if not conn.in_flight_requests: check in client_async.py#_poll() needs to use the same lock.

Still, missing path for reconnecting remains a problem. I haven't been able to find any code path which would reenable the connection in async_client poll loop.

After further investigation, I've come to believe that it is indeed the race condition, not network problems, which are causing this. In our case we are committing manually from the main thread.
The deadlock seems to occur because "protocol out of sync" is raised and connection is closed before in-flight-requests is populated. Close() would usually terminate the futures stored in in-flight-requests but does not do that because it's not yet populated.

When close is called 5 minutes later, in-flight-requests is populated but the short-circuit in conn.py#close() does not clear the futures. Besides locking that I recommend above, we should also clean the in-flight-requests and end futures there even when short-circuiting.

Interesting. Let me think a bit more about this. You went straight from 1.3.X to 1.4.5? did you ever try 1.4.4 or earlier?

We've tried to upgrade to 1.4.3 last year, then reverted to 1.3.5 because we had weird cluster issues. Nothing we could pin down, specifically, and not this. No one had time to investigate properly back then.

We are also seeing the same issue where the consumer is getting stuck after running for one/two days. The problem becomes worst since it is not causing re-balance. We are using 1.4.4.

See #1766 (comment) for a status update on this issue.

@vimal3271 1.4.4 is an older version, and one major source of deadlocks was fixed in 1.4.5, but there have been additional fixes to master since then, so please upgrade to master (or wait for the upcoming 1.4.6 release) and if you are still seeing issues then please file a new ticket. Happy to help, just don't want to spend time debugging/fixing issues that have already been solved on master.

Fixes have been merged to master. I'm going to close, but please reopen if the issue persists.

@isamaru is the issue resolved? if so what's the solution? Please help me out here
Even I am facing the same issue for Kafka-python library