Skip to content

Log warnings about high cpu usage when number of consumers is higher than partition count #572

Closed
@stefanth

Description

@stefanth

I have two consumers for a topic that only has one partition. When the idle consumer is running it prints out a constant stream of:
"_poll called with a non-zero timeout and sleep=False -- but there was nothing to do. This can cause high CPU usage during idle."

This is similar to issue #494 that had been fixed but it seems that commit b8c2097 broke it again. There is no way to pass the sleep=True parameter to the client object when calling the poll method of the KafkaConsumer object (in group.py). I tried changing line 465 of group.py to:
self._client.poll(timeout_ms, sleep=True)

and then everything worked correctly again. I don't see the reason for commit b8c2097 so I don't know if this would break anything else.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions