Skip to content

Kafka consumer not reading any message with zero timeout_ms poll value #2609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
berrfred opened this issue May 2, 2025 · 4 comments · Fixed by #2613
Closed

Kafka consumer not reading any message with zero timeout_ms poll value #2609

berrfred opened this issue May 2, 2025 · 4 comments · Fixed by #2613

Comments

@berrfred
Copy link

berrfred commented May 2, 2025

Zero timeout_ms poll value is working fine with 2.0.2 and 2.0.6 but not any longer with latest releases (2.1.x, 2.2.x).

Using a non zero value solves the issue but in my very specific environment (HPE NonStop) using a non zero timeout_ms poll value leads to a quite higher cpu consumption, experienced years ago with 2.0.2 and also checked after running for a while with 2.2.3 .
@dpkp dpkp mentioned this issue May 4, 2025
Merged
@dpkp dpkp closed this as completed in #2613 May 4, 2025
@berrfred
Copy link
Author

berrfred commented May 5, 2025

@dpkp Thank you so much for your support, unfortunately it does not seem to solve the 0 timeout_ms issue since I put a record in the queue at 08:35 but it was never read until I changed the timeout_ms from 0 to 100 at 08:38.

05/05/2025; 08:37:00.97; [I] $GMT22 ; no record available
05/05/2025; 08:37:06.13; [I] $GMT22 ; no record available
05/05/2025; 08:37:11.29; [I] $GMT22 ; no record available
05/05/2025; 08:37:16.37; [I] $GMT22 ; no record available
05/05/2025; 08:37:21.42; [I] $GMT22 ; no record available
05/05/2025; 08:37:26.46; [I] $GMT22 ; no record available
05/05/2025; 08:37:31.61; [I] $GMT22 ; no record available
05/05/2025; 08:37:36.65; [I] $GMT22 ; no record available
05/05/2025; 08:37:41.69; [I] $GMT22 ; no record available
05/05/2025; 08:37:46.73; [I] $GMT22 ; no record available
05/05/2025; 08:37:51.77; [I] $GMT22 ; no record available
05/05/2025; 08:37:56.82; [I] $GMT22 ; no record available

05/05/2025; 08:38:00.01; [W] $GMT22 ; statistics - saved
05/05/2025; 08:38:00.04; [W] $GMT22 ; parameters - reloaded

05/05/2025; 08:38:01.92; [I] $GMT22 ; no record available
05/05/2025; 08:38:07.08; [I] $GMT22 ; no record available
05/05/2025; 08:38:12.21; [I] $GMT22 ; no record available
05/05/2025; 08:38:17.32; [I] $GMT22 ; no record available
05/05/2025; 08:38:22.42; [I] $GMT22 ; no record available
05/05/2025; 08:38:27.53; [I] $GMT22 ; no record available
05/05/2025; 08:38:32.69; [W] $GMT22 ; read records - committed (1)
05/05/2025; 08:38:32.69; [W] $GMT22 ; read|oamds-mtonline|1|189231|b'no_check_1000200ffff33gfff'|10000
05/05/2025; 08:38:36.70; [I] $GMT22 ; nowait response for tag 10000
05/05/2025; 08:38:36.84; [I] $GMT22 ; no record available
05/05/2025; 08:38:41.94; [I] $GMT22 ; no record available
05/05/2025; 08:38:47.10; [I] $GMT22 ; no record available
05/05/2025; 08:38:52.24; [W] $GMT22 ; read records - committed (1)
05/05/2025; 08:38:52.24; [W] $GMT22 ; read|oamds-mtonline|1|189232|b'no_check_1000200ffff33gfff'|10001
05/05/2025; 08:38:56.25; [I] $GMT22 ; nowait response for tag 10001
05/05/2025; 08:38:56.38; [I] $GMT22 ; no record available

@berrfred
Copy link
Author

berrfred commented May 9, 2025

Release 2.2.6 - The issue is still present but it keeps working fine with 100ms timeout (drawback of unnecessary cpu consumption in my environment).

@berrfred
Copy link
Author

berrfred commented May 9, 2025

Once started the consumers and calling poll() with timeout_ms=0 (and facing the issue of not reading any message), changing the timeout_ms to 100 solves the issue since it starts reading messages on the next calls to poll() ... and apparently it keeps reading also after reverting the parameter back to 0.

Could not grab the debug logging yet but if this can help I'll try to reproduce it with corresponding logs in the next days.

@dpkp
Copy link
Owner

dpkp commented May 9, 2025

timeout_ms=0 should work, but you will need to call it many times before all of the connection + group setup is completed and messages start flowing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix KafkaConsumer.poll() with zero timeout dpkp/kafka-python
2 participants
@dpkp @berrfred