Skip to content

Kafka consumer not reading any message with zero timeout_ms poll value #2609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
berrfred opened this issue May 2, 2025 · 1 comment · Fixed by #2613
Closed

Kafka consumer not reading any message with zero timeout_ms poll value #2609

berrfred opened this issue May 2, 2025 · 1 comment · Fixed by #2613

Comments

@berrfred
Copy link

berrfred commented May 2, 2025

Zero timeout_ms poll value is working fine with 2.0.2 and 2.0.6 but not any longer with latest releases (2.1.x, 2.2.x).

Using a non zero value solves the issue but in my very specific environment (HPE NonStop) using a non zero timeout_ms poll value leads to a quite higher cpu consumption, experienced years ago with 2.0.2 and also checked after running for a while with 2.2.3 .
@dpkp dpkp mentioned this issue May 4, 2025
Merged
@dpkp dpkp closed this as completed in #2613 May 4, 2025
@berrfred
Copy link
Author

berrfred commented May 5, 2025

@dpkp Thank you so much for your support, unfortunately it does not seem to solve the 0 timeout_ms issue since I put a record in the queue at 08:35 but it was never read until I changed the timeout_ms from 0 to 100 at 08:38.

05/05/2025; 08:37:00.97; [I] $GMT22 ; no record available
05/05/2025; 08:37:06.13; [I] $GMT22 ; no record available
05/05/2025; 08:37:11.29; [I] $GMT22 ; no record available
05/05/2025; 08:37:16.37; [I] $GMT22 ; no record available
05/05/2025; 08:37:21.42; [I] $GMT22 ; no record available
05/05/2025; 08:37:26.46; [I] $GMT22 ; no record available
05/05/2025; 08:37:31.61; [I] $GMT22 ; no record available
05/05/2025; 08:37:36.65; [I] $GMT22 ; no record available
05/05/2025; 08:37:41.69; [I] $GMT22 ; no record available
05/05/2025; 08:37:46.73; [I] $GMT22 ; no record available
05/05/2025; 08:37:51.77; [I] $GMT22 ; no record available
05/05/2025; 08:37:56.82; [I] $GMT22 ; no record available

05/05/2025; 08:38:00.01; [W] $GMT22 ; statistics - saved
05/05/2025; 08:38:00.04; [W] $GMT22 ; parameters - reloaded

05/05/2025; 08:38:01.92; [I] $GMT22 ; no record available
05/05/2025; 08:38:07.08; [I] $GMT22 ; no record available
05/05/2025; 08:38:12.21; [I] $GMT22 ; no record available
05/05/2025; 08:38:17.32; [I] $GMT22 ; no record available
05/05/2025; 08:38:22.42; [I] $GMT22 ; no record available
05/05/2025; 08:38:27.53; [I] $GMT22 ; no record available
05/05/2025; 08:38:32.69; [W] $GMT22 ; read records - committed (1)
05/05/2025; 08:38:32.69; [W] $GMT22 ; read|oamds-mtonline|1|189231|b'no_check_1000200ffff33gfff'|10000
05/05/2025; 08:38:36.70; [I] $GMT22 ; nowait response for tag 10000
05/05/2025; 08:38:36.84; [I] $GMT22 ; no record available
05/05/2025; 08:38:41.94; [I] $GMT22 ; no record available
05/05/2025; 08:38:47.10; [I] $GMT22 ; no record available
05/05/2025; 08:38:52.24; [W] $GMT22 ; read records - committed (1)
05/05/2025; 08:38:52.24; [W] $GMT22 ; read|oamds-mtonline|1|189232|b'no_check_1000200ffff33gfff'|10001
05/05/2025; 08:38:56.25; [I] $GMT22 ; nowait response for tag 10001
05/05/2025; 08:38:56.38; [I] $GMT22 ; no record available

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix KafkaConsumer.poll() with zero timeout dpkp/kafka-python
1 participant
@berrfred