Skip to content

How to get current offsets of consumer group without subscribing #1501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
yoneal opened this issue May 19, 2018 · 5 comments
Closed

How to get current offsets of consumer group without subscribing #1501

yoneal opened this issue May 19, 2018 · 5 comments

Comments

@yoneal
Copy link

yoneal commented May 19, 2018

First of all thanks for this wonderful project!

In #1161 support for getting the end_offsets was added to get the high watermark of partitions without actually consuming. Is there a way to get the current offset of a consumer group, similar to the CURRENT-OFFSET column of the ff command:

./bin/kafka-consumer-groups.sh --bootstrap-server bootstrap:9092 --describe --group console-consumer-92539
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my_topic                      0          845771          845787          16         consumer-1-4acb3a97-d72e-4c68-805d-c9a8089b5a26   /10.128.0.17                   consumer-1

I seem to be missing it from the documentation..
@jeffwidman
Copy link
Contributor

currently to do this you'll need to directly use KafkaClient and pass the protocol calls directly. This is much easier if you use a broker >= 0.10.2 because then you don't have to identify the group/partition mapping, just use the functionality of KIP-88.

I'm working on a pull request to add this to the Datadog kafka consumer offsets check, so whenever I push that up you'll see how to do it.

Long term, this would go in the admin client, but no one has had the time to write that yet: #935

@cjdudleybw
Copy link

cjdudleybw commented Jan 28, 2019 •
edited
Loading

Has this moved forwards at all? Been trying to calculate lags using this API, and all I'm missing is the current offset, as of version 1.4.4.
Are there any other ways of establishing lag? Thanks!

EDIT:

Answered my own question; use a KafkaAdminClient, and call list_consumer_group_offsets(). Compare the results to results of KafkaConsumer.end_offsets()

@jeffwidman
Copy link
Contributor

@cjdudleybw see also #1673 (comment) including the linked source code for datadog for fetching the highwater offsets.

If you are monitoring / alerting on this, then you want to directly query the cluster for the high water offsets... IMO too dangerous to trust the KafkaConsumer.end_offsets() because one day there could be a bug in the consumer that prevents it from having an accurate view of the cluster (if the consumer is somehow deadlocked for example) and your alerts may not fire when you most need them.

@piperck
Copy link

piperck commented Jan 7, 2020

@jeffwidman
Hi jeff since #1673 fetching the highwater offsets solution can not direct run in 1.4.7?

KafkaAdminClient._send_request_to_node

has changed in version 1.4.7......

@jeffwidman
Copy link
Contributor

Please open a new ticket, this one is nearly a year old.

Also, there's a working example here of how to check offsets: https://github.com/DataDog/integrations-core/blob/master/kafka_consumer/datadog_checks/kafka_consumer/kafka_consumer.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yoneal @jeffwidman @piperck @cjdudleybw