@mumrah Got the thing working. Tested it also. Consumer re-balancing works like a charm. Used a built-in Kazoo recipe for doing this.

Will add test cases later. Closer to when you are about to merge it in.

Recipe I used - https://kazoo.readthedocs.org/en/latest/api/recipe/partitioner.html

Any plans to merge this @mahendra 's patch to the trunk?

@vihang yeah, it will be merged in a while. @mumrah had indicated that he is busy at work. wait for a while more. if you need the feature urgently, I would request you to checkout the branch and try it out. let me know if you face any issues...

This would be a valuable feature. Any update on when this will be merged? @mahendra what still needs to be addressed on your branch? Would you prefer we give feedback here or on your fork?

@timcherry do give feedback on my branch. I will update it so that @mumrah can pull a fully working branch.

The zookeeper support I added is not compliant with other Kafka Clients (Scala/Java etc.). I just used the kazoo recipe for partitioning resources.

Ideally the partitioning is compliant with at least the official Kafka clients. Why introduce another "standard"?

@mmlac I agree. I did it for the following reasons.

• The official Kafka partitioning protocol is going to change. There are discussions going on about it. I did not want to redo the thing again.
• I wanted a quick solution for one of my products and did not have enough time to read through the docs/source to figure it out. The documentation was very hazy about it.

If it is a must-have, we can start working on making it compliant to the official version. Would need some help on it though.

@mmlac @mahendra

Trying to be compatible with the Java/Scala consumer rebalancing is a difficult task. Very few, if any, of the clients support this.

And on top of that, there are currently plans within Kafka to simplify this rebalancing and remove ZK from the client side. See https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design and https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design for more details.

I absolutely agree that right now might not be a good time to implement this and I was just suggesting to i.e. maintain the current fix in a branch until the discussions in JIRA show a clear direction for these features and then implement this as a official solution for kafka-python.

@mahendra Great work nevertheless! I didn't want to suggest that your implementation is bad!

thanks @mmlac :-)

@mmlac We're needing a native or Python consumer with ZK support, too, and @mahendra 's fork looks promising. Do you have any idea when the consumer redesign is planned to happen?

There was a consumer redesign in 0.8.1 which allows the consumer offsets to be stored in ZK via Kafka. Is that what you're referring to or do you need access to ZK for another reason?

@wizzat We need ZK support to drive a high level consumer that can rebalance, right? If so, that is the "ZK support" I was referring to. If that hasn't been done by this branch or an existing tasks, can we chat on IRC or something about what needs to happen in code to make this possible? I and another will probably have come cycles to work on this.

I see where you're going. Yes, this patch looks promising but is quite old. I'll try to hop on IRC tomorrow when I get to work. I'm pretty sure I remember my creds on freenode.

@wizzat any updates/thoughts on this? Is now a better time to bring this up now that we have pypi uploads and versioning. :)

Looking at the diff for this, I think we'd have to reimplement it from scratch to be consistent with consumer improvements made since the patch was written. This pretty much sums it up:

class ZSimpleConsumer(object):
    ...

Also interested in seeing ZK support here.

I think most likely outcome is that this waits until support is added server-side. I think that is on the roadmap for kafka 0.9? see https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

thank you!

@dpkp I agree that's the most likely. I feel like we've been waiting forever for Kafka 0.9. :)

If we were to implement this support, would it have to be completely compatible with the JVM rebalancing?

Kafka 0.9.0.0 Group Coordination support has been added to KafkaConsumer in the 0.9 git branch. I'm planning to merge to master soon and will cut a new release after a bit more testing.

group coordination merged to master.