Consumer coordination is currently not supported by kafka-python. See Issue #38 . This feature requires either (1) custom client-side coordination via Zookeeper/Etcd/Consul, or (2) wait until kafka-server adds coordination API support. As much as I'd love to write an etcd/consul powered coordination layer, we'll probably end up waiting until kafka-server supports this. I believe that is expected in apache kafka 0.9.0 or possibly even 0.8.2

I emulate this by creating a SimpleConsumer with auto_commit=False and by calling fetch_last_known_offsets() before each read and commit() after.

a better emulation would be to assign consumers to partitions explicitly. As long as you have a consumer group in which no two consumers read from the same partition, you should not see duplicate message processing. The complexity arises when you want the client code to manage partition assignment dynamically, and additionally support add/removing consumers from the group.