Skip to content

Kafka: Offset commit (manual) failed #894

@KonstantinCodes

Description

@KonstantinCodes

My consumer always fails to commit the offset:

%4|1560006005.731|COMMITFAIL|rdkafka#consumer-20| [thrd:main]: Offset commit (manual) failed for 1/1 partition(s): Broker: Unknown member: commands[0]@1(Broker: Unknown member)

Here is my Config:

        consumer:
            dsn: "..."
            global:
                ### Make sure this is unique for each application / consumer group and does not change
                ### Otherwise, Kafka won't be able to track your last offset and will always start according to
                ### `auto.offset.reset` setting.
                ### See Kafka documentation regarding `group.id` property if you want to know more
                group.id: 'backend'
                metadata.broker.list: '...'
                security.protocol: sasl_ssl
                ssl.ca.location: '%kernel.project_dir%/config/kafka/ca.pem'
                sasl.username: '...'
                sasl.password: '...'
                sasl.mechanisms: "SCRAM-SHA-256"
                max.poll.interval.ms: '25000'
                enable.auto.commit: 'false'
                enable.auto.offset.store: 'false'
            topic:
                auto.offset.reset: beginning
            ### Commit async is true by default since version 0.9.9.
            ### It is suggested to set it to true in earlier versions since otherwise consumers become extremely slow,
            ### waiting for offset to be stored on Kafka before continuing.
            commit_async: true

Any ideas, why this is happening?

Tested with both heroku Kafka & CloudKarafka.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions