Skip to content

NoBrokersAvailable when using bootstrap_server list #1440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
thspinto opened this issue Mar 12, 2018 · 10 comments
Closed

NoBrokersAvailable when using bootstrap_server list #1440

thspinto opened this issue Mar 12, 2018 · 10 comments

Comments

@thspinto
Copy link

I'm getting the kafka.errors.NoBrokersAvailable: NoBrokersAvailable specifically when I use an URL list on the bootstrap_server configuration (e.g. kb1.server:9092,kb2.server:9092,kb3.server:9092).

Version Info:
Kafka: 1.0.1
kafka-python: 1.4.2

03/12/2018 19:15:29 - kafka.protocol.parser - DEBUG - Sending request FetchRequest_v4(replica_id=-1, max_wait_time=500, min_bytes=1, max_bytes=52428800, isolation_level=0, topics=[(topic='ModelEvent', partitions=[(partition=0, offset=0, max_bytes=1048576)]), (topic='Inspection', partitions=[(partition=0, offset=0, max_bytes=1048576)]), (topic='InspectionDumperTriggerTopic', partitions=[(partition=0, offset=0, max_bytes=1048576)])])
03/12/2018 19:15:29 - kafka.conn - DEBUG - <BrokerConnection node_id=0 host=10.6.3.105:9092 <connected> [IPv4 ('10.6.3.105', 9092)]> Request 8: FetchRequest_v4(replica_id=-1, max_wait_time=500, min_bytes=1, max_bytes=52428800, isolation_level=0, topics=[(topic='ModelEvent', partitions=[(partition=0, offset=0, max_bytes=1048576)]), (topic='Inspection', partitions=[(partition=0, offset=0, max_bytes=1048576)]), (topic='InspectionDumperTriggerTopic', partitions=[(partition=0, offset=0, max_bytes=1048576)])])
03/12/2018 19:15:29 - kafka.consumer.fetcher - DEBUG - Sending FetchRequest to node 1
03/12/2018 19:15:29 - kafka.protocol.parser - DEBUG - Sending request FetchRequest_v4(replica_id=-1, max_wait_time=500, min_bytes=1, max_bytes=52428800, isolation_level=0, topics=[(topic='UserEntity', partitions=[(partition=0, offset=0, max_bytes=1048576)]), (topic='OnboardingDumperTriggerTopic', partitions=[(partition=0, offset=1, max_bytes=1048576)])])
03/12/2018 19:15:29 - kafka.conn - DEBUG - <BrokerConnection node_id=1 host=10.6.3.56:9092 <connected> [IPv4 ('10.6.3.56', 9092)]> Request 5: FetchRequest_v4(replica_id=-1, max_wait_time=500, min_bytes=1, max_bytes=52428800, isolation_level=0, topics=[(topic='UserEntity', partitions=[(partition=0, offset=0, max_bytes=1048576)]), (topic='OnboardingDumperTriggerTopic', partitions=[(partition=0, offset=1, max_bytes=1048576)])])
03/12/2018 19:15:29 - kafka.consumer.fetcher - DEBUG - Sending FetchRequest to node 2
03/12/2018 19:15:29 - kafka.protocol.parser - DEBUG - Sending request FetchRequest_v4(replica_id=-1, max_wait_time=500, min_bytes=1, max_bytes=52428800, isolation_level=0, topics=[(topic='Onboarding', partitions=[(partition=0, offset=0, max_bytes=1048576)]), (topic='ContractDumperTriggerTopic', partitions=[(partition=0, offset=1, max_bytes=1048576)])])
03/12/2018 19:15:30 - kafka.conn - DEBUG - <BrokerConnection node_id=2 host=10.6.2.68:9092 <connected> [IPv4 ('10.6.2.68', 9092)]> Request 2: FetchRequest_v4(replica_id=-1, max_wait_time=500, min_bytes=1, max_bytes=52428800, isolation_level=0, topics=[(topic='Onboarding', partitions=[(partition=0, offset=0, max_bytes=1048576)]), (topic='ContractDumperTriggerTopic', partitions=[(partition=0, offset=1, max_bytes=1048576)])])
03/12/2018 19:15:30 - missioncontrolsonar.worker_manager - INFO - Return: start_shutdown(WorkerManager,<Task finished coro=<worker_loop() done, defined at /app/missioncontrolsonar/worker_manager.py:73> result=None>) = WorkerManager
03/12/2018 19:15:30 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished coro=<worker_loop() done, defined at /app/missioncontrolsonar/worker_manager.py:73> exception=NoBrokersAvailable()>
Traceback (most recent call last):
  File "/app/missioncontrolsonar/worker_manager.py", line 80, in worker_loop
    await task
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/app/missioncontrol/services/kafka/message_producer.py", line 32, in process
    return self.kafka_producer.flush(self.flush_timeout_secs)
  File "/app/missioncontrol/services/kafka/message_producer.py", line 44, in kafka_producer
    retries=3
  File "/usr/local/lib/python3.6/site-packages/kafka/producer/kafka.py", line 362, in __init__
    **self.config)
  File "/usr/local/lib/python3.6/site-packages/kafka/client_async.py", line 219, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/usr/local/lib/python3.6/site-packages/kafka/client_async.py", line 819, in check_version
    raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
@tvoinarovskyi
Copy link
Collaborator

Did it work properly with kafka-python==1.4.1? Can you check?

@thspinto
Copy link
Author

Same error on 1.4.1. I was using version 1.3.5 before and also got this error.

@tvoinarovskyi
Copy link
Collaborator

Then you just don't have connection to the cluster. Even telneting to those ports should not work.

@HansG89
Copy link

HansG89 commented Jun 22, 2018

Same problem here when using kafka-python consumer in docker container.
On the host system everything is fine. But when running in docker python:3.6 container the connection fails with exactly the same error message.
I'm using the brokers IP address in bootstrap_servers parameter.
Pinging this address from within the container works. Also a telnet connection to the required port is possible.

@ox123
Copy link

ox123 commented Jul 1, 2018

So, is anybody who knows this reason and have any ideas to resolve it?

@HansG89
Copy link

HansG89 commented Jul 2, 2018

Which OS your docker is running on? I'm using Ubuntu 18.04 and I figured out that the problem has something to do with dockers bridged network mode. So basically it looks like a routing problem.
When I start my container in host network mode, everything works like a charm.

@bmamouri
Copy link

The error message is very misguiding. You need to specify the api_version in your consumer connection:

consumer = KafkaConsumer(
    bootstrap_servers=settings.message_hub.brokers,
    api_version=(0, 11, 5),  # MUST be specified or NoBrokersAvailable error would be raised
)

@jeffwidman
Copy link
Contributor

@bmamouri there is more context in #1308 (comment) and the rest of that ticket...

@bmamouri
Copy link

bmamouri commented Nov 29, 2018 •
edited
Loading

@jeffwidman, you are quite right. Specifying the api_version resolve the connection problem, but the consumer does not necessarily work. I read the ticket you provided, and although I am using 1.4.4 I still can't consume any messages.

@Hassan-404
Copy link

@jeffwidman, by using your fix I run the consumer but it stuck on fetching records from specified topic, What can I do?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jeffwidman @tvoinarovskyi @thspinto @bmamouri @ox123 @HansG89 @Hassan-404