Skip to content

KafkaClient.send() always fails the first time, all subsequent send() calls succeed #1488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jeffwidman opened this issue May 9, 2018 · 1 comment
Assignees

Comments

@jeffwidman
Copy link
Contributor

jeffwidman commented May 9, 2018

The first time I call KafkaClient.send() it always immediately fails, even before I call poll(). Any subsequent calls will hang until I call poll(), then they will succeed.

This always happens on the first send() call to a new broker. So if send()/poll() are working as expected for node 1, and then I try sending to node 2, I will see the same behavior where the first call fails, then subsequent calls succeed.

Is this expected behavior or a bug?

Possibly related: #981 / #1278 / #1279

Example code:

import kafka.protocol as kp
from kafka.client_async import KafkaClient


kc = KafkaClient(bootstrap_servers=['kafka01.stg.local:9092'])

# first call to kc.send() always fails
gcr1 = kc.send(1, kp.commit.GroupCoordinatorRequest[0]('valid_group_name'))

gcr1.is_done  # always True even though I never called poll()
gcr1.failed()  # always True
gcr1.value  # `None`... the docs say this shold return a Response Struct or error?


# try it a second time:
gcr1 = kc.send(1, kp.commit.GroupCoordinatorRequest[0]('valid_group_name'))
gcr1.is_done  # always False until I call poll()

# will block until it gets a response... generally returns almost immediately
kc.poll(future=gcr1)

gcr1.value # always returns a Response struct
@dpkp
Copy link
Owner

dpkp commented May 25, 2018

This sounds correct to me. Until the node is connected / ready send will fail with NodeNotReadyError. You can check the exception on failure via gcr1.exception (not gcr1.value). This is because sends are still synchronous. See #981 re switching to fully async sends. Nonetheless, once the node is connected then the request can be sent and then you'll get a response future that requires calls to poll to pump the event loop for further processing.

@dpkp dpkp closed this as completed May 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants