Skip to content

Confluent-kafka-python, Tornado and GIL #100

@ghost

Description

Let's talk about usage of confluent-kafka-python Producer together with Tornado web server.

We know that Tornado is non blocking-server and has event loop for processing socket events.
We also know that librdkafka has several threads and its own event loop. And confluent-kafka-python is build on top of librdkafka.

When we call Producer.produce() from something inside Tornado it executes in main thread. Right? Then librdkafka pass data (with callback, probably) to another thread (event loop thread). And when message succefully recieved by Kafka, callback executed.

What happens when librdkafka call callback function from its event loop? Does it lock Python main thread with GIL?

Metadata

Metadata

Assignees

No one assigned

    Labels

    api-changeReferencing changes to the API speccomponent:producerIssues tied specifically to producer logic or code pathsenhancementRequesting a feature changestatus:wontfixFor issues being closed due to not being fixed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions