Skip to content

Message creation, encoding, and decoding performance #1210

Closed
@shargan

Description

@shargan

I'm working on a fairly-high-throughput Python project (150-200k msg/s to start) that uses the kafka-python protocol via aiokafka. While profiling to determine where our bottlenecks are, Message instantiation, encoding, and decoding stand out as areas ripe for improvement.

This is an area I'd like to contribute, but the design of kafka.protocol seems more oriented towards being a readable reference implementation than a high-performance implementation. Are updates in this area desirable even if they lead to superficially uglier code? Likewise, how important is backwards compatibility to you?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions