Skip to content

[Messenger][RFC] Kafka transport #35549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JustBlackBird opened this issue Feb 1, 2020 · 15 comments
Open

[Messenger][RFC] Kafka transport #35549

JustBlackBird opened this issue Feb 1, 2020 · 15 comments

Comments

@JustBlackBird
Copy link
Contributor

Description

At the moment the recommended way to interact with kafka from symfony messenger is using a bridge to enqueue. This is overcomplecated way and lacks of some features (e.g. one cannot easily set up serializer for enqueue's kafka transport via bundle config).

At the same time, an implementation of kafka transport without enqueue is quite simple, so may be it should be added to the messenger component (like AMQP, Doctrine and Redis)?

If so, I could try to implement kafka transport.

@alsar
Copy link

alsar commented Feb 3, 2020

@JustBlackBird have a look at this: https://github.com/KonstantinCodes/messenger-kafka
Would be great to have the Kafka as an official messenger transport.

@stof
Copy link
Member

stof commented Feb 11, 2020

Given that the implementation of the transport on top of ext-rdkafka is quite simple, I think it might indeed belong to the core, as a new messenger bridge.

@Steveb-p
Copy link
Contributor

Is anyone currently working on this, or can I take a shot at this?

@Steveb-p
Copy link
Contributor

Since no one responded I'll assume no 😄 I'll be providing a PoC PR during the weekend.

@doranych
Copy link

Is there any news about this topic?

@Steveb-p
Copy link
Contributor

Is there any news about this topic?

Sorry, there is always something else that occupies me 😞

Give me two days for it 😄

@KonstantinCodes
Copy link
Contributor

KonstantinCodes commented Dec 27, 2020

@Steveb-p Maybe I can just modify https://github.com/KonstantinCodes/messenger-kafka to work as a bridge?

Currently it's set up as a symfony-bundle but I'd be happy the change that :)

@Steveb-p
Copy link
Contributor

@KonstantinCodes yeah, that would work. I see you've been busy with this bundle and I haven't really done any Kafka related stuff since I've switched jobs, so 👍

@KonstantinCodes
Copy link
Contributor

@Steveb-p Cool, I opened a PR. Would it be possible that you take a look if it's okay, please?

@carsonbot
Copy link

Thank you for this suggestion.
There has not been a lot of activity here for a while. Would you still like to see this feature?

@gayansanjeewa
Copy link

Hey! yeah I'm one of the people (I'm sure there are many others) who got really frustrated about trying to set up the officially suggested enqueue transport. I really don't think there are good enough online materials to set it successfully with Kafka.

So I got really glad to see this issue is open. I'd just like to know what's the current progress of this and yeah @KonstantinCodes if you need any support to move your PR, I'd more than happy to help 🙂

@99hops
Copy link

99hops commented Aug 9, 2023

Hello guys, it's Aug 2023 and we are still anticipating on this one as a better alternative to RabbitMQ, please revisit this as it will be very much appreciated in our community. Thanks @stof @derrabus @nicolas-grekas ❤️

@xabbuh
Copy link
Member

xabbuh commented Aug 9, 2023

@nforced You can help by reviewing and testing #51070.

@AlexOstrovsky
Copy link

is there any update on this?

@chalasr
Copy link
Member

chalasr commented Apr 6, 2024

@AlexOstrovsky As Christian said above in #35549 (comment), anyone can help moving forward by participing to the discussion on the proposed implementation (reviewing, testing, suggesting improvements...).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.