Skip to content

HTTP Transport for Symfony Messenger #59901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Alidmo opened this issue Mar 3, 2025 · 11 comments
Closed

HTTP Transport for Symfony Messenger #59901

Alidmo opened this issue Mar 3, 2025 · 11 comments

Comments

@Alidmo
Copy link

Alidmo commented Mar 3, 2025

Description

I'd like to propose a new HTTP transport for Symfony Messenger to facilitate communication between microservices. After implementing a smaller, custom version in our project, I recognized the potential benefits of having this functionality in the core library—improving maintainability and scalability while avoiding domain-specific workarounds in a microservice architecture where all APIs are written in symfony.

Overview

This transport would allow microservices to communicate with a main application's Messenger as if it were local, using HTTP requests while preserving native trace and stamp functionality.

Key Features

  • HTTP-based communication between microservices and main application
  • Preservation of native Messenger traces and stamps across services
  • Ability to create stamps on the main application applicable to all APIs

Use Case

In a microservices architecture where:

  • A main application has Symfony Messenger installed and persists data
  • Other microservices also have Symfony Messenger installed
  • Microservices need to communicate with the main application's Messenger

Benefits

  1. Simplified communication between microservices
  2. Consistent Messenger experience across distributed systems
  3. Enhanced traceability and stamp functionality in a distributed environment

Questions

  1. Is this feature aligned with Symfony Messenger's roadmap?
  2. Are there any potential challenges or considerations we should discuss?
  3. Would you be interested in a pull request implementing this functionality?

I'm very excited and interested to provide more details or discuss this proposal further.

Example

No response

@ro0NL
Copy link
Contributor

ro0NL commented Mar 5, 2025

Do you have a POC in userland? I'd be curious how the HTTP routing is applied.

edit: i guess it can be done with stamps 😅

@Alidmo
Copy link
Author

Alidmo commented Mar 6, 2025

Do you have a POC in userland? I'd be curious how the HTTP routing is applied.

edit: i guess it can be done with stamps 😅

Yeah no but, we are going live with our own version at Oribi soon, but I keep concluding that this should've/could've been of a much better use in Symfony for everyone, and I need some reasons to live outside of work :)))

(For the routing, you can couple the extended messenger transport with a http service for now, then have your yaml pointing at an external dsn, not sure how it would be done by stamps, but ill nod and smile along like i know what the hell im doing anyway :))) )

edit 1: I don't have a POC in userland, I have one for the company, can make one pretty quick tho, love your cat.

@derrabus
Copy link
Member

derrabus commented Mar 6, 2025

I'm having a hard time understanding what this "HTTP transport" should do. How would I implement a worker that listens to this HTTP transport? If it's about signaling/triggering another service, wouldn't this go more into the direction of our RemoteEvent and Webhook components?

@Alidmo
Copy link
Author

Alidmo commented Mar 6, 2025

Well, this http transport, when in place and set for certain messages/transports, instead of passing this message internally in the messanger, will send them over to that dsn on http requests on another APIs messenger. (edit 2: while keeping the stamps/trace)

so kinda yes its a remote event, note that there's atm a gap in my knowledge how would RmoteEvent/Webhook components could work with messenger,

edit 1: I could be making a total fool of myself, but i don't really care about, please destroy my ego if im wrong, I don't think/know currently if this is possible with the messenger otherwise

@derrabus
Copy link
Member

derrabus commented Mar 6, 2025

you could destroy my ego as much as you want if im wrong

I don't know if there's been any doubt about that, but let me just make clear that it neither is nor has it ever been my intention to destroy anybody's ego with a GitHub issue comment. ☮

@Alidmo
Copy link
Author

Alidmo commented Mar 6, 2025

hehe, Just saying that I might have been all wrong from the beginning, like I didn't even know about these two components and if they could be implemented with the messenger allowing perservartion of the stamps/traces without having to share the knowledge between all micro services manually.

Could it ? isn't there a missing piece of jigsaw puzzle (radiohead vibes) here ?

Edit 1 : and I kinda built upon what was already existing, a standalone micro symfony messenger exposing http endpoints per fetch/ack/dispatch/consume, to which I added /reject

edit 2: that was an invitation to ask for your harshest comments on this, in peace ☮

@derrabus
Copy link
Member

derrabus commented Mar 6, 2025

Okay, but why don't you connect your microservices through a dedicated queue like for instance RabbitMQ then? What's the benefit of exposing an HTTP endpoint if we lose all the benefits of a queue if what we actually want is a queue?

@Alidmo
Copy link
Author

Alidmo commented Mar 6, 2025

I also thought that would've been a better use, but well, we have a series of external companies communicating through the messenger, that's one case, and second problem is that this has to have some domain/business logic involved, since its security and safety is regulated and some types of messages have to be encrypted.

in any scenario that you might decide to use symfony messenger between multiple symfony applications as the queue, this issue could solve the seemless integration problem, I think
edit 1: ^
edit 2: ^ this issue would allow symfony messenger to be more than just a queue in this scenario, to be a much more capable yet also customizable messaging system in a distributed system.

@ro0NL
Copy link
Contributor

ro0NL commented Mar 6, 2025

i considered this more from a query/sync POV btw.

so intead of working with a HTTP client, you'd dispatch a "request"

@Alidmo
Copy link
Author

Alidmo commented Mar 6, 2025

i considered this more from a query/sync POV btw.

so intead of working with a HTTP client, you'd dispatch a "request"

Yes, exactly, that's how we do it right now in our system. Everything that happens inside the messenger happens now through http requests between all applications, I've become bald (under investigation) but it works, the pain here is implementing new stamps and maintenance debt (is that even a thing)

So the thing is, in this scenario, you would cut the logic we have through the stamps and trace, and just communicate with the other symphony messenger through persisted messages, whereas if I had the knowledge transferred seamlessly (stamps/trace sent over, both messengers felt like they were in the same env, just communication partly through http) I wouldn't have to have distributed solutions for new stamps/rejection/failure or keeping workers alive and not terminated.(so dark).

Edit 1: I'm trying to convince myself in many ways that this is not a problem or something to solve, but I truly still believe this is indeed a feature that would improve the messenger's capabilities.

Edit 2: I'm still very enthusiastic about this, @derrabus if you think a "POC in userland" with some dummy functionality could help, please tell me, I can create one to showcase what I mean. Looking forward to it.

@Alidmo
Copy link
Author

Alidmo commented May 3, 2025

no interest at this point.

@Alidmo Alidmo closed this as not planned Won't fix, can't repro, duplicate, stale May 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants