Skip to content

[Messenger] Issues after migrating to AWS RabbitMQ #48241

Closed
php-amqp/php-amqp
#571
@xorik

Description

@xorik

Symfony version(s) affected

6.0

Description

We are switching from normal RabbitMQ server, to AWS-managed cluster.

After that periodically we have this error in log:

(AMQPException): Library error: a SSL error occurred

It happens on line $exchange->publish(...) in \Symfony\Component\Messenger\Bridge\Amqp\Transport\Connection::publishOnExchange

The problem is that after some time, the connection becomes "dead", but method Connection::clearWhenDisconnected doesn't detect is as disconnected.

It happened after 350 seconds, because they have some idle timeout.

I've provided a small script, that reproduces the issue: https://gist.github.com/xorik/24065aab0db094c99e746b277466fcba

Here's output:

Channel is connected!
Published!
Now wait for 360 seconds....
Channel is connected!
Exception (AMQPException): Library error: a SSL error occurred
#0 /tmp/1.php(37): AMQPExchange->publish('howdy-do', 'hello')
#1 {main}

I can give you credentials to a test server, if you want (PM https://twitter.com/xorik_dev/ or give me your contact).

I've tried to make persistent connection, adding heartbeat, but it won't help.

This is probably connected with #36538, but suggested fix doesn't help.

How to reproduce

  • Connect to a AWS RabbitMQ
  • publish a message
  • wait 350+ seconds
  • publish a message

Possible Solution

  • Actually the connection isn't 100% dead, so reconnect is not required and this code actually fixes the issue:
try {
    $exchange->publish($message, $routing_key);
} catch (\AMQPException) {
    $exchange->publish($message, $routing_key);
}
  • Alternative solution is perform some kind of ping or heartbeat, before publishing a message (improve method clearWhenDisconnected)

Additional Context

This is clearly not a Symfony issue, but I think Messenger should handle this issue, since this AWS RabbitMQ is quite popular, and there is no easy way to workaround it by developers, except creating a new transport and copy+pasting or extending bunch of Symfony classes.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions