Skip to content

[Messenger] Queue worker crashing on MessageDecodingFailedException #58510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
oniric85 opened this issue Oct 9, 2024 · 7 comments
Open

[Messenger] Queue worker crashing on MessageDecodingFailedException #58510

oniric85 opened this issue Oct 9, 2024 · 7 comments
Labels
Bug Messenger Status: Needs Review Status: Waiting feedback

Comments

@oniric85
Copy link

oniric85 commented Oct 9, 2024 •
edited
Loading

Symfony version(s) affected

7.1.3

Description

Symfony Messenger queue workers crash when a MessageDecodingFailedException is thrown.

How to reproduce

  • Stop your queue workers
  • Wrap an object instance in an Envelope and dispatch that object instance through Symfony Messenger
  • Change or rename the object class so the worker can't unserialize it any more
  • Start the worker, observe that it crashes

Possible Solution

Avoid terminating/crashing the worker when a MessageDecodingFailedException occurs. Not sure at which level catching the exception would be more appropriate though.

Additional Context

We are using Symfony Messenger to produce and consume messages from a MySQL 8 backed queue. We have implemented auto-restarting mechanism for queue workers of course but we are facing now a bit of a conundrum when dealing with MessageDecodingFailedException scenarios.

In some situations, where a burst of malformed messages appears, this can lead to the queue workers to crash and restart several times in a short amount of time reaching the limits configured for the systemd unit. While typically the policy we have in place is good for other scenarios it doesn't work very well in this case.

In this scenario, simply rejecting the message like it is done now and going on with the rest of the queue would seem like the most sane thing to do. It would also have the added benefit of reducing any negative impact on performance created by restarting the worker.

Worth noting that we are not using Symfony Messenger as part of a Symfony full installation but as a separate component though this seems like the expected behavior regardless of the way the component is used.

This is the stack trace:

In PhpSerializer.php line 121:
 [Symfony\Component\Messenger\Exception\MessageDecodingFailedException]
 Message class "XXX" not found during decoding.

 Exception trace:
   at /app/vendor/symfony/messenger/Transport/Serialization/PhpSerializer.php:121
  Symfony\Component\Messenger\Transport\Serialization\PhpSerializer::handleUnserializeCallback() at n/a:n/a
  unserialize() at /app/vendor/symfony/messenger/Transport/Serialization/PhpSerializer.php:93
  Symfony\Component\Messenger\Transport\Serialization\PhpSerializer->safelyUnserialize() at /app/vendor/symfony/messenger/Transport/Serialization/PhpSerializer.php:54
  Symfony\Component\Messenger\Transport\Serialization\PhpSerializer->decode() at /app/vendor/symfony/doctrine-messenger/Transport/DoctrineReceiver.php:134
  Symfony\Component\Messenger\Bridge\Doctrine\Transport\DoctrineReceiver->createEnvelopeFromData() at /app/vendor/symfony/doctrine-messenger/Transport/DoctrineReceiver.php:65
  Symfony\Component\Messenger\Bridge\Doctrine\Transport\DoctrineReceiver->get() at /app/vendor/symfony/doctrine-messenger/Transport/DoctrineTransport.php:40
  Symfony\Component\Messenger\Bridge\Doctrine\Transport\DoctrineTransport->get() at /app/vendor/symfony/messenger/Worker.php:102
  Symfony\Component\Messenger\Worker->run() at /app/vendor/symfony/messenger/Command/ConsumeMessagesCommand.php:244
  Symfony\Component\Messenger\Command\ConsumeMessagesCommand->execute() at /app/vendor/symfony/console/Command/Command.php:279
  Symfony\Component\Console\Command\Command->run() at /app/vendor/symfony/console/Application.php:1047
  Symfony\Component\Console\Application->doRunCommand() at /app/vendor/symfony/console/Application.php:316
  Symfony\Component\Console\Application->doRun() at /app/vendor/symfony/console/Application.php:167
  Symfony\Component\Console\Application->run() at /app/bin/console:40
@oniric85 oniric85 added the Bug label Oct 9, 2024
@xabbuh
Copy link
Member

xabbuh commented Oct 9, 2024

Can you please share the exception message and its stack trace that crashes your application?

@oniric85
Copy link
Author

Can you please share the exception message and its stack trace that crashes your application?

I've added the stack trace in the initial report.

@oniric85
Copy link
Author

@xabbuh added the stack trace and message. Does the ticket need more details?

@oniric85
Copy link
Author

shouldn't you avoid a burst of malformed messages by other means?

You are probably right with this assumption, but wouldn't it still make sense for the consumer of the messages to log the error, mark the message as failed and move on without erroring out?

@slknijnenburg
Copy link

Chiming in on this, I think there would be value in at least having an option to route messages that cannot be decoded to the failure transport, so that these can be manually inspected/processed. Right now there doesn't seem to be a viable way to do this, at least not when using the PhpSerializer, except for subclassing the Worker or Transport classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Messenger Status: Needs Review Status: Waiting feedback
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@slknijnenburg @xabbuh @oniric85 @carsonbot