[Messenger] Be able to get raw data when a message in not decodable by the PHP Serializer #39622

lyrixx · 2020-12-23T16:44:06Z

Q	A
Branch?	6.2
Bug fix?	yes
New feature?	yes
Deprecations?	no
Tickets
License	MIT
Doc PR

It's easier to review the pr with ?w=1

Why?

This PR aimed to handle properly messaged that are already failed and also not decodable anymore.

This use case occurs when a Message class is renamed / moved to another namespace

Reproducer

Publish many message of type App\Message\Foobar
Consume them all, but throw an Exception exception in the handler
Renamed Foobar to Foo

Before / After

command `messenger:failed:show`

Before

Without an ID

…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:show
There are 94 messages pending in the failure transport.

In PhpSerializer.php line 84:

  Message class "App\Message\Foobar" not found during decoding.


messenger:failed:show [--max MAX] [--transport [TRANSPORT]] [--stats] [--class-filter CLASS-FILTER] [--] [<id>]

And this message is lost

With an ID

>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:show 68
There are 93 messages pending in the failure transport.

In PhpSerializer.php line 84:

  Message class "App\Message\Foobar" not found during decoding.


messenger:failed:show [--max MAX] [--transport [TRANSPORT]] [--stats] [--class-filter CLASS-FILTER] [--] [<id>]

Message lost too

After

Without an ID

>…goire/dev/labs/symfony/symfony-5.2(remaned) bin/console  messenger:failed:show
There are 94 messages pending in the failure transport.
 ----- ------------------------ --------------------- ---------------------------------------
  Id    Class                    Failed at             Error
 ----- ------------------------ --------------------- ---------------------------------------
  62    __PHP_Incomplete_Class   2022-08-12 16:26:06   FoobarHandler is not implemented yet.
  63    __PHP_Incomplete_Class   2022-08-12 16:26:06   FoobarHandler is not implemented yet.
  64    __PHP_Incomplete_Class   2022-08-12 16:26:06   FoobarHandler is not implemented yet.

 // Showing first 50 messages.

 // Run messenger:failed:show {id} --transport=failed -vv to see message details.

No messages are lost

With an ID

>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:show 69
There are 92 messages pending in the failure transport.

Failed Message Details
======================

 ------------- ---------------------------------------
  Class         __PHP_Incomplete_Class
  Message Id    69
  Failed at     2022-08-12 16:26:06
  Error         FoobarHandler is not implemented yet.
  Error Code    0
  Error Class   Exception
  Transport     async
 ------------- ---------------------------------------

 Message history:
  * Message failed at 2022-08-12 16:25:58 and was redelivered
  * Message failed at 2022-08-12 16:25:59 and was redelivered
  * Message failed at 2022-08-12 16:26:02 and was redelivered
  * Message failed at 2022-08-12 16:26:06 and was redelivered


 [ERROR] The message could not be decoded.


 Re-run command with -vv to see more message & error details.

 Run messenger:failed:retry 69 --transport=failed to retry this message.
 Run messenger:failed:remove 69 --transport=failed to delete it.

messages not lost

With an ID an -vv

>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:show 69 -vv
There are 92 messages pending in the failure transport.

Failed Message Details
======================

 ------------- ---------------------------------------
  Class         __PHP_Incomplete_Class
  Message Id    69
  Failed at     2022-08-12 16:26:06
  Error         FoobarHandler is not implemented yet.
  Error Code    0
  Error Class   Exception
  Transport     async
 ------------- ---------------------------------------

 Message history:
  * Message failed at 2022-08-12 16:25:58 and was redelivered
  * Message failed at 2022-08-12 16:25:59 and was redelivered
  * Message failed at 2022-08-12 16:26:02 and was redelivered
  * Message failed at 2022-08-12 16:26:06 and was redelivered

Message:
========


 [ERROR] The message could not be decoded. See below an APPROXIMATIVE representation of the class.


__PHP_Incomplete_Class(App\Message\Foobar) {}

Exception:
==========

Exception^ {
  message: "FoobarHandler is not implemented yet."
  code: 0
  file: "/home/gregoire/dev/labs/symfony/messenger-class-not-found/src/MessageHandler/FoobarHandler.php"
  line: 12
  trace: {
    /home/gregoire/dev/labs/symfony/messenger-class-not-found/src/MessageHandler/FoobarHandler.php:12
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Middleware/HandleMessageMiddleware.php:95 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Middleware/SendMessageMiddleware.php:71 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Middleware/FailedMessageProcessingMiddleware.php:34 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Middleware/DispatchAfterCurrentBusMiddleware.php:68 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Middleware/RejectRedeliveredMessageMiddleware.php:41 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Middleware/AddBusNameStampMiddleware.php:37 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Middleware/TraceableMiddleware.php:43 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/MessageBus.php:73 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/TraceableMessageBus.php:41 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/RoutableMessageBus.php:54 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Worker.php:156 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Worker.php:105 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Messenger/Command/ConsumeMessagesCommand.php:223 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Console/Command/Command.php:309 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Console/Application.php:1019 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:94 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Console/Application.php:300 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:80 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Console/Application.php:172 { …}
    /home/gregoire/dev/github.com/lyrixx/symfony/src/Symfony/Component/Runtime/Runner/Symfony/ConsoleApplicationRunner.php:54 { …}
    ./vendor/autoload_runtime.php:29 { …}
    ./bin/console:11 { …}
  }
}

 Run messenger:failed:retry 69 --transport=failed to retry this message.
 Run messenger:failed:remove 69 --transport=failed to delete it.

screenshot with colors

messages not lost

command `messenger:failed:retry`

Before

>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:retry 69

 // Quit this command with CONTROL-C.

 // Re-run the command with a -vv option to see logs about consumed messages.

There are 92 messages pending in the failure transport.
To retry all the messages, run messenger:consume failed

In PhpSerializer.php line 84:

  Message class "App\Message\Foobar" not found during decoding.


messenger:failed:retry [--force] [--transport [TRANSPORT]] [--] [<id>...]

And the first message is lost

Same if I use an ID

After

Without ID

>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:retry

 // Quit this command with CONTROL-C.

 // Re-run the command with a -vv option to see logs about consumed messages.

There are 91 messages pending in the failure transport.
To retry all the messages, run messenger:consume failed

Failed Message Details
======================

 ------------- ---------------------------------------
  Class         __PHP_Incomplete_Class
  Message Id    63
  Failed at     2022-08-12 16:26:06
  Error         FoobarHandler is not implemented yet.
  Error Code    0
  Error Class   Exception
  Transport     async
 ------------- ---------------------------------------

 Message history:
  * Message failed at 2022-08-12 16:25:58 and was redelivered
  * Message failed at 2022-08-12 16:25:59 and was redelivered
  * Message failed at 2022-08-12 16:26:02 and was redelivered
  * Message failed at 2022-08-12 16:26:06 and was redelivered


 [ERROR] The message could not be decoded.


 Re-run command with -vv to see more message & error details.

In FailedMessagesRetryCommand.php line 176:

  The message with id "63" could not decoded, it can only be shown or removed.


messenger:failed:retry [--force] [--transport [TRANSPORT]] [--] [<id>...]

And the message is not lost

With an ID

>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:retry  63

 // Quit this command with CONTROL-C.

 // Re-run the command with a -vv option to see logs about consumed messages.

There are 91 messages pending in the failure transport.
To retry all the messages, run messenger:consume failed

Failed Message Details
======================

 ------------- ---------------------------------------
  Class         __PHP_Incomplete_Class
  Message Id    63
  Failed at     2022-08-12 16:26:06
  Error         FoobarHandler is not implemented yet.
  Error Code    0
  Error Class   Exception
  Transport     async
 ------------- ---------------------------------------

 Message history:
  * Message failed at 2022-08-12 16:25:58 and was redelivered
  * Message failed at 2022-08-12 16:25:59 and was redelivered
  * Message failed at 2022-08-12 16:26:02 and was redelivered
  * Message failed at 2022-08-12 16:26:06 and was redelivered


 [ERROR] The message could not be decoded.


 Re-run command with -vv to see more message & error details.

In FailedMessagesRetryCommand.php line 176:

  The message with id "63" could not decoded, it can only be shown or removed.


messenger:failed:retry [--force] [--transport [TRANSPORT]] [--] [<id>...]

command `messenger:failed:remove`

Before

>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:remove  63

In PhpSerializer.php line 84:

  Message class "App\Message\Foobar" not found during decoding.


messenger:failed:remove [--force] [--transport [TRANSPORT]] [--show-messages] [--] <id>...

The message is lost, but it's not an issue I guess :p

After

>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:remove  69


 [ERROR] The message with id "69" was not found.


>…abs/symfony/messenger-class-not-found(new) bin/console  messenger:failed:remove  79

Failed Message Details
======================

 ------------- ---------------------------------------
  Class         __PHP_Incomplete_Class
  Message Id    79
  Failed at     2022-08-12 16:26:06
  Error         FoobarHandler is not implemented yet.
  Error Code    0
  Error Class   Exception
  Transport     async
 ------------- ---------------------------------------

 Message history:
  * Message failed at 2022-08-12 16:25:58 and was redelivered
  * Message failed at 2022-08-12 16:25:59 and was redelivered
  * Message failed at 2022-08-12 16:26:02 and was redelivered
  * Message failed at 2022-08-12 16:26:06 and was redelivered


 [ERROR] The message could not be decoded.


 Re-run command with -vv to see more message & error details.

 Do you want to permanently remove this message? (yes/no) [no]:
 > yes


 [OK] Message with id 79 removed.

Extra

`messenger:consume` command

This command is not fixed on purpose: We keep the default behavior to keep a BC.

BUT I would like to use the same stamp system I introduce to avoid loosing a message.
Because ATM, when a DecodeException is thrown, the message is lost. Not cool!

>…goire/dev/labs/symfony/symfony-5.2(remaned) bin/console  messenger:consume failed


 [OK] Consuming messages from transports "failed".


 // The worker will automatically exit once it has received a stop signal via the messenger:stop-workers command.

 // Quit the worker with CONTROL-C.

 // Re-run the command with a -vv option to see logs about consumed messages.


In PhpSerializer.php line 92:

  Message class "App\Message\Foobar" not found during decoding.

src/Symfony/Component/Messenger/Transport/Serialization/PhpSerializer.php

src/Symfony/Component/Messenger/Command/FailedMessagesRetryCommand.php

src/Symfony/Component/Messenger/Command/FailedMessagesRemoveCommand.php

src/Symfony/Component/Messenger/Command/FailedMessagesShowCommand.php

lyrixx · 2021-05-10T14:46:39Z

ping @symfony/mergers : What do you think about this PR? IMHO this is really needed to be able to explore "dead" messages.

And what do you think about the new MessageDecodingFailedStamp class. IMHO, this is cleaner than before and more flexible.

Should I address comments and add some tests, or we gonna close it anyway?

maxhelias · 2021-05-19T07:50:04Z

Indeed, this could be very useful

nicolas-grekas · 2021-10-06T16:45:57Z

before reviewing, do you still want this in core? can you please rebase + fix review comments if yes?

lyrixx · 2021-10-06T17:28:15Z

Yes it would be very useful!

I'll rebase it ASAP 👍🏼

lyrixx · 2022-08-13T09:40:35Z

I rebased the PR. and added some tests.

src/Symfony/Component/Messenger/Command/FailedMessagesRetryCommand.php

src/Symfony/Component/Messenger/Transport/Serialization/PhpSerializer.php

src/Symfony/Component/Messenger/Command/FailedMessagesShowCommand.php

nicolas-grekas

LGTM after two minor CS changes. Thanks.

src/Symfony/Component/Messenger/Command/FailedMessagesRemoveCommand.php

…y the PHP Serializer

lyrixx · 2022-08-30T14:33:07Z

@nicolas-grekas I addressed your comments, replay all examples, and updated the PR description. I think it's OK

fabpot · 2022-09-01T02:09:39Z

Thank you @lyrixx.

This PR was merged into the 6.2 branch. Discussion ---------- Fix typo in method retrySpecificEnvelopes | Q | A | ------------- | --- | Branch? | 6.2 | Bug fix? | yes | New feature? | no | Deprecations? | no | Tickets | Fix for #39622 | License | MIT Commits ------- e5675ac fix typo in method retrySpecificEnvelopes

… of passing boolean flags (xabbuh) This PR was merged into the 6.2 branch. Discussion ---------- [Messenger] add dedicated method for disabling instead of passing boolean flags | Q | A | ------------- | --- | Branch? | 6.2 | Bug fix? | no | New feature? | yes (changes #39622) | Deprecations? | no | Tickets | | License | MIT | Doc PR | Commits ------- 99fac74 add dedicated method for disabling instead of passing boolean flags

jbdelhommeau · 2022-10-21T10:10:25Z

It will be planned to backport to 6.1 ?
We have this problem with the 5.4 and 6.1 !

OskarStark · 2022-10-21T11:31:49Z

No, it will be released in 6.2 only

lyrixx requested a review from sroze as a code owner December 23, 2020 16:44

carsonbot added Status: Needs Review Messenger Bug Feature labels Dec 23, 2020

lyrixx force-pushed the messenger-and-class-moved branch from 3423f53 to ca0627b Compare December 23, 2020 16:46

Korbeil reviewed Dec 23, 2020

View reviewed changes

src/Symfony/Component/Messenger/Transport/Serialization/PhpSerializer.php Outdated Show resolved Hide resolved

lyrixx force-pushed the messenger-and-class-moved branch from ca0627b to 25ce8d1 Compare December 23, 2020 18:01

nicolas-grekas added this to the 5.x milestone Dec 28, 2020

OskarStark reviewed Jan 30, 2021

View reviewed changes

fabpot modified the milestones: 5.4, 6.1 Oct 30, 2021

fabpot modified the milestones: 6.1, 6.2 May 20, 2022

lyrixx mentioned this pull request Jul 20, 2022

[Messenger] catch all exceptions while deserializing a message #46996

Closed

lyrixx force-pushed the messenger-and-class-moved branch 4 times, most recently from 6d37c94 to 292d6cf Compare August 13, 2022 09:05

lyrixx force-pushed the messenger-and-class-moved branch 2 times, most recently from 4c0d36f to 6e3065a Compare August 16, 2022 08:28

lyrixx requested review from OskarStark and removed request for sroze August 17, 2022 08:23

nicolas-grekas reviewed Aug 17, 2022

View reviewed changes

src/Symfony/Component/Messenger/Command/FailedMessagesRetryCommand.php Outdated Show resolved Hide resolved

src/Symfony/Component/Messenger/Transport/Serialization/PhpSerializer.php Outdated Show resolved Hide resolved

lyrixx force-pushed the messenger-and-class-moved branch from 6e3065a to 8ae4ed5 Compare August 30, 2022 13:44

nicolas-grekas reviewed Aug 30, 2022

View reviewed changes

src/Symfony/Component/Messenger/Command/FailedMessagesShowCommand.php Outdated Show resolved Hide resolved

nicolas-grekas approved these changes Aug 30, 2022

View reviewed changes

src/Symfony/Component/Messenger/Command/FailedMessagesRemoveCommand.php Outdated Show resolved Hide resolved

carsonbot added Status: Reviewed and removed Status: Needs Review labels Aug 30, 2022

lyrixx mentioned this pull request Aug 30, 2022

[Messenger] Message is lost if it cannot be processed by both handler and failed transport #36870

Closed

[Messenger] Be able to get raw data when a message in not decodable b…

34229af

…y the PHP Serializer

lyrixx force-pushed the messenger-and-class-moved branch from 8ae4ed5 to 34229af Compare August 30, 2022 14:10

fabpot approved these changes Sep 1, 2022

View reviewed changes

fabpot mentioned this pull request Sep 1, 2022

[Messenger] Be able to get raw data when a message in not decodable by … symfony/symfony-docs#17243

Closed

fabpot merged commit e788a49 into symfony:6.2 Sep 1, 2022

lyrixx deleted the messenger-and-class-moved branch September 1, 2022 07:11

Jeroeny mentioned this pull request Sep 1, 2022

Fix typo in method retrySpecificEnvelopes #47449

Merged

xabbuh mentioned this pull request Sep 2, 2022

[Messenger] add dedicated method for disabling instead of passing boolean flags #47460

Merged

fabpot mentioned this pull request Oct 24, 2022

Release v6.2.0-BETA1 #47972

Merged

lyrixx mentioned this pull request Feb 8, 2023

[Messenger] MessageDecodingFailedException should not delete message from queue #44117

Open

alex-dev mentioned this pull request Apr 17, 2023

[Messenger] Proper message decoding error handling #50043

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Messenger] Be able to get raw data when a message in not decodable by the PHP Serializer #39622

[Messenger] Be able to get raw data when a message in not decodable by the PHP Serializer #39622

lyrixx commented Dec 23, 2020 •

edited

Loading

lyrixx commented May 10, 2021

maxhelias commented May 19, 2021

nicolas-grekas commented Oct 6, 2021

lyrixx commented Oct 6, 2021

lyrixx commented Aug 13, 2022

nicolas-grekas left a comment

lyrixx commented Aug 30, 2022

fabpot commented Sep 1, 2022

jbdelhommeau commented Oct 21, 2022

OskarStark commented Oct 21, 2022

[Messenger] Be able to get raw data when a message in not decodable by the PHP Serializer #39622

[Messenger] Be able to get raw data when a message in not decodable by the PHP Serializer #39622

Conversation

lyrixx commented Dec 23, 2020 • edited Loading

Why?

Reproducer

Before / After

command messenger:failed:show

Before

Without an ID

With an ID

After

Without an ID

With an ID

With an ID an -vv

command messenger:failed:retry

Before

After

Without ID

With an ID

command messenger:failed:remove

Before

After

Extra

messenger:consume command

lyrixx commented May 10, 2021

maxhelias commented May 19, 2021

nicolas-grekas commented Oct 6, 2021

lyrixx commented Oct 6, 2021

lyrixx commented Aug 13, 2022

nicolas-grekas left a comment

Choose a reason for hiding this comment

lyrixx commented Aug 30, 2022

fabpot commented Sep 1, 2022

jbdelhommeau commented Oct 21, 2022

OskarStark commented Oct 21, 2022

lyrixx commented Dec 23, 2020 •

edited

Loading

command `messenger:failed:show`

command `messenger:failed:retry`

command `messenger:failed:remove`

`messenger:consume` command