Skip to content

[Messenger][Redis] Worker stops handling messages on first empty message #48166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jvmanji opened this issue Nov 9, 2022 · 5 comments
Closed

Comments

@jvmanji
Copy link
Contributor

jvmanji commented Nov 9, 2022

Symfony version(s) affected

5.4, 6.* and probably others

Description

We're using messenger with redis transport in one of our projects. On one of our staging environments, we trimmed one queue (probably in parallel, the consumer of that queue failed). This led us to a strange situation, having a message in XPENDING while not having it in the corresponding stream. As only IDs are being stored in XPENDING, while consuming, messenger can't decode the body. The whole problem is of course our fault and most probably should not happen, anyway the way messenger behaves is weird – it is returning an empty array, like there are no messages in the queue, while not XACK-ing that message, so it hangs in a strange loop. In my opinion, it should either XACK that message or throw an exception with enough information so that someone using the library can handle it manually. It took us a long time to discover what was going on, while messenger was hung in a strange situation, not consuming messages from the queue – because there are no errors being logged anywhere.

How to reproduce

I don't have a quick way to reproduce, but the problem lies in RedisReceiver.php:

    public function get(): iterable
    {
        $message = $this->connection->get();

        if (null === $message) {
            return [];
        }

        $redisEnvelope = json_decode($message['data']['message'] ?? '', true);

        if (null === $redisEnvelope) {
            return [];
        }

In my opinion, when $redisEnvelope is null, the library should either throw an exception with message ID or ACK this message, so it won't be handled any more.

Possible Solution

No response

Additional Context

No response

@jvmanji jvmanji added the Bug label Nov 9, 2022
@OskarStark OskarStark changed the title [MESSENGER][REDIS] Worker stops handling messages on first empty message [Messenger][Redis] Worker stops handling messages on first empty message Nov 17, 2022
@kayue
Copy link

kayue commented Dec 23, 2022

I ran into the same issue as well.

Basically Symfony uses XREADGROUP command to ensure the same message won't consume by more than one consumer

XREADGROUP GROUP symfony consumer COUNT 1 STREAMS messages 1 

If this command return null then it will completely block Symfony's queue due to this issue. Solution is to manually XACK it.

XACK messages symfony 1671011783792-0

@kayue
Copy link

kayue commented Dec 27, 2022

This is what the nil message look like

XREADGROUP GROUP symfony consumer COUNT 1 STREAMS messages 1 
1) 1) "messages"
   2) 1) 1) "1672049648252-0"
         2) (nil)

@chalasr
Copy link
Member

chalasr commented Dec 27, 2022

/cc @alexander-schranz

@alexander-schranz
Copy link
Contributor

That is an interesting case. If the message really does not longer exist but returns no content I think its fine to do a xdel on it, strange that redis still has a reference to it if xtrim did remove it, so it also could be a upstream issue of Redis itself.

@jvmanji Can you create a reproducable test case I think it should be possible inside the RedisExtIntegrationTest to create there a reproducer for the issue: https://github.com/symfony/symfony/blob/5.4/src/Symfony/Component/Messenger/Bridge/Redis/Tests/Transport/RedisExtIntegrationTest.php. There you can put messages into it, put it into pending state and do a xtrim what you did tell us did cause your issue.

jvmanji added a commit to jvmanji/symfony that referenced this issue Dec 31, 2022
jvmanji added a commit to jvmanji/symfony that referenced this issue Dec 31, 2022
jvmanji added a commit to jvmanji/symfony that referenced this issue Dec 31, 2022
jvmanji added a commit to jvmanji/symfony that referenced this issue Dec 31, 2022
@jvmanji
Copy link
Contributor Author

jvmanji commented Dec 31, 2022

@alexander-schranz, I added new test and also simple fix. If you'd comment out changes I did in src/Symfony/Component/Messenger/Bridge/Redis/Transport/RedisReceiver.php then the test will fail. Please let me know in the PR if you'd change anything.

jvmanji added a commit to jvmanji/symfony that referenced this issue Dec 31, 2022
jvmanji added a commit to jvmanji/symfony that referenced this issue Jan 2, 2023
nicolas-grekas added a commit that referenced this issue Apr 18, 2023
…ing messages on first empty message (jvmanji)

This PR was merged into the 5.4 branch.

Discussion
----------

[Messenger] [Redis] Fixed problem where worker stops handling messages on first empty message

| Q             | A
| ------------- | ---
| Branch?       |5.4 <!-- see below -->
| Bug fix?      | yes
| New feature?  | no <!-- please update src/**/CHANGELOG.md files -->
| Deprecations? |  no <!-- please update UPGRADE-*.md and src/**/CHANGELOG.md files -->
| Tickets       | Fix #48166 <!-- prefix each issue number with "Fix #", no need to create an issue if none exists, explain below instead -->
| License       | MIT
| Doc PR        | n/a <!-- required for new features -->

Fixed problem where worker stops handling messages on first empty message.

Commits
-------

ce103f1 [Messenger] [Redis] Fixed problem where worker stops handling messages on first empty message
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants