Skip to content

[Messenger] rate limited transport processes more messages than configured #57230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fritzmg opened this issue May 29, 2024 · 2 comments
Closed

[Messenger] rate limited transport processes more messages than configured #57230

fritzmg opened this issue May 29, 2024 · 2 comments
Labels
Bug Messenger Status: Needs Review

Comments

@fritzmg
Copy link
Contributor

fritzmg commented May 29, 2024

Symfony version(s) affected

6.4.x

Description

I was testing a rate limited mailer messenger transport configuration and noticed that more messages were consumed by the messenger than the rate limit actually allows - at least with a certain configuration.

How to reproduce

Create a configuration that looks like this for example:

framework:
    mailer:
        dsn: '%env(MAILER_DSN)%'

    rate_limiter:
        mailer:
            policy: fixed_window
            limit: 1
            interval: 1 minute

    messenger:
        transports:
            mailer: 
                dsn: 'doctrine://default?queue_name=mailer'
                rate_limiter: mailer

        routing:
            'Symfony\Component\Mailer\Messenger\SendEmailMessage': mailer

i.e. only allow 1 email per minute. Then execute messenger:setup-transports.

Then create a bunch of emails in your system. Unfortunately you cannot use the mailer:test command for that as it directly sends the email rather than using the Mailer service to send it via the messenger bus - so you'll have to create your own command/service/code that sends a bunch of email messages via the Mailer.

Now execute messenger:consume mailer to keep consuming messages for the mailer messenger transport. You will notice that at first only one email is sent. However, after every 60 seconds two emails are sent after that, instead of the expected one email per minute.

Possible Solution

No response

Additional Context

No response
@fritzmg fritzmg added the Bug label May 29, 2024
@fritzmg fritzmg changed the title [Mailer] rate limited mailer sends more messages than configured [Messenger] rate limited transport processes more messages than configured May 30, 2024
@ghost
Copy link

ghost commented May 30, 2024

Please create a reproducer app with a numbered list of steps to execute (setup script would be even better) so others can just git clone and execute the use case.

@fritzmg
Copy link
Contributor Author

fritzmg commented May 31, 2024 •
edited
Loading

Reproduction:

  1. Create new directory and enter.
  2. git clone git@github.com:fritzmg/symfony-issue-57230.git . (https://github.com/fritzmg/symfony-issue-57230)
  3. composer install
  4. Create .env.local
  5. Configure MAILER_DSN in .env.local (e.g. a Mailhog instance or similar)
  6. Create a database and configure DATABASE_URL in .env.local
  7. bin/console messenger:setup-transports
  8. bin/console app:mailer:test (puts 10 test emails into the messenger queue)
  9. bin/console messenger:consume mailer
  • At first only one email will arrive.
  • 60 seconds later 2 emails will arrive instead of one.
  • 60 seconds later 2 emails will arrive instead of one.
  • etc.

@Jean-Beru Jean-Beru mentioned this issue Nov 5, 2024
Merged
fabpot added a commit that referenced this issue Nov 9, 2024
@fabpot
bug #58763 [Messenger][RateLimiter] fix additional message handled wh…
91acfa8 
…en using a rate limiter (Jean-Beru)

This PR was merged into the 6.4 branch.

Discussion
----------

[Messenger][RateLimiter] fix additional message handled when using a rate limiter

| Q             | A
| ------------- | ---
| Branch?       | 6.4
| Bug fix?      | yes
| New feature?  | no
| Deprecations? | no
| Issues        | Fix #57230
| License       | MIT

Fix additional message handled by Messenger when using a rate limiter. A token was reserved but not consumed. See #57230

With the following configuration:
```yaml
framework:
  rate_limiter:
    test:
      policy: 'fixed_window'
      limit: 1
      interval: '10 seconds'

  messenger:
    transports:
      test:
        dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
        rate_limiter: 'test'
    routing:
      'App\Messenger\DoSomething': 'test'
```

Log generated by the MessageHandler:
```bash
$ bin/console messenger:consume test

 [OK] Consuming messages from transport "test".

 // The worker will automatically exit once it has received a stop signal via the messenger:stop-workers command.

 // Quit the worker with CONTROL-C.

 // Re-run the command with a -vv option to see logs about consumed messages.

09:13:48 WARNING   [app] Message handled
09:13:58 WARNING   [app] Message handled
09:13:58 WARNING   [app] Message handled # Duplicated
09:14:08 WARNING   [app] Message handled
09:14:08 WARNING   [app] Message handled # Duplicated
09:14:18 WARNING   [app] Message handled
09:14:18 WARNING   [app] Message handled # Duplicated
```

After fix:
```bash
bin/console messenger:consume test

 [OK] Consuming messages from transport "test".

 // The worker will automatically exit once it has received a stop signal via the messenger:stop-workers command.

 // Quit the worker with CONTROL-C.

 // Re-run the command with a -vv option to see logs about consumed messages.

09:18:54 WARNING   [app] Message handled
09:19:04 WARNING   [app] Message handled
09:19:14 WARNING   [app] Message handled
09:19:24 WARNING   [app] Message handled
```

Commits
-------

ec1b999 [Messenger][RateLimiter] fix additional message handled when using a rate limiter
@fabpot fabpot closed this as completed Nov 9, 2024
@xabbuh xabbuh added Messenger and removed Mailer labels Nov 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Messenger Status: Needs Review
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@fabpot @xabbuh @fritzmg @carsonbot