Hey!

I think @tienvx has recently worked with this code. Maybe they can help review this?

Cheers!

Carsonbot

Any update on this?

👍 on rate limiting for messenger transports 🙂

When using the Messenger component in order to asynchronously send emails via the Mailer component being able to limit the total amount of emails per time unit would be really helpful in hosting environments that impose a limit on how many emails you can send via their SMTP per hour or day for instance.

I just rebased the changed on the latest 6.1 branch, but I'm really looking for some input on whether these changes are actually going in the right direction (and will be accepted) before I will be spending some time on tests.

any news on this?

@nicolas-grekas Requested adjustments have been made, and I added two tests.

(the failed checks do not seem to be related to this change)

Thank you @bobvandevijver.

@bobvandevijver I'm trying out the messenger/rate limiter integration in 6.2 RC1, it seems like I have a race condition when consuming a rate limited queue (RabbitMq) with multiple consumers. In the setup below, instead of 1 email every 2 secs, I get multiple emails/sec processed.

framework:
  rate_limiter:
    email_output_limit:
      #Office 365 email limit is 30 mails per minute (per account)
      policy: 'fixed_window'
      limit: 1
      interval: '2 seconds'

messenger config:

Email_Job:
    dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
    failure_transport: failed_output
    rate_limiter: email_output_limit
    options:
        queues:
            email_job:
                arguments:
                    x-max-priority: 10
        exchange:
            name: email_job
            type: direct
    retry_strategy:
        max_retries: 3
        multiplier: 2

I have a lock setup as below, maybe this is the issue? Or perhaps there is some relation between the number of workers and the limits to be applied

###> symfony/lock ###
# Choose one of the stores below
LOCK_DSN='sqlite:///%kernel.project_dir%/var/lock.db'
###

This is all running on windows server 2019 😞
I've also tried token_bucket and mysql. Is it correct not to see a value in the db for tokens when the consumers are running? I assumed there would be an entry of some sort, but seems like an empty table

@gavinerickson I'm no expect on the implementation of the rate limiter/lock components, but I believe you might be using the wrong lock store for what you're looking as the DoctrineDbal lock store does not support blocking locks (see this table). Maybe you could try whether it works using a Flock based store, as you're using a local based lock anyways looking at the DSN you shared?

Otherwise I would recommend opening an issue (you can tag me with it) to discuss the issue further.

@bobvandevijver I'd tried flock initially too (and just sanity checked) - that's what I've used with rate limiter previously. I'll open an issue.