Skip to content

Idle workers eventually exhaust memory and crash #45386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Bilge opened this issue Feb 10, 2022 · 16 comments
Closed

Idle workers eventually exhaust memory and crash #45386

Bilge opened this issue Feb 10, 2022 · 16 comments
Labels
Bug Stalled Status: Needs Review

Comments

@Bilge
Copy link
Contributor

Bilge commented Feb 10, 2022 •
edited
Loading

Symfony version(s) affected

5.4.4

Description

An idle worker subscribed to one queue and processing no messages, running via bin/console messenger:consume -vv, will, eventually, simply crash and die with an error similar to the following:

16:22:32 CRITICAL [php] Fatal Error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 32768 bytes) ["exception" => Symfony\Component\ErrorHandler\Error\OutOfMemoryError^ { …}]

That is, a worker process that purportedly does absolutely nothing, quietly consumes more and more memory until it dies.

How to reproduce

Ensure at least one message queue is configured, e.g.

framework:
  messenger:
    reset_on_message: true

   transports:
      email: 'beanstalkd://127.0.0.1:11300?ttr=60&tube_name=email'

    routing:
      Symfony\Component\Mailer\Messenger\SendEmailMessage: email

Run the worker for the default (only) queue:

bin/console messenger:consume -vv

Eventually the process will crash. In my example, I am using memory_limit = 256M, but obviously smaller limits will allow quicker reproduction.

Possible Solution

A worker should not consume resources indefinitely, especially when it isn't even processing any messages! Most memory acquired should subsequently be dereferenced so it can be freed. Judicious use of unset() or null assignments may be necessary in some cases.

Additional Context

  • PHP 8.1/Win
  • Debug mode is on (dev environment).

Before I discovered the Symfony Messenger component, I wrote my own PHP queue worker and never had to consider running out of memory. It processes hundreds of thousands of messages per day and has an uptime of over 6 months with exactly the same PHP memory limits as the Symfony worker which dies within a few hours processing nothing. Clearly something is wrong here.
@Bilge Bilge added the Bug label Feb 10, 2022
@alexislefebvre
Copy link
Contributor

A workaround is explained in the documentation:

Don't Let Workers Run Forever
Some services (like Doctrine's EntityManager) will consume more memory over time. So, instead of allowing your worker to run forever, use a flag like messenger:consume --limit=10 to tell your worker to only handle 10 messages before exiting (then Supervisor will create a new process). There are also other options like --memory-limit=128M and --time-limit=3600.

Source: https://symfony.com/doc/5.4/messenger.html#deploying-to-production

@Bilge
Copy link
Contributor Author

Bilge commented Apr 14, 2022

  1. A workaround is just that.
  2. This is not a problem on Linux.

@alexislefebvre
Copy link
Contributor

This is not a problem on Linux? Can you please elaborate?

Does the issue occur on Windows in the prod environment? Maybe the memory leak is related to the debug code.

@Bilge
Copy link
Contributor Author

Bilge commented Apr 14, 2022

Fair point, it happens on Windows/debug and not on Linux/prod. Therefore the issue could be with the environment and not the platform.

@alexislefebvre
Copy link
Contributor

I thought to the debugger after seeing this other issue: #45974

@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@Bilge
Copy link
Contributor Author

Bilge commented Oct 29, 2022

Still relevant.

@carsonbot carsonbot removed the Stalled label Oct 29, 2022
@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@carsonbot
Copy link

Hello? This issue is about to be closed if nobody replies.

@alexislefebvre
Copy link
Contributor

  1. A workaround is just that.

Actually I don't know why I called this a “workaround”, this is a recommendation from the doc.

Can Symfony monitor and adjust the used memory to prevent an overflow automatically? From my understanding, this issue is too hard to tackle, and defining limits so that workers don't run indefinitely is a simple yet effective solution.

@carsonbot carsonbot removed the Stalled label Jul 4, 2023
@ostrolucky
Copy link
Contributor

We were running symfony workers for weeks without restarting, didn't have issues. Look for issue in your application. One common issue is that you are running it under dev environment, which saves extra debugging info. Another common issue is wrongly set up monolog.

@Bilge
Copy link
Contributor Author

Bilge commented Jul 4, 2023

One common issue is that you are running it under dev environment

Please don't post without reading.

@ostrolucky
Copy link
Contributor

ostrolucky commented Jul 4, 2023 •
edited
Loading

So your complaint is that it leaks memory in dev environment? I tried to fight it in symfony/monolog-bundle#165, but since then I've kinda accepted that dev will always leak memory. Also, seems similar to #48176

You might want to try to remove monolog-bundle to see if after you still have memory leak problem.

@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@carsonbot
Copy link

Friendly ping? Should this still be open? I will close if I don't hear anything.

@carsonbot
Copy link

Hey,

I didn't hear anything so I'm going to close it. Feel free to comment if this is still relevant, I can always reopen!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Stalled Status: Needs Review
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Bilge @ostrolucky @alexislefebvre @carsonbot