You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Current events pool implementation has a problem with too high memory consuption: current reset mechanism does not account for actual memory usage, instead we use AvgEventSize from the config, but actual event size may be different, depends on current server load. Additionally, the current implementation allocates memory when the application starts, which generally increases memory consumption unnecessarily.
I propose implement leveled event pool based on sync.Pool. Each level will store events based on the event size. The new implementation should also implement capacity logic and slow wait when exceeding capacity to avoid OOMs (as in the old implementation). In addition, we should maintain two different pool implementations (old and new) for gradual migration to the new implementation.
Each level will be responsible for its own size range, which will allow it to adapt to the incoming event size. The level boundaries will be calculated as follows: [2^(n-1), 2^n). For example:
Using a pool will allow automatic memory cleanup and better adapt to the application load. After implementing the new pool, we will be able to use GOGC variables to choose between the CPU (GC) and RAM, which didn't matter in the old implementation.
The text was updated successfully, but these errors were encountered:
Current events pool implementation has a problem with too high memory consuption: current reset mechanism does not account for actual memory usage, instead we use
AvgEventSize
from the config, but actual event size may be different, depends on current server load. Additionally, the current implementation allocates memory when the application starts, which generally increases memory consumption unnecessarily.I propose implement leveled event pool based on
sync.Pool
. Each level will store events based on the event size. The new implementation should also implement capacity logic and slow wait when exceeding capacity to avoid OOMs (as in the old implementation). In addition, we should maintain two different pool implementations (old and new) for gradual migration to the new implementation.Each level will be responsible for its own size range, which will allow it to adapt to the incoming event size. The level boundaries will be calculated as follows:
[2^(n-1), 2^n)
. For example:Using a pool will allow automatic memory cleanup and better adapt to the application load. After implementing the new pool, we will be able to use
GOGC
variables to choose between the CPU (GC) and RAM, which didn't matter in the old implementation.The text was updated successfully, but these errors were encountered: