This bug can be reproduced on my Linux environment. I think may need some time to figure it out

For now, I think this may related with the gc timing

import random
import time
from threading import Lock, Thread

import resource
import gc

class Cache:
    def __init__(self, cap):
        self.m = {}
        self.mu = Lock()
        self.cap = cap

    def get(self, key):
        with self.mu:
            try:
                v = self.m[key]
            except KeyError:
                return 0, False
            return v, True

    def set(self, key, value):
        with self.mu:
            self.m[key] = value
            if len(self.m) > self.cap:
                self.m.popitem()

def set_key(cache):
    c = 0
    while True:
        key = random.randint(0, 1000000000)
        cache.set(key, key)
        c += 1
        if c  == 100000:
            c = 0
            print(".", end="", flush=True)

def run():
    cache = Cache(500000)

    for _ in range(4):
        thread = Thread(target=set_key, args=[cache])
        thread.start()

    while True:
        time.sleep(1)
        with cache.mu:
            usage = resource.getrusage(resource.RUSAGE_SELF)

            rss_mb = usage.ru_maxrss
            print(f"Process RSS: {rss_mb:.2f} KB")
            # gc.collect()

if __name__ == "__main__":
    run()

if we run the gc.collect manually, the memory size will keep normal. Otherwise, the memory size will increase in 40MB in regular peroid

It seems to run normally in v3.13.0a3.

After 10-minute loops, the memory size remains at less than 200MB.

This is a good signal. I think I can bisect for the bad commit.

@Zheaoli Thanks for investigating the issue. I tested by manually triggering GC every 10 seconds and every 1 second. For the 10-second interval, the RSS stabilized around rss: 893140992, dict len: 500000. For the 1-second interval, the RSS stabilized around rss: 490344448, dict len: 500000. I also tested 3.13.0t without manually triggering GC, the memory usage still increases, though slower compared to 3.13.3t.

Got it. I think this bug is introduced into the codebase in #116336

But I don't have enough idea about how to fix this. Would you mind taking a little bit of help? @colesbury

@colesbury PTAL when you got time

I think this issue is still related with the GC. I count the free_keys_object time. The normal version is much more than the free thread version.

I think this issue is still related with the GC. I count the free_keys_object time. The normal version is much more than the free thread version.

This seems like a reasonable explanation for the issue. Hopefully, @colesbury can take a look when he has time, or if he's too busy, perhaps someone else familiar with free-threading could also help investigate.

I think we need better heuristics for the QSBR here. The repeated insertions and popitem() calls lead to the dictionary keys object getting reallocated every so often. The PyDictKeysObjects here are large here so keeping a bunch of them means a lot of extra memory usage in the example.

We "buffer" QSBR frees in two ways:

1. We only advance the global QSBR counter every 10 calls to _Py_qsbr_deferred_advance. This is to reduce contention on the shared counter for frequent calls, but that's probably too infrequent here given the size of the keys objects

2. We only process the delayed free requests every WORK_ITEMS_PER_CHUNK (254) calls, again to limit work for frequent calls.

In addition to periodic processing every 10 or 254 calls, I think we probably want to take into account the amount of memory that the QSBR subsystem is holding.

cc @mpage, @Yhg1s, @nascheme, @DinoV