Yes, this is expected and not a bug.

Closing this issue since this is expected.

To add a bit more context: if you have an inherently serial operation, like incrementing a counter or appending to a list, it's going to be slower when run with multiple threads on multiple CPUs than if you have some sort of coarse grained serialization (like the GIL) or run it on a single CPU.

@colesbury I have been reading GIL removal pep. And this issue has been closed as "not planned" , Do we have any plans to fix performance partially at least, especially in the cases of list.append(results) ?

No

Is this issue specific to .append or does it apply to .extend and slice insertions?

Is/should this be documented, warned against, somewhere other than here?

I don't think we need to specifically warn against it any more than we would warn against building a list via successive list.insert(0, item).

This is an inherently serial operation. If you try to perform a serial operation from multiple CPUs, you introduce lots of extra communication and everything gets slower. That's normal and not really specific to Python, just like quadratic behavior of some data structures is normal and not specific to Python.

@colesbury Thank you for the response. I understand the general principle about serial operations being slower with multiple threads. However, I want to clarify that my MRE doesn't use any threading at all - it's entirely single-threaded code.

The performance regression I'm observing is occurring in single-threaded code simply by running it in free-threaded Python mode. This suggests that there's a significant overhead for thread safety mechanisms even when no actual threading is used.

This seems important to document, since while experienced concurrent programmers will understand the underlying reasons, many Python users might not expect their unthreaded code to suddenly run 10-15x slower.

Would it be worth mentioning this behavior in the free-threading documentation - that even single-threaded code using mutable shared data structures may experience performance regressions due to the thread safety overhead?

Also, is there any scope for optimizing free-threaded mode under these conditions in future releases?

Sorry, I didn't pay close enough attention to the repro and focused on "shared list appends". Thanks for following up.

This appears to be some sort of performance regression related to GC. The list.append(results) isn't really slow itself, it just holds onto the generated combinations, which leads to the GC being scheduled. Without that list.append(), the objects are freed after each call to expensive_operation().

You can verify this by adding a import gc; gc.disable() at the top.

cc @nascheme, in case you want to take a look at this

Yeah, the difference in performance is almost purely due to the different behavior of the cyclic GC. With the list.append() you are quickly increasing the number of alive container objects and the GC runs many times. The free-threaded build doesn't have a generational collector and so it needs to do a full collection each time the increment in live objects threshold is exceeded. I need to investigate further to confirm but I suspect since you are using tuples, the default build GC is untracking those (since they don't contain containers) and so the need to perform full GC collections is greatly reduced. Needs more investigation to confirm that's all that's happening and there is not some other issue.

This is fixed in main now with:

• gh-132917: Check resident set size (RSS) before GC trigger. #133399
• gh-132917: Use RSS + swap for estimate of process memory usage #133464

@nascheme - should the fixes be backported to 3.14?

Oh, seems to be fixed in 3.14 as well.