enhance docs for critical sections #137334
Conversation
| :class:`list` and :class:`dict` because their public C-APIs | ||
| already use critical sections internally, with the notable | ||
| exception of :c:func:`PyDict_Next`, which requires critical section | ||
| to be acquired externally. |
There was a problem hiding this comment.
I'm not sure if I agree here. Unless I've done something wrong in NumPy. I handled this for lists using the PySequence API. I take a critical section on a list, convert to a sequence using PySequence_Fast, and then operate on the sequence under a critical section. The sequence APIs also bypass the internal critical sections and they're documented not to be thread-safe, so I think this is actually the only safe way to use PySequence_Fast correctly on the free-threaded build.
|
|
||
| * Calling :c:func:`PyEval_SaveThread` to detach the current thread. | ||
|
|
||
| * Recursively entering the critical section for the same object. |
There was a problem hiding this comment.
In 3.14, don't recursive critical sections not get suspended?
There was a problem hiding this comment.
it isn't exactly recursive, there is one more condition which is that the critical section should be the topmost held critical section otherwise it can still get suspended
There was a problem hiding this comment.
Ah I see. Does it make sense to document under exactly what circumstances recursive critical sections don't get released as of 3.14? And maybe document that it used to behave differently in 3.13?
There was a problem hiding this comment.
We talked about this a little during a meeting and we came to the conclusion that really the only thing that makes sense to document is that critical sections might be suspended if you call back into the C API, and whether or not that happens is implementation dependent.
| * Calling :c:func:`PyEval_SaveThread` to detach the current thread. | ||
|
|
||
| * Recursively entering the critical section for the same object. | ||
|
|
There was a problem hiding this comment.
It may be worth mentioning that these are exactly the same guarantees provided by the GIL.
|
The other danger we might want to mention is what we have been calling "re-entrancy". That's not quite the accurate word for it since re-entrancy is only one type of problem of that kind. The more general problem is that you call a Python API and it ends up mutating state that you didn't expect to change. I.e. modifying "pre-conditions" your logic sequence is depending on. Also known as "changing things under you" or "spooky action at a distance". In Python, there are many surprising ways this can happen. When I was working on thread-safety for typeobject.c, I discovered some that surprised me, even as a long-time Python user. A few examples:
Using a critical section (kind of obviously) doesn't protect you from this kind of thing. Why would it when this same problem can happen even if you never have more than a single thread. If it did try to protect you then likely the program would deadlock. I guess "potentially re-entrant" would be an okay description of this. I.e. if you call this API, it can basically do anything since it can start executing Python-level code (via a hash method, finalizer, etc). Then, it could very well re-enter the same function or method. |
📚 Documentation preview 📚: https://cpython-previews--137334.org.readthedocs.build/