Skip to content

gh-133931: Introduce _PyObject_XSetRefDelayed to replace Py_XSETREF #134377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

corona10
Copy link
Member

@corona10 corona10 commented May 20, 2025

_PyObject_XSetRefDelayed(PyObject **ptr, PyObject *value)
{
PyObject *old = *ptr;
FT_ATOMIC_STORE_PTR_RELEASE(*ptr, Py_NewRef(value));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default build implementation in pycore_pymem.h doesn't include the Py_NewRef().

My preference is to keep the Py_NewRef(value) in the callers, i.e.:

_PyObject_XSetRefDelayed(dictptr, Py_NewRef(value));

to match the semantics of Py_XSETREF.

@corona10
Copy link
Member Author

Interesting... CI passed in my local but not in the CI.. :(

@corona10
Copy link
Member Author

Ah okay..

test_aclose (test.test_asyncgen.TestUnawaitedWarnings.test_aclose) ... Assertion failed: (!_Py_IsImmortal(op)), function _Py_ExplicitMergeRefcount, file object.c, line 467.

@corona10 corona10 changed the title gh-133931: Introduce _PyObject_XSetRefDelayed to replace Py_XSETREF [WIP] gh-133931: Introduce _PyObject_XSetRefDelayed to replace Py_XSETREF May 21, 2025
@markshannon
Copy link
Member

Why do we need _PyObject_XSetRefDelayed? It looks like all uses of it are wrapped in critical sections where the old Py_XSETREF would work.

Also, what is "delayed"? Either choose a clearer name, or add an explanatory comment in the header file.

@corona10 corona10 changed the title [WIP] gh-133931: Introduce _PyObject_XSetRefDelayed to replace Py_XSETREF gh-133931: Introduce _PyObject_XSetRefDelayed to replace Py_XSETREF Jun 7, 2025
@corona10 corona10 marked this pull request as ready for review June 7, 2025 13:34
@corona10
Copy link
Member Author

corona10 commented Jun 7, 2025

Why do we need _PyObject_XSetRefDelayed? It looks like all uses of it are wrapped in critical sections where the old Py_XSETREF would work.

IIUC, the critical section is for the root object to prevent other threads from mutating the child object, and a delayed reference to prevent use-after-free from other threads that reference the child object.

@corona10 corona10 requested a review from colesbury June 7, 2025 13:42
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
@corona10 corona10 requested a review from kumaraditya303 June 10, 2025 11:18
@@ -100,6 +100,18 @@ static inline void _PyObject_XDecRefDelayed(PyObject *obj)
}
#endif

#ifdef Py_GIL_DISABLED
// This is the delayed-free equivalent of Py_XSETREF(), providing the same
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this sounds opposite, _PyObject_XSetRefDelayed is the delayed version whereas Py_XSETREF is the delayed free equivalent

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching

Co-authored-by: Kumar Aditya <kumaraditya@python.org>
_PyObject_XDecRefDelayed(old);
}
else {
Py_DECREF(old);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the object is immortal then decref has no effect

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah but it tracks the stat. I am not sure we need this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpython/Include/refcount.h

Lines 375 to 378 in 598aa7c

if (local == _Py_IMMORTAL_REFCNT_LOCAL) {
_Py_DECREF_IMMORTAL_STAT_INC();
return;
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But let's remove it, unitil we need this stat explicitly.

corona10 and others added 2 commits June 10, 2025 22:05
Co-authored-by: Victor Stinner <vstinner@python.org>
@efimov-mikhail
Copy link
Contributor

efimov-mikhail commented Jun 10, 2025

Could someone please explain to me, why _PyObject_XDecRefDelayed is called under critical section in this PR (and some others), but in Objects/dictobject.c decref_maybe_delay function is called after critical section, see

decref_maybe_delay((PyObject *)dict, true);

Which one is more correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants