Skip to content

gh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (in Objects/) #102218

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
Mar 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
f06d53a
Objects/dictobject.c
iritkatriel Feb 23, 2023
d5072bd
Objects/frameobject.c
iritkatriel Feb 23, 2023
c40fc27
Objects/genobject.c
iritkatriel Feb 23, 2023
13cb48c
Objects/object.c
iritkatriel Feb 23, 2023
fde54d4
Objects/odictobject.c
iritkatriel Feb 23, 2023
8fd66a8
Objects/typeobject.c
iritkatriel Feb 23, 2023
d83af14
Objects/weakrefobject.c
iritkatriel Feb 23, 2023
0298a04
Objects/exceptions.c
iritkatriel Feb 23, 2023
09f2ab6
put back the Py_DECREF(exc)
iritkatriel Feb 24, 2023
0fef293
Merge branch 'main' into fetch-restore-objects
iritkatriel Feb 24, 2023
9b5b62a
Merge branch 'main' into fetch-restore-objects
iritkatriel Feb 25, 2023
6378d56
Merge branch 'main' into fetch-restore-objects
iritkatriel Feb 26, 2023
e11d73e
Merge branch 'main' into fetch-restore-objects
iritkatriel Feb 27, 2023
eb45902
Merge branch 'main' into fetch-restore-objects
iritkatriel Feb 27, 2023
526e387
Merge branch 'main' into fetch-restore-objects
iritkatriel Feb 28, 2023
a579bc4
Merge branch 'main' into fetch-restore-objects
iritkatriel Feb 28, 2023
e5bf89d
Merge branch 'main' into fetch-restore-objects
iritkatriel Feb 28, 2023
0863f35
Merge branch 'main' into fetch-restore-objects
iritkatriel Mar 1, 2023
3f8b452
Merge branch 'main' into fetch-restore-objects
iritkatriel Mar 2, 2023
7a436c9
Merge branch 'main' into fetch-restore-objects
iritkatriel Mar 2, 2023
9b68505
remove redundant null check and obsolete comments (Mark's review comm…
iritkatriel Mar 3, 2023
c23baa2
Merge branch 'main' into fetch-restore-objects
iritkatriel Mar 3, 2023
44bead6
Merge branch 'main' into fetch-restore-objects
iritkatriel Mar 6, 2023
d592920
Merge branch 'main' into fetch-restore-objects
iritkatriel Mar 6, 2023
e136989
Merge branch 'main' into fetch-restore-objects
iritkatriel Mar 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Objects/dictobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ As a consequence of this, split keys have a maximum size of 16.
#include "pycore_dict.h" // PyDictKeysObject
#include "pycore_gc.h" // _PyObject_GC_IS_TRACKED()
#include "pycore_object.h" // _PyObject_GC_TRACK()
#include "pycore_pyerrors.h" // _PyErr_Fetch()
#include "pycore_pyerrors.h" // _PyErr_GetRaisedException()
#include "pycore_pystate.h" // _PyThreadState_GET()
#include "stringlib/eq.h" // unicode_eq()

Expand Down
39 changes: 13 additions & 26 deletions Objects/exceptions.c
Original file line number Diff line number Diff line change
Expand Up @@ -3781,16 +3781,13 @@ PyObject *
_PyErr_TrySetFromCause(const char *format, ...)
{
PyObject* msg_prefix;
PyObject *exc, *val, *tb;
PyTypeObject *caught_type;
PyObject *instance_args;
Py_ssize_t num_args, caught_type_size, base_exc_size;
PyObject *new_exc, *new_val, *new_tb;
va_list vargs;
int same_basic_size;

PyErr_Fetch(&exc, &val, &tb);
caught_type = (PyTypeObject *)exc;
PyObject *exc = PyErr_GetRaisedException();
PyTypeObject *caught_type = Py_TYPE(exc);
/* Ensure type info indicates no extra state is stored at the C level
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does all this stuff about "wrapping safely" make sense any more?
Since there are no "denomralized" exceptions, this should always work.
I think all this code is making sure that calling PyErr_NormalizeException is safe.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. Let me try to understand what's going on here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is about PyErr_NormalizeException. This function is called from only one place:

/* Helper that tries to ensure the reported exception chain indicates the
 * codec that was invoked to trigger the failure without changing the type
 * of the exception raised.       
 */
static void
wrap_codec_error(const char *operation,
                 const char *encoding)
{
    /* TrySetFromCause will replace the active exception with a suitably
     * updated clone if it can, otherwise it will leave the original
     * exception alone.
     */
    _PyErr_TrySetFromCause("%s with '%s' codec failed",
                           operation, encoding);
}

This is a classic use case for notes (PEP 678). It is trying to create an exception of the same type with additional info (and does a lot of stuff to check that it can indeed safely create a new exception of the same type).

I would just rewrite this whole thing to attach the additional info as a note (but not in this PR, it's unrelated).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A PR for that is at: #102407

* and that the type can be reinstantiated using PyErr_Format
*/
Expand All @@ -3810,31 +3807,30 @@ _PyErr_TrySetFromCause(const char *format, ...)
* more state than just the exception type. Accordingly, we just
* leave it alone.
*/
PyErr_Restore(exc, val, tb);
PyErr_SetRaisedException(exc);
return NULL;
}

/* Check the args are empty or contain a single string */
PyErr_NormalizeException(&exc, &val, &tb);
instance_args = ((PyBaseExceptionObject *)val)->args;
instance_args = ((PyBaseExceptionObject *)exc)->args;
num_args = PyTuple_GET_SIZE(instance_args);
if (num_args > 1 ||
(num_args == 1 &&
!PyUnicode_CheckExact(PyTuple_GET_ITEM(instance_args, 0)))) {
/* More than 1 arg, or the one arg we do have isn't a string
*/
PyErr_Restore(exc, val, tb);
PyErr_SetRaisedException(exc);
return NULL;
}

/* Ensure the instance dict is also empty */
if (!_PyObject_IsInstanceDictEmpty(val)) {
if (!_PyObject_IsInstanceDictEmpty(exc)) {
/* While we could potentially copy a non-empty instance dictionary
* to the replacement exception, for now we take the more
* conservative path of leaving exceptions with attributes set
* alone.
*/
PyErr_Restore(exc, val, tb);
PyErr_SetRaisedException(exc);
return NULL;
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block comment should go as well (if I'm correct about removing the above code)

Expand All @@ -3847,28 +3843,19 @@ _PyErr_TrySetFromCause(const char *format, ...)
* types as well, but that's quite a bit trickier due to the extra
* state potentially stored on OSError instances.
*/
/* Ensure the traceback is set correctly on the existing exception */
if (tb != NULL) {
PyException_SetTraceback(val, tb);
Py_DECREF(tb);
}

va_start(vargs, format);
msg_prefix = PyUnicode_FromFormatV(format, vargs);
va_end(vargs);
if (msg_prefix == NULL) {
Py_DECREF(exc);
Py_DECREF(val);
return NULL;
}

PyErr_Format(exc, "%U (%s: %S)",
msg_prefix, Py_TYPE(val)->tp_name, val);
Py_DECREF(exc);
PyErr_Format((PyObject*)Py_TYPE(exc), "%U (%s: %S)",
msg_prefix, Py_TYPE(exc)->tp_name, exc);
Py_DECREF(msg_prefix);
PyErr_Fetch(&new_exc, &new_val, &new_tb);
PyErr_NormalizeException(&new_exc, &new_val, &new_tb);
PyException_SetCause(new_val, val);
PyErr_Restore(new_exc, new_val, new_tb);
return new_val;
PyObject *new_exc = PyErr_GetRaisedException();
PyException_SetCause(new_exc, exc);
PyErr_SetRaisedException(new_exc);
return new_exc;
}
5 changes: 2 additions & 3 deletions Objects/frameobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -1308,7 +1308,6 @@ _PyFrame_LocalsToFast(_PyInterpreterFrame *frame, int clear)
/* Merge locals into fast locals */
PyObject *locals;
PyObject **fast;
PyObject *error_type, *error_value, *error_traceback;
PyCodeObject *co;
locals = frame->f_locals;
if (locals == NULL) {
Expand All @@ -1317,7 +1316,7 @@ _PyFrame_LocalsToFast(_PyInterpreterFrame *frame, int clear)
fast = _PyFrame_GetLocalsArray(frame);
co = frame->f_code;

PyErr_Fetch(&error_type, &error_value, &error_traceback);
PyObject *exc = PyErr_GetRaisedException();
for (int i = 0; i < co->co_nlocalsplus; i++) {
_PyLocals_Kind kind = _PyLocals_GetKind(co->co_localspluskinds, i);

Expand Down Expand Up @@ -1374,7 +1373,7 @@ _PyFrame_LocalsToFast(_PyInterpreterFrame *frame, int clear)
}
Py_XDECREF(value);
}
PyErr_Restore(error_type, error_value, error_traceback);
PyErr_SetRaisedException(exc);
}

void
Expand Down
64 changes: 16 additions & 48 deletions Objects/genobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,6 @@ void
_PyGen_Finalize(PyObject *self)
{
PyGenObject *gen = (PyGenObject *)self;
PyObject *res = NULL;
PyObject *error_type, *error_value, *error_traceback;

if (gen->gi_frame_state >= FRAME_COMPLETED) {
/* Generator isn't paused, so no need to close */
Expand All @@ -82,23 +80,22 @@ _PyGen_Finalize(PyObject *self)
PyObject *finalizer = agen->ag_origin_or_finalizer;
if (finalizer && !agen->ag_closed) {
/* Save the current exception, if any. */
PyErr_Fetch(&error_type, &error_value, &error_traceback);

res = PyObject_CallOneArg(finalizer, self);
PyObject *exc = PyErr_GetRaisedException();

PyObject *res = PyObject_CallOneArg(finalizer, self);
if (res == NULL) {
PyErr_WriteUnraisable(self);
} else {
Py_DECREF(res);
}
/* Restore the saved exception. */
PyErr_Restore(error_type, error_value, error_traceback);
PyErr_SetRaisedException(exc);
return;
}
}

/* Save the current exception, if any. */
PyErr_Fetch(&error_type, &error_value, &error_traceback);
PyObject *exc = PyErr_GetRaisedException();

/* If `gen` is a coroutine, and if it was never awaited on,
issue a RuntimeWarning. */
Expand All @@ -109,20 +106,19 @@ _PyGen_Finalize(PyObject *self)
_PyErr_WarnUnawaitedCoroutine((PyObject *)gen);
}
else {
res = gen_close(gen, NULL);
}

if (res == NULL) {
if (PyErr_Occurred()) {
PyErr_WriteUnraisable(self);
PyObject *res = gen_close(gen, NULL);
if (res == NULL) {
if (PyErr_Occurred()) {
PyErr_WriteUnraisable(self);
}
}
else {
Py_DECREF(res);
}
}
else {
Py_DECREF(res);
}

/* Restore the saved exception. */
PyErr_Restore(error_type, error_value, error_traceback);
PyErr_SetRaisedException(exc);
}

static void
Expand Down Expand Up @@ -648,39 +644,11 @@ _PyGen_SetStopIterationValue(PyObject *value)
int
_PyGen_FetchStopIterationValue(PyObject **pvalue)
{
PyObject *et, *ev, *tb;
PyObject *value = NULL;

if (PyErr_ExceptionMatches(PyExc_StopIteration)) {
PyErr_Fetch(&et, &ev, &tb);
if (ev) {
/* exception will usually be normalised already */
if (PyObject_TypeCheck(ev, (PyTypeObject *) et)) {
value = Py_NewRef(((PyStopIterationObject *)ev)->value);
Py_DECREF(ev);
} else if (et == PyExc_StopIteration && !PyTuple_Check(ev)) {
/* Avoid normalisation and take ev as value.
*
* Normalization is required if the value is a tuple, in
* that case the value of StopIteration would be set to
* the first element of the tuple.
*
* (See _PyErr_CreateException code for details.)
*/
value = ev;
} else {
/* normalisation required */
PyErr_NormalizeException(&et, &ev, &tb);
if (!PyObject_TypeCheck(ev, (PyTypeObject *)PyExc_StopIteration)) {
PyErr_Restore(et, ev, tb);
return -1;
}
value = Py_NewRef(((PyStopIterationObject *)ev)->value);
Py_DECREF(ev);
}
}
Py_XDECREF(et);
Py_XDECREF(tb);
PyObject *exc = PyErr_GetRaisedException();
value = Py_NewRef(((PyStopIterationObject *)exc)->value);
Py_DECREF(exc);
} else if (PyErr_Occurred()) {
return -1;
}
Expand Down
25 changes: 10 additions & 15 deletions Objects/object.c
Original file line number Diff line number Diff line change
Expand Up @@ -370,13 +370,12 @@ _PyObject_Dump(PyObject* op)
fflush(stderr);

PyGILState_STATE gil = PyGILState_Ensure();
PyObject *error_type, *error_value, *error_traceback;
PyErr_Fetch(&error_type, &error_value, &error_traceback);
PyObject *exc = PyErr_GetRaisedException();

(void)PyObject_Print(op, stderr, 0);
fflush(stderr);

PyErr_Restore(error_type, error_value, error_traceback);
PyErr_SetRaisedException(exc);
PyGILState_Release(gil);

fprintf(stderr, "\n");
Expand Down Expand Up @@ -860,25 +859,22 @@ set_attribute_error_context(PyObject* v, PyObject* name)
return 0;
}
// Intercept AttributeError exceptions and augment them to offer suggestions later.
PyObject *type, *value, *traceback;
PyErr_Fetch(&type, &value, &traceback);
PyErr_NormalizeException(&type, &value, &traceback);
// Check if the normalized exception is indeed an AttributeError
if (!PyErr_GivenExceptionMatches(value, PyExc_AttributeError)) {
PyObject *exc = PyErr_GetRaisedException();
if (!PyErr_GivenExceptionMatches(exc, PyExc_AttributeError)) {
goto restore;
}
PyAttributeErrorObject* the_exc = (PyAttributeErrorObject*) value;
PyAttributeErrorObject* the_exc = (PyAttributeErrorObject*) exc;
// Check if this exception was already augmented
if (the_exc->name || the_exc->obj) {
goto restore;
}
// Augment the exception with the name and object
if (PyObject_SetAttr(value, &_Py_ID(name), name) ||
PyObject_SetAttr(value, &_Py_ID(obj), v)) {
if (PyObject_SetAttr(exc, &_Py_ID(name), name) ||
PyObject_SetAttr(exc, &_Py_ID(obj), v)) {
return 1;
}
restore:
PyErr_Restore(type, value, traceback);
PyErr_SetRaisedException(exc);
return 0;
}

Expand Down Expand Up @@ -2190,9 +2186,8 @@ Py_ReprLeave(PyObject *obj)
PyObject *dict;
PyObject *list;
Py_ssize_t i;
PyObject *error_type, *error_value, *error_traceback;

PyErr_Fetch(&error_type, &error_value, &error_traceback);
PyObject *exc = PyErr_GetRaisedException();

dict = PyThreadState_GetDict();
if (dict == NULL)
Expand All @@ -2213,7 +2208,7 @@ Py_ReprLeave(PyObject *obj)

finally:
/* ignore exceptions because there is no way to report them. */
PyErr_Restore(error_type, error_value, error_traceback);
PyErr_SetRaisedException(exc);
}

/* Trashcan support. */
Expand Down
5 changes: 2 additions & 3 deletions Objects/odictobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -1556,10 +1556,9 @@ _PyODict_SetItem_KnownHash(PyObject *od, PyObject *key, PyObject *value,
res = _odict_add_new_node((PyODictObject *)od, key, hash);
if (res < 0) {
/* Revert setting the value on the dict */
PyObject *exc, *val, *tb;
PyErr_Fetch(&exc, &val, &tb);
PyObject *exc = PyErr_GetRaisedException();
(void) _PyDict_DelItem_KnownHash(od, key, hash);
_PyErr_ChainExceptions(exc, val, tb);
_PyErr_ChainExceptions1(exc);
}
}
return res;
Expand Down
10 changes: 4 additions & 6 deletions Objects/typeobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -4397,10 +4397,9 @@ static void
type_dealloc_common(PyTypeObject *type)
{
if (type->tp_bases != NULL) {
PyObject *tp, *val, *tb;
PyErr_Fetch(&tp, &val, &tb);
PyObject *exc = PyErr_GetRaisedException();
remove_all_subclasses(type, type->tp_bases);
PyErr_Restore(tp, val, tb);
PyErr_SetRaisedException(exc);
}
}

Expand Down Expand Up @@ -8445,10 +8444,9 @@ slot_tp_finalize(PyObject *self)
{
int unbound;
PyObject *del, *res;
PyObject *error_type, *error_value, *error_traceback;

/* Save the current exception, if any. */
PyErr_Fetch(&error_type, &error_value, &error_traceback);
PyObject *exc = PyErr_GetRaisedException();

/* Execute __del__ method, if any. */
del = lookup_maybe_method(self, &_Py_ID(__del__), &unbound);
Expand All @@ -8462,7 +8460,7 @@ slot_tp_finalize(PyObject *self)
}

/* Restore the saved exception. */
PyErr_Restore(error_type, error_value, error_traceback);
PyErr_SetRaisedException(exc);
}

static PyObject *
Expand Down
7 changes: 3 additions & 4 deletions Objects/weakrefobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -959,9 +959,8 @@ PyObject_ClearWeakRefs(PyObject *object)
if (*list != NULL) {
PyWeakReference *current = *list;
Py_ssize_t count = _PyWeakref_GetWeakrefCount(current);
PyObject *err_type, *err_value, *err_tb;
PyObject *exc = PyErr_GetRaisedException();

PyErr_Fetch(&err_type, &err_value, &err_tb);
if (count == 1) {
PyObject *callback = current->wr_callback;

Expand All @@ -980,7 +979,7 @@ PyObject_ClearWeakRefs(PyObject *object)

tuple = PyTuple_New(count * 2);
if (tuple == NULL) {
_PyErr_ChainExceptions(err_type, err_value, err_tb);
_PyErr_ChainExceptions1(exc);
return;
}

Expand Down Expand Up @@ -1010,7 +1009,7 @@ PyObject_ClearWeakRefs(PyObject *object)
Py_DECREF(tuple);
}
assert(!PyErr_Occurred());
PyErr_Restore(err_type, err_value, err_tb);
PyErr_SetRaisedException(exc);
}
}

Expand Down