-
-
Notifications
You must be signed in to change notification settings - Fork 11.1k
MAINT: Do proper cleanup in get_ufunc_arguments. #11260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@@ -575,6 +572,7 @@ get_ufunc_arguments(PyUFuncObject *ufunc, | |||
int i, nargs; | |||
int nin = ufunc->nin; | |||
int nout = ufunc->nout; | |||
int nop = ufunc->nargs; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nop should be used wherever ufunc->nargs appears
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. And fixed the failures by passing back retval
correctly.
40de0ff
to
31a4a29
Compare
errval = PyUFunc_GenericFunction(ufunc, args, kwds, mps); | ||
if (errval < 0) { | ||
for (i = 0; i < ufunc->nargs; i++) { | ||
PyArray_DiscardWritebackIfCopy(mps[i]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this no longer needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was impossible for it to ever be executed in the first place: the failure paths of PyUFunc_GenericFunction
and Py_UFuncGeneralizedFunction
both set op
(here called mps
) to NULL (after xdecref'ing).
But I guess it is possible that this was in fact a latent bug... @mattip, since you put the lines there, should they be moved up to the failure paths of the functions where the actual xdecref's are done? And, if so, is there a test that could be added to ensure this actually works as intended?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding a line
if (mps[0] != NULL) printf("\nufunc is %s\n", ufunc->name);
and running tests yeilds over 200 times this happens in tests, for comparison ufuncs (equal
, greater
, ...) but also for add
, inner1d
, and matrix_multiply
On the other hand, I do not immediately remember why we would need the PyArray_DiscardWritebackIfCopy
, but I vaguely remember a failing test that I tracked down to this call. Since then perhaps the logic has changed, I will have to go back and review where I put it in. In looking over this code, I see a few places more places with perhaps excessive use of PyArray_DiscardWritebackIfCopy
and NpyIter_Close
, I will go back and review it again.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mattip - it is not guaranteed that mps[i] == NULL
, just that it has been properly decref'd if there is a failure. The question is if we need the DiscardWriteBackIfCopy
in the function call (for cases, I guess, where there is a failure after the iterator is allocated).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is fine the way it is. What happened? I deprecate the old PyArray_XDECREF_ERR(obj)
in favor of the new PyArray_DiscardWritebackIfCopy(obj); Py_XDECREF(obj)
in changeset fd4e7b and mechanically replaced all the calls without checking whether the call to PyArray_DiscardWritebackIfCopy
was needed. Indeed in this case it was not.
Thank goodness for preservation of squashed commits in github.
@@ -762,16 +767,17 @@ get_ufunc_arguments(PyUFuncObject *ufunc, | |||
if (!strcmp(ufunc_name, "equal") || | |||
!strcmp(ufunc_name, "not_equal")) { | |||
/* Warn on non-scalar, return NotImplemented regardless */ | |||
assert(nin == 2); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason for this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I moved nin == 2
inside the if
statement above, so that ufuncs with other numbers of inputs are not checked. With the check just above, the assert seemed superfluous.
} | ||
#endif | ||
Py_DECREF(out_op[0]); | ||
Py_DECREF(out_op[1]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this do the full cleanup and goto fail
? Could add a fail_pre_kwargs
label, if we want to avoid unnecessary cleanup.
Of course, really the huge hack section of this function should be extracted from the rest of it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Initially I had a goto cleanup_out_op
and passed on a retval
, but then I found that it just became confusing, and might hinder ripping this whole part out (which, as you said, should be done, but better not in this PR...). Fortunately, here we know exactly what has been done up to this point (indeed, this is partially why I moved the nin == 2
to the if-statement; I wanted to be absolutely sure I decref'd just the right number of operands, even if someone gave their ufunc a name that matches but has more than 2 inputs).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More for completeness: See #11282
needs a rebase |
Instead of leaving it to the caller to clean up references to operands.
31a4a29
to
584e82f
Compare
Yes, rebase needed now that |
I think this could be in 1.15 but doesn't have to be. But having it in may be handy for review of #11282. |
I've marked this for 1.15, everyone happy with its current state? |
There's a memory leak I discovered in #11295 that will be easier to fix after this goes in - so I reckon we go for it |
OK, let's put this in. Thanks Marten. |
Instead of leaving it to the caller to clean up references to operands.
This is the promised follow-up to #11257, cleaning up references to operands on failure as well.