Skip to content

MAINT: Do proper cleanup in get_ufunc_arguments. #11260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 10, 2018

Conversation

mhvk
Copy link
Contributor

@mhvk mhvk commented Jun 6, 2018

Instead of leaving it to the caller to clean up references to operands.

This is the promised follow-up to #11257, cleaning up references to operands on failure as well.

@@ -575,6 +572,7 @@ get_ufunc_arguments(PyUFuncObject *ufunc,
int i, nargs;
int nin = ufunc->nin;
int nout = ufunc->nout;
int nop = ufunc->nargs;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nop should be used wherever ufunc->nargs appears

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. And fixed the failures by passing back retval correctly.

@mhvk mhvk force-pushed the ufunc-parsing-better-cleanup branch from 40de0ff to 31a4a29 Compare June 6, 2018 19:26
errval = PyUFunc_GenericFunction(ufunc, args, kwds, mps);
if (errval < 0) {
for (i = 0; i < ufunc->nargs; i++) {
PyArray_DiscardWritebackIfCopy(mps[i]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this no longer needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was impossible for it to ever be executed in the first place: the failure paths of PyUFunc_GenericFunction and Py_UFuncGeneralizedFunction both set op (here called mps) to NULL (after xdecref'ing).

But I guess it is possible that this was in fact a latent bug... @mattip, since you put the lines there, should they be moved up to the failure paths of the functions where the actual xdecref's are done? And, if so, is there a test that could be added to ensure this actually works as intended?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a line

if (mps[0] != NULL) printf("\nufunc is %s\n", ufunc->name);

and running tests yeilds over 200 times this happens in tests, for comparison ufuncs (equal, greater, ...) but also for add, inner1d, and matrix_multiply

On the other hand, I do not immediately remember why we would need the PyArray_DiscardWritebackIfCopy, but I vaguely remember a failing test that I tracked down to this call. Since then perhaps the logic has changed, I will have to go back and review where I put it in. In looking over this code, I see a few places more places with perhaps excessive use of PyArray_DiscardWritebackIfCopy and NpyIter_Close, I will go back and review it again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattip - it is not guaranteed that mps[i] == NULL, just that it has been properly decref'd if there is a failure. The question is if we need the DiscardWriteBackIfCopy in the function call (for cases, I guess, where there is a failure after the iterator is allocated).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine the way it is. What happened? I deprecate the old PyArray_XDECREF_ERR(obj) in favor of the new PyArray_DiscardWritebackIfCopy(obj); Py_XDECREF(obj) in changeset fd4e7b and mechanically replaced all the calls without checking whether the call to PyArray_DiscardWritebackIfCopy was needed. Indeed in this case it was not.

Thank goodness for preservation of squashed commits in github.

@@ -762,16 +767,17 @@ get_ufunc_arguments(PyUFuncObject *ufunc,
if (!strcmp(ufunc_name, "equal") ||
!strcmp(ufunc_name, "not_equal")) {
/* Warn on non-scalar, return NotImplemented regardless */
assert(nin == 2);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved nin == 2 inside the if statement above, so that ufuncs with other numbers of inputs are not checked. With the check just above, the assert seemed superfluous.

}
#endif
Py_DECREF(out_op[0]);
Py_DECREF(out_op[1]);
Copy link
Member

@eric-wieser eric-wieser Jun 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this do the full cleanup and goto fail? Could add a fail_pre_kwargs label, if we want to avoid unnecessary cleanup.

Of course, really the huge hack section of this function should be extracted from the rest of it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I had a goto cleanup_out_op and passed on a retval, but then I found that it just became confusing, and might hinder ripping this whole part out (which, as you said, should be done, but better not in this PR...). Fortunately, here we know exactly what has been done up to this point (indeed, this is partially why I moved the nin == 2 to the if-statement; I wanted to be absolutely sure I decref'd just the right number of operands, even if someone gave their ufunc a name that matches but has more than 2 inputs).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More for completeness: See #11282

@mattip
Copy link
Member

mattip commented Jun 7, 2018

needs a rebase

Instead of leaving it to the caller to clean up references
to operands.
@mhvk mhvk force-pushed the ufunc-parsing-better-cleanup branch from 31a4a29 to 584e82f Compare June 8, 2018 00:49
@mhvk
Copy link
Contributor Author

mhvk commented Jun 8, 2018

Yes, rebase needed now that axis is in! Done.

@mhvk
Copy link
Contributor Author

mhvk commented Jun 8, 2018

I think this could be in 1.15 but doesn't have to be. But having it in may be handy for review of #11282.

@charris
Copy link
Member

charris commented Jun 9, 2018

I've marked this for 1.15, everyone happy with its current state?

@eric-wieser
Copy link
Member

There's a memory leak I discovered in #11295 that will be easier to fix after this goes in - so I reckon we go for it

@charris charris merged commit 2125eca into numpy:master Jun 10, 2018
@charris
Copy link
Member

charris commented Jun 10, 2018

OK, let's put this in. Thanks Marten.

@mhvk mhvk deleted the ufunc-parsing-better-cleanup branch June 10, 2018 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants