Skip to content

BUG: np.lexsort MemoryError on empty array with non-standard strides #14228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
batterseapower opened this issue Aug 8, 2019 · 3 comments · Fixed by #14240
Closed

BUG: np.lexsort MemoryError on empty array with non-standard strides #14228

batterseapower opened this issue Aug 8, 2019 · 3 comments · Fixed by #14240
Labels
00 - Bug 06 - Regression component: numpy._core

Comments

@batterseapower
Copy link
Contributor

Numpy 1.17.0 raises MemoryError when using lexsort on an empty array with non-standard strides. If strides is the natural one, or if there is >= 1 element in the input, then it works as expected.

Reproducing code example:

import numpy as np

xs = np.array([], dtype='i8')
assert xs.strides == (8,)
assert np.lexsort((xs,)).shape[0] == 0 # Works

ys = np.array([0], dtype='i8')
ys.strides = (16,)
assert np.lexsort((ys,)).shape[0] == 1 # Works

xs.strides = (16,)
assert np.lexsort((xs,)).shape[0] == 0 # MemoryError

Numpy/Python version information:

1.17.0 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)]
@batterseapower
Copy link
Contributor Author

batterseapower commented Aug 8, 2019 •
edited
Loading

The problem is that in the needcopy case of PyArray_LexSort, a length 0 array causes failure:

numpy/numpy/core/src/multiarray/item_selection.c

Line 1519 in ea965e4

if (N == 0 || maxelsize == 0 || sizeof(npy_intp) == 0) {

No error code at the point when it fails, in which case the function assumes that the cause is memory exhaustion:

numpy/numpy/core/src/multiarray/item_selection.c

Line 1621 in ea965e4

PyErr_NoMemory();

The fix is probably to ensure that needcopy=false if N=0.

@mattip mattip changed the title np.lexsort MemoryError on empty array with non-standard strides BUG: np.lexsort MemoryError on empty array with non-standard strides Aug 8, 2019
@mattip
Copy link
Member

mattip commented Aug 8, 2019

Thanks for tracking this down. Is this a regression from previous versions?

Would you like to make a pull request? Please add a failing test to numpy/core/tests/test_regression.py (if it is a regression) or numpy/core/tests/test_multiarray.py near the existing lexsort tests, make sure the test fails with python runtests.py, then add the fix and make sure the test passes.

@batterseapower
Copy link
Contributor Author

batterseapower commented Aug 8, 2019 •
edited
Loading

It definitely does work in 1.13.1 and (I think) in 1.16 so this is a regression. I believe it was introduced in 03052b7 (#13691)

I will make a PR.

@seberg seberg added this to the 1.17.1 release milestone Aug 8, 2019
@batterseapower batterseapower mentioned this issue Aug 9, 2019
Merged
seberg pushed a commit to batterseapower/numpy that referenced this issue Aug 22, 2019
@batterseapower @seberg
BUG: don't fail when lexsorting some empty arrays (numpy#14228)
a062d07
charris pushed a commit to charris/numpy that referenced this issue Aug 23, 2019
@batterseapower @charris
BUG: don't fail when lexsorting some empty arrays (numpy#14228)
4452556
@charris charris mentioned this issue Aug 23, 2019
Merged
charris added a commit that referenced this issue Aug 23, 2019
@charris
Merge pull request #14338 from charris/backport-14240
66388aa 
BUG: don't fail when lexsorting some empty arrays (#14228)
maxwell-aladago pushed a commit to maxwell-aladago/numpy that referenced this issue Sep 6, 2019
@batterseapower @maxwell-aladago
BUG: don't fail when lexsorting some empty arrays (numpy#14228)
6447fe4
@WarrenWeckesser WarrenWeckesser mentioned this issue Apr 16, 2020
Closed
@eric-wieser eric-wieser mentioned this issue Apr 16, 2020
Closed
@charris charris removed the 09 - Backport-Candidate PRs tagged should be backported label Jul 19, 2020
@charris charris removed this from the 1.17.1 release milestone Jul 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug 06 - Regression component: numpy._core
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: Don't fail when lexsorting some empty arrays. batterseapower/numpy
4 participants
@batterseapower @seberg @charris @mattip