Skip to content

numpy.zeros((0, 2**31-1)) raises MemoryError #7819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cgohlke opened this issue Jul 10, 2016 · 11 comments
Closed

numpy.zeros((0, 2**31-1)) raises MemoryError #7819

cgohlke opened this issue Jul 10, 2016 · 11 comments

Comments

@cgohlke
Copy link
Contributor

cgohlke commented Jul 10, 2016

Using numpy-1.11.1 with the patch for issue #7813 on 64-bit Python, creating an array of size=0 several times raises MemoryError on a Windows 10 system with 32 GB RAM:

Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> a = numpy.zeros((0, 2**31-1))
>>> a.size
0
>>> a = numpy.zeros((0, 2**31-1))
>>> a = numpy.zeros((0, 2**31-1))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

According to the the VMMap tool, every call to numpy.zeros((0, 2**31-1)) allocates ~16GB on the heap.
I would not expect the creation of an array of size=0 to fail with MemoryError (64-bit) or ValueError (32-bit).

@cgohlke
Copy link
Contributor Author

cgohlke commented Jul 10, 2016

See also #7813 (comment)

@seberg
Copy link
Member

seberg commented Jul 10, 2016

Oh, is this a new bug or an old? Oh well, probably does not matter. I think it is supposed to allocate a single element for these arrays, though even that should really be unnecessary.

@charris
Copy link
Member

charris commented Jul 10, 2016

@seberg Old. There is a comment in the code that some memory is needed:

        /*
         * Allocate something even for zero-space arrays
         * e.g. shape=(0,) -- otherwise buffer exposure
         * (a.data) doesn't work as it should.
         */

But that path is never taken currently. I'll put up a patch.

@charris
Copy link
Member

charris commented Jul 10, 2016

See #7820.

@charris
Copy link
Member

charris commented Jul 10, 2016

If someone will review #7820 and #7817 I'll try to get a 1.11.2rc1 out tomorrow.

@juliantaylor
Copy link
Contributor

hm there is nothing worth releasing on the branch, why the rush?
this issue is super unimportant

@charris
Copy link
Member

charris commented Jul 10, 2016

There is also a fix for a failing test in pending scipy release.

@charris
Copy link
Member

charris commented Jul 10, 2016

@juliantaylor See #7515. However, the release schedule is negotiable.

@cgohlke
Copy link
Contributor Author

cgohlke commented Jul 10, 2016

#7813 causes scipy.test() to crash on 32 bit Python 2.6-3.4 for Windows. That's how I discovered the issues.

@cgohlke
Copy link
Contributor Author

cgohlke commented Jul 12, 2016

Thanks for the fix and backport @charris.

Still, I would not expect numpy.zeros((0, 2**31-1)) to raise ValueError: array is too big; arr.size * arr.dtype.itemsize is larger than the maximum possible size. on 32 bit but that is a different issue.

@cgohlke cgohlke closed this as completed Jul 12, 2016
@seberg
Copy link
Member

seberg commented Jul 13, 2016

Well, it is a bit of a theoretical construct and there are some things (like stride/size after view) which may misbehave. So I think it is correct to just disallow it. I admit the error message might be a bit confusing in this specific case.
However, I doubt there is any reason to have a (0, huge) array in a place where a (1, huge) array can never occur, so it also seems unlikely for anyone to run into it, unless when designing weird tests or just trying out edge cases for fun.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants