ENH: Improved performance of PyArray_FromAny for sequences of array-like #13399

superbobry · 2019-04-24T13:18:34Z

Prior to this commit

np.array([array_like])

would recursively copy each element of array_like. This is due to the fact that
setArrayFromSequence only special-cased lists of NumPy arrays, any other
object was treated as a sequence even if it supported buffer or __array*__
interfaces. See tensorflow/tensorflow#27692 for details.

The commit generalizes the special-case in setArrayFromSequence to any
array-like, i.e. a buffer or an object with __array__, __array_interface__
__array_struct__.

mattip · 2019-04-26T22:03:15Z

Running

python -mtimeit -s "import numpy as np; b=memoryview(np.arange(1000))" \
       "a = np.array([b] * 100)"

shows a marked improvement. Before

100 loops, best of 3: 2.6 msec per loop

After

10000 loops, best of 3: 71.9 usec per loop

numpy/core/src/multiarray/ctors.c

mattip · 2019-04-26T22:23:04Z

numpy/core/src/multiarray/ctors.c

+            return NULL;
+        }
+    }
+


For reviewers: the original code had a check clause at this point if (tmp != Py_NotImplemented){ } which has been folded into the logic below instead

mattip · 2019-04-26T22:36:00Z

LGTM. Maybe could add a benchmark to benchmarks/benchmarks/bench_core.Core.time*.

eric-wieser · 2019-04-27T00:44:51Z

I worry that this needs to handle broken ctypes buffer formats in the same way we do elsewhere. Will try to look at the diff later

superbobry · 2019-04-27T08:13:13Z

I was also thinking of addressing #8562 but I hesitated because buffer interface allows for ~free shape queries whereas __array__ potentially allocates a new array which seems wasteful.

@mattip I've added a benchmark, could you have another look please?

numpy/core/src/multiarray/ctors.c

mattip · 2019-04-27T16:59:44Z

I was also thinking of addressing #8562

Great, but that should be a follow-on PR, not part of this one, unless it is a one-line fix

mattip · 2019-04-30T12:38:47Z

@eric-wieser since this PR reuses existing code, could we push off possible broken ctypes PEP 3118 handling to a different issue/PR? This seems like a clear win for more common memoryview and __array__ use cases

numpy/core/src/multiarray/ctors.c

eric-wieser · 2019-05-06T00:33:37Z

@mattip: At a cursory glance, I was worried this introduced a regression in the following case:

class PaddedStruct(ctypes.Structure):
    _fields_ = [('a', ctypes.c_uint8), ('b', ctypes.c_uint16)]

data = (PaddedStruct * 4)()
np.array(data)  # ok
np.array(list(data))
# ValueError: invalid literal for int() with base 10: b'\x00\x00\x00\x00'

Testing seems to show that the error is produced both before and after this patch, so there's no need to worry about it here.

numpy/core/src/multiarray/ctors.c

Prior to this commit np.array([array_like]) would recursively copy each element of array_like. This is due to the fact that setArrayFromSequence only special-cased lists of NumPy arrays, any other object was treated as a sequence even if it supported buffer or __array*__ interfaces. See tensorflow/tensorflow#27692 for details. The commit generalizes the special-case in setArrayFromSequence to any array-like, i.e. a buffer or an object with __array__, __array_interface__ __array_struct__.

mattip · 2019-05-26T05:25:05Z

I created issue #13628 about np.array(list(ctypes-struct)) which does not need the padding to fail.

This looks ready to merge, any more comments?

mattip · 2019-05-26T11:32:20Z

Thanks @superbobry

mattip · 2019-05-28T17:45:21Z

This seems to have broken pandas, where import numpy as np, pandas as pd; np.array([pd.DataFrame()]) used to work and now raises ValueError: could not broadcast input array from shape (0,0) into shape (0)

Fixup of gh-13399 to avoid a regression with pandas. The actual output after this is the same as before, but incorrect with respect to the output shape. However, it seems a bit larger and an issue to look at after the 1.17 release. * BUG: test, fix regression for array([pandas.DataFrame()]) * BUG: refactor to make sure tmp is DECREFfed

…rray-like" This reverts commit 71fc59d which was part of numpygh-13399 and the followup commit ba53a63 from numpygh-13663. The issue is that we mix Sequence protocol and `__array__` usage when coercing arrays. It seems easiest to revert this for the 1.17.x release and make a larger breaking change in the 1.18.x release cycle. Since this only occurs for stranger array-likes. This revert is limited to the 1.17.x branch for now.

…rray-like" This reverts commit 71fc59d which was part of numpygh-13399 and the followup commit ba53a63 from numpygh-13663. The issue is that we mix Sequence protocol and `__array__` usage when coercing arrays. It seems easiest to revert this for the 1.18.x release and make a larger breaking change in the 1.19.x release cycle. Since this only occurs for stranger array-likes.

superbobry force-pushed the np.array-list-of-array-like branch from 5cf5ee7 to edcb829 Compare April 24, 2019 13:19

charris added 01 - Enhancement component: numpy._core labels Apr 24, 2019

superbobry force-pushed the np.array-list-of-array-like branch from edcb829 to 0187811 Compare April 24, 2019 13:56

mattip reviewed Apr 26, 2019

View reviewed changes

numpy/core/src/multiarray/ctors.c Show resolved Hide resolved

mattip reviewed Apr 26, 2019

View reviewed changes

eric-wieser self-requested a review April 27, 2019 00:47

superbobry force-pushed the np.array-list-of-array-like branch from 0187811 to 1b57708 Compare April 27, 2019 08:27

mattip reviewed Apr 27, 2019

View reviewed changes

numpy/core/src/multiarray/ctors.c Show resolved Hide resolved

superbobry force-pushed the np.array-list-of-array-like branch from 1b57708 to 9f8826a Compare May 1, 2019 11:38

eric-wieser reviewed May 6, 2019

View reviewed changes

numpy/core/src/multiarray/ctors.c Show resolved Hide resolved

eric-wieser reviewed May 6, 2019

View reviewed changes

numpy/core/src/multiarray/ctors.c Show resolved Hide resolved

superbobry force-pushed the np.array-list-of-array-like branch from 9f8826a to 2363084 Compare May 12, 2019 20:26

superbobry force-pushed the np.array-list-of-array-like branch from 2363084 to 71fc59d Compare May 19, 2019 19:06

mattip mentioned this pull request May 26, 2019

BUG: converting list of ctypes.Structure to ndarray #13628

Open

mattip merged commit 95aacf5 into numpy:master May 26, 2019

superbobry mentioned this pull request May 26, 2019

Converting Tensor to Numpy array extremely slow in TF 2.0 tensorflow/tensorflow#27692

Closed

mattip mentioned this pull request May 28, 2019

CI Failing - Linux py37_np_dev - test_constructor_list_frames pandas-dev/pandas#26546

Closed

mattip mentioned this pull request May 28, 2019

BUG: regression in setArrayFromSequence #13659

Closed

This was referenced May 29, 2019

BUG: regression for array([pandas.DataFrame()]) #13663

Merged

BUG: np.array cleared errors occured in PyMemoryView_FromObject #13507

Merged

stonebig mentioned this pull request May 30, 2019

release 2019-02 follow-up winpython/winpython#724

Closed

12 tasks

seberg mentioned this pull request Jul 22, 2019

Regression: Change in behaviour for np.array() on class that defines __array__ #13958

Closed

seberg mentioned this pull request Jul 24, 2019

REV: "ENH: Improved performance of PyArray_FromAny for sequences of array-like" #14109

Merged

charris mentioned this pull request Nov 26, 2019

REV: "ENH: Improved performance of PyArray_FromAny for sequences of array-like #14983

Merged

charris mentioned this pull request Dec 2, 2019

REV: "ENH: Improved performance of PyArray_FromAny for sequences of a…" #15027

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

ENH: Improved performance of PyArray_FromAny for sequences of array-like #13399

ENH: Improved performance of PyArray_FromAny for sequences of array-like #13399

Uh oh!

superbobry commented Apr 24, 2019

Uh oh!

mattip commented Apr 26, 2019

Uh oh!

Uh oh!

mattip Apr 26, 2019

Uh oh!

mattip commented Apr 26, 2019

Uh oh!

eric-wieser commented Apr 27, 2019

Uh oh!

superbobry commented Apr 27, 2019 •

edited

Loading

Uh oh!

Uh oh!

mattip commented Apr 27, 2019

Uh oh!

mattip commented Apr 30, 2019

Uh oh!

Uh oh!

eric-wieser commented May 6, 2019

Uh oh!

Uh oh!

mattip commented May 26, 2019

Uh oh!

mattip commented May 26, 2019

Uh oh!

mattip commented May 28, 2019

Uh oh!

Uh oh!

Uh oh!

ENH: Improved performance of PyArray_FromAny for sequences of array-like #13399

ENH: Improved performance of PyArray_FromAny for sequences of array-like #13399

Uh oh!

Conversation

superbobry commented Apr 24, 2019

Uh oh!

mattip commented Apr 26, 2019

Uh oh!

Uh oh!

mattip Apr 26, 2019

Choose a reason for hiding this comment

Uh oh!

mattip commented Apr 26, 2019

Uh oh!

eric-wieser commented Apr 27, 2019

Uh oh!

superbobry commented Apr 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mattip commented Apr 27, 2019

Uh oh!

mattip commented Apr 30, 2019

Uh oh!

Uh oh!

eric-wieser commented May 6, 2019

Uh oh!

Uh oh!

mattip commented May 26, 2019

Uh oh!

mattip commented May 26, 2019

Uh oh!

mattip commented May 28, 2019

Uh oh!

Uh oh!

superbobry commented Apr 27, 2019 •

edited

Loading