NEP: issue deprecation warning when creating ragged array (NEP 34) #15119

mattip · 2019-12-16T17:47:09Z

Redo gh-14794 ~~but add a np.allow_object sentinel to retain legacy behaviour.~~

Also see gh-15118 which is tangentially connected to this PR, where the context argument now serves dual purposes.

seberg

LGTM, just some nitpicks. Others may have opinions about using context for this purpose, but it is convenient. Mainly waiting for a clear decision the whole thing I guess.

numpy/core/numerictypes.py

numpy/core/src/multiarray/common.c

numpy/core/src/multiarray/ctors.c

numpy/core/src/multiarray/multiarraymodule.c

numpy/core/tests/test_multiarray.py

mattip · 2019-12-30T14:27:40Z

waiting on gh-15083 to resolve the name and need for the singleton

mattip · 2020-01-13T22:00:42Z

It turns out downstream libraries (I checked pandas, scipy) adapted to NEP 34 even without the sentinel. Once this is ready and reviewed, I would like to try again to see if it causes less noise this time

seberg

LGTM (minor things). IIRC this is pretty much what we had before, right? We probably still need to (re-) add the release notes?

numpy/core/tests/test_multiarray.py

numpy/core/tests/test_deprecations.py

numpy/core/_internal.py

doc/release/upcoming_changes/template.rst

eric-wieser · 2020-01-14T12:44:56Z

numpy/core/src/multiarray/ctors.c

-        if (is_object) {
+        if (is_object != DISCOVERED_OK) {
+            if (is_object == DISCOVERED_RAGGED && requested_dtype == NULL) {
+                /* NumPy 1.18, 2019-11-01 */


Should this be 19 now?

Looks like this was missed? I can fix it up though.

numpy/core/src/multiarray/ctors.c

seberg

Sorry for dragging this out, but I am considering if we should try to spell out/target the error message for the unexperienced user who likely just got it by accident. We could even add that other tools may exist to handle such data, but not sure about that?

seberg · 2020-01-16T15:08:10Z

numpy/core/src/multiarray/ctors.c

-        if (is_object) {
+        if (is_object != DISCOVERED_OK) {
+            if (is_object == DISCOVERED_RAGGED && requested_dtype == NULL) {
+                /* NumPy 1.18, 2019-11-01 */


Looks like this was missed? I can fix it up though.

seberg · 2020-01-16T15:21:59Z

numpy/core/src/multiarray/ctors.c

+                /* NumPy 1.18, 2019-11-01 */
+                if (DEPRECATE("Creating an ndarray from ragged nested sequences "
+                    "and automatic object dtype detection is deprecated. Use "
+                    "dtype=object or an exact dtype") < 0)


Now that Eric pointed out that we may want to include "ragged", I am thinking we should actually bloat this up a lot more. Maybe even create a VisibleDeprecationWarning (although that is a bit annoying).

The reason is that we are currently aiming the error at users who want to keep the old behaviour, when actually most users who see it are likely people who simply mistyped and thus have a bug. For them "ragged" is probably not helpful except as a google term.

It is a bit long I know, but am considering something like:

You are attempting to create an array, or doing an array operation, with uneven shape (ragged). This is not supported and may be an input error such as np.array([[1], [1, 2]]) where the first list is shorter than the second. If you wish to create an array containing sequences of different length you must create an array with dtype=object in the future.

Changed to a VisibleDeprecationWarning, extended the message, rebased off master, and fixed the deprecation version in e11d57a.

Although in my mind the difference between a VisibleDeprecationWarning and an Error is negligible: emitting a VisibleDeprecationWarning is so annoying it means there is no real deprecation period, users will, in practice, have to fix their code immediately.

Searching for "ragged nested sequences numpy" the first hit is NEP 34, so I think this text is good enough.

mattip · 2020-01-16T20:34:53Z

Should be squashed before merging so we can revert a single changeset.

seberg

Good point about the googlebility of "ragged nested sequences". I am OK with this, just thoughts. About the VisibleDeprecationWarning if anyone else has an opinion, please just note. I think I am still a little bit in favor: The assumption is that in the vast majority of (non-library) use cases, it currently is actually a silent error. If anyone thinks that is not true, we should first do the normal deprecation.

seberg · 2020-01-16T20:36:21Z

numpy/core/src/multiarray/ctors.c

-        if (is_object) {
+        if (is_object != DISCOVERED_OK) {
+            static PyObject * visibleDeprecationWarning = NULL;
+            if (visibleDeprecationWarning == NULL) {


npy_cache_import includes this if (and is guaranteed to be inlined)

seberg · 2020-01-16T20:36:32Z

numpy/core/src/multiarray/ctors.c

+            if (is_object == DISCOVERED_RAGGED && requested_dtype == NULL) {
+                /* NumPy 1.19, 2019-11-01 */
+                if (PyErr_WarnEx(visibleDeprecationWarning, "Creating an "
+                    " ndarray from ragged nested sequences (which is a "


Suggested change

" ndarray from ragged nested sequences (which is a "

"ndarray from ragged nested sequences (which is a "

Ack, forgot to press submit: Maybe rather than the bracketed explanation, put in the example: such as '[[1], [2, 3]]' where '[2, 3]' is longer than '[1]'? I would hope that is more useful.

Not sure though, and I am fine with it as is (we can always revise as well)

mattip · 2020-01-16T21:11:21Z

Squashed to one commit

seberg · 2020-01-21T18:32:54Z

Thanks matti, let me put this in. I feel the error message could be nicer, but do not have a good idea right now and its easy to fix up. (Will squash some whitespace ony changes, so not waiting for it).

…cation dimension=None makes no forced dimension conversion handle numpy.VisibleDeprecationWarning and future ValueError produced by creating a numpy.ndarray with elements of varying shape or type introduced by numpy/numpy#15119 Also see https://numpy.org/neps/nep-0034-infer-dtype-is-object.html

idanmiara · 2023-06-13T11:37:18Z

Thanks for this fix! Just pointing out to whoever googled their way here later:
The VisibleDeprecationWarning was replaced by ValueError in numpy v1.24:
https://numpy.org/devdocs/release/1.24.0-notes.html

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

mattip added 07 - Deprecation component: NEP labels Dec 16, 2019

mattip force-pushed the nep-0034-impl branch from c6083ad to bc04f1a Compare December 16, 2019 19:14

mattip mentioned this pull request Dec 16, 2019

COMPAT: np 1.18 wants explicit dtype=object) pandas-dev/pandas#30035

Closed

seberg approved these changes Dec 20, 2019

View reviewed changes

mattip force-pushed the nep-0034-impl branch from b2dfa83 to 34425f7 Compare January 13, 2020 19:39

mattip mentioned this pull request Jan 13, 2020

NEP: document reimplementation of NEP 34 #15323

Merged

mattip requested a review from seberg January 14, 2020 03:20

seberg reviewed Jan 14, 2020

View reviewed changes

numpy/core/tests/test_multiarray.py Outdated Show resolved Hide resolved

numpy/core/tests/test_deprecations.py Outdated Show resolved Hide resolved

numpy/core/_internal.py Show resolved Hide resolved

doc/release/upcoming_changes/template.rst Show resolved Hide resolved

seberg added the 56 - Needs Release Note. Needs an entry in doc/release/upcoming_changes label Jan 14, 2020

eric-wieser reviewed Jan 14, 2020

View reviewed changes

numpy/core/src/multiarray/ctors.c Outdated Show resolved Hide resolved

seberg approved these changes Jan 16, 2020

View reviewed changes

mattip force-pushed the nep-0034-impl branch from 4e89930 to e11d57a Compare January 16, 2020 20:26

seberg reviewed Jan 16, 2020

View reviewed changes

NEP: issue deprecation warning when creating ragged array (NEP 34)"

22ac3c7

mattip force-pushed the nep-0034-impl branch from e11d57a to 22ac3c7 Compare January 16, 2020 21:10

mattip removed the 56 - Needs Release Note. Needs an entry in doc/release/upcoming_changes label Jan 19, 2020

mattip requested a review from seberg January 19, 2020 23:32

STY: Tiny whitespaces fixes

8ff45dc

seberg merged commit 287a9d0 into numpy:master Jan 21, 2020

seberg changed the title ~~NEP, ENH: implement NEP34 (deprecate ragged object array creation)~~ NEP: issue deprecation warning when creating ragged array (NEP 34) Jan 21, 2020

pllim mentioned this pull request Jan 21, 2020

TST: VisibleDeprecationWarning causing numpy-dev failures astropy/astropy#9879

Closed

mattip mentioned this pull request Jan 22, 2020

"Controlled" creation of object arrays #15047

Closed

QuLogic mentioned this pull request May 1, 2020

Prepare for ragged array warnings in NumPy 1.19 matplotlib/matplotlib#17289

Merged

6 tasks

kmantel mentioned this pull request Jul 1, 2020

Fix numpy 1.19 deprecations PrincetonUniversity/PsyNeuLink#1697

Merged

f380cedric mentioned this pull request Sep 4, 2020

DeprecatedWarning: apply_along_axis and dtype=object #17249

Closed

mattip deleted the nep-0034-impl branch November 2, 2020 08:29

MarieStLaurent mentioned this pull request Nov 10, 2021

Visible Deprecation Warning creating ndarray from ragged nested sequences noahbenson/neuropythy#24

Closed

TransMux mentioned this pull request Jun 14, 2022

VisibleDeprecationWarning(Creating an ndarray from ragged nested sequences) When Saving data cache datawhalechina/torch-rechub#22

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NEP: issue deprecation warning when creating ragged array (NEP 34) #15119

NEP: issue deprecation warning when creating ragged array (NEP 34) #15119

mattip commented Dec 16, 2019 •

edited

Loading

seberg left a comment

mattip commented Dec 30, 2019

mattip commented Jan 13, 2020

seberg left a comment

eric-wieser Jan 14, 2020

mattip Jan 14, 2020

seberg Jan 16, 2020

seberg left a comment

seberg Jan 16, 2020

seberg Jan 16, 2020

mattip Jan 16, 2020

mattip Jan 16, 2020

mattip commented Jan 16, 2020

seberg left a comment

seberg Jan 16, 2020

mattip Jan 16, 2020

seberg Jan 16, 2020

seberg Jan 16, 2020

mattip Jan 16, 2020

mattip commented Jan 16, 2020

seberg commented Jan 21, 2020

idanmiara commented Jun 13, 2023

	" ndarray from ragged nested sequences (which is a "
	"ndarray from ragged nested sequences (which is a "

NEP: issue deprecation warning when creating ragged array (NEP 34) #15119

NEP: issue deprecation warning when creating ragged array (NEP 34) #15119

Conversation

mattip commented Dec 16, 2019 • edited Loading

seberg left a comment

Choose a reason for hiding this comment

mattip commented Dec 30, 2019

mattip commented Jan 13, 2020

seberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattip commented Jan 16, 2020

seberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattip commented Jan 16, 2020

seberg commented Jan 21, 2020

idanmiara commented Jun 13, 2023

mattip commented Dec 16, 2019 •

edited

Loading