ENH: Show subclass type in dtype repr and str of structured arrays #5483

ahaldane · 2015-01-22T20:45:42Z

Hello, this is an enhancement that clarifies (or solves?) #3581.

I discussed it on the mailing list in a message "Re: structured arrays, recarrays, and record arrays" on Jan 19 2015. I didn't get any replies, but hopefully that merely means "no opinion" rather than "bad idea". I though maybe implementing it would draw more interest.

What it does: For structured arrays, if the dtype is not np.void then print the dtype as (base_dtype, dtype). This is the standard form for subclassed dtypes as documented in reference/arrays.dtypes, although that page says this form is discouraged. (Perhaps that discouragement should also be removed?).

This solves the problem that array objects may "look" identical even though their dtypes differ, as happened in issue 3581.

Old Behavior:

>>> a = np.array([(1,'ABC'), (2, "DEF")], dtype=[('foo', int), ('bar', 'S4')])
>>> np.rec.array(a)
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', 'S4')])
>>> a.view(np.recarray)
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', 'S4')])

New Behavior:

>>> a = np.array([(1,'ABC'), (2, "DEF")], dtype=[('foo', int), ('bar', 'S4')])
>>> np.rec.array(a)
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=(record, [('foo', '<i8'), ('bar', 'S4')]))
>>> a.view(np.recarray)
rec.array([(1, 'ABC'), (2, 'DEF')], 
      dtype=[('foo', '<i8'), ('bar', 'S4')])

Now, probably instead of saying simply "record" in these examples, it should say something like "numpy.record" or "np.record". Any preference?

charris · 2015-01-22T22:44:02Z

numpy/core/src/multiarray/descriptor.c

+    if(dtype->type_num == NPY_VOID && dtype->typeobj != &PyVoidArrType_Type) {
+        PyObject *typestr = PyUString_FromString(dtype->typeobj->tp_name);
+
+        PyObject *ret = PyUString_FromString("(");


|They all return NULL or -1 if an exception occurs.

I wonder if ret should be checked for NULL? It's possible an error could occur and we wouldn't know it.

charris · 2015-01-22T23:08:54Z

One thing that is likely a problem is that the dtype string needs to be parseable by the dtype constructor. That would affect pickling and other things.

charris · 2015-01-22T23:10:52Z

And I think that is why the tests are failing.

ahaldane · 2015-01-22T23:14:33Z

Yes, sorry, I completely forgot to run the tests!

Indeed, the test fails because "NameError: name 'record' is not defined". I think having it listed as numpy.record will fix this. I will figure out how to do this later.

ahaldane · 2015-01-23T17:53:48Z

Whew, that was trickier than I expected.

The problem was that dtype->typeobj->tp_name does not reliably give you the name of the class in module.name form (If it did, my code before would have worked). I expected tp_name to be "numpy.core.records.record", but instead it was just "record".

As documented here, tp_name will be of form "module.typename" for (properly created) "statically allocated" types (ie, all the builtin numpy types). In contrast, for "dynamically allocated" types (ie, created in python code), tp_name is simply the typename (no "module.").

As far as I can tell, the reliable way to get a string of form "module.typename" is to get the __module__ and __name__ strings from the object's attributes and paste them together.

I'm pretty sure all the other uses of tp_name in descriptor.c are OK, assuming that user-defined types are always defined in C and are therefore statically allocated.

The next complication is that __module__ and __name__ do not make sense by default for numpy.record and numpy.recarray, so I set them manually in records.py. This also makes the type string look nicer:

Setup:

>>> arr = np.array([(1,'a'), (2,'b')], dtype=[('one', int), ('two', 'S1')])
>>> rec = np.rec.array(arr)

Old:

>>> type(arr), type(rec)
(numpy.ndarray, numpy.core.records.recarray)

New:

>>> type(arr), type(rec)
(numpy.ndarray, numpy.recarray)

Also, I think it's possible that this pull might be a bug fix rather than just an enhancement, because the repr of a dtype does not give back the same dtype without this fix. For example,

Old:

>>> rec.dtype.type
numpy.core.records.record
>>> eval(repr(rec.dtype)).type
numpy.void

New:

>>> rec.dtype.type
numpy.record
>>> eval(repr(rec.dtype)).type
numpy.record

I though this might screw up pickling in the old code, and that pickling+unpickling would cause the numpy.core.records.record type to be lost. But actually pickling retains the type attribute properly, so that's not a problem. I haven't yet found an existing clear case where the missing base_type causes a problem.

ahaldane · 2015-01-23T22:14:24Z

Thanks @charris for your comments. I've made changes you suggested.

I don't see your comments on this page anymore (I don't know why), but I replied to them. You can read my replies here: ahaldane@d2bf24b

charris · 2015-01-24T01:17:24Z

numpy/core/records.py

@@ -215,6 +215,10 @@ def _createdescr(self, byteorder):
 class record(nt.void):
    """A data-type scalar that allows field access as attribute lookup.
    """
+
+    __name__ = 'record'


Please add a comment explaining why this is done. Same below.

charris · 2015-01-24T01:18:10Z

LGTM modulo comment nitpick.

This is a modification to the dtype str and repr functions what helps solve numpy#3581. I discussed it on the mailing list in a message "Re: structured arrays, recarrays, and record arrays" on Jan 19 2015. I didn't get any replies, but hopefully that merely means "no opinion" rather than "bad idea". What it does: For structured arrays, if the dtype is not np.void then print the dtype as `(base_dtype, dtype)`. New Behavior: >>> a = np.array([(1,'ABC'), (2, "DEF")], dtype=[('foo', int), ('bar', 'S4')]) >>> np.rec.array(a) rec.array([(1, 'ABC'), (2, 'DEF')], dtype=(numpy.record, [('foo', '<i8'), ('bar', 'S4')])) >>> a.view(np.recarray) rec.array([(1, 'ABC'), (2, 'DEF')], dtype=[('foo', '<i8'), ('bar', 'S4')])

ahaldane · 2015-01-24T17:09:09Z

Done.

Actually changing the name and module isn't absolutely necessary now that the C code is right, but it's still nicer with them simplified this way.

charris · 2015-01-24T17:41:16Z

Merged, thanks @ahaldane.

ENH: Show subclass type in dtype repr and str of structured arrays

In numpy#5483, I solved the problem that a "recarray" and a "record array" (nomenclature defined in numpy#5482) looked identical by making sure that a type's subclass was listed in the repr. However, recarrays are still represented using the function 'rec.array' even though this function technically creates record arrays, not recarrays. So I have updated recarray.__repr__. Setup: >>> a = np.array([(1,'ABC'), (2, "DEF")], dtype=[('foo', int), ('bar', 'S4')]) >>> recordarr = np.rec.array(a) >>> recarr = a.view(np.recarray) Behavior after numpy#5483: >>> recordarr rec.array([(1, 'ABC'), (2, 'DEF')], dtype=(numpy.record, [('foo', '<i8'), ('bar', 'S4')])) >>> recarr rec.array([(1, 'ABC'), (2, 'DEF')], dtype=[('foo', '<i8'), ('bar', 'S4')]) New Behavior: >>> recordarr rec.array([(1, 'ABC'), (2, 'DEF')], dtype=[('foo', '<i8'), ('bar', '|S4')]) >>> recarr array([(1, 'ABC'), (2, 'DEF')], dtype=[('foo', '<i8'), ('bar', 'S4')]).view(numpy.recarray)

charris reviewed Jan 22, 2015
View reviewed changes

ahaldane force-pushed the dtype_showtype branch from df3db29 to 1d6c3a0 Compare January 23, 2015 17:53

ahaldane force-pushed the dtype_showtype branch 4 times, most recently from bae39b0 to 745feb7 Compare January 23, 2015 22:08

charris reviewed Jan 24, 2015
View reviewed changes

ahaldane force-pushed the dtype_showtype branch from 745feb7 to 86efa7d Compare January 24, 2015 17:06

charris added a commit that referenced this pull request Jan 24, 2015

Merge pull request #5483 from ahaldane/dtype_showtype

7ce93ba

ENH: Show subclass type in dtype repr and str of structured arrays

charris merged commit 7ce93ba into numpy:master Jan 24, 2015

ahaldane deleted the dtype_showtype branch January 25, 2015 19:15

ahaldane mentioned this pull request Jan 29, 2015

BUG: recarray __repr__ gives inaccurate representation #5523

Merged

ahaldane mentioned this pull request Dec 9, 2015

numpy.ndarray.__repr__ shorts incorrect/inconsistent dtype for unions #6802

Closed

ahaldane mentioned this pull request Jan 15, 2018

BUG/ENH: Improve output for structured non-void types #10381

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Show subclass type in dtype repr and str of structured arrays #5483

ENH: Show subclass type in dtype repr and str of structured arrays #5483

ahaldane commented Jan 22, 2015

charris Jan 22, 2015

charris commented Jan 22, 2015

charris commented Jan 22, 2015

ahaldane commented Jan 22, 2015

ahaldane commented Jan 23, 2015

ahaldane commented Jan 23, 2015

charris Jan 24, 2015

charris commented Jan 24, 2015

ahaldane commented Jan 24, 2015

charris commented Jan 24, 2015

ENH: Show subclass type in dtype repr and str of structured arrays #5483

ENH: Show subclass type in dtype repr and str of structured arrays #5483

Conversation

ahaldane commented Jan 22, 2015

charris Jan 22, 2015

Choose a reason for hiding this comment

charris commented Jan 22, 2015

charris commented Jan 22, 2015

ahaldane commented Jan 22, 2015

ahaldane commented Jan 23, 2015

ahaldane commented Jan 23, 2015

charris Jan 24, 2015

Choose a reason for hiding this comment

charris commented Jan 24, 2015

ahaldane commented Jan 24, 2015

charris commented Jan 24, 2015