This is problematic with something like this:

>>> x.view((np.byte, x.nbytes))  # this should usually work
ValueError: new type not compatible with array.
>>> x.view((np.byte, x.nbytes * 2))  # ???
array([116,   0,   0,   0, 101,   0,   0,   0, 115,   0,   0,   0, 116,
         0,   0,   0], dtype=int8)

Although it seems that it is sometimes correct:

>>> b = buffer(x)
>>> len(b) == x.nbytes
True
>>> list(b)
>>> list(buffer(x))
['t', '\x00', 'e', '\x00', 's', '\x00', 't', '\x00']

Why does unicode seem to have different binary representations depending on how you look at it? Aren't both buffer and .view((np.byte,...)) supposed to simply expose the underlying memory?

Python 2,7 can be compiled with either 16 or 32 bit unicode whereas Numpy always uses 32 bit unicode when storing unicode strings in order to avoid problems.

Numpy always uses 32 bit unicode when storing unicode strings

Except in np.unicode_, when it somehow both does and doesn't?

I'll guess that np.unicode_ aliases the Python type. My Python is compiled to use UCS4

/usr/include/python2.7/pyconfig-64.h:#define Py_UNICODE_SIZE 4

and

In [1]: np.unicode_("test").itemsize
Out[1]: 16

I believe Microsoft itself uses wide characters (16 bits) for unicode.

I'll guess that np.unicode_ aliases the Python type

If you mean np.unicode_ is unicode, then that's not the case. Perhaps unicode_ internally stores a unicode.

I believe Microsoft itself uses wide characters (16 bits) for unicode.

Indeed - I'm only seeing this for that reason, but I think the difference in storage between arrays and numpy scalars is screwy

The numpy scalar type looks to subclass unicode in order to add some attributes.

In [16]: isinstance(np.unicode_("test"), unicode)
Out[16]: True

See numpy/core/numerictypes.py, lines 262-275

    if sys.version_info[0] >= 3:
        if name == 'bytes_':
            char = 'S'
            base = 'bytes'
        elif name == 'str_':
            char = 'U'
            base = 'str'
    else:
        if name == 'string_':
            char = 'S'
            base = 'string'
        elif name == 'unicode_':
            char = 'U'
            base = 'unicode'

Yep, you're right, that is the reason

Pretty sure this was fixed by #15385, now all unicode objects in numpy are UCS4.