Skip to content

ValueError: low level cast function is for unequal type numbers for object types #2798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

cgohlke
Copy link
Contributor

@cgohlke cgohlke commented Dec 7, 2012

This was reported on the numpy-discussion mailing list.

The following code using np.object_ data types works with numpy 1.5.1 but fails with 1.6.2. Other data types, np.float64 for example, seem to work.

In [1]: import numpy as np

In [2]: np.array(['a'], dtype='O').astype(('O', [('name', 'O')]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-f4a62241f13c> in <module>()
----> 1 np.array(['a'], dtype='O').astype(('O', [('name', 'O')]))

ValueError: low level cast function is for unequal type numbers

In [3]: np.array([16], dtype='d').astype(('d', [('name', 'd')]))
Out[3]: array([ 16.])

These downstream issues could be related:

I'm not entirely sure the proposed patch is correct/best, but it seems to fix this and the h5py issue.

@rgommers
Copy link
Member

rgommers commented Dec 8, 2012

This (base_dtype, new_dtype) way of specifying dtypes is discouraged according to http://docs.scipy.org/doc/numpy-dev/reference/arrays.dtypes.html#arrays-dtypes but still valid. So should indeed be fixed.

There are really too many ways to create dtypes.....

@cgohlke
Copy link
Contributor Author

cgohlke commented Dec 16, 2012

Reply by Andrew Collette, author of h5py, on the numpy discussion mailing list on 13 Dec 22:57:

"

the following code using np.object_ data types works with numpy 1.5.1
but fails with 1.6.2. Is this intended or a regression? Other data
types, np.float64 for example, seem to work.

I am also seeing this problem; there was a change to how string types
are handled in h5py 2.1.0 which triggers this bug. It's a serious
inconvenience, as people can't do e.g. np.copy()
any more.

These downstream issues could be related:

http://code.google.com/p/h5py/issues/detail?id=217

Yes, this seems to be the cause of issue 217.
"

@certik
Copy link
Contributor

certik commented Dec 18, 2012

@cgohlke, could you please write a test for this too?

@certik
Copy link
Contributor

certik commented Dec 18, 2012

I am not very familiar with this part of numpy. But if Ralf or others are giving the patch +1, I am ok to merge this to 1.7. It seems simple enough that hopefully things will not break.

@certik
Copy link
Contributor

certik commented Dec 21, 2012

@cgohlke either that, or maybe numpy/core/tests/test_multiarray.py.

@certik
Copy link
Contributor

certik commented Dec 27, 2012

@rgommers, are you +1 with this patch? All tests pass (the only failure is the old python 3.1 which was removed from Travis and we fixed it in our .travis.yml in the meantime) and it looks ok to me.

certik pushed a commit to certik/numpy that referenced this pull request Dec 27, 2012
@certik certik mentioned this pull request Dec 27, 2012
@certik
Copy link
Contributor

certik commented Dec 27, 2012

Backported in #2859.

@rgommers
Copy link
Member

@certik looks reasonable, but I'm not familiar with this code either. @mwiebe, @charris could one of you have a quick look?

@charris
Copy link
Member

charris commented Dec 27, 2012

I'm not familiar with that code either, but it looks like the patch makes the object type special, just as datetime and timedelta are. That 'feels' like it might be the right thing but I can't say much more without a longer look at the code. @cgohlke It might be nice to have a comment in the code explaining the reason. That said, I tend to trust Christoph's judgement in these things.

@certik
Copy link
Contributor

certik commented Dec 27, 2012

@cgohlke, or you can explain your reasoning in the commit log. For example to me, as somebody who is not familiar with this part of the code, I have no idea what your patch does, or is intending to do. All I can see is that it fixes some problem with astype().

@charris
Copy link
Member

charris commented Dec 28, 2012

The Travis failure is due to the missing Python 3.1

@certik
Copy link
Contributor

certik commented Dec 28, 2012

@cgohlke I would like to release rc1 today or early tomorrow. If you could please address the reviews, let's get it in and release. If not, then it will have to wait for rc2, or go into 1.8.

@njsmith
Copy link
Member

njsmith commented Dec 28, 2012

Looking at the code some, this patch is definitely not a complete fix for whatever's going on here... there are only two call sites for get_cast_transfer_function, and one of them calls it specifically to retrieve a cast function in case the src and dst dtypes have different type nums (dtype_transfer.c:3638)! So either the check in get_cast_transfer_function is totally wrong, or the code that calls it is.

I suspect the correct fix is to just delete the check and error message entirely. This function is just responsible for fixing up alignment/byte-order and delegating to the lower-level functions that actually know how to do casting for different types, so having a check on the types here is reaaally fishy.

@njsmith
Copy link
Member

njsmith commented Dec 28, 2012

Well, stuck up the just-delete-the-check patch as #2863.

@njsmith njsmith closed this in 678ad43 Dec 28, 2012
njsmith added a commit that referenced this pull request Dec 28, 2012
At least, I hope it's spurious. Certainly it's creating a spurious
error message, is unexpected by callers (one of whom specifically
makes the *opposite* check before calling get_cast_transfer_function),
and even if it is a useful check for some reason I can't see, it
certainly doesn't belong in this function (which is otherwise just
taking care of byte-swapping and alignment issues and doesn't know
anything about dtypes). And worst case, we'll have turned an exception
into a crash; even if I'm wrong, this shouldn't cause any code to go
from working to not working, just from broken to slightly-more-broken.

Test and original diagnosis by @cgohlke.

Fixes gh-2798.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants