Skip to content

BUG: Integer overflow warning applies to scalars but not arrays #8987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eric-wieser opened this issue Apr 25, 2017 · 9 comments
Open

BUG: Integer overflow warning applies to scalars but not arrays #8987

eric-wieser opened this issue Apr 25, 2017 · 9 comments

Comments

@eric-wieser
Copy link
Member

eric-wieser commented Apr 25, 2017

Simple demo:

In [1]: np.uint8(0) - np.uint8(1)
RuntimeWarning: overflow encountered in ubyte_scalars
Out[1]: 255

In [2]: np.uint8(0)[...] - np.uint8(1)[...]
Out[2]: 255

Should we even warn at all for unsigned integers?

@eric-wieser eric-wieser changed the title BUG: Integer overflow error applies to scalars but not arrays BUG: Integer overflow warning applies to scalars but not arrays Apr 25, 2017
@mwtoews
Copy link
Contributor

mwtoews commented Sep 6, 2017

Digging into the mail list from 7 Apr 2009, @rkern writes:

Unlike true floating point errors (where the hardware FPU sets a flag whenever it does an atomic operation that overflows), we need to implement the integer overflow detection ourselves. We do it on the scalars, but not arrays because it would be too slow to implement for every atomic operation on arrays.

I suspect this is still true, and it's too expensive to check on arrays. I suggest improving the numpy.seterr docs to make it clear that number errors can only be detected on scalar types, and not array types.

@insulsa
Copy link

insulsa commented Apr 24, 2018

This is an exact duplicate of #593.

My thoughts were that numpy.seterr was ugly designed. We could make use of some new interface, apart from numpy.seterr, to explicitly define the behavior of integer overflows among choices of

  1. to wrap,
  2. to raise an exception,
  3. to wrap unsigned but raise an exception for signed (Because according to C, unsigned overflow is mandated to wrap, but signed overflow is UB.),

And naming this something like numpy.setintwrap, maybe allow separate settings for scalars and arrays, while replacing numpy.seterr for setfloaterr with no effect on integers.

@charris
Copy link
Member

charris commented Apr 24, 2018

To do integer overflow right, you need hardware support. Modern hardware does not provide that support. Some ancient hardware did, but it fell out of favor.

@njsmith
Copy link
Member

njsmith commented Apr 24, 2018

Eh, you can do it, it just has some penalty in speed. If we added it for arrays then I think we'd also want to have a dedicated option in seterr (no need to add a separate setintwrap function), and make sure that if errors are disabled then we take a fast-path that disables the checking entirely, so that it actually works as a speed/safety tradeoff knob. All of which seems doable and reasonable to me; integer overflow causes a lot of silent miscalculations.

@mattip
Copy link
Member

mattip commented Apr 26, 2018

related to issue pandas-dev/pandas#15557 , issue #5657, issue #5166,

This was referenced Apr 26, 2018
oleksiyskononenko added a commit to h2oai/datatable that referenced this issue Apr 18, 2020
On Windows numpy's default integer type is `numpy.int32`, and not `numpy.int64` like on Mac and Linux. To have the same column types on different platforms and to prevent a known numpy overflow numpy/numpy#8987, `dtype=numpy.int64` was explicitly specified when creating numpy arrays. 

Since numpy tests don't fail on Windows anymore, adjust AppVeyor config file to install pandas and numpy when building/testing datatable.

WIP for #2301
@mattany
Copy link

mattany commented Dec 17, 2021

I noticed that numpy.prod overflows silently as well. Not sure if this issue covers that already or if I should open a new issue suggesting an overflow error for that as well.

@gmmarconi
Copy link

Looks like the issue is still present with int64.

image

@oussjarrousse
Copy link

If I was not using hypothesis given and the integer strategy to test a function I would have not noticed this issue. sum on an array should trigger an overflow signal. please prioritise!

@mattip
Copy link
Member

mattip commented Jun 6, 2025

Eh, you can do it, it just has some penalty in speed

Over at #29129 I showed an opt-in path for those who want overflow checking (in this case inside the casting ufuncs used by ndarray.astype). The speed penalty would only be paid by those who opt-in to the check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants