Skip to content

BUG: test_arange() fails on Dask due to illegal dtype #338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
crusaderky opened this issue Jan 30, 2025 · 6 comments · Fixed by data-apis/array-api-compat#246
Closed

BUG: test_arange() fails on Dask due to illegal dtype #338

crusaderky opened this issue Jan 30, 2025 · 6 comments · Fixed by data-apis/array-api-compat#246

Comments

@crusaderky
Copy link
Contributor

test_arange has started failing on dask:
https://github.com/data-apis/array-api-compat/actions/runs/13034089416/job/36360099375?pr=244

exc = NotImplementedError('An error occurred while calling the arange method registered to the numpy backend.\nOriginal Message: Can not use auto rechunking with object dtype. We are unable to estimate the size in bytes of object data')

This is unrelated to the recent changes to dask in array-api-compat.
The problem is that the test calls da.arange(..., dtype=object), which is not a valid dtype according to the Array API standard.

@crusaderky crusaderky changed the title test_arange() fails on Dask due to illegal dtype BUG: test_arange() fails on Dask due to illegal dtype Jan 30, 2025
@ev-br
Copy link
Member

ev-br commented Jan 30, 2025

FWIW, I find it hard to believe it's unrelated to recent dask related changes. The test, while quite hairy,

  • has no reason to generate dtype=object
  • was not generating dtype=object until recently
  • today, does not generate dtype=object on anything other than dask.

@crusaderky
Copy link
Contributor Author

Not a regression in array-api-compat, nor array-api-tests, nor dask.
I could not find out what started it.

>>> start, stop, step = 0., -9_131_138_316_486_228_481, -92_233_720_368_547_759
>>> da.arange(start, stop, step)

NotImplementedError: An error occurred while calling the arange method registered to the numpy backend.
Original Message: Can not use auto rechunking with object dtype. We are unable to estimate the size in bytes of object data

dask does not support float parameters.

dask internally multiplies step * 100 to divine the dtype, which in turn sends step beyond the limit of int64, which in turn triggers edge case behaviour on behalf of numpy:

>>> np.arange(start, stop, step * 100)
array([0.0], dtype=object)

@ev-br
Copy link
Member

ev-br commented Jan 30, 2025

Ouch. So the right fix is to limit the range for step that moster of a hypothesis test, apparently.

@ev-br
Copy link
Member

ev-br commented Jan 30, 2025

A first shot could be to add one more assume, like assume(abs(step) < M // 2) to
https://github.com/data-apis/array-api-tests/blob/master/array_api_tests/test_creation_functions.py#L133
?

Generally adding assumes makes it more likely to get hypothesis health check warnings, but maybe we'll be lucky here.

@crusaderky
Copy link
Contributor Author

The input is legal though. This is a bug in dask.
The last person to touch it was... myself, seven years ago: dask/dask#3722

@crusaderky
Copy link
Contributor Author

dask/dask#11707 fixes it upstream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants