Skip to content

ulab array corruption with slices #4753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
kevinjwalters opened this issue May 12, 2021 · 5 comments
Closed

ulab array corruption with slices #4753

kevinjwalters opened this issue May 12, 2021 · 5 comments
Labels
bug nordic ulab Related to the ulab module
Milestone
Long term

Comments

@kevinjwalters
Copy link

kevinjwalters commented May 12, 2021 •
edited
Loading

I noticed this on CircuitPython 6.2.0 on a CLUE (nRF52840):

Adafruit CircuitPython 6.2.0 on 2021-04-05; Adafruit CLUE nRF52840 Express with nRF52840
>>> import ulab
>>> import ulab.numerical
>>> import ulab.filter
>>> data = ulab.ones(10000)
>>> fir_taps = ulab.array([0.15, 0.2, 0.3, 0.2, 0.15])
>>> filtered_data = ulab.filter.convolve(data, fir_taps)[6:-6]
>>> ulab.numerical.sum(filtered_data)
9992.0
>>> len(filtered_data)
9992
>>> a = []  ### append to an array to write and use up free memory
>>> for idx in range(1_000_000):
...     a.append("AYMABTU")
...
...
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
MemoryError: memory allocation failed, allocating 131072 bytes
>>> del a
>>> ulab.numerical.sum(filtered_data)  ### no longer 9992.0 !!
9882.01
>>> ulab.numerical.sum(data)  ### although this one is ok
10000.0
>>> filtered_data[0:10]
array([2.98023e-08, 0.0, 1.19209e-07, 0.0, -6.61744e-24, 0.0, -2.64698e-23, 0.0, 1.70141e+38, 0.0], dtype=float32)

I'd guess it's related to premature GC of the ndarray returned by ulab.filter.convolve(). That would be inappropriate as the slice is a view onto that data as I understand it.
@kevinjwalters
Copy link
Author

@v923z may be interested in this.

@tannewt tannewt added the nordic label May 13, 2021
@tannewt tannewt added this to the Long term milestone May 13, 2021
@tannewt tannewt added the ulab Related to the ulab module label May 13, 2021
@v923z
Copy link

v923z commented May 13, 2021

@kevinjwalters

That would be inappropriate as the slice is a view onto that data as I understand it.

This is correct, but perhaps the first question is, whether this has anything to do with ulab, and convolve. I believe, especially, if this is a problem with the garbage collection, we should strip this to the most trivial example that produces such behaviour. Something like this:

import math

_data = [1.0] * 10000
data = _data[6:-6]
sum(data)
a = []
for idx in range(1_000_000):
    a.append("AYMABTU")

del a
sum(data)

If this still works, I would then do

_data = np.array([1.0] * 10000)
data = _data[6:-6]
np.sum(data)
a = []
for idx in range(1_000_000):
    a.append("AYMABTU")

del a
np.sum(data)

Do you get similar results, if you try to initialise a = ['AYMABTU'] * 1_000_000?

Also, when you have this

ulab.numerical.sum(filtered_data)  ### no longer 9992.0 !!
9882.01

can you find out, where the data are corrupted? At the beginning, or the end of the array?

@kevinjwalters
Copy link
Author

First part:

Adafruit CircuitPython 6.2.0 on 2021-04-05; Adafruit CLUE nRF52840 Express with nRF52840
>>>
>>> import math
>>> _data = [1.0] * 10000
>>> data = _data[6:-6]
>>> sum(data)
9988.0
>>> a = []
>>> for idx in range(1_000_000):
...     a.append("AYMABTU")
...
...
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
MemoryError: memory allocation failed, allocating 131072 bytes
>>> del a
>>> sum(data)
9988.0

Although perhaps this is closer to my example:

>>> import math
>>> _data = [1.0] * 10000
>>> data = _data[6:-6]
>>> del _data
>>> sum(data)
9988.0
>>> a = []
>>> for idx in range(1_000_000):
...     a.append("AYMABTU")
...
...
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
MemoryError: memory allocation failed, allocating 131072 bytes
>>> del a
>>> sum(data)
9988.0

I added a del _data here too:

Adafruit CircuitPython 6.2.0 on 2021-04-05; Adafruit CLUE nRF52840 Express with nRF52840
>>> import ulab as np
>>> _data = np.array([1.0] * 10000)
>>> data = _data[6:-6]
>>> del _data
>>> np.sum(data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'sum'
>>> np.numerical.sum(data)
9988.0
>>> a = []
>>> for idx in range(1_000_000):
...     a.append("AYMABTU")
...
...
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
MemoryError: memory allocation failed, allocating 131072 bytes
>>> del a
>>> np.numerical.sum(data)
9988.0

@jepler jepler mentioned this issue May 13, 2021
Closed
@v923z
Copy link

v923z commented May 13, 2021

@kevinjwalters Thanks for bringing up the issue; it has been fixed in v923z/micropython-ulab#388, and the fix backported to the legacy branch: https://github.com/v923z/micropython-ulab/releases/tag/1.7.8.

@jepler Thanks for digging to the root of the problem!

@dhalbert dhalbert mentioned this issue May 13, 2021
Merged
@dhalbert
Copy link
Collaborator

Fixed as reported above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug nordic ulab Related to the ulab module
Projects
None yet
Milestone
Long term
Development

No branches or pull requests
4 participants
@tannewt @v923z @dhalbert @kevinjwalters