Skip to content

esp32s3 memory allocation fails after #12141 #12237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nspsck opened this issue Aug 15, 2023 · 7 comments
Closed

esp32s3 memory allocation fails after #12141 #12237

nspsck opened this issue Aug 15, 2023 · 7 comments
Labels
bug port-esp32

Comments

@nspsck
Copy link
Contributor

nspsck commented Aug 15, 2023

This is related to #12075 , #11853 and #12141.

After the new feature (dynamic memory allocation) added to the gc by #12141, I was not able to allocate a big memory block (roughly 252KB) in my C-Module using neither gc_collect(), malloc() nor heap_caps_malloc(). So I went digging, and found out, that this new feature does not seems to do what it promises, unless I have totally missunderstood what it promises.

Take the following code as a example:

# Code:
from micropython import mem_info

buf1 = bytearray(64 * 1024)
mem_info()
buf2 = bytearray(64 * 1024)
mem_info()
buf3 = bytearray(55 * 1024)
mem_info()

try:
    buf4 = bytearray(55 * 1024)
    mem_info()
except MemoryError as e:
    print("***********************************")
    print(e)
    mem_info()
    print("***********************************")

# Output: 
stack: 656 out of 15360
GC: total: 129984, used: 67536, free: 62448, max new split: 122880
 No. of 1-blocks: 23, 2-blocks: 6, max blk sz: 4096, max free sz: 3774
stack: 656 out of 15360
GC: total: 249984, used: 133088, free: 116896, max new split: 10240
 No. of 1-blocks: 24, 2-blocks: 6, max blk sz: 4096, max free sz: 3774
stack: 656 out of 15360
GC: total: 249984, used: 189440, free: 60544, max new split: 10240
 No. of 1-blocks: 25, 2-blocks: 6, max blk sz: 4096, max free sz: 3404
***********************************
memory allocation failed, allocating 56320 bytes
stack: 656 out of 15360
GC: total: 249984, used: 189600, free: 60384, max new split: 10240
 No. of 1-blocks: 30, 2-blocks: 6, max blk sz: 4096, max free sz: 3404
***********************************

The gc-total doesn't seem to grow on demand after a certain amount. And even it says there are 60KB free memory available, it can not allocate 55KB of memory. I have (not so carefully) read through the whole conversation on #12141, but I think 55KB is not considered as a large block by the team.

Also, I probably totally miss understood what this line does, does that mean the memory will grow at least 2KB at a time?

size_t needed = failed_alloc + MAX(2048, failed_alloc * 13 / 512);

I also tried the following:

# Code:
from micropython import mem_info

buf1 = bytearray(64 * 1024)
mem_info()
buf2 = bytearray(64 * 1024)
mem_info()
buf3 = bytearray(55 * 1024)
mem_info()

my_list = []

try:
    for i in range(100):
        my_list.append(bytearray(2 * 1024))
except MemoryError as e:
    print("***********************************")
    print(e)
    mem_info()
    print("***********************************")

# Output:
stack: 656 out of 15360
GC: total: 129984, used: 67568, free: 62416, max new split: 122880
 No. of 1-blocks: 23, 2-blocks: 6, max blk sz: 4096, max free sz: 3755
stack: 656 out of 15360
GC: total: 249984, used: 133120, free: 116864, max new split: 10240
 No. of 1-blocks: 24, 2-blocks: 6, max blk sz: 4096, max free sz: 3755
stack: 656 out of 15360
GC: total: 249984, used: 189472, free: 60512, max new split: 10240
 No. of 1-blocks: 25, 2-blocks: 6, max blk sz: 4096, max free sz: 3404
***********************************
memory allocation failed, allocating 2048 bytes
stack: 656 out of 15360
GC: total: 267328, used: 260128, free: 7200, max new split: 3328
 No. of 1-blocks: 65, 2-blocks: 6, max blk sz: 4096, max free sz: 108
***********************************

The behavior is the same as above. But I could allocate a little bit more than before.
@nspsck nspsck added the bug label Aug 15, 2023
@dpgeorge
Copy link
Member

What hardware are you using, and what build/board are you using for the firmware? In particular, does your hardware have PSRAM?

Were you previously able to allocate more memory, prior to #12141 being merged?

@nspsck
Copy link
Contributor Author

nspsck commented Aug 15, 2023

What hardware are you using, and what build/board are you using for the firmware? In particular, does your hardware have PSRAM?

I have tested it on a Lolin S3 using the ESP32-S3-WROOM-1 module with 8MB PSRAM, 16MB Flash (MON16R8), I set the MICROPY_BOARD_VARIANT flag to spiram-oct.

Were you previously able to allocate more memory, prior to #12141 being merged?

Yes, I was able to use gc_collect() in the C Module to allocate 252KB of Memory. malloc() worked back then, but causes some random issues. heap_caps_malloc() could not allocate that amount.
I also have tested the function on the previous version of micropython, and it worked just fine.

# Code:
from micropython import mem_info

buf1 = bytearray(64 * 1024)
mem_info()
buf2 = bytearray(64 * 1024)
mem_info()
buf3 = bytearray(55 * 1024)
mem_info()

my_list = []

try:
    for i in range(100):
        my_list.append(bytearray(2 * 1024))
except MemoryError as e:
    print("***********************************")
    print(e)
    mem_info()
    print("***********************************")

mem_info()

# Output: 
stack: 656 out of 15360
GC: total: 4255616, used: 67744, free: 4187872
 No. of 1-blocks: 25, 2-blocks: 5, max blk sz: 4096, max free sz: 261678
stack: 656 out of 15360
GC: total: 4255616, used: 133312, free: 4122304
 No. of 1-blocks: 26, 2-blocks: 5, max blk sz: 4096, max free sz: 257582
stack: 656 out of 15360
GC: total: 4255616, used: 189648, free: 4065968
 No. of 1-blocks: 27, 2-blocks: 5, max blk sz: 4096, max free sz: 254062
stack: 656 out of 15360
GC: total: 4255616, used: 396592, free: 3859024
 No. of 1-blocks: 128, 2-blocks: 5, max blk sz: 4096, max free sz: 241189

@dpgeorge
Copy link
Member

I have tested it on a Lolin S3 using the ESP32-S3-WROOM-1 module with 8MB PSRAM, 16MB Flash (MON16R8), I set the MICROPY_BOARD_VARIANT flag to spiram-oct.

If you have 8MB of PSRAM then something is wrong, because "max new split" should be reporting a much larger number.

I just tried the latest master code on a UM_PROS3 (ESP32-S3 with 8MB SPIRAM, but not octal) and here is the output:

MicroPython v1.20.0-363-g91674c41b on 2023-08-15; ProS3 with ESP32-S3
Type "help()" for more information.
>>> 
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== from micropython import mem_info
=== 
=== mem_info()
=== buf1 = bytearray(64 * 1024)
=== mem_info()
=== buf2 = bytearray(64 * 1024)
=== mem_info()
=== buf3 = bytearray(55 * 1024)
=== mem_info()
=== 
=== try:
===     buf4 = bytearray(55 * 1024)
===     mem_info()
=== except MemoryError as e:
===     print("***********************************")
===     print(e)
===     mem_info()
===     print("***********************************")
=== 
stack: 640 out of 15360
GC: total: 64000, used: 2176, free: 61824, max new split: 8257536
 No. of 1-blocks: 17, 2-blocks: 4, max blk sz: 23, max free sz: 3800
stack: 640 out of 15360
GC: total: 129984, used: 67440, free: 62544, max new split: 8126464
 No. of 1-blocks: 12, 2-blocks: 4, max blk sz: 4096, max free sz: 3813
stack: 640 out of 15360
GC: total: 260032, used: 133008, free: 127024, max new split: 7995392
 No. of 1-blocks: 13, 2-blocks: 4, max blk sz: 4096, max free sz: 4032
stack: 640 out of 15360
GC: total: 260032, used: 189344, free: 70688, max new split: 7995392
 No. of 1-blocks: 14, 2-blocks: 4, max blk sz: 4096, max free sz: 4032
stack: 640 out of 15360
GC: total: 260032, used: 245696, free: 14336, max new split: 7995392
 No. of 1-blocks: 15, 2-blocks: 4, max blk sz: 4096, max free sz: 512
>>>

You can see it all works as expected.

So it must be something to do with the octal SPIRAM build in combination with the hardware you are using.

@dpgeorge
Copy link
Member

I set the MICROPY_BOARD_VARIANT flag to spiram-oct.

What is the exact command you used to build the board?

@nspsck
Copy link
Contributor Author

nspsck commented Aug 15, 2023

The command I used was:

make BOARD=GENERIC_S3  MICROPY_BOARD_VARIANT=spiram-oct deploy

Meanwhile I tested it on a official devkit from espressif, the ESP32-S3-DevKitC-1 N32R8v, I saw the same behavior.

@dpgeorge
Copy link
Member

Please try:

make BOARD=GENERIC_S3  BOARD_VARIANT=spiram-oct deploy

@nspsck
Copy link
Contributor Author

nspsck commented Aug 15, 2023

That worked! Thank you very much!

@nspsck nspsck closed this as completed Aug 15, 2023
@nspsck nspsck mentioned this issue Aug 28, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug port-esp32
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dpgeorge @nspsck