Summary

This issue was originally reported on Discord by ondrej1024: on RPI_PICO_W a TLS connection via micropython-lib umqtt.simple would fail consistently on reconnect with OSError(ENOMEM). A workaround was to explicitly close the old socket before connecting again. This doesn't seem like it should be necessary.

Was able to generalise into the new test case in this PR: extmod/ssl_noleak. This fails when trying to allocate sequential SSL socket objects (one at a time) on rp2 and also other ports.

Different fixes were needed, in separate commits in this PR. First approach was replaced with a new approach at suggestion of @dpgeorge, see inlinecomments below.

Changes:

• For rp2 and other mbedTLS bare metal ports, TLS nodes are "tracked" memory allocations associated with the SSL socket object. If an SSL socket object runs its finaliser during garbage collection then the "tracked" nodes can't be immediately freed (GC is locked) so they were left for the next GC. As a result, allocating a new SSL socket could fail even though enough memory was available (but not yet freed). Fix is to support gc_free() during a GC sweep by tracking gc collection separately from "heap is locked".
• This change led to a double-free assertion failure during gc_sweep_all (soft-reset), as the memory buffer might already be free. Fix for this was to split gc_sweep into two phases - one which runs finalisers and then one which frees memory. This works around other potential finaliser issues when memory has already been freed, too. Due to the sparse nature of FTB entries, this could be optimised to more than trade-off adding a second pass, with only a small code size increase.
• Similarly, GC mutex needed to be made recursive on ports which support MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL. Have added an optional generic recursive mutex interface.
• For esp32 something similar happens where the small SSL socket object in the MicroPython heap holds large buffers outside MicroPython in the ESP-IDF heap. Allocatinga new SSL socket object could fail to allocate new buffers as the ESP-IDF heap is too full, even if there are SSL socket objects eligible for garbage collection that would free up lots of ESP-IDF heap when garbage collected. Fix is to run a gc_collect() and retry if SSL socket setup fails.

Testing

• New unit test extmod/ssl_noleak fails consistently on master (using RPI_NANO_W and ESP32_GENERIC_C3 boards). Probably also fails on other ports using MICROPY_MBEDTLS_CONFIG_BARE_METAL, unless they have huge amounts of RAM.
• With fixes in this PR, new test passes (as well as all other unit tests).
• Separate commit enables mbedTLS "bare metal" memory management in the coverage build, to cover these changes and increase the coverage of MicroPython-specific code (the default for unix port being libc calloc and free).

Performance

Performance benchmark run on PYB_V11 with all cache init commented out:

diff of scores (higher is better)
N=168 M=100                ./pybv11_master.txt -> ./pybv11_pr.txt         diff      diff% (error%)
bm_chaos.py                    240.23 ->     242.18 :      +1.95 =  +0.812% (+/-0.00%)
bm_fannkuch.py                  48.62 ->      48.98 :      +0.36 =  +0.740% (+/-0.01%)
bm_fft.py                     1479.57 ->    1492.25 :     +12.68 =  +0.857% (+/-0.00%)
bm_float.py                   4082.97 ->    4080.69 :      -2.28 =  -0.056% (+/-0.00%)
bm_hexiom.py                    31.85 ->      31.80 :      -0.05 =  -0.157% (+/-0.01%)
bm_nqueens.py                 2840.25 ->    2866.36 :     +26.11 =  +0.919% (+/-0.01%)
bm_pidigits.py                 372.74 ->     376.82 :      +4.08 =  +1.095% (+/-0.36%)
bm_wordcount.py                 30.88 ->      30.88 :      +0.00 =  +0.000% (+/-0.01%)
core_import_mpy_multi.py       361.54 ->     363.66 :      +2.12 =  +0.586% (+/-0.01%)
core_import_mpy_single.py       57.34 ->      57.34 :      +0.00 =  +0.000% (+/-0.00%)
core_locals.py                  24.96 ->      24.96 :      +0.00 =  +0.000% (+/-0.00%)
core_qstr.py                    78.47 ->      78.46 :      -0.01 =  -0.013% (+/-0.00%)
core_str.py                     12.56 ->      12.56 :      +0.00 =  +0.000% (+/-0.00%)
core_yield_from.py             220.01 ->     220.00 :      -0.01 =  -0.005% (+/-0.01%)
misc_aes.py                    268.31 ->     268.31 :      +0.00 =  +0.000% (+/-0.01%)
misc_mandel.py                2069.87 ->    2090.56 :     +20.69 =  +1.000% (+/-0.01%)
misc_pystone.py               1366.78 ->    1366.45 :      -0.33 =  -0.024% (+/-0.01%)
misc_raytrace.py               246.10 ->     248.10 :      +2.00 =  +0.813% (+/-0.01%)
viper_call0.py                 332.95 ->     332.96 :      +0.01 =  +0.003% (+/-0.01%)
viper_call1a.py                324.59 ->     324.59 :      +0.00 =  +0.000% (+/-0.00%)
viper_call1b.py                252.76 ->     252.76 :      +0.00 =  +0.000% (+/-0.01%)
viper_call1c.py                254.68 ->     254.67 :      -0.01 =  -0.004% (+/-0.01%)
viper_call2a.py                319.64 ->     319.65 :      +0.01 =  +0.003% (+/-0.01%)
viper_call2b.py                222.30 ->     222.30 :      +0.00 =  +0.000% (+/-0.01%)

Trade-offs and Alternatives

• For future performance gains we could look at adding a similar "fast path" optimisation to the ATB lookups in GC, and/or expanding the ATB and FTB table entries to word size (currently individual bytes).
• For bare metal, @dpgeorge pointed out that it's possible mbedTLS doesn't need "tracked node" allocations (i.e. if mbedTLS structures always contain pointers to the head of each buffer, then they could be tracked as normal GC objects). However the mbedTLS implementation has used "tracked node" allocations for as long as it's been in MicroPython, and seems like good insurance rather than auditing every mbedTLS allocation.
• For ESP32, it's tempting to try and extend the "GC and then retry" behaviour to all ESP-IDF heap allocations. However, this is much more fiddly and has potential thread safety issues for allocations made from non-Python threads. Most of the large ESP-IDF memory allocations happen as part of mbedTLS, at least.