Skip to content

Conversation

AJMansfield
Copy link
Contributor

@AJMansfield AJMansfield commented Aug 29, 2025

Summary

This PR implements the required functionality to allow the garbage collector to run user-defined __del__ methods as finalisers.

Testing

I've added full-coverage tests of this feature that I've tested for sensitivity across both the Unix port and my RP2040 and RP2350 boards. All tests pass on my RP2 boards, and with one very difficult exception noted below all tests also pass on unix.

tests/basics/class_del_postadded.py only passes on the unix builds I've built with DEBUG=1; but fails on my DEBUG=0 builds. I suspect that this is an underlying garbage-collector bug, rather than a bug with this implementation of __del__ specifically. So far I've failed to discover exactly why it is this occurs, though I've confirmed using the debugger on a non-debug non-stripped build that everything up to and including the call to mp_obj_malloc_var_with_finaliser does run as expected.

Trade-offs and Alternatives

This PR claims two additional bits from the mp_obj_type_t.flags field, for MP_TYPE_FLAG_IS_INSTANCED and MP_TYPE_FLAG_HAS_FINALISER. The latter bit is required in order to track whether instances should be allocated using the _with_finaliser allocators, while the former is required to track whether or it's safe to allow the __del__ method to be inserted into a class after-the-fact. It's possible that MP_TYPE_FLAG_IS_INSTANCED could be eliminated by forbidding this type of insertion entirely.

Alternatively, almost all of the code complexity in this PR could be eliminated by unconditionally allocating objects from user-defined classes with mp_obj_malloc_var_with_finaliser; however the performance penalty that this would introduce on classes that don't use it leads me to favor this more-complex approach.

Copy link

codecov bot commented Aug 29, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.39%. Comparing base (af745a5) to head (84519e0).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #18005   +/-   ##
=======================================
  Coverage   98.38%   98.39%           
=======================================
  Files         171      171           
  Lines       22296    22314   +18     
=======================================
+ Hits        21937    21955   +18     
  Misses        359      359           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AJMansfield
Copy link
Contributor Author

AJMansfield commented Aug 29, 2025

Note that 105f475 is common with #18004; this feature was the main impetus for finding that cleanup.

Copy link

Code size report:

   bare-arm:   +12 +0.021% 
minimal x86:   +16 +0.008% 
   unix x64:  +120 +0.014% standard
      stm32:  +104 +0.026% PYBV10
     mimxrt:  +104 +0.028% TEENSY40
        rp2:  +120 +0.013% RPI_PICO_W
       samd:  +132 +0.049% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:  +268 +0.059% VIRT_RV32

Cleans up 2423493, and fixes builds
that include LFS but not MICROPY_ENABLE_FINALISER.

Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
@AJMansfield
Copy link
Contributor Author

AJMansfield commented Aug 29, 2025

Remaining TODO:

  • Find a means to discriminate MICROPY_ENABLE_FINALISER to avoid test failures on minimal ports.
  • Determine the root cause for the Unix-specific test failures and fix it.
  • Revise documentation to communicate this capability instead of warning its absence.
  • Develop and test performance benchmarks to profile runtime impact.

@dlech
Copy link
Contributor

dlech commented Aug 29, 2025

Other than for freeing resources obtained through FFI, what use case would it have in MicroPython?

@AJMansfield
Copy link
Contributor Author

AJMansfield commented Aug 29, 2025

Other than for freeing resources obtained through FFI, what use case would it have in MicroPython?

What use cases do finalisers have in MicroPython at all? Anything you could want to do with them in C, you could have a perfectly valid reason to want to do with python. __del__ isn't just about letting you close or destruct non-python external resources, it's also for managing the lifetimes of other python objects in designs that involve caching or ECS-like systems.

The original impetus for this feature was related to @agatti's current work implementing machine.Pin on the unix port, in order to preserve the standard machine.Pin(*a) is machine.Pin(*a) invariant with pins backed by /dev/gpiochipX ioctls. Not all device files necessarily even exist yet when the interpreter starts, so they really do need to be dynamically allocated, and freed too when user code no longer holds a reference, in order to not leak memory over multiple hotplug cycles of a gpio device.

I've got another library I've been developing that would benefit from this, that's for managing I2C devices --- with efficient transaction queueing, caching, and automatic fusion of adjacent transactions.

When a sqlite returns a handle to a stored procedure it's memoized, if you're using a python-level ORM object to manage that procedure, it ought to be able to set a __del__ method that instructs sqlite to discard that entry from its stored procedure table, too.

More broadly, this applies to any sorts of code that uses ECS/SoA data layout, i.e. with user-defined type with a non-pointer value indexing some kind of table; or when doing any kind of database that needs to track reach-ability across non-trivial foreign-key relationships.

In many cases, context managers are the "clean code approved" way to manage external lifetimes --- but if every problem was one with lifetime boundaries that were that simple, we'd already all just be writing Rust instead of Python. Plus, to implement a context manager that's robust against bugs like e.g. #2621, that too also needs to make use of __del__, as a second-resort cleanup behind __exit__.

I'm currently working on an implementation of the weakref module, which is also a tool often used in caching and ECS-like situations. With __del__, it should be possible to do this as part of micropython's python-language standard library, without necessarily requiring other c-language primitives (except maybe one for de/obfuscating object pointers from the garbage collector).

@AJMansfield
Copy link
Contributor Author

AJMansfield commented Aug 29, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants