Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:   -96 -0.052% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
        rp2:    +8 +0.002% PICO

Thanks @smurfix -- I investigated a different approach to a similar problem in #8922

I'm curious how this comes up in the non-frozen case? i.e. how do you end up with two on-filesystem hierarchies.

That said, I'm not sure I realised that MicroPython's behavior here is actually different to CPython, and if I understand correctly, the approach used in this PR actually improves CPython compatibility? (And would solve the frozen scenario I outlined in my PR).

Sorry I didn't see you'd already mentioned this already in #10813.

I agree with what you said there, but there needs to be some thought as to maintaining CPython compatibility.

In particular consider this case (which I think is likely a common case for what you're trying to improve):

.frozen:
    foo
        __init__.py   `from .a import x; from .b import y`
        a.py    `def x(): print(".frozen x()")`
        b.py  `def y(): print(".frozen y()")`

user does "import foo; foo.x"

Now the user creates on the filesystem

dev
    foo
        a.py  `def x(): print("dev x()")`

(i.e. they've patched a.py for development, but left the rest of the foo module unchanged. this could be uasyncio/core.py for example)

Then

import sys
sys.path.insert(0, "dev")  # make dev searched before .frozen

import foo
foo.x()

On CPython, this will ignore the "dev" directory entirely. On current MicroPython, and this PR, it will ignore frozen/foo/__init__.py (i.e. import foo gets the empty module from dev/foo).

We could make it a requirement that you must also override all __init__.py, i.e. you now add dev/foo/__init__.py identical to frozen/foo/__init__.py but this will now fail on the from .b import y.

(In summary, I think this PR works well as long as you're entirely working with namespace packages, but as soon as __init__.py gets involved, it's very confusing. MicroPython currently essentially only supports non-namespace packages. Realistically, I think many of the scenarios where you want this ability to override frozen code are for non-namespace packages).

IMHO whether CPython does or does not ignore the "dev" directory, and/or whatever subhierarchy still is accessible because it assumes your packages are namespace-y, is mostly immaterial because on CPython you simply create a copy of the whole "foo" subtree when you need to edit it, point your PYTHONPATH to it, and you're set.

We can't do that on MicroPython because doing so requires too much RAM.

this PR works well as long as you're entirely working with namespace packages

Well, this patch essentially namespace-ifies all packages, and thus behaves exactly the same as CPython – except for the fact that it doesn't require __init__.py to either not exist anywhere, or to contain the magic __path__ = __import__('pkgutil').extend_path(__path__, __name__) compatibility stanza (which BTW you could use to get your dev example to work on CPython).

I don't think there's much confusion with __init__.[m]py. Conceptually you simply put the directories on top of each other and the first file that's found in any of them gets used, with the single proviso that foo/__init__.[m]py always overrides foo.[m]py – which in practice would be (a) rare, and (b) exactly what you want (supersede a small in-ROM package with a larger one in the Flash filesystem).

Realistically, I think many of the scenarios where you want this ability to override frozen code are for non-namespace packages

Given that currently there aren't any namespace-y packages on MicroPython, you're correct by definition. however, the chance that somebody has a foo/bar/baz.py lying around on their filesystem which (a) is in their usys.path before .frozen and which they don't want to override the version from ROM are small enough that documenting this change of behavior in the release notes should be way more than sufficient to ward off any problems.

I can't lay claim to a solution, but here are a few thoughts.

1. By definition the problem is one for experienced developers who can compile a build.
2. Therefore could we consider a build option? In this case the option might override standard Python behaviour.
3. Given that we are talking about large projects it's likely that .mpy files will be required. In the developer tree a .mpy should be preferred over a .py.
4. Any solution needs to consider an __init__.py that uses Damien's lazy loader (as per uasyncio).