So, first thing is that classic Py packages require __init__.py to be in a package dir, it's ~ error if it's not there. But recent funky novelty is "namespace packages" which abuses dirs w/o __init__.py for its purposes. I don't even bother to wrap my mind of details what ns packages are, intuitively, it's a search path with a backtrack - whereas classical rules say that it's error if submodules not found in a top-level package, with ns package, you backtrack and continue. We don't need that. But now we need to can't tell user it's error that a dir lacks __init__.py, we must say that some pundit said it's namespace package dir, and we don't support that stuff. Authoritative spec: http://www.python.org/dev/peps/pep-0420/#specification

Agree that package import is not the cleanest thing.

As you suggest, if we try to get standard packages working properly, and then just error out if we detect a namespace package, then I think that's a good way to have things initially. It's nice that namespace packages must not have __init__.py, so that gives a way to distinguish them from normal packages, by just looking at the filesystem.

Where do I start next? http://docs.python.org/3/library/functions.html#__import__ :

When the name variable is of the form package.module, normally, the top-level package (the name up till the first dot) is returned, not the module named by name.

Note that sys.modules contains both package and package.module, so there's additional PITA how to do caching right, not just do loading right.

Ok, I pushed changes to get basic package imports work. Lot more to do.

Thank you!

Ok, going further. http://docs.python.org/3.3/library/functions.html#__import__ :

When the name variable is of the form package.module, normally, the top-level package (the name up till the first dot) is returned, not the module named by name. However, when a non-empty fromlist argument is given, the module named by name is returned.

Implemented in fb7f943.

Ok, now it's fair to say that basic package imports indeed work, only "advanced" left.

Next one is: relative imports, http://legacy.python.org/dev/peps/pep-0328/ , http://docs.python.org/3/library/functions.html#__import__ level arg.

So, it turns out these are not implemented on compiler level (I added at least a1aba36 to clarify that). So, leaving implementation on import() level to those who actually will need them, but added check to throw NotImplementedError.

So, looking into CPython stdlib, relative imports are used widely - which is not surprising, because they resolve real import scoping problem. So, @dpgeorge , can you please schedule making them supported on compiler level. Not a big priority - I probably want be looking into further imports work near-term.

oops, please ignore - was already implemented in 635543c ;-)

Currently from pkg import mod doesn's work where import pkg.mod does.

Latest case is partially resolved. Full resolution likely will require similar effort as implementing relative imports (there're issues how package paths resolved relative to different package modules).

Thanks for your work on this.

Actually - it was different issue, kind of recursive import. Should be fixed now, regardless of relative import stuff.

Ok, basic relative imports are now implemented. There're likely various corner cases in it yet.

This also should mark complete implementation of module-from-package import support. But not symbols-from-modules - likely, only top-level symbols can be imported now. And I wonder, how far Python goes with that? Can it import unbound method from a class? Bound method from an object?

Overall, code is tangles and spread out between runtime.c and builtinimport.c, requires heavy refactoring (and that's not talking about verification for correctness).

Ok, I finally came to real-world cases when namespace package are needed and thus learned details of them a bit ;-).

The usecase actually pretty simple. Consider for example that Python3 has http.server and http.client modules, being under common "http" package. The unit of distribution and installation is a package, so we would need to force installation of both http.server and http.client at the same, and that's clear not acceptable exactly for us!

So, idea of namespace packages is simple - just get rid of empty __init__.py file which will let install different archives at the same directory without having conflicts and ambiguities among their files. This is now enabled.

But it goes beyond that, allowing different package portions be installed at different paths as appearing in sys.path. This is not implemented. Per above, this requires recursive traversal of path with backtracking. CPython optimizes this by caching suitable subpaths at each namespace package's __path__ attribute. And yet it requires namespace request to immediately react to sys.path changes. This last thing we definitely don't want to implement, and even multiple dirs probably should wait for real-world usecases, in the meantime we we can just require that all namespace packages were installed into single dir from sys.path.

So, namespaces packages are cool things, using them a lot in micropython-lib. But uPy currently picks up first suitable dir as a namespace package, whereas CPy doesn't have conflicts when for example in current dir there's a subdir named as stdlib module - it will import stdlib package. But uPy will pick up this random subdir.

Just wanted to note this as a known issue.

Example: micropython/micropython-lib@6224013

What logic does CPy use to get around this?

From memory of reading PEP, and based on behavior seen, it would be something like prioritizing normal modules/package over namespace packages (or perhaps over empty namespace packages). That would require building a candidate complete path list first, or scanning sys.path twice. We know where that (well, order of magnitude more such things) led CPython - to people complaining that CPython startup times are awful. So, not feeling like going there ;-)

Relative imports currently currently appear to be broken on the board?

I have created a test package as

pkg/
  __init__.py
  test.py

pkg/__init__.py contains

from .test import hello

def hello_pkg():
    print('Hello, Pkg!')

pkg/test.py contains

def hello():
    print("Hello, World!")

importing pkg using the unix port works as expected, but on the board I get all lights on crash.

If I adjust __init__.py to use absolute import it works as expected on the board.

@cjbarnes18: The issue with relative imports now has its own issue at #872.

As of now, there're no known issues with package imports remaining, closing.