gh-69605: Add module autocomplete to PyREPL #129329

tomasr8 · 2025-01-27T00:55:24Z

DPO discussion thread: https://discuss.python.org/t/looking-for-feedback-on-adding-import-autocomplete-to-pyrepl/82281

Adds a module autocomplete functionality to the PyREPL. Some examples first:

import <tab>
import foo<tab>
import foo.<tab>
import foo as bar, baz<tab>

from <tab>
from foo<tab>
from foo import <tab>
from foo import bar<tab>
from foo import (bar as baz, qux<tab>

(Check the tests to see a complete list of what is and is not supported)

The implementation is based on the original patch by @vadmium and adapted to PyREPL.

It can autcomplete both import and from ... import statements as long as the import fragment is valid. It uses a tokenizer/parser-based approach so that it can correctly recognize more complex contructs such as from foo import (bar as baz, qux<tab>.

Module search is done by pkgutil.iter_modules.

I wasn't sure where exactly to put this - for now it hooks into ReadlineAlikeReader.get_completions, let me know if there's a better place for it!

Note that, if there are any import completions available, normal completions are turned off (so that e.g. import m<tab> will not suggest import map().

Feedback welcome :)

Issue: Readline completion of module names in import statements #69605

picnixz · 2025-01-27T06:37:51Z

I have a branch which adds a filtering logic to the completer (see https://discuss.python.org/t/repl-introduce-generic-filters-to-filter-auto-completion-matches/77553) and I wondered whether we could expand the Completer interface in order to have a more generic hook category. Both your work and mine can be orthogonal but we might take this opportunity to make the completer class more generic (I haven't looked at the PR in details as I'm travelling now)

tomasr8 · 2025-01-27T09:34:30Z

Right, I remember seeing your dpo post before! I think it's a good idea :) With this PR I decided to target PyREPL since it is less stable than rlcompleter and thus it's easier to expand/make changes. Though we could definitely think about unifying the interface with that of rlcompleter. For now the API is a bit ad hoc so I'd welcome some discussion in that direction 🙂

tomasr8 · 2025-01-27T09:48:44Z

Lib/_pyrepl/readline.py

@@ -161,6 +170,11 @@ def get_completions(self, stem: str) -> list[str]:
            result.sort()
        return result

+    def get_module_completions(self) -> list[str]:
+        completer = ModuleCompleter(namespace={'__package__': '_pyrepl'})  # TODO: namespace?


One thing I wasn't sure about is how to handle relative imports in regards to __package__. In PyREPL, it is set to _pyrepl so I replicate that here but maybe we want it to be configurable?

hugovk · 2025-01-30T12:23:22Z

On macOS, type import re<tab>:

>>> import re
[ complete but not unique ]

Press <tab> again:

>>> import re
re        reprlib   readline  resource  requests

Then change your mind and press ctrl+C, and you get:

>>> import re
KeyboardInterrupt   readline  resource  requests
>>>
re        reprlib   readline  resource  requests

I'd expect something more like:

>>> import re
KeyboardInterrupt
>>>

Press ctrl+C a couple more times:

>>> import re
KeyboardInterrupt   readline  resource  requests
>>>
KeyboardInterrupt   readline  resource  requests
>>>
KeyboardInterrupt   readline  resource  requests
>>>
re        reprlib   readline  resource  requests

I'd expect something more like:

>>> import re
KeyboardInterrupt
>>>
KeyboardInterrupt
>>>
KeyboardInterrupt
>>>

tomasr8 · 2025-01-30T13:11:12Z

Answering here what I wrote on Discord - the ctrl+C behaviour is present for the existing name/attribute autocomplete as well, this PR is not changing that. I think it's something we could add in a separate PR though!

gaogaotiantian · 2025-02-25T01:23:15Z

I only did a quick look at the code. Are you trying to import the module during completion?

tomasr8 · 2025-02-25T08:30:49Z

I only did a quick look at the code. Are you trying to import the module during completion?

Yes I try to import it to ensure it's a valid package and so that I can easily list the submodules (e.g. import foo.<tab>). IPython does something similar. Is it a concern?

Lib/_pyrepl/completing_reader.py

Lib/_pyrepl/readline.py

gaogaotiantian · 2025-02-25T16:45:59Z

Yes I try to import it to ensure it's a valid package and so that I can easily list the submodules (e.g. import foo.<tab>). IPython does something similar. Is it a concern?

I think so. Implicitly import a module during auto-completion seems a bit unintentional to me. For example, importing torch could take 2~3 seconds, and to user it would be a freeze during completion. Some libraries monkeypatch stdlib or even Python during import, and that's some hidden side effect. Personally I don't like this implicit behavior. Also, will the import only triggered if . is in the text? Or for all possible completion? Say I do import t<tab> does it import all the modules starting with a t?

tomasr8 · 2025-02-25T21:23:25Z

Say I do import t does it import all the modules starting with a t?

This PR uses pkgutil.iter_modules so just typing import t<tab> does not actually import anything. The results of pkgutil.iter_modules are also cached so the top-level packages are only searched once.

I do import in case you type import torch.x<tab>, but I think I might be able to avoid that.

What I think I cannot avoid is an import in case you type from foo import x<tab>. In order for me to know which objects are available for import, I need to actually import the module. I don't think there is a way around that? What do you think?

gaogaotiantian · 2025-02-25T22:06:08Z

Personally, I would prefer if the auto-completer never imports any module without my knowledge, even if that means it can't do its job in some cases. It's okay if it completes when the module already imported, but importing a new module during auto-completion seems wrong to me. I think we should at least get opinions from a few other core devs about this, because in my mind this is a relatively large new behavior.

tomasr8 · 2025-02-25T22:20:12Z

I think we should at least get opinions from a few other core devs about this, because in my mind this is a relatively large new behavior.

I admit I do not fully understand the implications of this so I think getting more opinions is a good idea :) Do you want me to start a discussion on Discord?

gaogaotiantian · 2025-02-25T22:26:54Z

Discord or/and dpo maybe? We should at least get some opinions from repl owners.

tomasr8 · 2025-02-25T22:46:18Z

ok! It's getting a bit late here, so I'll do it tomorrow 🙂

tomasr8 · 2025-03-10T15:28:37Z

I updated the PR following the discussion on DPO: https://discuss.python.org/t/looking-for-feedback-on-adding-import-autocomplete-to-pyrepl/82281

Modules are no longer imported. Modules are discovered using a combination of pkgutil.iter_modules and submodule_search_locations.
Removed support for module attributes. This relied on actually importing the modules. We can add this once we have decided in which way we should discover module attributes.

pablogsal · 2025-04-19T01:16:46Z

Lib/_pyrepl/readline.py

@@ -161,6 +169,11 @@ def get_completions(self, stem: str) -> list[str]:
            result.sort()
        return result

+    def get_module_completions(self) -> list[str]:
+        completer = ModuleCompleter(namespace={'__package__': '_pyrepl'})  # TODO: namespace?


Either remove the TODO or we should handle this differently

I removed the TODO and left a comment. Inside pyrepl, __package__ is set to _pyrepl so the module completer should use the same value when resolving relative packages.

pablogsal · 2025-04-19T01:20:19Z

Lib/_pyrepl/readline.py

+    def get_module_completions(self) -> list[str]:
+        completer = ModuleCompleter(namespace={'__package__': '_pyrepl'})  # TODO: namespace?
+        line = self.get_line()
+        return completer.get_completions(line)


Can this raise? What do we want to do if any of the pkgutil raises for any reason?

It shouldn't unless iter_modules or find_spec raise. Since those can call code from user-supplied finders, they can raise an arbitrary exception. I could add a simple except Exception block and simply return no completions if that happens. To the user it would look like no completions are available rather than outright crashing the repl.

Added in fd81999

pablogsal · 2025-04-19T01:21:57Z

Lib/_pyrepl/readline.py

+        """Global module cache"""
+        if not self._global_cache or self._curr_sys_path != sys.path:
+            self._curr_sys_path = sys.path[:]
+            self._global_cache = list(pkgutil.iter_modules())


Nothing to do here but I am a bit concerned about how this can perform for a lot of packages

See this #129329 (comment) with timings

pablogsal · 2025-04-19T01:23:39Z

Lib/_pyrepl/readline.py

+        if self.code.rstrip().endswith('import') and self.code.endswith(' '):
+            return Result(from_name=self.parse_empty_from_import(), name='')
+        if self.code.rstrip().endswith('from') and self.code.endswith(' '):
+            return Result(from_name='')


This calls code.rstrip() a lot, maybe is worth setting that as an attribute or just always use self.code = code.rstrip(). What do you think?

At the very least maybe set it as a local so you don't strip twice

I added it as a local since it's only called in 3 places

pablogsal · 2025-04-19T01:25:56Z

Lib/_pyrepl/readline.py

@@ -161,6 +169,11 @@ def get_completions(self, stem: str) -> list[str]:
            result.sort()
        return result

+    def get_module_completions(self) -> list[str]:
+        completer = ModuleCompleter(namespace={'__package__': '_pyrepl'})  # TODO: namespace?


Is this creating one of this guys per request? Should we cache the ModuleCompleter?

it is, and yes we should cache it. Updated in 5c11124 (I also moved the completer to a separate file to avoid having to reorder everything when I added it to the ReadlineConfig)

Also changed in f4e290a so that each Reader gets a new ModuleCompleter instance (with its own module cache)

pablogsal · 2025-04-19T01:26:55Z

@tomasr8 can you try to play a bit with some edge cases (tons of packages, etc) to see how the performance is for extreme circumstances? I want to be aware of the ways this can "break"

tomasr8 · 2025-04-19T06:48:31Z

Thanks for taking the time to review this, I really appreciate it!

@tomasr8 can you try to play a bit with some edge cases (tons of packages, etc) to see how the performance is for extreme circumstances? I want to be aware of the ways this can "break"

When it comes to performance, the biggest bottleneck is the call to pkgutil.iter_modules() to find all top-level packages.
(the result is cached so we pay the cost only the first time you hit TAB).

Here are some timings for lots of packages. I installed the top 1000 pypi packages (at least those that can be installed on 3.14 so 836 packages installed):

~ pip freeze | wc -l
836

The timings are on a laptop with Intel ultra 9 185H CPU and a new-ish SSD (relevant because pkgutil.iter_modules interacts with the file system).

For those ~800 packages in a single location, pkgutil.iter_modules takes about 0.03s:

>>> import time, pkgutil
>>> start = time.time(); list(pkgutil.iter_modules()); time.time() - start
0.03072071075439453

Typing import <tab> takes about 0.03s as well so most of the cost is really inside pkgutil.iter_modules:

>>> import <tab>
0.03162884712219238

Subsequent completion requests are much faster since the results are cached:

>>> import <tab>
0.002469778060913086

Another thing I tried is multiple search locations (i.e. multiple sys.path entries):

5 different locations and ~4000 packages total:

`pkgutil.iter_modules`: 0.3535459041595459s
Initial `import <tab>`: 0.3592982292175293s
Subsequent `import <tab>`: 0.0021576881408691406s

10 different locations and ~8000 packages total:

`pkgutil.iter_modules`: 0.6836235523223877s
Initial `import <tab>`: 0.6652779579162598s
Subsequent `import <tab>`: 0.0002522468566894531s

The initial search is about 0.35s and 0.68s respectively, while the subsequent ones are again very cheap.

hugovk · 2025-04-23T10:58:29Z

I still get this failure locally:

======================================================================
FAIL: test_import_completions (test.test_pyrepl.test_pyrepl.TestPyReplModuleCompleter.test_import_completions) (code='import path\t\n')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/hugo/github/python/cpython/main/Lib/test/test_pyrepl/test_pyrepl.py", line 938, in test_import_completions
    self.assertEqual(output, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: 'import path' != 'import pathlib'
- import path
+ import pathlib
?            +++

It's because my env isn't a clean slate because I've pip installed some packages for testing on 3.14, and a dependency is interfering:

Python 3.14.0a7+ (heads/completer:f4e290a03f1, Apr 23 2025, 13:32:14) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import path<tab>
pathlib       pathvalidate

❯ ./python.exe -m pip freeze | grep path
pathvalidate==3.2.0

I don't think this needs to block initial merge before the freeze, but would be good to fix.

(PS there's a merge conflict on some imports.)

tomasr8 · 2025-04-23T12:24:12Z

Weird, I tested this locally with some scripts on sys.path and it passed for me. Sorry about that, I'll take a look later today!

tomasr8 · 2025-04-23T20:41:58Z

Ok so iter_modules also gets FileFinders from sys.path so I had to change the approach but it should work now. Tested locally with pathvalidate installed and the tests are passing.

hugovk · 2025-04-24T08:49:06Z

Passing now, thanks!

pablogsal · 2025-04-25T01:24:37Z

LGTM, fantastic job @tomasr8

tomasr8 added 2 commits January 27, 2025 00:19

Add module autocomplete to PyREPL

b3bcd67

Add news entry

bcd3527

tomasr8 requested review from pablogsal, lysnikolaou and ambv as code owners January 27, 2025 00:55

bedevere-app bot added the awaiting review label Jan 27, 2025

bedevere-app bot mentioned this pull request Jan 27, 2025

Readline completion of module names in import statements #69605

Open

Merge branch 'main' into completer

df3e8ec

tomasr8 commented Jan 27, 2025

View reviewed changes

Merge branch 'main' into completer

48ee6ad

tomasr8 added the topic-repl Related to the interactive shell label Feb 22, 2025

hugovk reviewed Feb 25, 2025

View reviewed changes

tomasr8 added 4 commits March 8, 2025 14:52

Remove attribute completion, never import modules

0917d24

Add type annotations

589cf63

fix some mypy issues

62d0b55

Pass explicit None to find_spec

46ca249

neutrinoceros mentioned this pull request Mar 9, 2025

ENH: ensure that dir(astropy) lists subpackages astropy/astropy#17598

Merged

1 task

tomasr8 added 3 commits April 17, 2025 16:38

Merge branch 'main' into completer

75e4b55

Do not suggest modules which are not legal identifiers

3c13f86

Make the tests more robust

8eb656f

pablogsal reviewed Apr 19, 2025

View reviewed changes

tomasr8 added 3 commits April 19, 2025 08:29

Remove todo comment

7a2fde0

Move to a separate file and cache ModuleCompleter

5c11124

Avoid calling rstrip more than once

10da15b

tomasr8 added 3 commits April 19, 2025 09:01

Catch exceptions

fd81999

Fix tests

8fba3d3

Every Reader has its own ModuleCompleter instance

f4e290a

tomasr8 requested a review from pablogsal April 20, 2025 18:48

tests: Only look for modules in the stdlib

602121d

pablogsal approved these changes Apr 25, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Apr 25, 2025

pablogsal merged commit c3a7118 into python:main Apr 25, 2025
51 checks passed

bedevere-app bot removed the awaiting merge label Apr 25, 2025

tomasr8 deleted the completer branch April 25, 2025 07:31

pllim mentioned this pull request Apr 25, 2025

Revisit dir(astropy) implementation when Python 3.14 is minversion astropy/astropy#18055

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-69605: Add module autocomplete to PyREPL #129329

gh-69605: Add module autocomplete to PyREPL #129329

tomasr8 commented Jan 27, 2025 •

edited

Loading

picnixz commented Jan 27, 2025

tomasr8 commented Jan 27, 2025

tomasr8 Jan 27, 2025

hugovk commented Jan 30, 2025

tomasr8 commented Jan 30, 2025

gaogaotiantian commented Feb 25, 2025

tomasr8 commented Feb 25, 2025

gaogaotiantian commented Feb 25, 2025

tomasr8 commented Feb 25, 2025

gaogaotiantian commented Feb 25, 2025

tomasr8 commented Feb 25, 2025

gaogaotiantian commented Feb 25, 2025

tomasr8 commented Feb 25, 2025

tomasr8 commented Mar 10, 2025

pablogsal Apr 19, 2025

tomasr8 Apr 19, 2025

pablogsal Apr 19, 2025

tomasr8 Apr 19, 2025

tomasr8 Apr 19, 2025

pablogsal Apr 19, 2025

tomasr8 Apr 19, 2025

pablogsal Apr 19, 2025

tomasr8 Apr 19, 2025

pablogsal Apr 19, 2025

tomasr8 Apr 19, 2025

tomasr8 Apr 19, 2025 •

edited

Loading

pablogsal commented Apr 19, 2025

tomasr8 commented Apr 19, 2025 •

edited

Loading

hugovk commented Apr 23, 2025

tomasr8 commented Apr 23, 2025

tomasr8 commented Apr 23, 2025

hugovk commented Apr 24, 2025

pablogsal commented Apr 25, 2025

gh-69605: Add module autocomplete to PyREPL #129329

gh-69605: Add module autocomplete to PyREPL #129329

Conversation

tomasr8 commented Jan 27, 2025 • edited Loading

picnixz commented Jan 27, 2025

tomasr8 commented Jan 27, 2025

Choose a reason for hiding this comment

hugovk commented Jan 30, 2025

tomasr8 commented Jan 30, 2025

gaogaotiantian commented Feb 25, 2025

tomasr8 commented Feb 25, 2025

gaogaotiantian commented Feb 25, 2025

tomasr8 commented Feb 25, 2025

gaogaotiantian commented Feb 25, 2025

tomasr8 commented Feb 25, 2025

gaogaotiantian commented Feb 25, 2025

tomasr8 commented Feb 25, 2025

tomasr8 commented Mar 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomasr8 Apr 19, 2025 • edited Loading

Choose a reason for hiding this comment

pablogsal commented Apr 19, 2025

tomasr8 commented Apr 19, 2025 • edited Loading

hugovk commented Apr 23, 2025

tomasr8 commented Apr 23, 2025

tomasr8 commented Apr 23, 2025

hugovk commented Apr 24, 2025

pablogsal commented Apr 25, 2025

tomasr8 commented Jan 27, 2025 •

edited

Loading

tomasr8 Apr 19, 2025 •

edited

Loading

tomasr8 commented Apr 19, 2025 •

edited

Loading