gh-131507: Add support for syntax highlighting in PyREPL #133247

ambv · 2025-05-01T11:02:32Z

This is a much improved version of gh-131562. It uses the tokenizer for better speed and pattern matching for more robust handling of soft keywords. While it still won't hit all cases correctly, it's better than idlelib's regular expression-based colorizer and unlike our glorious PEG parser it supports incomplete input, which is crucial for an interactive shell.

Relatedly, pasting support was tweaked to be way faster. Now the entire contents of Frankenstein can be pasted within 3 seconds both on Unix and Windows as long as bracketed pasting is supported by the terminal. This is a necessary tweak for syntax highlighting not to cripple performance of pastes above 5kB.

There is experimental support for theming through _colorize.set_theme() that's mentioned in "What's New" but otherwise undocumented so far.

Issue: Syntax highlighting in PyREPL #131507

📚 Documentation preview 📚: https://cpython-previews--133247.org.readthedocs.build/

…ted paste

bedevere-bot · 2025-05-01T11:31:56Z

🤖 New build scheduled with the buildbot fleet by @ambv for commit ffebbbe 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

tomasr8

Sorry for being too nitpicky, but I still find the blue used for keywords to be too dark when coupled with dark-themed terminals (e.g. #131562 (comment)). Do you think we could use a more contrasting color?

Lib/_pyrepl/utils.py

ambv · 2025-05-01T14:20:33Z

I disagree about the blue color, the fact that Ubuntu colors it too dim is not actionable for the Python project. Adjust your terminal.

ambv · 2025-05-01T15:19:46Z

The refleak failures are unrelated, see #133258.

bedevere-bot · 2025-05-01T17:41:42Z

🤖 New build scheduled with the buildbot fleet by @ambv for commit 9b60382 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

chris-eibl · 2025-05-02T06:36:30Z

@ambv: your changes in this PR regarding chunked reading in Windows are more or less what I did in #132889, so maybe I should close my PR?

chris-eibl · 2025-05-02T06:49:17Z

Having a closer look, I see you are only reading in chunks in case of getpending, so my PR might have added value. Let's merge this first and then I can rebase my PR if it is worthwile to do chunked reading in the regular case, too?

ambv · 2025-05-02T08:36:57Z

@chris-eibl I simplified the Unix case to remove the additional buffering since it complicates the codebase. Your PR was adding that same buffering to Windows. Given that we now achieve the same performance without double-buffering, I'd say we don't need the other PR.

Viicos · 2025-05-02T08:41:48Z

Lib/_colorize.py

+    type ColorTag = (
+        Literal["PROMPT"]
+        | Literal["KEYWORD"]
+        | Literal["BUILTIN"]
+        | Literal["COMMENT"]
+        | Literal["STRING"]
+        | Literal["NUMBER"]
+        | Literal["OP"]
+        | Literal["DEFINITION"]
+        | Literal["SOFT_KEYWORD"]
+        | Literal["RESET"]
+    )


hugovk · 2025-05-02T08:40:20Z

Doc/whatsnew/3.14.rst

+Syntax highlighting in PyREPL
+-----------------------------


Let's list this in the "Summary – release highlights".

hugovk · 2025-05-02T08:52:24Z

Lib/_colorize.py

+        "STRING": colors.GREEN,
+        "NUMBER": colors.YELLOW,
+        "OP": colors.RESET,
+        "DEFINITION": colors.BOLD_WHITE,


Looks good on macOS with dark theme:

But I think we should avoid bold white for light theme:

Instead, we can add BOLD = "\x1b[1m" to ANSIColors with no specific colour.

Suggested change

"DEFINITION": colors.BOLD_WHITE,

"DEFINITION": colors.BOLD,

Then we get a bold version of whatever colour the user's terminal has set:

hugovk · 2025-05-02T08:56:00Z

Lib/_pyrepl/commands.py

-        self.reader.dirty = True
+class perform_bracketed_paste(Command):
+    def do(self) -> None:
+        done = "\x1b[201~"


Perhaps:

Suggested change

done = "\x1b[201~"

done = "\x1b[201~" # Disable bracketed paste

hugovk · 2025-05-02T08:57:10Z

Lib/_pyrepl/reader.py

            wrapcount = (sum(char_widths) + prompt_len) // self.console.width
-            trace("wrapcount = {wrapcount}", wrapcount=wrapcount)
+            # trace("wrapcount = {wrapcount}", wrapcount=wrapcount)


Delete? Or useful to keep?

Suggested change

# trace("wrapcount = {wrapcount}", wrapcount=wrapcount)

hugovk · 2025-05-02T09:17:06Z

Lib/_pyrepl/utils.py

+    if (
+        msg.startswith("unterminated string literal")
+        or msg.startswith("unterminated f-string literal")
+        or msg.startswith("unterminated t-string literal")
+        or msg.startswith("EOF in multi-line string")
+        or msg.startswith("unterminated triple-quoted f-string literal")
+        or msg.startswith("unterminated triple-quoted t-string literal")
+    ):


Suggested change

if (

msg.startswith("unterminated string literal")

or msg.startswith("unterminated f-string literal")

or msg.startswith("unterminated t-string literal")

or msg.startswith("EOF in multi-line string")

or msg.startswith("unterminated triple-quoted f-string literal")

or msg.startswith("unterminated triple-quoted t-string literal")

):

if msg.startswith(

(

"unterminated string literal",

"unterminated f-string literal",

"unterminated t-string literal",

"EOF in multi-line string",

"unterminated triple-quoted f-string literal",

"unterminated triple-quoted t-string literal",

)

):

Single call with a tuple can be 3x as fast:

❯ python3 -m timeit -s 'msg = "unterminated triple-quoted t-string literal (detected at line 2)"' 'msg.startswith("unterminated string literal") or msg.startswith("unterminated f-string literal") or msg.startswith("unterminated t-string literal") or msg.startswith("EOF in multi-line string") or msg.startswith("unterminated triple-quoted f-string literal") or msg.startswith("unterminated triple-quoted t-string literal")' 5000000 loops, best of 5: 99.8 nsec per loop ❯ python3 -m timeit -s 'msg = "unterminated triple-quoted t-string literal (detected at line 2)"' 'msg.startswith(("unterminated string literal", "unterminated f-string literal", "unterminated t-string literal", "EOF in multi-line string", "unterminated triple-quoted f-string literal", "unterminated triple-quoted t-string literal"))' 10000000 loops, best of 5: 33.9 nsec per loop

hugovk · 2025-05-02T09:20:40Z

Lib/_pyrepl/utils.py

+        # the next call to `disp_str()` will revive it.
+        chars[-1] += _colorize.theme["RESET"]
+
+    # trace("disp_str({buffer}) = {s}, {b}", buffer=repr(buffer), s=chars, b=char_widths)


?

Suggested change

# trace("disp_str({buffer}) = {s}, {b}", buffer=repr(buffer), s=chars, b=char_widths)

skirpichev · 2025-05-02T09:43:23Z

Sorry for being too nitpicky, but I still find the blue used for keywords to be too dark when coupled with dark-themed terminals

I think that the default theme polishing can be addressed in following pr(s?), as per pr description:

There is experimental support for theming through _colorize.set_theme() that's mentioned in "What's New" but otherwise undocumented so far.

ambv added 15 commits April 29, 2025 18:05

pythongh-131507: Add support for syntax highlighting in PyREPL

e921a80

Add Blurb

fb95911

Fix irrelevant Windows tests

b428513

Replace idlelib.colorizer with a faster solution

2bdcd06

Slurp the entire input buffer before refreshing display during bracke…

8c70c45

…ted paste

Implement getpending() on Windows

4d7ae36

Adapt tests

9585bd6

Support soft keywords (fight fire with fire)

b1f2557

Fix test

01e1129

Remove unnecessary import

20eff49

Add test for prev_next_window

8d3648a

Windows: bracketed pasting of 448692 chars done in 2.38s ✊

656fea3

Remove colors from Windows low-level console tests

dac8961

Fix lint and stuff

7891fa7

Add experimental theming support for syntax highlighting and the prompt

362a21b

ambv requested review from hugovk, pablogsal and lysnikolaou as code owners May 1, 2025 11:02

bedevere-app bot mentioned this pull request May 1, 2025

Syntax highlighting in PyREPL #131507

Open

bedevere-app bot added the awaiting core review label May 1, 2025

Fix lint

ffebbbe

ambv added 🔨 test-with-buildbots Test PR w/ buildbots; report in status section topic-repl Related to the interactive shell labels May 1, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 1, 2025

tomasr8 reviewed May 1, 2025

View reviewed changes

Lib/_pyrepl/utils.py Show resolved Hide resolved

Lib/_pyrepl/utils.py Outdated Show resolved Hide resolved

ambv added 2 commits May 1, 2025 19:17

Merge branch 'main' into pyrepl-syntax-highlighting-tokens

388e494

Add t-string support to syntax highlighting

9b60382

ambv added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 1, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 1, 2025

Viicos reviewed May 2, 2025

View reviewed changes

hugovk reviewed May 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-131507: Add support for syntax highlighting in PyREPL #133247

gh-131507: Add support for syntax highlighting in PyREPL #133247

ambv commented May 1, 2025 •

edited by github-actions bot

Loading

bedevere-bot commented May 1, 2025

tomasr8 left a comment

ambv commented May 1, 2025

ambv commented May 1, 2025

bedevere-bot commented May 1, 2025

chris-eibl commented May 2, 2025

chris-eibl commented May 2, 2025

ambv commented May 2, 2025

Viicos May 2, 2025

hugovk May 2, 2025

hugovk May 2, 2025

hugovk May 2, 2025

hugovk May 2, 2025

hugovk May 2, 2025

hugovk May 2, 2025

skirpichev commented May 2, 2025

	done = "\x1b[201~"
	done = "\x1b[201~" # Disable bracketed paste

-    if (
-        msg.startswith("unterminated string literal")
-        or msg.startswith("unterminated f-string literal")
-        or msg.startswith("unterminated t-string literal")
-        or msg.startswith("EOF in multi-line string")
-        or msg.startswith("unterminated triple-quoted f-string literal")
-        or msg.startswith("unterminated triple-quoted t-string literal")
-    ):
+    if msg.startswith(
+        (
+            "unterminated string literal",
+            "unterminated f-string literal",
+            "unterminated t-string literal",
+            "EOF in multi-line string",
+            "unterminated triple-quoted f-string literal",
+            "unterminated triple-quoted t-string literal",
+        )
+    ):

gh-131507: Add support for syntax highlighting in PyREPL #133247

Are you sure you want to change the base?

gh-131507: Add support for syntax highlighting in PyREPL #133247

Conversation

ambv commented May 1, 2025 • edited by github-actions bot Loading

bedevere-bot commented May 1, 2025

tomasr8 left a comment

Choose a reason for hiding this comment

ambv commented May 1, 2025

ambv commented May 1, 2025

bedevere-bot commented May 1, 2025

chris-eibl commented May 2, 2025

chris-eibl commented May 2, 2025

ambv commented May 2, 2025

Viicos May 2, 2025

Choose a reason for hiding this comment

hugovk May 2, 2025

Choose a reason for hiding this comment

hugovk May 2, 2025

Choose a reason for hiding this comment

hugovk May 2, 2025

Choose a reason for hiding this comment

hugovk May 2, 2025

Choose a reason for hiding this comment

hugovk May 2, 2025

Choose a reason for hiding this comment

hugovk May 2, 2025

Choose a reason for hiding this comment

skirpichev commented May 2, 2025

ambv commented May 1, 2025 •

edited by github-actions bot

Loading