GH-132439: REPL on Windows swallows characters entered via AltGr #132440

chris-eibl · 2025-04-12T14:31:29Z

E.g. on my keyboard with German layout, { is usually entered via pressing the AltGr key and 7, i.e. AltGr+7.

Likewise, }, [, ], \ and some more can only be entered via AltGr.

But since #128388 / #128389 these are swallowed by the REPL on Windows and can no longer be entered.

This happens in legacy Windows terminals, where the virtual terminal mode is turned off (e.g. cmd.exe).

In virtual terminal mode there are other issues, see #131878.

Issue: New REPL on Windows swallows characters entered via AltGr #132439

chris-eibl · 2025-04-12T14:34:57Z

Lib/_pyrepl/windows_console.py

+                # Do not swallow characters that have been entered via AltGr:
+                # Windows internally converts AltGr to CTRL+ALT, see
+                # https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-vkkeyscanw
+                if not key_event.dwControlKeyState & CTRL_ACTIVE:


This fixes the issue for me. Control-← and Control-→ word-skipping keybindings and Alt-d to kill-word or Alt-Backspace to backward-kill-word mentioned in #128389 are still working.

chris-eibl · 2025-04-12T14:42:29Z

Lib/_pyrepl/windows_console.py

                if block:
                    continue

                return None
            elif self.__vt_support:
                # If virtual terminal is enabled, scanning VT sequences
-                self.event_queue.push(rec.Event.KeyEvent.uChar.UnicodeChar)
+                self.event_queue.push(raw_key)


A missed opportunity when main was merged during development of #124119.

Which @sergey-miryanov takes care of here as part of #131901.

So maybe I should remove this one, but I'd really like to keep the other two raw_key changes (not only because I'd have to adapt almost all tests :)

IMHO you should merge my branch here :)

I can undo the change - then you won't have conflicts. I don't think we should (partially) merge between our two PRs?

Yeah, I think you are right - we shouldn't merge our PRs. You can keep your changes - I'm OK if here will be conflict.

chris-eibl · 2025-04-12T14:46:39Z

Lib/_pyrepl/windows_console.py

                        return Event(evt="key", data="\033")  # keymap.py uses this for meta
-                    return Event(evt="key", data=key, raw=key)
+                    return Event(evt="key", data=key, raw=raw_key)


Looking at the diff , previously

return Event( evt="key", data=code, raw=rec.Event.KeyEvent.uChar.UnicodeChar )

was used, which in the new code should have been raw_key?

Actually, I was wondering why this change did not break anything, but AFAICT the raw member is not used in the whole code base except

cpython/Lib/_pyrepl/unix_console.py

Line 513 in 5d8e432

def getpending(self):

where the two getpending implementations dutifully respect e.raw += e.raw :)

When searching in the code base for getpending I found

cpython/Lib/_pyrepl/windows_console.py

Lines 518 to 521 in 5d8e432

def getpending(self) -> Event:

"""Return the characters that have been typed but not yet

processed."""

return Event("key", "", b"")

which clearly seems to be a bug to me? Because even though WindowsConsole._read_input() only reads one Windows INPUT_RECORD per call, get_event could have put something into self.event_queue.

I've addressed this in https://github.com/python/cpython/pull/132889/files#r2059235071.

chris-eibl · 2025-04-12T14:48:34Z

Lib/_pyrepl/windows_console.py

@@ -459,22 +459,26 @@ def get_event(self, block: bool = True) -> Event | None:
                        key = f"ctrl {key}"
                    elif key_event.dwControlKeyState & ALT_ACTIVE:
                        # queue the key, return the meta command
-                        self.event_queue.insert(Event(evt="key", data=key, raw=key))
+                        self.event_queue.insert(Event(evt="key", data=key, raw=raw_key))


Unsure about this one, but raw_key seems plausible?
@paulie4

I'm not sure that key or raw_key is OK here. key is a str here, raw_key is unicode symbol (maybe with two or more code points) - both are not a byte that expected. Maybe we should encode raw_key here? But I don't know how to test it.

Yeah, these code paths return a string instead of bytes either way. But as written in #132440 (comment), Event.raw is not used anywhere.

I fully agree that it should be bytes:

cpython/Lib/_pyrepl/console.py

Lines 40 to 44 in 5d8e432

@dataclass

class Event:

evt: str

data: str

raw: bytes = b""

It is almost everywhere used incorrectly in windows_console.py.

Btw, raw_key is always "\x00" in the mentioned code block

elif key == "\x00": # Handle special keys like arrow keys and translate them into the appropriate command key = VK_MAP.get(key_event.wVirtualKeyCode) if key: if key_event.dwControlKeyState & CTRL_ACTIVE: key = f"ctrl {key}" elif key_event.dwControlKeyState & ALT_ACTIVE: # queue the key, return the meta command self.event_queue.insert(Event(evt="key", data=key, raw=raw_key)) return Event(evt="key", data="\033") # keymap.py uses this for meta return Event(evt="key", data=key, raw=raw_key)

because

raw_key = key = key_event.uChar.UnicodeChar

a few lines above. I tend to believe that Event.raw is / was a debug help, since it serves (to me) no obvious purpose ...

chris-eibl · 2025-04-12T14:53:03Z

Adding @vstinner, @eendebakpt and @paulie4 , since they were involved in #128389 and already talked about AltGr.

chris-eibl · 2025-04-12T14:56:22Z

PS: switching to an english keyboard layout, I can use the AltGr key like right-Alt, since it is not used in this keyboard layout :)

chris-eibl · 2025-04-12T17:15:46Z

The failing of Ubuntu (free-threading) is definitely unrelated, since this is a Windows specific change.

tomasr8 · 2025-04-17T19:21:45Z

Do you think it'd be possible to add some tests for this change to ensure we don't accidentally regress again?

chris-eibl · 2025-04-17T19:43:44Z

Yeah, I've already thought about it, but am still unsure how to best do it.

ATM, I am thinking of mocking

cpython/Lib/_pyrepl/windows_console.py

Line 411 in b87189d

def _read_input(self, block: bool = True) -> INPUT_RECORD | None:

to be able to test

cpython/Lib/_pyrepl/windows_console.py

Lines 426 to 432 in b87189d

    
               def get_event(self, block: bool = True) -> Event | None: 
        
                   """Return an Event instance.  Returns None if |block| is false 
        
                   and there is no event pending, otherwise waits for the 
        
                   completion of an event.""" 
        
                   while self.event_queue.empty(): 
        
                       rec = self._read_input(block)

AFAICT, all the existing tests just mock get_event, but for this, IMHO get_event itself must be tested.
Mocking _read_input seems to be the best way forward.

WDYT?

tomasr8 · 2025-04-17T20:17:22Z

Agreed that we shouldn't be mocking get_event since that's where we're making changes. _read_input seems like a good option, we can capture some real inputs and then replay them with it.

chris-eibl · 2025-04-20T10:16:41Z

Lib/test/test_pyrepl/test_windows_console.py

        self.assertEqual(reader.cxy, (2, 3))
+        con.restore()


Without this fix I get a garbaged terminal after running the tests.

chris-eibl · 2025-04-20T10:21:54Z

Lib/test/test_pyrepl/test_windows_console.py

+        con.restore()
+
+
+class WindowsConsoleGetEventTests(TestCase):


I've printed the INPUT_RECORDs I get for

a lot of keys (or combination thereof)

and for different keyboard layouts

with virtual terminal support disabled or enabled (by running either in a legacy console or a Windows terminal)

chris-eibl · 2025-04-20T10:28:41Z

Lib/test/test_pyrepl/test_windows_console.py

+        self.assertEqual(self.get_event([None]), None)
+        self.assertEqual(self.mock.call_count, 1)
+
+    def test_WINDOW_BUFFER_SIZE_EVENT(self):


I know this is not PEP 8 compliant, but IMHO here it makes more sense to use things like WINDOW_BUFFER_SIZE_EVENT or LEFT_ALT_PRESSED.

E.g. using lower/upper case m / M instead m_key_lower etc.

Especially, test_m_LEFT_ALT_PRESSED_and_LEFT_CTRL_PRESSED would be much harder to write to not be misleading, e.g. test_m_key_lower_left_alt_pressed_and_left_ctr_pressed. Or test_left_LEFT_CTRL_PRESSED.

chris-eibl · 2025-04-20T11:03:19Z

@sergey-miryanov since you are working on REPL issues on Windows, too: can you have a look at my PR?
And do some more manual tests with this PR in a legacy console? The virtual terminal fixes are anyway part of your #131901.

sergey-miryanov · 2025-04-20T12:18:43Z

@chris-eibl

Python 3.11.9 from cmd or ps - ctrl+backspace removes whole words, alt (left or right) + [{}] - inputs just [{}]
Python 3.13.1 from cmd or ps - ctrl+backspace removes symbol by symbol, alt (left or right) + [{}] - inputs just [{}]
Your branch from cmd or ps - ctrl+backspace removes symbol by symbol, alt (left or right) + [ - inputs nothing, alt + [[ (or [[[ - I'm not sure sometime two repeats sometimes 3 repeats) - inputs [, also alt + [] - inputs ].

sergey-miryanov · 2025-04-20T12:22:17Z

Also on your branch if I put altgr + m - it inputs nothing.

chris-eibl · 2025-04-20T13:22:06Z

And do some more manual tests with this PR in a legacy console

Can you also verify, that for cmd the VT is disabled?
My PR won't be any help if VT is enabled.
Depending on how the Windows Terminal is configured, it might be possible, that you are never in the legacy case?
For me, I have to manually start a cmd. If I start it via the Windows Terminal (Eingabeauffordernung in German),
a cmd instance in VT mode is started :-O

What is your keyboard layout?
Does your keybord feature the AltGr key?
If not, does the right Alt key behave like AltGr when you switch to a keyboard layout that uses it (like German)?
How does the main branch behave in these cases? I.e. without my changes.

Also on your branch if I put altgr + m - it inputs nothing.

What does AltGr + m do in your editor? It should do the same in the REPL. If your keyboard layout does not feature anything on AltGr + m, then it is normal, that nothing happens. See https://en.wikipedia.org/wiki/AltGr_key which keyboards do what.

I've tested e.g. a French layout, too. For me, everything works well in the legacy terminal.
My PR doesn't touch VT, so everything is unchanged there.

That's your PR that fixes the VT for me 🚀

chris-eibl · 2025-04-20T13:23:12Z

@sergey-miryanov: and many thanks for testing ❤️

chris-eibl · 2025-04-20T15:00:12Z

Merge with main to see whether that solves the GHA hiccup.

[Edit] - it did fix GHA.

sergey-miryanov · 2025-04-20T15:38:34Z

Previously I started it via cmd.exe but windows 11 has option about what terminal host to use - legacy or the new one. I disabled VT now - but get the same results. Later I will try to add German keyboard layout and test again.

Btw, I have some strange case - I can't exit from multiline mode:

chris-eibl · 2025-04-20T16:10:59Z

Oh, I am running out of ideas now. I am on Windows 10, but can't believe this is the culprit.

Seems that not a sinlge thing that is working for me is working for you.

After all, my PR should not induce any change in behaviour, except

in addition to one of the Alt keys a Ctrl key is pressed, too, and furthermore one of the keys with a blue letter according to the keyboard diagrams in https://en.wikipedia.org/wiki/AltGr_key
or the AltGr key is used together with such a "blue" key. Windows converts AltGr automatically into CTRL+ALT, so basically this is the same as above.
both of the above (e.g. AltGr + m or Left Ctrl + Left Alt + m will only do something in an appropriate keyboard layout (e.g. in German, this will result in µ). Otherwise, nothing happens (e.g. when I switch to an English keyboard layout).

Since the Alt or Ctrl code path together with e.g. arrow keys is unchanged

            elif key == "\x00":
                # Handle special keys like arrow keys and translate them into the appropriate command
                key = VK_MAP.get(key_event.wVirtualKeyCode)

(admittetly except for the raw_key change which doesn't change the behaviour for me), all those should remain working (and do for me).

What happens if you omit my fix?

Maybe other Windows users can test, too, so we have better coverage, in case this depends on more things we do not yet know.

sergey-miryanov · 2025-04-20T17:33:58Z

Maybe I misinformed you.
With DEU German (Germany) layout in cmd with vt=False I have:

main branch - if I input altgr+8 or altgr+9 or altgr+m - nothing happens
your branch - if I input as above - [, ], µ inserted.

With DEU German (Germany) layout in ps with vt=True I have:
2. main and your branch - if I input as above - [, ] inserted, and for altgr+m I switched to unknown (for me) mode when nothing typed if I press any key, if I press Esc I got error as in #131901

I hope this is helps you. Let me know if you need to test something else.

chris-eibl · 2025-04-20T18:35:58Z

Oh, that helps a lot - because you can reproduce my fix in legacy terminals.

As said, my PR cannot help with VT mode quirks (and is not intended to fix it), that's why we need your #131901 - which looks promising, too, but we shouldn't mix them.

sergey-miryanov · 2025-04-21T06:07:53Z

Lib/test/test_pyrepl/test_windows_console.py

+    VK_OEM_7 = 0xDE
+
+    # State of control keys: https://learn.microsoft.com/en-us/windows/console/key-event-record-str
+    RIGHT_ALT_PRESSED = 0x0001


windows_console.py has ALT_ACTIVE as bitwise or of 0x01 and 0x02 - I believe it is better to move these constants to file and use them. WDYT?

I've placed it there in the beginning and then later removed it to keep the diff in windows_console.py small.
I'd happily move it if other reviewers prefer that, too.

Likewise for CTRL_ACTIVE.

sergey-miryanov

LGTM

Misc/NEWS.d/next/Library/2025-04-12-16-29-42.gh-issue-132439.3twrU6.rst

Lib/test/test_pyrepl/test_windows_console.py

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

chris-eibl added 2 commits April 12, 2025 16:01

do not swallow characters that have been entered via AltGr

59d71b0

blurb it

cca6155

chris-eibl requested review from pablogsal, lysnikolaou and ambv as code owners April 12, 2025 14:31

bedevere-app bot added the awaiting review label Apr 12, 2025

bedevere-app bot mentioned this pull request Apr 12, 2025

New REPL on Windows swallows characters entered via AltGr #132439

Open

chris-eibl commented Apr 12, 2025

View reviewed changes

chris-eibl added OS-windows 3.13 bugs and security fixes 3.14 new features, bugs and security fixes topic-repl Related to the interactive shell labels Apr 12, 2025

unrelated stuff worth to do (?)

1e61540

chris-eibl commented Apr 12, 2025

View reviewed changes

chris-eibl requested a review from vstinner April 12, 2025 14:49

picnixz added needs backport to 3.13 bugs and security fixes and removed 3.13 bugs and security fixes 3.14 new features, bugs and security fixes labels Apr 12, 2025

chris-eibl added 3 commits April 19, 2025 08:30

Merge remote-tracking branch 'origin/main' into fix_altgr

c60a727

fix missing con.restore() in test_multiline_ctrl_z

ed035e5

add WindowsConsoleGetEventTests

69e8779

chris-eibl commented Apr 20, 2025

View reviewed changes

Merge branch 'main' into fix_altgr

8ee232c

sergey-miryanov reviewed Apr 21, 2025

View reviewed changes

sergey-miryanov approved these changes Apr 21, 2025

View reviewed changes

bedevere-app bot added awaiting core review and removed awaiting review labels Apr 21, 2025

StanFromIreland reviewed Apr 22, 2025

View reviewed changes

Misc/NEWS.d/next/Library/2025-04-12-16-29-42.gh-issue-132439.3twrU6.rst Outdated Show resolved Hide resolved

sergey-miryanov mentioned this pull request Apr 24, 2025

Brackets "[" and "]" writing or inserting does not work in Python 3.13 interactive shell (Portuguese keyboard) #132871

Closed

This was referenced Apr 24, 2025

Pasting in new REPL is slow #130328

Open

GH-130328: pasting in new REPL is slow on Windows #132884

Merged

chris-eibl commented Apr 25, 2025

View reviewed changes

Lib/test/test_pyrepl/test_windows_console.py Outdated Show resolved Hide resolved

chris-eibl and others added 2 commits April 25, 2025 21:40

Apply suggestions from code review

e67452b

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Update Lib/test/test_pyrepl/test_windows_console.py

396f48e

	def getpending(self) -> Event:
	"""Return the characters that have been typed but not yet
	processed."""
	return Event("key", "", b"")

GH-132439: REPL on Windows swallows characters entered via AltGr #132440

Are you sure you want to change the base?

GH-132439: REPL on Windows swallows characters entered via AltGr #132440

Conversation

chris-eibl commented Apr 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergey-miryanov Apr 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chris-eibl Apr 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chris-eibl commented Apr 12, 2025

chris-eibl commented Apr 12, 2025

chris-eibl commented Apr 12, 2025

tomasr8 commented Apr 17, 2025

chris-eibl commented Apr 17, 2025

tomasr8 commented Apr 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chris-eibl commented Apr 20, 2025

sergey-miryanov commented Apr 20, 2025

sergey-miryanov commented Apr 20, 2025

chris-eibl commented Apr 20, 2025

chris-eibl commented Apr 20, 2025

chris-eibl commented Apr 20, 2025 • edited Loading

sergey-miryanov commented Apr 20, 2025

chris-eibl commented Apr 20, 2025

sergey-miryanov commented Apr 20, 2025

chris-eibl commented Apr 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergey-miryanov left a comment

Choose a reason for hiding this comment

chris-eibl commented Apr 12, 2025 •

edited

Loading

sergey-miryanov Apr 20, 2025 •

edited

Loading

chris-eibl Apr 20, 2025 •

edited

Loading

chris-eibl commented Apr 20, 2025 •

edited

Loading