@zooba, @vstinner, do you mind to take a look at this PR?

I keep trying to review and I keep getting confused 😆

Perhaps we should revise this entire piece of functionality, do one big conversion (and hope that anyone writing more than 32KB to the console in one go has enough RAM), and loop over that? Then we're only dealing with UTF-16 pairs, which ought to be simpler.

The RAM constraint should be gone with Windows 8+ (gh-121940), just the "interactivity" / Ctrl-C interruptability (if can get to WaitForMultipleObjectsEx would solve that and is where I've been trying to aim recently, but also lots of steps to get there).

The RAM constraint should be gone with Windows 8+

I was referring to the memory we have to allocate to store a UTF-16 version of the string. If someone decides to write 2GB worth of ASCII in one go, we'll need another 4GB to store the UTF-16 encoded version, even before we get to passing it to the console APIs.

This PR consists of three parts:

• fixed _find_last_utf8_boundary();
• _wchar_to_utf8_count() to calculate the number of partially written bytes;
• numerous other fixes, mainly missed PyErr_NoMemory().

Even if the 32KiB limit is gone, there is still the 2GiB or 4GiB limit, so the code will stay almost the same. It cannot help. Unless we get rid of MultiByteToWideChar() and use double conversion with Python UTF-8 and UTF-16 codecs, but this would be a much larger change, and the code may be less efficient.

I think it's the binary search to find the converted length that upsets me the most.

Does it get any simpler if we arbitrarily cut at 32K bytes, count back to a complete UTF-8 character, and then write from there? Rather than trying to fit within 32KB worth of UTF-16, which is no longer a real limit on supported platforms.

No, it will not get much simpler.

Note that the code to find the initial length was already here. I only fixed some corner cases in it.

We still need binary search to map the number of actually written wchars to the number of UTF-8 bytes. Even if we do as you suggested, this will only save ~10 lines of code (relatively simple in comparison of the rest of this PR). We can do this in the following PR. For now, keeping this code means less changes.

Yeah, that's annoying... but you're right.

Okay, I have no further objections. Thanks for your work on this.

Thank you for your review @zooba.

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

GH-127325 is a backport of this pull request to the 3.13 branch.

GH-127326 is a backport of this pull request to the 3.12 branch.

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 Refleaks 3.12 has failed when building commit c3bb32d.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1125/builds/735) and take a look at the build logs.
4. Check if the failure is related to this commit (c3bb32d) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1125/builds/735

Failed tests:

• test_complex

Failed subtests:

• test_truediv - test.test_complex.ComplexTest.test_truediv

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.12.cstratak-RHEL8-x86_64.refleak/build/Lib/test/test_complex.py", line 107, in test_truediv
    self.check_div(complex(random(), random()),
  File "/home/buildbot/buildarea/3.12.cstratak-RHEL8-x86_64.refleak/build/Lib/test/test_complex.py", line 84, in check_div
    self.assertClose(q, y)
  File "/home/buildbot/buildarea/3.12.cstratak-RHEL8-x86_64.refleak/build/Lib/test/test_complex.py", line 77, in assertClose
    self.assertCloseAbs(x.imag, y.imag, eps)
  File "/home/buildbot/buildarea/3.12.cstratak-RHEL8-x86_64.refleak/build/Lib/test/test_complex.py", line 72, in assertCloseAbs
    self.assertTrue(abs((x-y)/y) < eps)
AssertionError: False is not true

test_truediv - test.test_complex.ComplexTest.test_truediv

It's unrelated and I cannot reproduce the issue on the buildbot (nor on my laptop). I don't know what's going on.