GH-133711: Enable UTF-8 mode by default (PEP 686) #133712

AA-Turner · 2025-05-08T21:21:32Z

Issue: Enable UTF-8 mode by default (implement PEP 686) #133711

📚 Documentation preview 📚: https://cpython-previews--133712.org.readthedocs.build/

Doc/library/os.rst

Doc/library/sys.rst

zooba · 2025-05-09T10:19:05Z

Doc/whatsnew/3.15.rst

+* Python UTF-8 mode is now enabled by default.
+  It may be disabled with by setting :envvar:`PYTHONUTF8=0 <PYTHONUTF8>` as
+  an environment variable or by using the :option:`-X utf8=0 <-X>` flag.
+  See :pep:`686` for further details.


I feel like we can probably put some more explanation in here, such as that it affects TextIOWrapper and hence open(). The current description doesn't sound as scary as it needs to, in my opinion.

Along the lines of: "Python UTF-8 mode is now enabled by default. This means that (files/console/etc.) will now use UTF-8 regardless of system settings, unless specifically overridden in code (typically with an encoding= argument)."

To be clear, it's nothing new. But we shouldn't assume that everyone already knows what UTF-8 mode implies. There are many more people out there who haven't ever thought about it than those who are waiting for it to be the default.

Another effect of the UTF-8 Mode is that Python ignores the locale encoding.

I've expanded the What's New entry, please see https://cpython-previews--133712.org.readthedocs.build/en/133712/whatsnew/3.15.html#other-language-changes

Doc/using/windows.rst

vstinner · 2025-05-10T07:25:22Z

Doc/whatsnew/3.15.rst

@@ -75,7 +75,30 @@ New features
 Other language changes
 ======================

-
+* Python now uses UTF-8_ as the default encoding, independent of the system's


You might mention the UTF-8 Mode earlier since it has other side effects documented in the UTF-8 Mode section, such as changing sys.stdout error handler and ignoring the locale encoding.

Lib/test/test_cmd_line.py

# Conflicts: # Doc/whatsnew/3.15.rst

Lib/test/test_cmd_line.py

# Conflicts: # Lib/test/test_cmd_line.py

vstinner · 2025-05-28T14:50:55Z

test_regrtest failed:

I wrote #134839 to fix test_regrtest when PYTHONUTF8=1 env var is set.

vstinner · 2025-05-28T15:25:01Z

I can also reproduce the [test_readline] issue in the main branch using the fr_FR locale and the command:
PYTHONUTF8=1 LANG=fr_FR ./python -m test -v test_readline -u all

I wrote #134841 to fix test_readline for the Python UTF-8 Mode.

vstinner · 2025-05-28T16:31:58Z

buildbot/wasm32-wasi Non-Debug PR — Build done.

Configure host Python fails with:

checking for /dev/ptmx... not set
configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

It seems to be unrelated to this PR.

AA-Turner · 2025-05-29T16:52:07Z

buildbot/wasm32-wasi Non-Debug PR — Build done.

Configure host Python fails with:
checking for /dev/ptmx... not set
configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling
It seems to be unrelated to this PR.

cc @brettcannon, this builder seems to be consistently failing since ~15/05.

vstinner

LGTM.

vstinner · 2025-05-30T12:07:02Z

Doc/library/os.rst

+
+   Python UTF-8 mode is now enabled by default (:pep:`686`).
+   It may be disabled with by setting :envvar:`PYTHONUTF8=0 <PYTHONUTF8>` as
+   an environment variable or by using the :option:`-X utf8=0 <-X>` flag.


Suggested change

an environment variable or by using the :option:`-X utf8=0 <-X>` flag.

an environment variable or by using the :option:`-X utf8=0 <-X>` command line option.

vstinner · 2025-05-30T12:36:04Z

Include/cpython/initconfig.h

+       Enabled by default (equal to 1; PEP 686), or if Py_UTF8Mode=1,
+       or if "-X utf8=1" or PYTHONUTF8=1.


Suggested change

Enabled by default (equal to 1; PEP 686), or if Py_UTF8Mode=1,

or if "-X utf8=1" or PYTHONUTF8=1.

Enabled by default (equal to 1; PEP 686), or if Py_UTF8Mode=1,

or if "-X utf8=1" or PYTHONUTF8=1.

Enable UTF-8 mode by default (PEP 686)

fc19177

AA-Turner requested a review from methane May 8, 2025 21:21

AA-Turner requested review from gpshead, FFY00 and ericsnowcurrently as code owners May 8, 2025 21:21

bedevere-app bot mentioned this pull request May 8, 2025

Enable UTF-8 mode by default (implement PEP 686) #133711

Open

bedevere-app bot added the awaiting core review label May 8, 2025

AA-Turner requested a review from vstinner May 8, 2025 21:21

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

vstinner reviewed May 9, 2025

View reviewed changes

Doc/library/os.rst Outdated Show resolved Hide resolved

Doc/library/os.rst Show resolved Hide resolved

Doc/library/sys.rst Outdated Show resolved Hide resolved

zooba reviewed May 9, 2025

View reviewed changes

AA-Turner added 3 commits May 10, 2025 01:47

Victor's documentation comments

c706cfa

Fix test_cmd_line

f095ab0

Improve What's New entry

ff066b1

AA-Turner force-pushed the utf-8-mode branch from 036555d to ff066b1 Compare May 10, 2025 02:09

AA-Turner and others added 2 commits May 10, 2025 03:20

Add references to UTF-8 and Unicode

abfb864

Merge branch 'main' into utf-8-mode

f40b986

FFY00 removed their request for review May 10, 2025 02:28

vstinner reviewed May 10, 2025

View reviewed changes

Doc/using/windows.rst Show resolved Hide resolved

vstinner reviewed May 10, 2025

View reviewed changes

methane reviewed May 10, 2025

View reviewed changes

Lib/test/test_cmd_line.py Outdated Show resolved Hide resolved

AA-Turner added 2 commits May 10, 2025 23:50

Check stream buffer type in test_python_legacy_windows_stdio

3226e1d

Merge branch 'main' into utf-8-mode

b83c966

# Conflicts: # Doc/whatsnew/3.15.rst

This comment was marked as resolved.

Sign in to view

methane reviewed May 10, 2025

View reviewed changes

Lib/test/test_cmd_line.py Outdated Show resolved Hide resolved

methane reviewed May 11, 2025

View reviewed changes

Lib/test/test_cmd_line.py Outdated Show resolved Hide resolved

StanFromIreland mentioned this pull request May 11, 2025

gh-133677: Fix tests when running in non-UTF-8 locale #133865

Merged

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

AA-Turner added 2 commits May 26, 2025 10:32

Merge branch 'main' into utf-8-mode

b56e01b

# Conflicts: # Lib/test/test_cmd_line.py

line length

3da6bf2

AA-Turner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 26, 2025

This comment was marked as outdated.

Sign in to view

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 26, 2025

This comment was marked as resolved.

Sign in to view

Merge branch 'main' into utf-8-mode

9b3ebf7

AA-Turner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2025

This comment was marked as duplicate.

Sign in to view

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2025

vstinner approved these changes May 30, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels May 30, 2025

mostafaammer added this to lavitaconnect@MOSTAFAAMMER May 31, 2025

	an environment variable or by using the :option:`-X utf8=0 <-X>` flag.
	an environment variable or by using the :option:`-X utf8=0 <-X>` command line option.

		Enabled by default (equal to 1; PEP 686), or if Py_UTF8Mode=1,
		or if "-X utf8=1" or PYTHONUTF8=1.

Uh oh!

GH-133711: Enable UTF-8 mode by default (PEP 686) #133712

Are you sure you want to change the base?

GH-133711: Enable UTF-8 mode by default (PEP 686) #133712

Uh oh!

Conversation

AA-Turner commented May 8, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

zooba May 9, 2025

Choose a reason for hiding this comment

Uh oh!

zooba May 9, 2025

Choose a reason for hiding this comment

Uh oh!

vstinner May 9, 2025

Choose a reason for hiding this comment

Uh oh!

AA-Turner May 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner May 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

This comment was marked as resolved.

vstinner commented May 28, 2025

Uh oh!

vstinner commented May 28, 2025

Uh oh!

This comment was marked as duplicate.

vstinner commented May 28, 2025

Uh oh!

AA-Turner commented May 29, 2025

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner May 30, 2025

Choose a reason for hiding this comment

Uh oh!

vstinner May 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AA-Turner commented May 8, 2025 •

edited by github-actions bot

Loading