Skip to content

Recording and restoring the LC_NUMERIC locale setting with LC_ALL=C.UTF-8 no longer works with Python 3.13.3 #133967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bouweandela opened this issue May 13, 2025 · 13 comments
Labels
3.13 bugs and security fixes extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error

Comments

@bouweandela
Copy link

bouweandela commented May 13, 2025

Bug report

Bug description:

We have project where we record the current value of locale.LC_NUMERIC, change it to a different value to use a library that needs a particular locale setting, and then restore it to the original value. This stopped working with Python 3.13.3. I suspect this may be a bug, but please correct me if I misunderstood the functionality.

Example with Python 3.13.3:

$ docker run -it python:3.13.3 
Python 3.13.3 (main, May  9 2025, 23:49:05) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, locale
>>> os.environ
environ({'PATH': '/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOSTNAME': '2e43b79a7135', 'TERM': 'xterm', 'GPG_KEY': '7169605F62C751356D054A26A821E680E5FA6305', 'PYTHON_VERSION': '3.13.3', 'PYTHON_SHA256': '40f868bcbdeb8149a3149580bb9bfd407b3321cd48f0be631af955ac92c0e041', 'HOME': '/root', 'LC_CTYPE': 'C.UTF-8'})
>>> os.environ["LC_ALL"] = "C.UTF-8"
>>> locale.setlocale(locale.LC_ALL, "")
'C.UTF-8'
>>> locale.getlocale(locale.LC_NUMERIC)
('en_US', 'UTF-8')
>>> locale.setlocale(locale.LC_NUMERIC, locale.getlocale(locale.LC_NUMERIC))
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    locale.setlocale(locale.LC_NUMERIC, locale.getlocale(locale.LC_NUMERIC))
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/locale.py", line 615, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

The same example works fine with earlier versions of Python, for example 3.13.2:

$ docker run -it python:3.13.2 
Python 3.13.2 (main, Apr  8 2025, 04:27:11) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, locale
>>> os.environ
environ({'PATH': '/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOSTNAME': '64800af765e4', 'TERM': 'xterm', 'GPG_KEY': '7169605F62C751356D054A26A821E680E5FA6305', 'PYTHON_VERSION': '3.13.2', 'PYTHON_SHA256': 'd984bcc57cd67caab26f7def42e523b1c015bbc5dc07836cf4f0b63fa159eb56', 'HOME': '/root', 'LC_CTYPE': 'C.UTF-8'})
>>> os.environ["LC_ALL"] = "C.UTF-8"
>>> locale.setlocale(locale.LC_ALL, "")
'C.UTF-8'
>>> locale.getlocale(locale.LC_NUMERIC)
('C', 'UTF-8')
>>> locale.setlocale(locale.LC_NUMERIC, locale.getlocale(locale.LC_NUMERIC))
'C.UTF-8'

It looks like this may have been caused by the changes in #129647.

CPython versions tested on:

3.13

Operating systems tested on:

No response

Linked PRs

@bouweandela bouweandela added the type-bug An unexpected behavior, bug, or error label May 13, 2025
@picnixz picnixz added extension-modules C modules in the Modules dir 3.13 bugs and security fixes labels May 15, 2025
@mvkashyap
Copy link

I'm at the CPython sprints right now, so I can take a look at this.

@mvkashyap
Copy link

While I'm unable to reproduce your exact issue, I'm debugging a slightly similar issue that I'm having on my machine. Instead of the error, I'm getting this:

Type "help", "copyright", "credits" or "license" for more information.
>>> import os, locale
>>> os.environ
environ({'PATH': '/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOSTNAME': '64800af765e4', 'TERM': 'xterm', 'GPG_KEY': '7169605F62C751356D054A26A821E680E5FA6305', 'PYTHON_VERSION': '3.13.2', 'PYTHON_SHA256': 'd984bcc57cd67caab26f7def42e523b1c015bbc5dc07836cf4f0b63fa159eb56', 'HOME': '/root', 'LC_CTYPE': 'C.UTF-8'})
>>> os.environ["LC_ALL"] = "C.UTF-8"
>>> locale.setlocale(locale.LC_ALL, "")
'C.UTF-8'
>>> locale.getlocale(locale.LC_NUMERIC)
('en_US, UTF-8)
>>> locale.setlocale(locale.LC_NUMERIC, ('C', 'UTF-8'))
'en_US.UTF-8'

@bouweandela
Copy link
Author

Thank you for taking the time to investigate! Are you using the Python containers from Dockerhub? In my tests with those containers, the problem only appears with Python 3.13.3, while for you, it seems to partially appear in 3.13.2, but without the crash when trying to set locale.LC_NUMERIC. Could you try if you can reproduce it with Python 3.13.3?

@mvkashyap
Copy link

Ah, this is for 3.13.3!

@valeriupredoi
Copy link

many thanks @mvkashyap 🍺

@serhiy-storchaka
Copy link
Member

This was caused by #129646.

The locale.aliases file from X.org distribution (https://cgit.freedesktop.org/xorg/lib/libX11/tree/nls/locale.alias.pre) maps locale C.utf8 to en_US.UTF-8. The problem is that en_US.UTF-8 may not exist on all platforms, especially on the old ones. This is a congenital defect in the locale module design -- that it uses a static platform-independent mapping.

cc @malemburg

@malemburg
Copy link
Member

I think the problem is different: the en_US locale files may not have been generated on the base system, whereas the C locale should always be present.

In any case, we should probably special case the C local mappings in the script Tools/i18n/makelocalealias.py to not have them alias to en_US (even though that is what they normally refer to).

@malemburg
Copy link
Member

BTW: I do wonder why the locale files don't exist in the official Python docker image.

Note that locale.getlocale()returns ('en_US', 'UTF-8') in the 3.13.3 image, but the /usr/share/locale is missing the en_US subdir. In the 3.12.3 image, it returns ('C', 'UTF-8'), which is present. This image is also missing the en_US subdir.

This suggests that the official images are somewhat broken... where can this be reported ?

@malemburg
Copy link
Member

Checking using locale -a, only the locales C, C.utf8 and POSIX are available in those images, so I guess they are based on a minimal installation of Debian.

@serhiy-storchaka
Copy link
Member

In any case, we should probably special case the C local mappings in the script Tools/i18n/makelocalealias.py to not have them alias to en_US (even though that is what they normally refer to).

This is what I came to. It is not clear what to do with the C.ISO8859-1 -> en_US.ISO8859-1 mapping which has existed for a long time, but at least it is not so common as C.UTF-8.

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 10, 2025
…-8' (pythonGH-135347)

(cherry picked from commit 0f866cb)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 10, 2025
…-8' (pythonGH-135347)

(cherry picked from commit 0f866cb)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@malemburg
Copy link
Member

It is not clear what to do with the C.ISO8859-1 -> en_US.ISO8859-1 mapping

Let's leave that as it is, since it hasn't caused any reported issues, AFAIK.

@Yhg1s
Copy link
Member

Yhg1s commented Jun 10, 2025

en_US.ISO8859-1 is much more likely to exist. It still makes me very uncomfortable (as mentioned on Discord) because we're changing the encoding in a patch release, but it's not entirely clear to me when and how this could affect people.

@serhiy-storchaka
Copy link
Member

For some people changing the encoding was a bug fix, for others a regression. This is a problem with such approach -- there may be different default encodings for the same locale on different platforms, but the locale module will use the same mapping. There are even conflicts between glibc and X.org mappings, so this problem is not Python specific.

serhiy-storchaka added a commit that referenced this issue Jun 11, 2025
…F-8' (GH-135347) (GH-135349)

(cherry picked from commit 0f866cb)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this issue Jun 11, 2025
…F-8' (GH-135347) (GH-135350)

(cherry picked from commit 0f866cb)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@github-project-automation github-project-automation bot moved this from Todo to Done in Locale issues 🗺 Jun 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.13 bugs and security fixes extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error
Projects
Development

No branches or pull requests

7 participants