Skip to content

gh-64243: Implement locale.getlocale fall back in gettext.find #131477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

gh-64243: Implement locale.getlocale fall back in gettext.find #131477

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fcce7bd
Initial
StanFromIreland Mar 19, 2025
03ca427
Correct NEWS
StanFromIreland Mar 19, 2025
f9851e1
Tomas's feedback
StanFromIreland Mar 31, 2025
4f37342
Tomas's suggestion + typo spotted while testing
StanFromIreland Apr 5, 2025
c85d0c2
Serhiy's review
StanFromIreland Apr 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions Doc/library/gettext.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,9 +130,10 @@ install themselves in the built-in namespace as the function :func:`!_`.
strings, where each string is a language code.

If *localedir* is not given, then the default system locale directory is used.
[#]_ If *languages* is not given, then the following environment variables are
searched: :envvar:`LANGUAGE`, :envvar:`LC_ALL`, :envvar:`LC_MESSAGES`, and
:envvar:`LANG`. The first one returning a non-empty value is used for the
[#]_ If *languages* is not given, then the environment variable :envvar:`LANGUAGE`
is searched, it falls back to :func:`locale.setlocale`, which in turn falls
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mentioning locale.setlocale() here looks weird, as we do not change the current locale. It is just an implementation detail. Maybe just say "the current locale"?

"... it falls back to the current locale or to the environment variables ..."?

back to the environment variables :envvar:`LC_ALL`, :envvar:`LC_MESSAGES`, and
:envvar:`LANG` where the first one returning a non-empty value is used for the
*languages* variable. The environment variables should contain a colon separated
list of languages, which will be split on the colon to produce the expected list
of language code strings.
Expand All @@ -147,6 +148,9 @@ install themselves in the built-in namespace as the function :func:`!_`.
of all file names, in the order in which they appear in the languages list or
the environment variables.

.. versionchanged:: next
:func:`locale.setlocale` is used to generate *languages* if *languages* is
not provided.

.. function:: translation(domain, localedir=None, languages=None, class_=None, fallback=False)

Expand Down
17 changes: 11 additions & 6 deletions Lib/gettext.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
import operator
import os
import sys
import locale


__all__ = ['NullTranslations', 'GNUTranslations', 'Catalog',
Expand Down Expand Up @@ -229,7 +230,6 @@ def func(n):


def _expand_lang(loc):
import locale
loc = locale.normalize(loc)
COMPONENT_CODESET = 1 << 0
COMPONENT_TERRITORY = 1 << 1
Expand Down Expand Up @@ -491,11 +491,16 @@ def find(domain, localedir=None, languages=None, all=False):
localedir = _default_localedir
if languages is None:
languages = []
for envar in ('LANGUAGE', 'LC_ALL', 'LC_MESSAGES', 'LANG'):
val = os.environ.get(envar)
if val:
languages = val.split(':')
break
if val := os.environ.get('LANGUAGE'):
languages = val.split(':')
elif (loc := locale.setlocale(locale.LC_MESSAGES)) != (None, None):
languages = [".".join(filter(None, loc))]
else:
for envar in ('LC_ALL', 'LC_MESSAGES', 'LANG'):
val = os.environ.get(envar)
if val:
languages = val.split(':')
break
if 'C' not in languages:
languages.append('C')
# now normalize and expand the languages
Expand Down
30 changes: 23 additions & 7 deletions Lib/test/test_gettext.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -736,17 +736,33 @@ def create_mo_file(self, lang):
f.write(GNU_MO_DATA)
return mo_file

def test_find_with_env_vars(self):
# test that find correctly finds the environment variables
# when languages are not supplied
mo_file = self.create_mo_file("ga_IE")
def _for_all_vars(self, mo_file, locale):
for var in ('LANGUAGE', 'LC_ALL', 'LC_MESSAGES', 'LANG'):
self.env.set(var, 'ga_IE')
self.env.set(var, locale)
result = gettext.find("mofile",
localedir=os.path.join(self.tempdir, "locale"))
self.assertEqual(result, mo_file)
self.assertEqual(mo_file, result)
self.env.unset(var)

@unittest.mock.patch("locale.setlocale", return_value=(None, None))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that LANGUAGE takes priority over the current locale. This is not tested.

def test_find_with_env_vars(self, patch_getlocale):
# test that find correctly finds the environment variables
# when languages are not supplied
mo_file = self.create_mo_file("ga_IE")
self._for_all_vars(mo_file, "ga_IE")
self._for_all_vars(mo_file, "ga_IE.UTF-8")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test that the encoding is not ignored, and "ga_IE.UTF-8.mo" is found.

self._for_all_vars(mo_file, "es_ES:ga_IE:fr_FR")
self._for_all_vars(mo_file, "ga_IE@euro")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test also with a non-euro modifier (@euro is handled specially), e.g. "ca_ES@valencia" or "ks_IN@devanagari". Test that the modifier is not ignored. For @euro, test also with encoding.

Test also "C" and "C.UTF-8".


def test_process_vars_override(self):
mo_file = self.create_mo_file("ga_IE")
with unittest.mock.patch("locale.setlocale", return_value=('ga_IE', 'UTF-8')):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

locale.setlocale() returns a string, not a tuple. This is why you need also tests with a real, not mocked locale.setlocale().

Test also with modifiers, "C" and "C.UTF-8".

Test that locale overrides 'LC_ALL', 'LC_MESSAGES', 'LANG', but 'LANGUAGE' overrides locale.

result = gettext.find("mofile", localedir=os.path.join(self.tempdir, "locale"))
self.assertEqual(mo_file, result)
with unittest.mock.patch("locale.setlocale", return_value=('ga_IE', None)):
result = gettext.find("mofile", localedir=os.path.join(self.tempdir, "locale"))
self.assertEqual(mo_file, result)

def test_find_with_languages(self):
# test that passed languages are used
self.env.set('LANGUAGE', 'pt_BR')
Expand Down Expand Up @@ -798,7 +814,7 @@ def test__all__(self):
unittest.main()


# For reference, here's the .po file used to created the GNU_MO_DATA above.
# For reference, here's the .po file used to create the GNU_MO_DATA above.
#
# The original version was automatically generated from the sources with
# pygettext. Later it was manually modified to add plural forms support.
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2025-03-19-17-30-00.gh-issue-64243.fuheq3.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Implement a fall back to :func:`locale.getlocale` in :func:`gettext.find` if
*languages* is not provided and :envvar:`LANGUAGE` is not set.
Loading