Skip to content

bpo-43510: Implement PEP 597 opt-in EncodingWarning. #19481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 51 commits into from
Mar 29, 2021
Merged

bpo-43510: Implement PEP 597 opt-in EncodingWarning. #19481

merged 51 commits into from
Mar 29, 2021

Conversation

methane
Copy link
Member

@methane methane commented Apr 12, 2020 •
edited
Loading

See PEP 597.

  • Add -X warn_default_encoding and PYTHONWARNDEFAULTENCODING.
  • Add EncodingWarning
  • Add io.text_encoding()
  • open(), TextIOWrapper() emits EncodingWarning when encoding is omitted and warn_default_encoding is enabled.
  • _pyio.TextIOWrapper() uses UTF-8 as fallback default encoding used when failed to import locale module. (used during building Python)
  • bz2, configparser, gzip, lzma, pathlib, tempfile modules use io.text_encoding().
  • Add documents
  • Add what's new entry

bpo-43510

@methane methane force-pushed the open-encoding branch 4 times, most recently from c37295e to 6f72b38 Compare April 13, 2020 09:22
@methane
Raise a warning when encoding is omitted
a3c014b 
The warning is raised only in dev mode.

pathlib uses the new `io.text_encoding` helper function.
Other libraries will follow.
@methane
add test
050bd1b
@methane
wrap encoding=None with text_encoding.
939f4a0
@methane methane changed the title PEP 597: Raise a warning when encoding is omitted PEP 597: Emit EncodingWarning when encoding is omitted Jan 30, 2021
@methane methane changed the title PEP 597: Emit EncodingWarning when encoding is omitted bpo-43510: Add encoding="locale" and optional EncodingWarning Mar 16, 2021
methane and others added 8 commits March 23, 2021 14:48
@methane @CAM-Gerlach
Update Doc/library/io.rst
54c7dc6 
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
@methane @CAM-Gerlach
Update Doc/library/io.rst
5b2830b 
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
@methane @CAM-Gerlach
Apply suggestions from code review
14f2a6e 
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
@methane
Move EncodingWarnings
06e2a32
@methane
fix comment
27d49d2
@methane
fix text_encoding() docstring
80f4644
@methane
update what's new
6ad0e7f
@methane
fix doc build
73b27f1
CAM-Gerlach
CAM-Gerlach reviewed Mar 23, 2021
View reviewed changes
@CAM-Gerlach
Copy link
Member

Thanks @methane ! I've replied to your comments with updated suggestions to reflect your feedback.

@methane @CAM-Gerlach
Update Doc/library/io.rst
c149d65 
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
methane
methane commented Mar 24, 2021
View reviewed changes
@methane @CAM-Gerlach
Apply suggestions from code review
4eb7655 
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
methane
methane commented Mar 24, 2021
View reviewed changes
@methane @CAM-Gerlach
Apply suggestions from code review
e3bce76 
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
CAM-Gerlach
CAM-Gerlach reviewed Mar 24, 2021
View reviewed changes
@methane @CAM-Gerlach
Update Doc/library/io.rst
c089fd7 
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
@methane
Copy link
Member Author

methane commented Mar 25, 2021

I will merge this PR next week if there are no objections.

@methane methane merged commit 4827483 into python:master Mar 29, 2021
@methane methane deleted the open-encoding branch March 29, 2021 03:28
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 LTO + PGO 3.x has failed when building commit 4827483.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/96/builds/910) and take a look at the build logs.
  4. Check if the failure is related to this commit (4827483) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/96/builds/910

Failed tests:

  • test_tools
  • test_ftplib

Failed subtests:

  • test_storlines - test.test_ftplib.TestFTPClass
  • test_reindent_file_with_bad_encoding - test.test_tools.test_reindent.ReindentTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

  • test_concurrent_futures: 2 min 38 sec
  • test_shelve: 2 min 30 sec
  • test_mailbox: 1 min 57 sec
  • test_dbm: 1 min 52 sec
  • test_tarfile: 1 min 36 sec
  • test_multiprocessing_spawn: 1 min 34 sec
  • test_tokenize: 1 min 33 sec
  • test_dbm_gnu: 1 min 28 sec
  • test_unparse: 1 min 28 sec
  • test_asyncio: 1 min 17 sec

1 test failed:
test_tools

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

2 re-run tests:
test_ftplib test_tools

Total duration: 6 min 19 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_ftplib.py", line 623, in test_storlines
    self.check_data(self.server.handler_instance.last_received_data, RETR_DATA)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_ftplib.py", line 494, in check_data
    self.assertEqual(len(received), len(expected))
AssertionError: 12019 != 12018


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_tools/test_reindent.py", line 29, in test_reindent_file_with_bad_encoding
    rc, out, err = assert_python_ok(self.script, '-r', bad_coding_path)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/python', '-X', 'faulthandler', '-I', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Tools/scripts/reindent.py', '-r', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/bad_coding.py']

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 LTO 3.x has failed when building commit 4827483.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/507/builds/896) and take a look at the build logs.
  4. Check if the failure is related to this commit (4827483) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/507/builds/896

Failed tests:

  • test_tools

Failed subtests:

  • test_reindent_file_with_bad_encoding - test.test_tools.test_reindent.ReindentTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

  • test_mailbox: 4 min 33 sec
  • test_peg_generator: 3 min 37 sec
  • test_concurrent_futures: 2 min 56 sec
  • test_shelve: 2 min 27 sec
  • test_multiprocessing_spawn: 1 min 50 sec
  • test_tokenize: 1 min 49 sec
  • test_unparse: 1 min 37 sec
  • test_lib2to3: 1 min 25 sec
  • test_asyncio: 1 min 21 sec
  • test_multiprocessing_forkserver: 1 min 9 sec

1 test failed:
test_tools

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_tools

Total duration: 7 min 15 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/test_tools/test_reindent.py", line 29, in test_reindent_file_with_bad_encoding
    rc, out, err = assert_python_ok(self.script, '-r', bad_coding_path)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/python', '-X', 'faulthandler', '-I', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Tools/scripts/reindent.py', '-r', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/bad_coding.py']

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x has failed when building commit 4827483.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/146/builds/1014) and take a look at the build logs.
  4. Check if the failure is related to this commit (4827483) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/146/builds/1014

Failed tests:

  • test_locale

Failed subtests:

  • test_getsetlocale_issue1813 - test.test_locale.TestMiscellaneous

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

395 tests OK.

10 slowest tests:

  • test_mmap: 9 min 37 sec
  • test_largefile: 5 min 57 sec
  • test_io: 5 min 39 sec
  • test_unparse: 3 min 12 sec
  • test_multiprocessing_spawn: 2 min 16 sec
  • test_concurrent_futures: 2 min 3 sec
  • test_tokenize: 1 min 46 sec
  • test_capi: 1 min 46 sec
  • test_peg_generator: 1 min 39 sec
  • test_asyncio: 1 min 31 sec

1 test failed:
test_locale

31 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxlimited test_xxtestfuzz test_zipfile64

1 re-run test:
test_locale

Total duration: 19 min 35 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_locale.py", line 567, in test_getsetlocale_issue1813
    locale.setlocale(locale.LC_CTYPE, loc)
  File "D:\buildarea\3.x.bolen-windows10\build\lib\locale.py", line 610, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

@methane
Copy link
Member Author

methane commented Mar 29, 2021 •
edited by bedevere-bot
Loading

test_locale fail is unrelating to this PR. See bpo-37945.

@CAM-Gerlach
Copy link
Member

Thanks @methane ! Excited to see this in Python 3.10.

@alexdelorenzo alexdelorenzo mentioned this pull request Apr 16, 2021
Open
chris-allan added a commit to chris-allan/omero-web that referenced this pull request Oct 10, 2022
@chris-allan
Update to a suitable version of the concurrent-log-handler
5dbc09c 
We have a vendored version of ConcurrentLogHandler 0.9.1 [1] from
omero-py that we use.  This version is from 2013, has a few bugs, and
doesn't work properly with the encoding changes [2,3] made in Python
3.10.

A new version [4] whose lineage is that of the now unmaintained
ConcurrentLogHandler is now available which resolves several of the
aforementioned bugs, supports Python 3.10, and is drop in replacement
API compatible with the previous version.  This commit adds a dependency
for this new version and swaps the default logger class for it in
configuration.

  1. https://pypi.org/project/ConcurrentLogHandler/
  2. https://peps.python.org/pep-0597/
  3. python/cpython#19481
  4. https://pypi.org/project/concurrent-log-handler/
@chris-allan chris-allan mentioned this pull request Oct 10, 2022
Merged
@AlexWaygood AlexWaygood mentioned this pull request Oct 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vstinner vstinner vstinner left review comments

@CAM-Gerlach CAM-Gerlach CAM-Gerlach left review comments

@gpshead gpshead Awaiting requested review from gpshead

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@methane @CAM-Gerlach @bedevere-bot @vstinner @the-knights-who-say-ni