Skip to content

gh-118718: Incorrect decoding of preamble in email parser #134384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

gh-118718: Incorrect decoding of preamble in email parser #134384

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3bc8212
email: Correctly decode using policy UTF-8 attribute in BytesParser
GGyll May 20, 2025
29f3802
Added test for message_from_binary_file using utf-8
GGyll May 20, 2025
64d2bd3
Update test_bytes_parser_uses_policy_utf8_setting test
GGyll May 20, 2025
4bbea35
📜🤖 Added by blurb_it.
blurb-it[bot] May 20, 2025
bfcef66
removed extra whitespace in test_email.py
GGyll May 21, 2025
b9d0d15
Merge branch 'main' into bugfix/Incorrect-decoding-of-preamble-in-ema…
GGyll May 27, 2025
4ea72b6
Merge branch 'main' into bugfix/Incorrect-decoding-of-preamble-in-ema…
GGyll May 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Lib/email/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,8 @@ def parse(self, fp, headersonly=False):
parsing after reading the headers or not. The default is False,
meaning it parses the entire contents of the file.
"""
fp = TextIOWrapper(fp, encoding='ascii', errors='surrogateescape')
encoding = "utf-8" if getattr(self.parser.policy, "utf8", False) else "ascii"
fp = TextIOWrapper(fp, encoding=encoding, errors='surrogateescape')
try:
return self.parser.parse(fp, headersonly)
finally:
Expand Down
38 changes: 38 additions & 0 deletions Lib/test/test_email/test_email.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3990,6 +3990,44 @@ def test_bytes_parser_on_exception_does_not_close_file(self):
fp)
self.assertFalse(fp.closed)

def test_bytes_parser_uses_policy_utf8_setting(self):
m = """
From: Nathaniel Nameson <nathan@nameson.com>
To: Ned Sampleson <ned@sampleson.com>
Subject: Sample message
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="i-am-boundary"

This is the préamble. It is to be ignored, though it
is a handy place for mail composers to include an
explanatory note to non-MIME compliant readers.

--i-am-boundary
Content-type: text/plain; charset=us-ascii

This is explicitly typed plain ASCII text.
It DOES end with a linebreak.

--i-am-boundary
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

This should be correctly encapsulated: Un petit café ?

--i-am-boundary--
This is the epilogue. It is also to be ignored.

""".lstrip()
M_BYTES = BytesIO(m.encode())

msg = email.message_from_binary_file(M_BYTES, policy=email.policy.default.clone(utf8=True))
for i, part in enumerate(msg.iter_parts(), 1):
_ = part.as_string()

msg_string = msg.as_string()
self.assertIn("This is the préamble.", msg_string)
self.assertIn("Un petit café", msg_string)

def test_parser_does_not_close_file(self):
with openfile('msg_02.txt', encoding="utf-8") as fp:
email.parser.Parser().parse(fp)
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2025-05-20-19-37-09.gh-issue-118718.dyhtAS.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Fix incorrect decoding of preamble in BytesParser
Contributed by Gustaf Gyllensporre.
Loading