[3.13] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133944

serhiy-storchaka · 2025-05-12T18:02:27Z

If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal().

_PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal().
(cherry picked from commit 9f69a58)

Issue: Use-after-free in unicode_escape decoder with error handler #133767

…der with an error handler (pythonGH-129648) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

serhiy-storchaka · 2025-05-13T13:37:06Z

Removing _PyUnicode_DecodeUnicodeEscapeInternal and _PyBytes_DecodeEscape breaks the C ABI. We have the following options:

Remove them anyway. They are not for use in the user code, and are guarded against this. You have to try hard to use it in your code.
Add stub functions which:
- Always raise an exception.
- Work as before, but may ignore invalid escape sequences (set first_escape_char to NULL).
- Work as before, but may raise an exception if found an invalid escape sequence.
- Work as before, but may set first_escape_char to some other point in input if found an invalid escape sequences.

May means that it depends on the error handler and if an encoding error happens before an invalid escape sequence.

serhiy-storchaka · 2025-05-14T09:39:44Z

@Yhg1s, what do you prefer?

encukou · 2025-05-19T10:35:35Z

I'm not @Yhg1s, but, I'd vote for making them raise.

serhiy-storchaka · 2025-05-19T11:58:16Z

The next beta is close, so I implemented a simple, but with minimal impact, solution. The users of _PyUnicode_DecodeUnicodeEscapeInternal() may lose some warnings if they use non-strict error handler. Nothing changed for users of _PyBytes_DecodeEscape().

Anything simpler can break the user code if they actually use these functions. Anything more complex increases probability of adding bugs in non-tested code.

encukou

This looks good, thank you!

miss-islington-app · 2025-05-20T12:47:01Z

Thanks @serhiy-storchaka for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9, 3.10, 3.11, 3.12.
🐍🍒⛏🤖

miss-islington-app · 2025-05-20T12:47:05Z

Sorry, @serhiy-storchaka and @encukou, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6279eb8c076d89d3739a6edb393e43c7929b429d 3.12

miss-islington-app · 2025-05-20T12:47:09Z

Sorry, @serhiy-storchaka and @encukou, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6279eb8c076d89d3739a6edb393e43c7929b429d 3.11

miss-islington-app · 2025-05-20T12:47:12Z

Sorry, @serhiy-storchaka and @encukou, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6279eb8c076d89d3739a6edb393e43c7929b429d 3.10

miss-islington-app · 2025-05-20T12:47:16Z

Sorry, @serhiy-storchaka and @encukou, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6279eb8c076d89d3739a6edb393e43c7929b429d 3.9

…der with an error handler (pythonGH-129648) (pythonGH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-app · 2025-05-20T14:42:44Z

GH-134337 is a backport of this pull request to the 3.12 branch.

…der with an error handler (pythonGH-129648) (pythonGH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-app · 2025-05-20T15:13:03Z

GH-134341 is a backport of this pull request to the 3.11 branch.

bedevere-app · 2025-05-20T15:24:14Z

GH-134345 is a backport of this pull request to the 3.10 branch.

…der with an error handler (pythonGH-129648) (pythonGH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) (cherry picked from commit 0c33e5b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…er with an error handler (pythonGH-129648) (pythonGH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) (cherry picked from commit 0c33e5b) (cherry picked from commit 8b528ca) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-app · 2025-05-20T15:27:58Z

GH-134346 is a backport of this pull request to the 3.9 branch.

…th an error handler (GH-129648) (GH-133944) (#134337) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8)

## Summary Fix use-after-free vulnerability in the unicode-escape decoder with non-strict error handlers. ## Details - **CVE**: CVE-2025-4516 - **Severity**: Medium - **Issue**: Use-after-free crash when using `bytes.decode("unicode_escape", error="ignore|replace")` ## Changes - Add CVE-2025-4516.patch from upstream merged PRs - Python 3.12: [PR #134337](python/cpython#134337) - Python 3.13: [PR #133944](python/cpython#133944) - Increment epoch to 2 for both packages ## Status - ✅ Python 3.12: Upstream patch merged and applied - ✅ Python 3.13: Upstream patch merged and applied - ⏳ Python 3.9, 3.10, 3.11: Waiting for upstream PRs to be merged ## Testing CI will validate that: - Patches apply cleanly - Packages build successfully - Tests pass ## References - [CVE-2025-4516 Details](https://www.cve.org/CVERecord?id=CVE-2025-4516) - [Security Advisory](https://mail.python.org/archives/list/security-announce@python.org/thread/L75IPBBTSCYEF56I2M4KIW353BB3AY74/) - Related to: chainguard-dev/internal-dev#12589

…th an error handler (GH-129648) (GH-133944) (GH-134341) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…th an error handler (GH-129648) (GH-133944) (GH-134345) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) (cherry picked from commit 0c33e5b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…h an error handler (GH-129648) (GH-133944) (#134346) * [3.9] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) (GH-133944) If the error handler is used, a new bytes object is created to set as the object attribute of UnicodeDecodeError, and that bytes object then replaces the original data. A pointer to the decoded data will became invalid after destroying that temporary bytes object. So we need other way to return the first invalid escape from _PyUnicode_DecodeUnicodeEscapeInternal(). _PyBytes_DecodeEscape() does not have such issue, because it does not use the error handlers registry, but it should be changed for compatibility with _PyUnicode_DecodeUnicodeEscapeInternal(). (cherry picked from commit 9f69a58) (cherry picked from commit 6279eb8) (cherry picked from commit a75953b) (cherry picked from commit 0c33e5b) (cherry picked from commit 8b528ca) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

serhiy-storchaka requested review from pablogsal and lysnikolaou as code owners May 12, 2025 18:02

serhiy-storchaka added needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels May 12, 2025

bedevere-app bot mentioned this pull request May 12, 2025

Use-after-free in unicode_escape decoder with error handler #133767

Closed

bedevere-app bot added the type-security A security issue label May 12, 2025

bedevere-app bot mentioned this pull request May 12, 2025

gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler #129648

Merged

bedevere-app bot added the awaiting core review label May 12, 2025

serhiy-storchaka enabled auto-merge (squash) May 12, 2025 18:03

serhiy-storchaka requested a review from Yhg1s May 13, 2025 13:37

Add stub functions.

99204fd

serhiy-storchaka disabled auto-merge May 19, 2025 12:03

jamie-albert mentioned this pull request May 19, 2025

python-3.10/11/12/13/CVE-2025-4516 adv update wolfi-dev/advisories#18858

Merged

mcepl mentioned this pull request May 19, 2025

[3.12] Fix use-after-free in the unicode-escape decoder with error handler (GH-133767) #134255

Closed

encukou approved these changes May 20, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels May 20, 2025

encukou merged commit 6279eb8 into python:3.13 May 20, 2025
39 checks passed

bedevere-app bot removed the awaiting merge label May 20, 2025

miss-islington-app bot assigned encukou May 20, 2025

bedevere-app bot removed the needs backport to 3.12 only security fixes label May 20, 2025

bedevere-app bot removed the needs backport to 3.11 only security fixes label May 20, 2025

bedevere-app bot removed the needs backport to 3.10 only security fixes label May 20, 2025

bedevere-app bot removed the needs backport to 3.9 only security fixes label May 20, 2025

dakaneye mentioned this pull request May 28, 2025

python-3.12, python-3.13: Apply CVE-2025-4516 patch wolfi-dev/os#54620

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[3.13] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133944

[3.13] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133944

serhiy-storchaka commented May 12, 2025 •

edited by bedevere-app bot

Loading

Uh oh!

serhiy-storchaka commented May 13, 2025 •

edited

Loading

Uh oh!

serhiy-storchaka commented May 14, 2025

Uh oh!

encukou commented May 19, 2025

Uh oh!

serhiy-storchaka commented May 19, 2025

Uh oh!

encukou left a comment

Uh oh!

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

bedevere-app bot commented May 20, 2025

Uh oh!

bedevere-app bot commented May 20, 2025

Uh oh!

bedevere-app bot commented May 20, 2025

Uh oh!

bedevere-app bot commented May 20, 2025

Uh oh!

Uh oh!

Uh oh!

[3.13] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133944

[3.13] gh-133767: Fix use-after-free in the unicode-escape decoder with an error handler (GH-129648) #133944

Conversation

serhiy-storchaka commented May 12, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka commented May 14, 2025

Uh oh!

encukou commented May 19, 2025

Uh oh!

serhiy-storchaka commented May 19, 2025

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

miss-islington-app bot commented May 20, 2025

Uh oh!

bedevere-app bot commented May 20, 2025

Uh oh!

bedevere-app bot commented May 20, 2025

Uh oh!

bedevere-app bot commented May 20, 2025

Uh oh!

bedevere-app bot commented May 20, 2025

Uh oh!

Uh oh!

serhiy-storchaka commented May 12, 2025 •

edited by bedevere-app bot

Loading

serhiy-storchaka commented May 13, 2025 •

edited

Loading