Skip to content

gh-135410: Use a critical section around StringIO.__next__ #135412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 12, 2025

Conversation

ZeroIntensity
Copy link
Member

@ZeroIntensity ZeroIntensity commented Jun 11, 2025

I'm generally not a fan of adding locks around __next__, because concurrent iteration isn't generally a real use case, but readlines calls into __next__.

@corona10
Copy link
Member

Let's consider the following options.

  1. Prevent segfault only.
  2. Guarantee thread safety also. But IIRC, we do not guarantee all stdlib iterations.

@ZeroIntensity
Copy link
Member Author

Yeah, I totally agree on that. There was an issue similar to this a few weeks ago, where a TextIOWrapper.__next__ didn't have a critical section on it, which ended up causing crashes. I originally rejected it, but I was eventually convinced that it is worth doing for IO objects.

@corona10
Copy link
Member

corona10 commented Jun 12, 2025

I am fine with if this change will not regress overall performance. But if the object is shared by multiple threads user should care about the correctness if the object itself does not gurentee it.
cc @colesbury

Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Without the fix, the test fails randomly with:

FAIL: test_concurrent_use (test.test_memoryio.PyStringIOPickleTest.test_concurrent_use)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_memoryio.py", line 742, in test_concurrent_use
    self.assertIsNone(cm.exc_value)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: UnboundLocalError("cannot access local variable 'dec_flags' where it is not associated with a value") is not None

So the test works as expected!

@vstinner
Copy link
Member

But if the object is shared by multiple threads user should care about the correctness if the object itself does not gurentee it.

Currently, the doc says nothing about io.StringIO: https://docs.python.org/dev/library/io.html#multi-threading

@kumaraditya303 kumaraditya303 merged commit e6c3039 into python:main Jun 12, 2025
47 checks passed
@kumaraditya303 kumaraditya303 added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jun 12, 2025
@miss-islington-app
Copy link

Thanks @ZeroIntensity for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

@miss-islington-app
Copy link

Thanks @ZeroIntensity for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 12, 2025
…ythonGH-135412)

(cherry picked from commit e6c3039)

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
@miss-islington-app
Copy link

Sorry, @ZeroIntensity and @kumaraditya303, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker e6c3039cb39e68ae9af9ddcaca341c5af8f9cf23 3.13

@bedevere-app
Copy link

bedevere-app bot commented Jun 12, 2025

GH-135425 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Jun 12, 2025
kumaraditya303 pushed a commit that referenced this pull request Jun 12, 2025
…H-135412) (#135425)

gh-135410: use a critical section around `StringIO.__next__` (GH-135412)
(cherry picked from commit e6c3039)

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL 3.14 (tier-1) has failed when building commit 8cb7d9a.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1733/builds/261) and take a look at the build logs.
  4. Check if the failure is related to this commit (8cb7d9a) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1733/builds/261

Failed tests:

  • test_memoryio

Failed subtests:

  • test_concurrent_use - test.test_memoryio.PyStringIOPickleTest.test_concurrent_use

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.14.itamaro-centos-aws.nogil/build/Lib/test/test_memoryio.py", line 742, in test_concurrent_use
    self.assertIsNone(cm.exc_value)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: UnboundLocalError("cannot access local variable 'dec_flags' where it is not associated with a value") is not None

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit e6c3039.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/1566) and take a look at the build logs.
  4. Check if the failure is related to this commit (e6c3039) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/1566

Failed tests:

  • test_memoryio

Failed subtests:

  • test_concurrent_use - test.test_memoryio.PyStringIOPickleTest.test_concurrent_use
  • test_concurrent_use - test.test_memoryio.PyStringIOTest.test_concurrent_use

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/test_memoryio.py", line 742, in test_concurrent_use
    self.assertIsNone(cm.exc_value)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: UnboundLocalError("cannot access local variable 'dec_flags' where it is not associated with a value") is not None

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.14 (tier-1) has failed when building commit 8cb7d9a.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1774/builds/185) and take a look at the build logs.
  4. Check if the failure is related to this commit (8cb7d9a) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1774/builds/185

Failed tests:

  • test_memoryio

Failed subtests:

  • test_concurrent_use - test.test_memoryio.PyStringIOPickleTest.test_concurrent_use
  • test_concurrent_use - test.test_memoryio.PyStringIOTest.test_concurrent_use

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.14.itamaro-centos-aws.refleak.nogil/build/Lib/test/test_memoryio.py", line 742, in test_concurrent_use
    self.assertIsNone(cm.exc_value)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: UnboundLocalError("cannot access local variable 'dec_flags' where it is not associated with a value") is not None

@ZeroIntensity ZeroIntensity deleted the fix-stringio branch June 12, 2025 13:29
@ZeroIntensity
Copy link
Member Author

Those buildbot failures look related. I'll quickly investigate and if I can't figure it out in the next half an hour or so, I'll put up a revert.

@vstinner
Copy link
Member

IMO this change was merged too quickly and I suggest to revert it.

  • I was awaiting for @colesbury to review the change.
  • @corona10 asked for benchmarks, this request was ignored.
  • I pointed out that the documentation says nothing about io.StringIO thread safety. I would prefer to address this point at the same time.
  • And now, Refleak buildbots are failing. It seems like the Python implementation (_pyio) behaves differently than the C implementation (io).

I suggest to revert the two changes (main and 3.14 branches) and spend more time to design this fix. I don't think that there is any urgency to fix this issue.

@ZeroIntensity
Copy link
Member Author

Sorry for the hassle everyone! I didn't get a chance to reply to any comments before it was merged. I'll put up a revert while we discuss this.

ZeroIntensity added a commit to ZeroIntensity/cpython that referenced this pull request Jun 12, 2025
corona10 pushed a commit that referenced this pull request Jun 12, 2025
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 12, 2025
…xt__` (pythonGH-135412)" (pythonGH-135439)

This reverts commit e6c3039.
(cherry picked from commit 7343135)

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
corona10 pushed a commit that referenced this pull request Jun 12, 2025
…ext__` (GH-135412)" (GH-135439) (gh-135449)

Revert "gh-135410: use a critical section around `StringIO.__next__` (GH-135412)" (GH-135439)

This reverts commit e6c3039.
(cherry picked from commit 7343135)

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants