Skip to content

Inconsistent Popen.communicate() behavior if stdin/stdout/stderr is closed PIPE #131064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pekkaklarck opened this issue Mar 10, 2025 · 1 comment
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@pekkaklarck
Copy link

pekkaklarck commented Mar 10, 2025

Bug report

Bug description:

The behavior with Popen.communicate() if stdin/stdout/stderr are PIPEs and they are closed before calling communicate() is rather strange.

If both stdout and stderr are PIPEs and one (or both) is closed, communicate() succeeds. If only one of them is a PIPE and it is closed, the behavior changes and there's a ValueError instead:

>>> from subprocess import Popen, PIPE
>>>
>>> p = Popen(['python', '-V'], stdout=PIPE, stderr=PIPE)
>>> p.stdout.close()
>>> p.communicate()
(b'', b'')
>>>
>>> p = Popen(['python', '-V'], stdout=PIPE)
>>> p.stdout.close()
>>> p.communicate()
Traceback (most recent call last):
  File "<python-input-7>", line 1, in <module>
    p.communicate()
    ~~~~~~~~~~~~~^^
  File "/usr/lib/python3.14/subprocess.py", line 1207, in communicate
    stdout = self.stdout.read()
ValueError: read of closed file

If stdin is a closed PIPE, the behavior is pretty much the opposite of the above. If other streams are not PIPEs, communicate() succeeds, but if the are other PIPEs, we get a ValueError:

>>> from subprocess import Popen, PIPE
>>>
>>> p = Popen(['python', '-c', 'pass'], stdin=PIPE)
>>> p.stdin.close()
>>> p.communicate()
(None, None)
>>> 
>>> p = Popen(['python', '-c', 'pass'], stdin=PIPE, stdout=PIPE)
>>> p.stdin.close()
>>> p.communicate()
Traceback (most recent call last):
  File "<python-input-14>", line 1, in <module>
    p.communicate()
    ~~~~~~~~~~~~~^^
  File "/usr/lib/python3.14/subprocess.py", line 1220, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/subprocess.py", line 2081, in _communicate
    self.stdin.flush()
    ~~~~~~~~~~~~~~~~^^
ValueError: flush of closed file

This inconsistency seems to be caused by an optimization that leads to two different code paths depending on the number of PIPEs. I needed to spend some time to understand why our tests behaved seemingly inconsistently after code was refactored.

It would be easy to fix problems by changing code like if self.stdout: to if self.stdout and not self.stdout.closed:. I guess it could be argued that raising a ValueError is the correct way to handle these situations, but in that case it should be raised always.

Tested with Python 3.14 alpha 5. Occurs also with earlier ones.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

@pekkaklarck
Copy link
Author

pekkaklarck commented Mar 10, 2025

If this is agreed to be a bug, I can provide a PR fixing these inconsistencies so that closed PIPEs are always handled gracefully. In that case we needed to decide should None or an empty string/bytes (depending on self.text_mode) be returned if stdout or stderr is a closed PIPE. A naïve fix would return None, but the current un-optimized code path returns an empty string/bytes and changing it could cause backwards compatibility problems.

@ZeroIntensity ZeroIntensity added the stdlib Python modules in the Lib dir label Mar 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

2 participants