gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror #136796

rani-pinchuk · 2025-07-19T11:41:42Z

This is a small fix as suggested in #135427 - using PyErr_WriteUnraisable() instead of PyErr_Clear().

The fix includes a test which checks that the DeprecationWarning is provided when using fork or forkpty within a thread, also when running with -Werror.

@encukou

Issue: -Werror doesn't error out on DeprecationWarning for os.fork() #135427

…ds with -Werror

python-cla-bot · 2025-07-19T11:41:46Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2025-07-19T11:41:46Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2025-07-19T11:45:45Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

rani-pinchuk · 2025-07-19T11:57:25Z

I have created a NEWS entry for this fix, but actually this change has indeed a little impact on Python users, so maybe skip news tag is more appropriate here.

…he exception.

…nto fix-issue-135427

rani-pinchuk · 2025-07-19T15:52:52Z

Many tests are failing with a warning - as now the DeprecationWarning is no longer hidden (and as I unedrstand it, the tests run with the equivalent of -Werror).

Many of the failing tests do not fail where one would expect to have more than one thread running. For example, in test_uuid.testIssue8621.

When printing there the number of threads using threading.active_count() we get 1. However, when investigating further, the number of threads are actually 2 - just that one of the threads is not a python thread, and therefore not counted by threading.active_count().

Therefore, we should find a way to suppress the warning of running a fork within a thread in all the tests that fail. Petr suggests to consider using the sys.unraisablehook (see https://vstinner.github.io/sys-unraisablehook-python38.html) in all the places where the tests fail.

If anyone knows where this extra thread comes from, it will be appreciated if this is explained :-)

gpshead · 2025-07-19T17:51:33Z

If anyone knows where this extra thread comes from, it will be appreciated if this is explained :-)

That is ironically exactly why we have this warning - threads in a process can come from everywhere including external libraries outside of our control.

I do think you're on the right track in terms of how to enable this to be something -Werror captures. As a potentially disruptive behavior change for some test suites, it isn't a bug fix - but it'd make sense in 3.15. Having a news entry is indeed appropriate.

using sys.unraisablehook in these parts of the test suites is an interesting idea.

…ead. Started to fix tests.

…sed.

…line

…24.02 run

kumaraditya303 · 2025-07-22T08:54:03Z

Lib/test/test_asyncio/test_unix_events.py

@@ -1182,6 +1182,7 @@ async def runner():
 @support.requires_fork()
 class TestFork(unittest.IsolatedAsyncioTestCase):

+    @warnings_helper.ignore_warnings(category=DeprecationWarning)  # gh-135427


The whole class has fork tests, maybe add decorator to class itself?

Yes. I am still learning... I tried to avoid putting decorator to the classes themselves, as I thought that it is more correct that if someone adds a new test, that developer should see the depracation warning.

But what you suggest will save many lines of code.

And also it seems that in some cases the test may run with forks or without them, so I am not sure that it is correct to decorate the functions themselves. For example, ProcessPoolExecutorTest extends ExecutorTest but also ThreadPoolExecutorTest extends ExecutorTest. So the tests in ExecutorTest must be decorated only if it is extended by ProcessPoolExecutorTest.

Lib/test/test_asyncio/test_unix_events.py

… warnings

…ific decorator to ignore only the fork in thread deprecation warnings

rani-pinchuk · 2025-07-24T20:49:14Z

@encukou, @gpshead - I am not sure, but maybe we should indeed create two PRs. The one that is here for 3.15. And in the second PR we will not raise exceptions for these deprecation warnings when using -Werror, but just show the warnings. This other PR maybe then can go in 3.14 - so at least the deprecation warnings are shown in that release when we have -Werror.

I think that the differences between the two PRs will only be the change in Modules/posixmodule.c but all the decorators to ignore the warnings should be the same. So in the other PR the change will be as shown in this earlier commit.

What do you think?

encukou

Sorry for the delay!

Thank you for tackling this, and finding all the places that need the suppression! I think this looks good, just needs a bit of polish/simplification.

Modules/posixmodule.c

Misc/NEWS.d/next/Library/2025-07-19-11-53-19.gh-issue-135427.iJM_X2.rst

Lib/test/support/warnings_helper.py

encukou · 2025-07-31T12:00:57Z

I think this should go in 3.15 only, and go through the full testing period.
There's no removal date set for these DeprecationWarnings; if people see them one release later, there's not that much harm done.
The 3.14 branch is only accepts critical fixes for 3.14.0, and after that bugfixes for 3.14.1. Emitting new warnings in bugfix releases tends to be messy.

Co-authored-by: Petr Viktorin <encukou@gmail.com>

…JM_X2.rst Co-authored-by: Petr Viktorin <encukou@gmail.com>

encukou

Thank you! This is looking good, but needs a bit more polish yet.

encukou · 2025-08-11T10:00:55Z

.gitignore

+
+# Mypy symlinks (generated by Misc/mypy/make_symlinks.py)
+Misc/mypy/_colorize.py
+Misc/mypy/_pyrepl
+Misc/mypy/token.py


This doesn't look related to this PR. Could you revert this, and the removal of the files themselves?

encukou · 2025-08-11T10:43:18Z

Lib/test/_test_multiprocessing.py

+        # gh-135427
+        # In some of the tests, a forked child forks another child of itself. In that case, using
+        # warnings_helper.ignore_warnings decorator does not actually ignore the warning from that
+        # child of child, and a warnings_helper.ignore_warnings exception is raised.
+        with warnings.catch_warnings():
+            warnings.filterwarnings('ignore',
+                                    message=".*fork.*may lead to deadlocks in the child.*",
+                                    category=DeprecationWarning)
+            super().setUpClass()
+            cls.pool = cls.Pool(4)


These setUp methods should be able to use the context manager too. Or is there something I'm missing?

Suggested change

# gh-135427

# In some of the tests, a forked child forks another child of itself. In that case, using

# warnings_helper.ignore_warnings decorator does not actually ignore the warning from that

# child of child, and a warnings_helper.ignore_warnings exception is raised.

with warnings.catch_warnings():

warnings.filterwarnings('ignore',

message=".*fork.*may lead to deadlocks in the child.*",

category=DeprecationWarning)

super().setUpClass()

cls.pool = cls.Pool(4)

with warnings_helper.ignore_fork_in_thread_deprecation_warnings():

super().setUpClass()

cls.pool = cls.Pool(4)

The repeated comment could go in ignore_fork_in_thread_deprecation_warnings's docstring.

encukou · 2025-08-11T10:53:08Z

changed, 7314 insertions(+), 3271 deletions(-)

This file looks added by mistake.

bedevere-bot · 2025-08-11T10:54:09Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit 8b699d8 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136796%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

pythongh-135427: Fix DeprecationWarning for os.fork when run in threa…

6622c55

…ds with -Werror

rani-pinchuk requested a review from gpshead as a code owner July 19, 2025 11:41

bedevere-app bot added the awaiting review label Jul 19, 2025

bedevere-app bot mentioned this pull request Jul 19, 2025

-Werror doesn't error out on DeprecationWarning for os.fork() #135427

Open

Merge branch 'main' into fix-issue-135427

7b13bdc

📜🤖 Added by blurb_it.

bf97fe0

rani-pinchuk added 2 commits July 19, 2025 15:34

pythongh-135427: check that exception occured, and also clear later t…

3482918

…he exception.

Merge remote-tracking branch 'refs/remotes/origin/fix-issue-135427' i…

ce90d77

…nto fix-issue-135427

rani-pinchuk added 4 commits July 20, 2025 15:06

Behavior changed to raise an exception when fork is used within a thr…

1a20748

…ead. Started to fix tests.

Seems like test_multiprocessing fork deprecation warnings are suppres…

9bead0e

…sed.

Warning of deprecated fork are suppressed in more tests.

d5336dc

WIP - suppressing warnings in tests that fail in the tests running on…

ae3a5ed

…line

rani-pinchuk requested review from rhettinger, a team, 1st1, asvetlov, kumaraditya303 and willingc as code owners July 20, 2025 15:54

rhettinger removed their request for review July 21, 2025 06:06

rani-pinchuk added 3 commits July 21, 2025 16:56

WIP - continuing suppressing warnings in tests.

f768f5d

WIP - fix imports.

6308e19

WIP - more suppressing of warnings

8955136

rani-pinchuk requested a review from vsajip as a code owner July 21, 2025 15:50

WIP - fix ignore_warnings to wrap also async functions

8731cba

rani-pinchuk added 4 commits July 21, 2025 21:12

WIP - more suppress of warnings.

41e8337

WIP - more suppress of warnings.

359b0ed

WIP - test.test_multiprocessing_fork gives no errors on local ubuntu …

5277d4c

…24.02 run

WIP - more suppression of warnings

204c642

kumaraditya303 reviewed Jul 22, 2025

View reviewed changes

rani-pinchuk and others added 4 commits July 22, 2025 21:13

WIP - adding more suppressing of the DeprecatinoWarning

1b44537

WIP - more suppress of warnings.

c43b3c0

Merge branch 'main' into fix-issue-135427

5d7ae0c

WIP - more suppress of warnings.

868ffd6

rani-pinchuk commented Jul 22, 2025

View reviewed changes

Lib/test/test_asyncio/test_unix_events.py Outdated Show resolved Hide resolved

rani-pinchuk added 3 commits July 24, 2025 21:54

Add dedicated decorator to ignore only the fork in thread deprecation…

e1398d8

… warnings

Replace the generic decorators to ignore deprecation errors with spec…

b58b2ca

…ific decorator to ignore only the fork in thread deprecation warnings

Linting issue

8d832ac

encukou reviewed Jul 31, 2025

View reviewed changes

Modules/posixmodule.c Outdated Show resolved Hide resolved

Misc/NEWS.d/next/Library/2025-07-19-11-53-19.gh-issue-135427.iJM_X2.rst Outdated Show resolved Hide resolved

Lib/test/support/warnings_helper.py Outdated Show resolved Hide resolved

rani-pinchuk and others added 8 commits July 31, 2025 20:11

Update Modules/posixmodule.c

a0fc743

Co-authored-by: Petr Viktorin <encukou@gmail.com>

Update Misc/NEWS.d/next/Library/2025-07-19-11-53-19.gh-issue-135427.i…

5d1d553

…JM_X2.rst Co-authored-by: Petr Viktorin <encukou@gmail.com>

Simply ignore_warning functions

5a80e4a

Linting fix

52f6f97

Remove Misc/mypy/_colorize.py - should be a symlink

36147a1

Remove Misc/mypy/_token.py and Misc/mypy/_pyrelp - should be a symlink

baab332

Ignore the symlinked files

244869e

Fix warning for asymc function

8b699d8

encukou reviewed Aug 11, 2025

View reviewed changes

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 11, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 11, 2025

Uh oh!

gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror #136796

Are you sure you want to change the base?

gh-135427: Fix DeprecationWarning for os.fork when run in threads with -Werror #136796

Conversation

rani-pinchuk commented Jul 19, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

python-cla-bot bot commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Jul 19, 2025

Uh oh!

bedevere-app bot commented Jul 19, 2025

Uh oh!

rani-pinchuk commented Jul 19, 2025

Uh oh!

rani-pinchuk commented Jul 19, 2025

Uh oh!

gpshead commented Jul 19, 2025

Uh oh!

kumaraditya303 Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

rani-pinchuk Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rani-pinchuk commented Jul 24, 2025

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

encukou commented Jul 31, 2025

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

encukou Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

encukou Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

encukou Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Aug 11, 2025

Uh oh!

Uh oh!

rani-pinchuk commented Jul 19, 2025 •

edited by bedevere-app bot

Loading

python-cla-bot bot commented Jul 19, 2025 •

edited

Loading