Skip to content

gh-132413: Fix crash in _datetime when used at shutdown #132665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 60 commits into
base: main
Choose a base branch
from

Conversation

neonene
Copy link
Contributor

@neonene neonene commented Apr 18, 2025

Create a module when it is not importable.

This patch gives the interpreter state struct an additional pointer to the _datetime module state. Currently, the module state can be reached via a weak reference to the module, which seems problematic because the weakref counts the active module as invalid earlier at shutdown.

(No different than before logically, but the attempt would need to be withdrawn if there should be a reasonable case that the introduced pointer becomes a dangling pointer or one to a reallocated region. #132599 is an alternative.)

(This does not suppress ImportError raised when running the strptime methods at shutdown, which is almost traditional.)

@python-cla-bot
Copy link

python-cla-bot bot commented Apr 18, 2025

All commit authors signed the Contributor License Agreement.

CLA signed

Copy link
Member

@ZeroIntensity ZeroIntensity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be storing more things in the interpreter dictionary. It's relatively slow, and at a glance, probably a thread-safety issue in the way it's being used here.

The better solution would be to drop the interpreter dictionary usage here entirely, and just store the module state on the interpreter state.

cc @ericsnowcurrently

@neonene
Copy link
Contributor Author

neonene commented Apr 20, 2025

It sounds like the concern is not about the issue that needs to be fixed even on 3.13. The current module state access may not be so fast, but I doubt this PR makes it slower. My question for alternatives is that how much we need to care about the thread-safety after finalize_modules() finishes in pylifecycle.c.

@ZeroIntensity
Copy link
Member

I meant an existing thread-safety issue, but you're right, let's not address that here. _datetime is a special module, it's fine to make it special on the interpreter state.

  • It's significantly less complex to do this through the interpreter state.
  • Manual invocation of PyModule_FromDefAndSpec/PyModule_ExecDef like this is very likely going to introduce other bugs. I don't think we have any tests for either of those functions, let alone during finalization.

@neonene
Copy link
Contributor Author

neonene commented Apr 20, 2025

I agree to the risk of manual invocation during finalization.

Two versions work fine:

  • This PR: ABI changed. Module state is added to the interpreter state.

  • #132599: ABI not changed. Backportable.

Both require the interpreter dict. I failed to add a module object to the interpreter state (bad reloading).

@ZeroIntensity
Copy link
Member

The interpreter state isn't part of the public ABI, we can (and do) change it in backports.

@neonene neonene marked this pull request as ready for review April 26, 2025 11:23
@ZeroIntensity
Copy link
Member

Let me know when you're ready for a re-review.

@neonene
Copy link
Contributor Author

neonene commented May 1, 2025

Ready for reviewing. Though, there seems to be fundamental fixes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants