Skip to content

gh-87634: deprecate cached_property locking, add lock kwarg #98123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Doc/faq/programming.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1939,7 +1939,7 @@ This example shows the various techniques::
# Do not cache this because old results
# can be out of date.

@cached_property
@cached_property(lock=False)
def location(self):
"Return the longitude/latitude coordinates of the station"
# Result only depends on the station_id
Expand Down
2 changes: 1 addition & 1 deletion Doc/howto/descriptor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1500,7 +1500,7 @@ instance dictionary to function correctly:
class CP:
__slots__ = () # Eliminates the instance dict

@cached_property # Requires an instance dict
@cached_property(lock=False) # Requires an instance dict
def pi(self):
return 4 * sum((-1.0)**n / (2.0*n + 1.0)
for n in reversed(range(100_000)))
Expand Down
24 changes: 22 additions & 2 deletions Doc/library/functools.rst
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The :mod:`functools` module defines the following functions:
.. versionadded:: 3.9


.. decorator:: cached_property(func)
.. decorator:: cached_property(func, *, lock=True)

Transform a method of a class into a property whose value is computed once
and then cached as a normal attribute for the life of the instance. Similar
Expand All @@ -69,7 +69,7 @@ The :mod:`functools` module defines the following functions:
def __init__(self, sequence_of_numbers):
self._data = tuple(sequence_of_numbers)

@cached_property
@cached_property(lock=False)
def stdev(self):
return statistics.stdev(self._data)

Expand All @@ -86,6 +86,19 @@ The :mod:`functools` module defines the following functions:
The cached value can be cleared by deleting the attribute. This
allows the *cached_property* method to run again.

By default, a ``cached_property`` instance contains an
:class:`~threading.RLock` which prevents multiple threads from executing the
cached property concurrently. This was intended to ensure that the cached
property would execute only once per instance, but it also prevents
concurrent execution on different instances of the same class, causing
unnecessary and excessive serialization of updates. This locking behavior is
deprecated and will be removed in Python 3.14. It can be avoided by passing
``lock=False``.

With ``lock=False``, ``cached_property`` provides no guarantee that it will
run only once per instance, if accessed in multiple threads. An application
wanting such guarantees must handle its own locking explicitly.

Note, this decorator interferes with the operation of :pep:`412`
key-sharing dictionaries. This means that instance dictionaries
can take more space than usual.
Expand All @@ -112,6 +125,13 @@ The :mod:`functools` module defines the following functions:

.. versionadded:: 3.8

.. versionchanged:: 3.12
The ``lock`` argument was added.

.. deprecated-removed:: 3.12 3.14
The locking behavior of ``cached_property`` is deprecated and will be
removed in 3.14. Use ``lock=False`` to avoid the deprecated behavior.


.. function:: cmp_to_key(func)

Expand Down
11 changes: 11 additions & 0 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,17 @@ Deprecated
:exc:`ImportWarning`).
(Contributed by Brett Cannon in :gh:`65961`.)

* The locking behavior of :class:`~functools.cached_property` is deprecated and
will be removed in Python 3.14. The locking is class-wide and thus
inefficient if many instances of the same class with a cached property are
used across multiple threads (only a single instance across all instances of
the class can update its cached property at once). Integrating locking into
:class:`~functools.cached_property` was a mistake; the application should
determine its need for synchronization guarantees and lock as needed to meet
them. Use ``@cached_property(lock=False)`` to create a cached property with
no locking and avoid the deprecation warning.
(Contributed by Carl Meyer in :gh:`87634`.)


Pending Removal in Python 3.13
------------------------------
Expand Down
2 changes: 2 additions & 0 deletions Lib/ensurepip/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ def _run_pip(args, additional_paths=None):
sys.executable,
'-W',
'ignore::DeprecationWarning',
'-W',
'ignore::PendingDeprecationWarning',
'-c',
code,
]
Expand Down
66 changes: 50 additions & 16 deletions Lib/functools.py
Original file line number Diff line number Diff line change
Expand Up @@ -963,13 +963,34 @@ def __isabstractmethod__(self):


class cached_property:
def __init__(self, func):
def __init__(self, func=None, *, lock=True):
self.func = func
self.attrname = None
self.__doc__ = func.__doc__
self.lock = RLock()
if func is not None:
self.__doc__ = func.__doc__
if lock:
import warnings
warnings._deprecated(
"lock=True default behavior of cached_property",
"Locking cached_property is deprecated; use @cached_property(lock=False)",
remove=(3, 14)
)
self.lock = RLock() if lock else None

def __call__(self, func=None):
if self.func is not None:
raise TypeError(f"{type(self).__name__!r} object is not callable")
self.func = func
if func is not None:
self.__doc__ = func.__doc__
return self

def __set_name__(self, owner, name):
if self.func is None:
raise TypeError(
"Cannot assign cached_property(lock=...) directly to instance, "
"it must decorate a function."
)
if self.attrname is None:
self.attrname = name
elif name != self.attrname:
Expand All @@ -984,6 +1005,11 @@ def __get__(self, instance, owner=None):
if self.attrname is None:
raise TypeError(
"Cannot use cached_property instance without calling __set_name__ on it.")
if self.func is None:
raise TypeError(
"Cannot assign cached_property(lock=...) directly to instance, "
"it must decorate a function."
)
try:
cache = instance.__dict__
except AttributeError: # not all objects have __dict__ (e.g. class defines slots)
Expand All @@ -994,19 +1020,27 @@ def __get__(self, instance, owner=None):
raise TypeError(msg) from None
val = cache.get(self.attrname, _NOT_FOUND)
if val is _NOT_FOUND:
with self.lock:
# check if another thread filled cache while we awaited lock
val = cache.get(self.attrname, _NOT_FOUND)
if val is _NOT_FOUND:
val = self.func(instance)
try:
cache[self.attrname] = val
except TypeError:
msg = (
f"The '__dict__' attribute on {type(instance).__name__!r} instance "
f"does not support item assignment for caching {self.attrname!r} property."
)
raise TypeError(msg) from None
if self.lock is None:
val = self._fill(instance, cache)
else:
with self.lock:
# check if another thread filled cache while we awaited lock
val = cache.get(self.attrname, _NOT_FOUND)
if val is _NOT_FOUND:
val = self._fill(instance, cache)
return val

def _fill(self, instance, cache):
val = self.func(instance)
try:
cache[self.attrname] = val
except TypeError:
msg = (
f"The '__dict__' attribute on {type(instance).__name__!r} instance "
f"does not support item assignment for caching {self.attrname!r} property."
)
raise TypeError(msg) from None
return val


__class_getitem__ = classmethod(GenericAlias)
8 changes: 4 additions & 4 deletions Lib/ipaddress.py
Original file line number Diff line number Diff line change
Expand Up @@ -755,12 +755,12 @@ def overlaps(self, other):
other.network_address in self or (
other.broadcast_address in self)))

@functools.cached_property
@functools.cached_property(lock=False)
def broadcast_address(self):
return self._address_class(int(self.network_address) |
int(self.hostmask))

@functools.cached_property
@functools.cached_property(lock=False)
def hostmask(self):
return self._address_class(int(self.netmask) ^ self._ALL_ONES)

Expand Down Expand Up @@ -1390,7 +1390,7 @@ def __init__(self, address):
self.netmask = self.network.netmask
self._prefixlen = self.network._prefixlen

@functools.cached_property
@functools.cached_property(lock=False)
def hostmask(self):
return self.network.hostmask

Expand Down Expand Up @@ -2094,7 +2094,7 @@ def __init__(self, address):
self.netmask = self.network.netmask
self._prefixlen = self.network._prefixlen

@functools.cached_property
@functools.cached_property(lock=False)
def hostmask(self):
return self.network.hostmask

Expand Down
2 changes: 1 addition & 1 deletion Lib/pdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ def check(self):
traceback.print_exc()
sys.exit(1)

@functools.cached_property
@functools.cached_property(lock=False)
def _details(self):
import runpy
return runpy._get_module_details(self)
Expand Down
2 changes: 1 addition & 1 deletion Lib/platform.py
Original file line number Diff line number Diff line change
Expand Up @@ -847,7 +847,7 @@ class uname_result(
except when needed.
"""

@functools.cached_property
@functools.cached_property(lock=False)
def processor(self):
return _unknown_as_blank(_Processor.get())

Expand Down
Loading