Skip to content

Max recursion during unpickling with a proxy object #133015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gbrawn opened this issue Apr 26, 2025 · 7 comments
Open

Max recursion during unpickling with a proxy object #133015

gbrawn opened this issue Apr 26, 2025 · 7 comments
Labels
extension-modules C modules in the Modules dir pending The issue will be closed if no feedback is provided stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@gbrawn
Copy link

gbrawn commented Apr 26, 2025

Bug report

Bug description:

Description of Problem

A wrapper class was needed to wrap an underlying object while exposing the underlying object's attributes. https://stackoverflow.com/questions/68926132/creation-of-a-class-wrapper-in-python

When creating a wrapper object for an object to be placed on a multiprocessing queue, if the wrapper object overrides the getattr method, and the object is retrieved from the queue, a max recursion depth error is observed. The expected behavior was for the wrapper object to be returned and have the underlying object exposed in the process that was handling the object.

Example Snippet

from multiprocessing import Queue

class Foo:
    """
    Any old object
    """
    def __init__(self, var: int):
        self._var = var

    @property
    def var(self) -> int:
        return self._var

class FooWrapper:
    """
    Wrapper class to expose underlying object attrs
    """
    def __init__(self, foo: Foo):
        self.foo = foo

    def __getattr__(self, name):
        return getattr(self.foo, name)

if __name__=="__main__":
    queue = Queue()
    foo = Foo(2)
    foo_wrapper = FooWrapper(foo)
    queue.put(foo_wrapper)
    message_wrapper = queue.get()  # This fails due to max recursion depth reached

>>
    return getattr(self.foo, name)
                   ^^^^^^^^
  [Previous line repeated 994 more times]
RecursionError: maximum recursion depth exceeded

CPython versions tested on:

3.11

Operating systems tested on:

Linux, Windows

@gbrawn gbrawn added the type-bug An unexpected behavior, bug, or error label Apr 26, 2025
@picnixz picnixz added extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir and removed extension-modules C modules in the Modules dir labels Apr 26, 2025
@picnixz picnixz changed the title Max recursion depth reached when queue.get() is called on object with __getattr__ override Max recursion depth reached when multiprocessing.queue.Queue.get() is called on object with __getattr__ override Apr 26, 2025
@picnixz
Copy link
Member

picnixz commented Apr 26, 2025

How does it fare with a plain queue.Queue? does it fail as well?

@gbrawn
Copy link
Author

gbrawn commented Apr 26, 2025

Just tested it and it appears to work as intended.

from queue import Queue

class Foo:
    """
    Any old object
    """
    def __init__(self, var: int):
        self._var = var

    @property
    def var(self) -> int:
        return self._var

class FooWrapper:
    """
    Wrapper class to expose underlying object attrs
    """
    def __init__(self, foo: Foo):
        self.foo = foo

    def __getattr__(self, name):
        return getattr(self.foo, name)

if __name__=="__main__":
    queue = Queue()
    foo = Foo(2)
    foo_wrapper = FooWrapper(foo)
    queue.put(foo_wrapper)
    message_wrapper = queue.get()
    print(f"{message_wrapper.var}")
>>>
2

@brianschubert
Copy link
Contributor

This happens during unpickling. Here's a simplified reproducer without multiprocessing.Queue:

import pickle

class Wrapper:
    def __init__(self, x):
        self.x = x

    def __getattr__(self, name):
        return getattr(self.x, name)

b = pickle.dumps(Wrapper("foo"))
pickle.loads(b)  # <-- recursion here

@picnixz picnixz changed the title Max recursion depth reached when multiprocessing.queue.Queue.get() is called on object with __getattr__ override Max recursion during unpickling with a proxy object Apr 26, 2025
@brianschubert
Copy link
Contributor

brianschubert commented Apr 26, 2025

The recursion happens during this call of PyObject_GetOptionalAttr, which is checking for the existence of __setstate__ on the instance being built. This calls __getattr__, and since the instance dict hasn't been populated yet, the lookup of self.x inside __getattr__ results in an infinitely recursing __getattr__ call.

@picnixz picnixz added the extension-modules C modules in the Modules dir label Apr 26, 2025
@picnixz
Copy link
Member

picnixz commented Apr 26, 2025

  • Does it also affect the Python implementation of pickle?
  • Do you want to make a PR for that?

@brianschubert
Copy link
Contributor

brianschubert commented Apr 26, 2025

Ya, the Python implementation is also affected (recurses in the "same spot" here).

However, looking at the pickle docs, this seems to be expected behavior. The docs have this warning about relying on __init__ being called before __getattr__ (as the code above does):

At unpickling time, some methods like __getattr__(), __getattribute__(), or __setattr__() may be called upon the instance. In case those methods rely on some internal invariant being true, the type should implement __new__() to establish such an invariant, as __init__() is not called when unpickling an instance.

So, I think this can be considered a bug in the user code and not something CPython needs to fix.

@picnixz picnixz added the pending The issue will be closed if no feedback is provided label Apr 26, 2025
@picnixz
Copy link
Member

picnixz commented Apr 26, 2025

Indeed, thanks for pointing out the docs. Is there a way to determine which methods can be called during unpickling (by default)? if not, let's close this issue; otherwise, let's improve the wording in the docs (to me, it seems as if we say "we know of some methods but others can also be called so up to you to find which ones")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
extension-modules C modules in the Modules dir pending The issue will be closed if no feedback is provided stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
Status: No status
Development

No branches or pull requests

3 participants