-
-
Notifications
You must be signed in to change notification settings - Fork 31.8k
WeakSet is not pickleable #74876
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
WeakSet contains the __reduce__ method, but it isn't pickleable (and never was), because the pickle state contains the value of the __dict__ dict attribute which contains a reference to unpickleable local function _remove(). >>> pickle.dumps(weakref.WeakSet())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: Can't pickle local object 'WeakSet.__init__.<locals>._remove' I wondering whether WeakSet should be made pickleable or the __reduce__ method should be removed. __reduce__() can be used also for copying, but there are no tests for this feature. |
When considering whether to remove a method from long published code, if the method isn't broken, our guidance should come from whether user's have actually taken advantage of __reduce__. Determining that answer involves searching published code (Google's code search used to be a good tool for this, now we've got Github's code search tools). In general, the burden is high for removing an existing feature (even if untested); otherwise, we risk breaking people's code for no good reason other than the joy that comes with churning code to solve an invented problem (one that has never arisen in real code and has never been reported by an actual user). When considering whether to add pickle support, the bar is much lower. Roughly the question is amounts to balancing the potential benefits (whether someone might need to pickle a weakset someday even though we have no evidence that anyone has ever wanted this) versus the costs (risk of introducing bugs, creating cross-version incompatabilities, increasing future maintenance costs, increasing the total code volume, etc). If adding pickling capability is easy and clean, then it seems reasonable. On the other hand, if it is even slightly tricky, we should skip adding a feature that no one has ever asked for. The |
The method is broken (and always was). Pickling doesn't work because the state contains unpickleable object. Copying works incorrectly, since the pickle state contains references to the original WeakSet and it overrides the __dict__ of the copy, making its state inconsistent. If the purpose of implementing the __reduce__ method was the support of the copying, the __reduce__ method should be fixed or the copying should be implemented with implementing the __copy__ and __deepcopy__ methods. However there is a subtle moment with pickling WeakSet (as well as with pickling weakref.proxy, see bpo-18068). The problem is that if you pickled a WeakSet, but not hard references to its items, the items will be disappeared just after unpickling, since they have no hard references. This may confuse. If you pickle also hard refereces to the items, a WeakSet can be pickled and unpickled as expected (after fixing __reduce__). |
That seems contrary to the original intent of a WeakSet. I recommend not going down the path of adding pickle support. If copying is also broken, then it is reasonable to either fix it or remove it depending on whether a fix is straight-forward. |
Copying is broken too. But instead of failing it produces incorrect result: two WeakSets share the same content. >>> import weakref, copy
>>> class A(float): pass
...
>>> s = {A(1)}
>>> ws = weakref.WeakSet(s)
>>> ws2 = copy.copy(ws)
>>> list(ws)
[1.0]
>>> list(ws2)
[1.0]
>>> x = A(2)
>>> ws.add(x)
>>> list(ws)
[1.0, 2.0]
>>> list(ws2)
[1.0, 2.0] |
Hello, I am using iPyParallel (or rather, attempting to) to support parallelizing a complex task. I have successfully used iPyParallel elsewhere in another project. Unfortunately, in this project, I have this error:
I have tried to identify what the object is so I can come up with a workaround (iPyParallel allows you to instantiate remotely so pickling is not required)... iPyParallel provides two workarounds to help with pickling, being Dill and CloudPickle. Dill complains: Per the above request:
I am formally requesting this :) Thanks for considering! Meanwhile I'll try to ID the object and see if I can remove it... |
How does PR #93732 look for this? |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
The text was updated successfully, but these errors were encountered: