Skip to content

WeakSet is not pickleable #74876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
serhiy-storchaka opened this issue Jun 17, 2017 · 7 comments · May be fixed by #93732
Open

WeakSet is not pickleable #74876

serhiy-storchaka opened this issue Jun 17, 2017 · 7 comments · May be fixed by #93732
Labels
3.11 only security fixes 3.12 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Jun 17, 2017

BPO 30691
Nosy @Yhg1s, @birkenfeld, @rhettinger, @pitrou, @serhiy-storchaka

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-06-17.18:54:10.235>
labels = ['library']
title = 'WeakSet is not pickleable'
updated_at = <Date 2017-06-18.06:09:54.301>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-06-18.06:09:54.301>
actor = 'rhettinger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-06-17.18:54:10.235>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 30691
keywords = []
message_count = 4.0
messages = ['296243', '296256', '296262', '296264']
nosy_count = 5.0
nosy_names = ['twouters', 'georg.brandl', 'rhettinger', 'pitrou', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue30691'
versions = []

Linked PRs

@serhiy-storchaka
Copy link
Member Author

WeakSet contains the __reduce__ method, but it isn't pickleable (and never was), because the pickle state contains the value of the __dict__ dict attribute which contains a reference to unpickleable local function _remove().

>>> pickle.dumps(weakref.WeakSet())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: Can't pickle local object 'WeakSet.__init__.<locals>._remove'

I wondering whether WeakSet should be made pickleable or the __reduce__ method should be removed. __reduce__() can be used also for copying, but there are no tests for this feature.

@serhiy-storchaka serhiy-storchaka added the stdlib Python modules in the Lib dir label Jun 17, 2017
@rhettinger
Copy link
Contributor

I wondering whether WeakSet should be made pickleable or
the __reduce__ method should be removed.

When considering whether to remove a method from long published code, if the method isn't broken, our guidance should come from whether user's have actually taken advantage of __reduce__. Determining that answer involves searching published code (Google's code search used to be a good tool for this, now we've got Github's code search tools).

In general, the burden is high for removing an existing feature (even if untested); otherwise, we risk breaking people's code for no good reason other than the joy that comes with churning code to solve an invented problem (one that has never arisen in real code and has never been reported by an actual user).

When considering whether to add pickle support, the bar is much lower. Roughly the question is amounts to balancing the potential benefits (whether someone might need to pickle a weakset someday even though we have no evidence that anyone has ever wanted this) versus the costs (risk of introducing bugs, creating cross-version incompatabilities, increasing future maintenance costs, increasing the total code volume, etc).

If adding pickling capability is easy and clean, then it seems reasonable. On the other hand, if it is even slightly tricky, we should skip adding a feature that no one has ever asked for. The
weak reference containers have long been a source of bugs, some of which were challenging to fix.

@serhiy-storchaka
Copy link
Member Author

The method is broken (and always was). Pickling doesn't work because the state contains unpickleable object. Copying works incorrectly, since the pickle state contains references to the original WeakSet and it overrides the __dict__ of the copy, making its state inconsistent. If the purpose of implementing the __reduce__ method was the support of the copying, the __reduce__ method should be fixed or the copying should be implemented with implementing the __copy__ and __deepcopy__ methods.

However there is a subtle moment with pickling WeakSet (as well as with pickling weakref.proxy, see bpo-18068). The problem is that if you pickled a WeakSet, but not hard references to its items, the items will be disappeared just after unpickling, since they have no hard references. This may confuse. If you pickle also hard refereces to the items, a WeakSet can be pickled and unpickled as expected (after fixing __reduce__).

@rhettinger
Copy link
Contributor

If you pickle also hard refereces to the items,
a WeakSet can be pickled and unpickled as expected (after
fixing __reduce__).

That seems contrary to the original intent of a WeakSet. I recommend not going down the path of adding pickle support.

If copying is also broken, then it is reasonable to either fix it or remove it depending on whether a fix is straight-forward.

@serhiy-storchaka
Copy link
Member Author

serhiy-storchaka commented Jun 11, 2022

Copying is broken too. But instead of failing it produces incorrect result: two WeakSets share the same content.

>>> import weakref, copy
>>> class A(float): pass
... 
>>> s = {A(1)}
>>> ws = weakref.WeakSet(s)
>>> ws2 = copy.copy(ws)
>>> list(ws)
[1.0]
>>> list(ws2)
[1.0]
>>> x = A(2)
>>> ws.add(x)
>>> list(ws)
[1.0, 2.0]
>>> list(ws2)
[1.0, 2.0]

@AA-Turner AA-Turner added type-bug An unexpected behavior, bug, or error 3.11 only security fixes 3.10 only security fixes 3.12 only security fixes labels Jun 11, 2022
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jun 11, 2022
@hugovk hugovk removed the 3.10 only security fixes label Apr 7, 2023
@webberfc
Copy link

webberfc commented Jun 19, 2023

Hello, I am using iPyParallel (or rather, attempting to) to support parallelizing a complex task. I have successfully used iPyParallel elsewhere in another project. Unfortunately, in this project, I have this error:

File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\client\view.py", line 334, in map_async
return self.map(f, *sequences, **kwargs)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\client\view.py", line 53, in sync_results
ret = f(self, *args, **kwargs)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\client\view.py", line 37, in save_ids
ret = f(self, *args, **kwargs)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\client\view.py", line 1338, in map
return pf.map(*sequences)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\client\remotefunction.py", line 332, in map
return self(*sequences, __ipp_mapping=True)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\client\remotefunction.py", line 81, in sync_view_results
return f(self, *args, **kwargs)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\client\remotefunction.py", line 272, in call
pf = PrePickled(self.func)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\serialize\serialize.py", line 46, in init
self.buffers = serialize_object(obj)
File "C:\Users\Simba\AppData\Local\Programs\Python\Python310\lib\site-packages\ipyparallel\serialize\serialize.py", line 119, in serialize_object
buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL))
AttributeError: Can't pickle local object 'WeakSet.init.._remove'

I have tried to identify what the object is so I can come up with a workaround (iPyParallel allows you to instantiate remotely so pickling is not required)... iPyParallel provides two workarounds to help with pickling, being Dill and CloudPickle. Dill complains:
TypeError: no default __reduce__ due to non-trivial __cinit__
And CloudPickle is complaining about a thread, which I will have to look into, I thought this version was a single thread version up until the call to map_async.

Per the above request:

If adding pickling capability is easy and clean, then it seems reasonable. On the other hand, if it is even slightly tricky, we should skip adding a feature that no one has ever asked for.

I am formally requesting this :) Thanks for considering! Meanwhile I'll try to ID the object and see if I can remove it...

@hugovk
Copy link
Member

hugovk commented Apr 26, 2025

How does PR #93732 look for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
Status: No status
Status: No status
Development

Successfully merging a pull request may close this issue.

5 participants