multiprocessing.managers: fix TypeVar usage
#7938
Conversation
#7928 `dict()` and `list()` just return empty dictionaries and lists (respectively) if no arguments are supplied: ```python >>> from multiprocessing.managers import SyncManager >>> with SyncManager() as s: ... print(s.dict()) ... {} ```
stdlib/multiprocessing/managers.pyi
Outdated
| @overload | ||
| def dict(self, sequence: Mapping[_KT, _VT]) -> _dict[_KT, _VT]: ... | ||
| @overload | ||
| def dict(self) -> _dict[Any, Any]: ... |
There was a problem hiding this comment.
This is another instance of the issue @erictraut found in microsoft/pyright#3501 (comment), though here it's on a function, not a constructor.
There was a problem hiding this comment.
Yeah, empty collections are tough :(
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Don't have a strong opinion on what to do here, but observations:
- As Eric pointed out, with changes like this, we potentially make it so users silently get
Anytypes, instead of getting warned by pyright's strict mode about "unknown" types. - The parameter types here are wrong. multiprocessing is a maze but as far as I can tell these should accept the same arguments as the
listanddictconstructors. For example,dict()allows kwargs andlist()allows any iterable. - The return types are wrong: these methods return special proxy objects, not actual dicts or lists.
Some examples from playing with them:
>>> m.list({1, 2, 3})
<ListProxy object, typeid 'list' at 0xffff933192b0>
>>> l = _
>>> type(l)
<class 'multiprocessing.managers.ListProxy'>
>>> len(l)
3
>>> str(l)
'[1, 2, 3]'
>>> l[0]
1
>>> m.dict(x=3)
<DictProxy object, typeid 'dict' at 0xffff933cd040>
>>> d=_
>>> d["x"]
3
>>> d["x"] = 5
>>> d["x"]
5
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
2 similar comments
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
The code in question is doing this: sequence_dict: Dict[int, List[int]] = manager.dict()
hash_dict: Dict[int, int] = manager.dict()This was the best way of annotating that code prior to this PR, given that in typeshed we didn't acknowledge the existence of these classes previously. But it will have to be updated if this PR is merged. |
This comment was marked as resolved.
This comment was marked as resolved.
This comment has been minimized.
This comment has been minimized.
| if sys.version_info < (3, 7): | ||
| def has_key(self, k: _KT) -> bool: ... | ||
|
|
||
| class BaseListProxy(BaseProxy, MutableSequence[_T]): |
There was a problem hiding this comment.
Missing __contains__ and reverse
There was a problem hiding this comment.
Both inherited from MutableSequence
| @@ -66,6 +65,62 @@ class ValueProxy(BaseProxy, Generic[_T]): | |||
| if sys.version_info >= (3, 9): | |||
| def __class_getitem__(cls, item: Any) -> GenericAlias: ... | |||
|
|
|||
| class DictProxy(BaseProxy, MutableMapping[_KT, _VT]): | |||
There was a problem hiding this comment.
Also __contains__, clear, popitem, setdefault, update.
(Or are those inherited? In that case, why do we have __len__?)
There was a problem hiding this comment.
All inherited from MutableMapping. MutableMapping.__len__ is an abstract method, so must be overridden here in order to make this class concrete.
| def pop(self, __key: _KT) -> _VT: ... | ||
| @overload | ||
| def pop(self, __key: _KT, __default: _VT | _T) -> _VT | _T: ... | ||
| def keys(self) -> list[_KT]: ... # type: ignore[override] |
There was a problem hiding this comment.
This is because of https://github.com/python/cpython/blob/518595652759462b13904df767f69b8cc2c61143/Lib/multiprocessing/managers.py#L51 (took me a while to figure out)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
|
Diff from mypy_primer, showing the effect of this PR on open source code: optuna (https://github.com/optuna/optuna)
+ tests/samplers_tests/tpe_tests/test_sampler.py:1044: error: Incompatible types in assignment (expression has type "DictProxy[Any, Any]", variable has type "Dict[int, List[int]]")
+ tests/samplers_tests/tpe_tests/test_sampler.py:1045: error: Incompatible types in assignment (expression has type "DictProxy[Any, Any]", variable has type "Dict[int, int]")
|
#7928
dict()andlist()just return empty dictionaries and lists (respectively) if no arguments are supplied: