builtins: Audit bytes arguments #7631
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As a followup from python#7589 (comment), I audited all occurrences of bytes in builtins.pyi by reading the corresponding C code on CPython main. Most use the C buffer protocol, so _typeshed.ReadableBuffer is the right type. A few check specifically for bytes and bytearray.
JelleZijlstra
commented
Apr 16, 2022
| @@ -200,7 +200,7 @@ _NegativeInteger = Literal[-1, -2, -3, -4, -5, -6, -7, -8, -9, -10, -11, -12, -1 | |||
|
|
|||
| class int: | |||
| @overload | |||
| def __new__(cls: type[Self], __x: str | bytes | SupportsInt | SupportsIndex | SupportsTrunc = ...) -> Self: ... | |||
| def __new__(cls: type[Self], __x: str | ReadableBuffer | SupportsInt | SupportsIndex | SupportsTrunc = ...) -> Self: ... | |||
| @overload | |||
| def __new__(cls: type[Self], __x: str | bytes | bytearray, base: SupportsIndex) -> Self: ... | |||
There was a problem hiding this comment.
>>> int(memoryview(b"0xdeadbeef"), 16)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: int() can't convert non-string with explicit base
>>> int(memoryview(b"123"))
123
Showing that the first overload accepts buffers but the second doesn't.
There was a problem hiding this comment.
| @@ -223,7 +223,7 @@ class int: | |||
| @classmethod | |||
| def from_bytes( | |||
| cls: type[Self], | |||
| bytes: Iterable[SupportsIndex] | SupportsBytes, # TODO buffer object argument | |||
| bytes: Iterable[SupportsIndex] | SupportsBytes | ReadableBuffer, | |||
There was a problem hiding this comment.
>>> int.from_bytes([1, 2, 3])
66051
>>> int.from_bytes(memoryview(b"123"))
3224115
| ) -> bool: ... | ||
| if sys.version_info >= (3, 8): | ||
| def expandtabs(self, tabsize: SupportsIndex = ...) -> bytes: ... | ||
| else: | ||
| def expandtabs(self, tabsize: int = ...) -> bytes: ... | ||
|
|
||
| def find( | ||
| self, __sub: bytes | SupportsIndex, __start: SupportsIndex | None = ..., __end: SupportsIndex | None = ... | ||
| self, __sub: ReadableBuffer | SupportsIndex, __start: SupportsIndex | None = ..., __end: SupportsIndex | None = ... | ||
| ) -> int: ... | ||
| if sys.version_info >= (3, 8): | ||
| def hex(self, sep: str | bytes = ..., bytes_per_sep: SupportsIndex = ...) -> str: ... |
There was a problem hiding this comment.
>>> b"xy".hex(memoryview(b"x"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: sep must be str or bytes.
>>> b"xy".hex(bytearray(b"x"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: sep must be str or bytes.
| def __len__(self) -> int: ... | ||
| def __iter__(self) -> Iterator[int]: ... | ||
| def __hash__(self) -> int: ... | ||
| @overload | ||
| def __getitem__(self, __i: SupportsIndex) -> int: ... | ||
| @overload | ||
| def __getitem__(self, __s: slice) -> bytes: ... | ||
| def __add__(self, __s: bytes) -> bytes: ... | ||
| def __add__(self, __s: ReadableBuffer) -> bytes: ... |
There was a problem hiding this comment.
>>> b"x" + memoryview(b"y")
b'xy'
| @@ -667,14 +679,14 @@ class bytearray(MutableSequence[int], ByteString): | |||
| @overload | |||
| def __setitem__(self, __s: slice, __x: Iterable[SupportsIndex] | bytes) -> None: ... | |||
| def __delitem__(self, __i: SupportsIndex | slice) -> None: ... | |||
| def __add__(self, __s: bytes) -> bytearray: ... | |||
| def __iadd__(self: Self, __s: Iterable[int]) -> Self: ... | |||
There was a problem hiding this comment.
This was wrong; ba += [1, 2, 3] fails
| @@ -1352,7 +1364,7 @@ def open( | |||
| closefd: bool = ..., | |||
| opener: _Opener | None = ..., | |||
| ) -> IO[Any]: ... | |||
| def ord(__c: str | bytes) -> int: ... | |||
| def ord(__c: str | bytes | bytearray) -> int: ... | |||
There was a problem hiding this comment.
>>> ord(memoryview(b"x"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: ord() expected string of length 1, but memoryview found
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
1 similar comment
|
According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉 |
srittau
approved these changes
Apr 16, 2022
| def join(self, __iterable_of_bytes: Iterable[ByteString | memoryview]) -> bytes: ... | ||
| def ljust(self, __width: SupportsIndex, __fillchar: bytes = ...) -> bytes: ... | ||
| def join(self, __iterable_of_bytes: Iterable[ReadableBuffer]) -> bytes: ... | ||
| def ljust(self, __width: SupportsIndex, __fillchar: bytes | bytearray = ...) -> bytes: ... |
There was a problem hiding this comment.
Unfortunately, this will also accept memoryview at the moment, but having it more explicit can't hurt.
There was a problem hiding this comment.
That's a mypy bug :)
There was a problem hiding this comment.
It's working as documented. In the past when reviewing I've always asked people to remove bytearray from argument types due to that.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As a followup from #7589 (comment),
I audited all occurrences of bytes in builtins.pyi by reading the corresponding C code
on CPython main.
Most use the C buffer protocol, so _typeshed.ReadableBuffer is the right type. A few
check specifically for bytes and bytearray.