Skip to content

Adds TypeGuard to filter #6140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

Adds TypeGuard to filter #6140

wants to merge 3 commits into from

Conversation

sobolevn
Copy link
Member

@sobolevn sobolevn commented Oct 9, 2021

Couple of questsions:

  1. How can I test that it works?

I was testing it with local code:

from typing import Iterator, Iterable, Callable, TypeVar, Generic, TypeGuard, Any, List, Optional, overload

_T = TypeVar('_T')
R = TypeVar('R')


class filter(Iterator[_T], Generic[_T]):
    @overload
    def __init__(self, __function: None, __iterable: Iterable[_T | None]) -> None: ...
    @overload
    def __init__(self, __function: Callable[[R], TypeGuard[_T]], __iterable: Iterable[R]) -> None: ...
    @overload
    def __init__(self, __function: Callable[[_T], Any], __iterable: Iterable[_T]) -> None: ...
    def __iter__(self) -> Iterator[_T]: ...
    def __next__(self) -> _T: ...

a: List[Optional[int]]
def is_int_typeguard(a: object) -> TypeGuard[int]: pass
cc = filter(is_int_typeguard, a)
reveal_type(cc)  # N: Revealed type is "ex.filter[Union[builtins.int, None]]"
  1. This is the first TypeGuard in builtins.pyi, are there any extra steps?

Closes #5661

@github-actions

This comment has been minimized.

Sign in to view
@JelleZijlstra
Copy link
Member

From the mypy-primer output, this seems to expose some bugs in mypy's TypeGuard implementation.

Also you can test with mypy's --custom-typeshed-dir option.

@sobolevn
Copy link
Member Author

Almost all of the errors above happen because when calling filter(returns_bool, items) with def returns_bool(a) -> bool, it matches def __init__(self, __function: Callable[[R], TypeGuard[_T]], __iterable: Iterable[R]) -> None: ... overload for some reason. Look like a bug in mypy to me.

And that's why inference is broken.

@hauntsaninja
Copy link
Collaborator

IIRC at least at some point mypy treated TypeGuard exactly the same as bool, instead of as a subtype. Could be a pretty easy fix.

(Also worth noting that in typeshed's CI, we use latest mypy release instead of mypy master)

@sobolevn
Copy link
Member Author

Also worth noting that in typeshed's CI, we use latest mypy release instead of mypy master

The same happens with master, I've just checked.
I will open a new issue in mypy. I hopefully will be able to fix it. 👍

@sobolevn sobolevn mentioned this pull request Oct 10, 2021
Closed
@sobolevn
Copy link
Member Author

Done: python/mypy#11307

@sobolevn sobolevn mentioned this pull request Oct 11, 2021
Closed
@sobolevn
Copy link
Member Author

@hauntsaninja @JelleZijlstra any chance to get this PR merged, please? python/mypy#11314 🙂

JelleZijlstra
JelleZijlstra approved these changes Nov 7, 2021
View reviewed changes
Copy link
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mypy PR got merged, thanks!

This looks good but I'm going to retrigger CI just in case.

@JelleZijlstra JelleZijlstra reopened this Nov 7, 2021
@github-actions

This comment has been minimized.

Sign in to view
@JelleZijlstra
Copy link
Member

I'm a bit worried about the mypy-primer results here, I'll have to look at it more.

@github-actions

This comment has been minimized.

Sign in to view
@JelleZijlstra
Copy link
Member

Looking at the freqtrade ones:

  • the first one does len(list(filter(lambda x: x["name"] == searchname, columns))) == 1 where columns: List (they got to turn on --disallow-any-generics). It produces Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]". That error makes little sense to me.
  • The others are around a line roi_list = list(filter(lambda x: x <= trade_dur, self.minimal_roi.keys())), where self.minimal_roi: Dict.
  • So maybe there's a bug in mypy around using Dict/List without generic arguments.

This was referenced Nov 18, 2021
Open
Closed
@JelleZijlstra
Copy link
Member

Let's see if mypy got any better.

@JelleZijlstra JelleZijlstra reopened this Dec 21, 2021
@github-actions
Copy link
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

zulip (https://github.com/zulip/zulip)
+ zerver/views/documentation.py:244: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]

jax (https://github.com/google/jax)
+ jax/core.py:798: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]
+ jax/core.py:799: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]
+ jax/experimental/host_callback.py:983: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]
+ jax/experimental/host_callback.py:1030: error: Argument 1 to "aval_to_xla_shapes" has incompatible type "_T"; expected "AbstractValue"  [arg-type]
+ jax/experimental/jax2tf/impl_no_xla.py:323: error: Argument 1 to "join" of "str" has incompatible type "List[bool]"; expected "Iterable[str]"  [arg-type]
+ jax/experimental/jax2tf/tests/tf_test_util.py:220: error: Argument 1 to "tuple" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]

aiortc (https://github.com/aiortc/aiortc)
+ src/aiortc/sdp.py:294: error: Incompatible return value type (got "_T", expected "RTCRtpCodecParameters")
+ src/aiortc/sdp.py:294: error: Argument 1 to "next" has incompatible type "filter[_T]"; expected "SupportsNext[_T]"
+ src/aiortc/rtcpeerconnection.py:66: error: "_T" has no attribute "mimeType"
+ src/aiortc/rtcpeerconnection.py:67: error: "_T" has no attribute "parameters"
+ src/aiortc/rtcpeerconnection.py:74: error: "bool" has no attribute "parameters"
+ src/aiortc/rtcpeerconnection.py:75: error: Argument 1 to "append" of "list" has incompatible type "bool"; expected "RTCRtpCodecParameters"
+ src/aiortc/rtcpeerconnection.py:657: error: Incompatible types in assignment (expression has type "_T", variable has type "RTCRtpTransceiver")

optuna (https://github.com/optuna/optuna)
+ optuna/cli.py:555: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ optuna/cli.py:556: error: Argument 1 to "_format_output" has incompatible type "List[_T]"; expected "Union[List[Dict[Tuple[str, str], Any]], Dict[Tuple[str, str], Any]]"
+ optuna/importance/_base.py:110: error: Incompatible return value type (got "OrderedDict[str, str]", expected "Dict[str, BaseDistribution]")
+ optuna/importance/_base.py:114: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ optuna/importance/_base.py:134: error: "_T" has no attribute "distributions"

freqtrade (https://github.com/freqtrade/freqtrade)
+ freqtrade/persistence/migrations.py:15: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ freqtrade/strategy/interface.py:808: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ freqtrade/strategy/interface.py:811: error: Value of type variable "SupportsRichComparisonT" of "max" cannot be "_T"
+ freqtrade/strategy/interface.py:812: error: Incompatible return value type (got "Tuple[_T, Any]", expected "Tuple[Optional[int], Optional[float]]")

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/ext/intersphinx.py: note: In function "_resolve_reference_in_domain_by_target":
+ sphinx/ext/intersphinx.py:308:40: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ sphinx/ext/intersphinx.py:311:43: error: Invalid index type "_T" for "Dict[str, Tuple[str, str, str, str]]"; expected type "str"

edgedb (https://github.com/edgedb/edgedb)
+ edb/schema/utils.py:787:9: error: List comprehension has incompatible type List[Tuple[_T, int]]; expected List[Tuple[Object, int]]
+ edb/schema/utils.py:787:66: error: Argument 2 to "get_nq_name" has incompatible type "_T"; expected "Object"
+ edb/schema/utils.py:793:20: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ edb/schema/utils.py:799:13: error: Value of type "object" is not indexable
+ edb/schema/utils.py:800:37: error: Value of type "object" is not indexable
+ edb/schema/utils.py:801:13: error: Value of type "object" is not indexable
+ edb/schema/utils.py:805:13: error: Value of type "_T" is not indexable

rclip (https://github.com/yurijmikhalevich/rclip)
+ rclip/main.py:138: error: Value of type "_T" is not indexable

rotki (https://github.com/rotki/rotki)
+ rotkehlchen/chain/ethereum/interfaces/ammswap/ammswap.py:719: error: Incompatible types in assignment (expression has type "_T", variable has type "ChecksumAddress")
+ rotkehlchen/chain/ethereum/modules/sushiswap/sushiswap.py:143: error: Incompatible types in assignment (expression has type "_T", variable has type "ChecksumAddress")
+ rotkehlchen/chain/ethereum/modules/uniswap/uniswap.py:186: error: Incompatible types in assignment (expression has type "_T", variable has type "ChecksumAddress")
+ rotkehlchen/chain/ethereum/modules/uniswap/uniswap.py:296: error: Incompatible types in assignment (expression has type "_T", variable has type "ChecksumAddress")
+ rotkehlchen/externalapis/covalent.py:214: error: Argument 1 to "convert_transaction_from_covalent" has incompatible type "bool"; expected "Dict[str, Any]"
+ rotkehlchen/chain/avalanche/manager.py:67: error: Value of type "bool" is not indexable

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/csv/__init__.py:78: error: "_T" has no attribute "name"
+ ibis/backends/csv/__init__.py:82: error: "_T" has no attribute "schema"
+ ibis/backends/csv/__init__.py:91: error: "_T" has no attribute "schema"
+ ibis/backends/csv/__init__.py:92: error: Dict entry 0 has incompatible type "_T": "Any"; expected "Node": "Any"

Tanjun (https://github.com/FasterSpeeding/Tanjun)
+ tanjun/conversion.py:646: error: Incompatible types in "yield from" (actual type "bool", expected type "Snowflake")
+ tanjun/conversion.py:650: error: Incompatible types in "yield from" (actual type "bool", expected type "Snowflake")

graphql-core (https://github.com/graphql-python/graphql-core)
+ src/graphql/utilities/print_schema.py:60: error: Argument 1 to "map" has incompatible type "Callable[[GraphQLDirective], str]"; expected "Callable[[bool], str]"
+ src/graphql/utilities/print_schema.py:61: error: Argument 1 to "map" has incompatible type "Callable[[GraphQLNamedType], str]"; expected "Callable[[bool], str]"
+ src/graphql/validation/rules/provided_required_arguments.py:52: error: "bool" has no attribute "name"
+ src/graphql/validation/rules/provided_required_arguments.py:52: error: Value expression in dictionary comprehension has incompatible type "bool"; expected type "Union[GraphQLArgument, InputValueDefinitionNode]"

spark (https://github.com/apache/spark)
+ python/pyspark/pandas/frame.py:2232: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]
+ python/pyspark/pandas/frame.py:2237: error: Argument 1 to "loads" has incompatible type "_T"; expected "Union[str, bytes]"  [arg-type]
+ python/pyspark/pandas/frame.py:2245: error: Argument 2 to "scol_for" has incompatible type "_T"; expected "str"  [arg-type]

core (https://github.com/home-assistant/core)
+ homeassistant/components/picnic/coordinator.py:97: error: Argument 1 to "next" has incompatible type "filter[_T]"; expected "SupportsNext[Dict[Any, Any]]"  [arg-type]
+ homeassistant/components/fitbit/sensor.py:408: error: Incompatible types in assignment (expression has type "_T", variable has type "Optional[Dict[str, str]]")  [assignment]
+ homeassistant/components/fitbit/sensor.py:409: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]

prefect (https://github.com/PrefectHQ/prefect)
+ src/prefect/core/flow.py:291: error: Argument 1 to "list" has incompatible type "filter[bool]"; expected "Iterable[Task]"
+ src/prefect/core/flow.py:1111: error: "_T" has no attribute "map_states"
+ src/prefect/core/flow.py:1114: error: "_T`2727" object is not iterable
+ src/prefect/core/flow.py:1148: error: "_T" has no attribute "is_cached"
+ src/prefect/core/flow.py:1150: error: "_T" has no attribute "is_mapped"
+ src/prefect/core/flow.py:1151: error: "_T" has no attribute "map_states"
+ src/prefect/core/flow.py:1155: error: "_T" has no attribute "map_states"
+ src/prefect/core/flow.py:1159: error: Incompatible types in assignment (expression has type "List[<nothing>]", variable has type "List[_T]")
+ src/prefect/core/flow.py:1159: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/prefect/core/flow.py:1159: note: Consider using "Sequence" instead, which is covariant

porcupine (https://github.com/Akuli/porcupine)
+ porcupine/settings.py:916: error: Incompatible return value type (got "List[bool]", expected "List[str]")  [return-value]

dragonchain (https://github.com/dragonchain/dragonchain)
+ dragonchain/lib/dao/api_key_dao.py:89:25: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ dragonchain/lib/dao/api_key_dao.py:90:28: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ dragonchain/lib/dao/api_key_dao.py:93:78: error: Argument 1 to "get_json_from_object" has incompatible type "_T"; expected "str"
+ dragonchain/lib/dao/api_key_dao.py:95:5: error: Incompatible types in assignment (expression has type "_T", variable has type "_T")
+ dragonchain/lib/dao/api_key_dao.py:97:27: error: Value of type "_T" is not indexable
+ dragonchain/lib/dao/api_key_dao.py:97:31: error: "_T" has no attribute "find"
+ dragonchain/lib/dao/api_key_dao.py:98:78: error: Argument 1 to "get_json_from_object" has incompatible type "_T"; expected "str"

@JelleZijlstra
Copy link
Member

Still no good. I'm guessing this is also affected by python/mypy#11428. We can reopen this once mypy's TypeGuard support improves.

@layday layday mentioned this pull request Dec 28, 2021
Closed
@Jasha10 Jasha10 mentioned this pull request Dec 29, 2021
Merged
GRBurst added a commit to GRBurst/slack-logger-python that referenced this pull request Jul 17, 2023
@GRBurst
chore: rewrite filter to list comprehension to satisfy mypy
1d9ca12 
seems to be a bug in mypy, see python/typeshed#6140

Signed-off-by: GRBurst <GRBurst@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JelleZijlstra JelleZijlstra JelleZijlstra approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Make filter() use TypeGuard
3 participants
@sobolevn @JelleZijlstra @hauntsaninja