Skip to content

Fix Match.group problems #3172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 8, 2019
Merged

Fix Match.group problems #3172

merged 5 commits into from
Aug 8, 2019

Conversation

srittau
Copy link
Collaborator

@srittau srittau commented Aug 7, 2019

  • group() may return None values
  • group() allows to mix int and str arguments
  • Use concrete return type
  • Merge overloads

Fixes #3171

Returning Optional is potentially problematic. It is often obvious from a regexp whether a certain group exists (every time it is non-optional). Many users may rely on group() not returning an Optional. Nevertheless, returning AnyStr is clearly wrong. One alternative is returning Any, but that would lose any type checking ability. Personally, I believe this patch is the best solution, despite the problems it may cause.

* group() may return None values
* group() allows to mix int and str arguments
* Use concrete return type
* Merge overloads
@JelleZijlstra
Copy link
Member

Why do you say it is "clearly wrong" to return AnyStr? As far as I can tell, the only problem with that is that we don't handle the None case. I can see arguments either way on that one, but it's probably best to err on the side of type safety here and include None in the return type.

Could you also make the arguments positional-only?

  File "<ipython-input-44-e82d21244346>", line 1, in <module>
    m.group(group1='1')
TypeError: group() takes no keyword arguments

@srittau
Copy link
Collaborator Author

srittau commented Aug 7, 2019

Well, "clearly wrong" -> "not type-safe in the general case". Will change them to positional only.

@@ -525,10 +525,16 @@ class Match(Generic[AnyStr]):

def expand(self, template: AnyStr) -> AnyStr: ...

def group(self, __group: Literal[0] = ...) -> AnyStr: ...
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing the @overload decorator

@srittau srittau merged commit 3f6cec9 into python:master Aug 8, 2019
@srittau srittau deleted the match-group branch August 8, 2019 14:59
@ilevkivskyi
Copy link
Member

This causes 19 errors in mypy self-check and 45 errors in the Python 3 part of our internal code base (which is actually relatively small).

TBH, I am not sure the Optional part in the second overload is really worth it. What do you think about (partially) reverting this?

@srittau
Copy link
Collaborator Author

srittau commented Aug 16, 2019

I wonder why our tests did not catch this, since the mypy self-test is run as part of the build. I'm open to changing it back, although just removing Optional can problems in theory as well. For example, a code analysis tool making use of type annotations could complain about the "impossible" branch in code like this:

if m.group(12) is None:
    ...

While I don't think such a tool exists yet, I'd prefer to use Any if we decide to change it.

@ilevkivskyi
Copy link
Member

Actually I was wrong about number of errors internally, it is not 45 it is 110. So I think it makes sense reverting this in some form.

As for the --warn-unreachable problem, this problem indeed exists, but as practice shows, it is still quite rare with the old signature. For example it caused only couple errors in mypy itself. So I think reverting to AnyStr is a safe bet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Wrong return type for Match.group()
3 participants