Skip to content

Document how to write test-cases for type annotations using mypy (in subprocess) #11506

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
sirosen opened this issue Nov 9, 2021 · 10 comments · Fixed by python/typing#1071
Closed

Document how to write test-cases for type annotations using mypy (in subprocess) #11506

sirosen opened this issue Nov 9, 2021 · 10 comments · Fixed by python/typing#1071
Labels
documentation

Comments

@sirosen
Copy link

sirosen commented Nov 9, 2021

Documentation

When writing a python package which publishes type information, it's very easy to mess up the annotations you present to your users. For example, a decorator can easily lose type information if written carelessly:

def mydeco(f: Callable) -> Callable:
    ...

The solution, "obviously", is to test your annotations. But how?
Although there are several packages which add pytest-mypy integrations, there is no guidance from mypy.

I propose the addition of a new doc on "testing package annotations" in the miscellaneous section, which demonstrates a simple test helper which writes provided code to files and runs mypy on those files, for use with test suites.

To seed things, here's what I wrote recently to do this:

import os
import re

import pytest

# mypy's only python-callable API
# see:
# https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application
from mypy import api as mypy_api


@pytest.fixture
def _in_tmp_path(tmp_path):
    cur = os.getcwd()
    try:
        os.chdir(tmp_path)
        yield
    finally:
        os.chdir(cur)


@pytest.fixture
def run_reveal_type(tmp_path, _in_tmp_path):
    content_path = tmp_path / "reveal_type_test.py"

    def func(code_snippet, *, preamble: str = ""):
        content_path.write_text(preamble + f"reveal_type({code_snippet})")
        result = mypy_api.run(["reveal_type_test.py"])
        stdout = result[0]
        # mypy reveal_type shows notes like
        #   note: Revealed type is "Union[builtins.str, None]"
        # find and extract with a regex
        match = re.search(r'note: Revealed type is "([^"]+)"', stdout)
        assert match is not None
        return match.group(1)

    return func

I'm leveraging pytest's fixture system here, but I'd be happy to write the necessary tmpdir bits to make this compatible with unittest or other frameworks.

I would be enthusiastic about documenting this and putting together a PR, provided that the maintainers here like the idea (and find the proposed testing helper to document acceptable).
@erictraut
Copy link

Thanks for the suggestion. Improving the quality of type information in packages benefits everyone, and investments in tools and documentation are welcome!

A few additional thoughts...

There has recently been a push to consolidate type-related documentation in the python/typing repo. The documentation that you're proposing might make more sense to post there. I guess it depends on how mypy-specific it is.

In addition to testing the correctness of type information, there's also the issue of "type completeness" — i.e. is a package missing type information for portions of its interface? If you're interested in validating both the correctness and completeness of your package's type annotations, I added a feature to pyright that you might be interested in checking out. For details, refer to this documentation.

@sirosen
Copy link
Author

sirosen commented Nov 9, 2021

I wasn't aware of the shift towards docs on the typing repo. I think even though the current doc that I'd submit is mypy-specific, putting it there leaves room for the doc to expand to cover pyright, pyre, etc.

Do you think it makes sense to convert the doc on "type completeness" to a "testing type information" doc, and make completeness and inspection of specific types two separate sections?

@erictraut
Copy link

@srittau is the main person behind the recent improvements to the typing repo. I think he has a vision for the layout of the docs, so he'd be in the best position to answer your question.

From my perspective, I think it might be preferable to have one document that provides guidance to library authors about what is expected for typing within a "py.typed" package and a separate document that discusses tools, tests, automation, and other best practices. If others agree with that split, we could move the section about pyright's "--verifytypes" feature out of the existing document and into the new document that focuses on tools & tests.

@sobolevn
Copy link
Member

sobolevn commented Nov 9, 2021

As the main maintainer of https://github.com/typeddjango/pytest-mypy-plugins I am totally interested in helping with this doc 🙂

@srittau
Copy link
Contributor

srittau commented Nov 9, 2021

Just briefly (I haven't read through all comments here yet): I suggested a documentation structure in python/typing#845. @shannonzhu also provided an initial PR for a structure in python/typing#919 (which I also haven't found time to look at yet). I agree with Eric that we should generally split general typing documentation and type checker specific documentation. Two exception that come to mind: Tutorials should pick a type checker (most likely mypy) for experimentation and we could point out specific type checker quirks as footnotes or in callout boxes, if necessary.

@hauntsaninja
Copy link
Collaborator

Not quite what you're asking for, but mypy has a tool called stubtest that is pretty good at testing that type annotations match runtime objects. It's particularly useful if you're bundling pyi files, since those have a tendency to diverge.

If possible, running mypy with stricter settings on package code is also useful (e.g. --disallow-any-generics would complain about your example)

In general, what you're suggesting is very similar to how mypy's tests run. Could be great to try and factor out the relevant logic from there.

@sirosen
Copy link
Author

sirosen commented Nov 10, 2021 •
edited
Loading

Thanks everyone for thoughts and ideas! It sounds like this would come together as a page with a structure like

Tools for Working with Type Annotations
- Generating Annotations
  - stubgen
  - monkeytype? others?
- Testing Annotation Accuracy
  - reveal_type + subprocess
  - pytest-mypy-plugins
  - mypy stubtest
- Annotation Coverage
  - pyright --verifytypes
  - mypy --html-report / --txt-report

I can write a first draft sometime soon -- and I'll keep it all brief to make review and incremental improvements easier. But I'm fine with waiting for python/typing#919 to land before starting on this if that simplifies things.

@hauntsaninja
Copy link
Collaborator

Some prior art:
https://mypy.readthedocs.io/en/latest/stubgen.html
https://mypy.readthedocs.io/en/latest/installed_packages.html#creating-pep-561-compatible-packages

@ikonst
Copy link
Contributor

ikonst commented Nov 11, 2021

Similar approaches I've seen / taken:

The last case provides somewhat weaker guarantees, but I like the elegance of needing no additional tooling. For example, the following ensures that a decorator isn't a typing black hole:

@my_decorator()
def f(a: int) -> str:
    return str(a)

f.warm_up(42)
f.warm_up("42")  # type: ignore [arg-type]
f.warm_up(b=42)  # type: ignore [call-arg]
r1: str = f(42)
r2: int = f(42)  # type: ignore [assignment]

@sirosen sirosen mentioned this issue Dec 21, 2021
Merged
sirosen added a commit to sirosen/typing that referenced this issue Feb 12, 2022
@sirosen
Document how to test type annotations
9f1abd0 
This is a slightly broader refactor than just testing. It also
consolidates information about checking type coverage/completeness.

This originates from a thread on the mypy tracker [1]. In terms of
presentation, the goal is to present guidance and offer up several
options, many of which were proposed by contributors to that thread.

Several of the goals from that thread were not achieved here,
including documentation covering stubgen and monkeytype, stubtest, and
potentially more. However, the document is written such that it should
be possible to add a section on "Generating Annotations" as was
planned earlier.

[1]: python/mypy#11506
sirosen added a commit to sirosen/typing that referenced this issue Feb 12, 2022
@sirosen
Document how to test type annotations
86ee99c 
This is a slightly broader refactor than just testing. It also
consolidates information about checking type coverage/completeness.

This originates from a thread on the mypy tracker [1]. In terms of
presentation, the goal is to present guidance and offer up several
options, many of which were proposed by contributors to that thread.

Several of the goals from that thread were not achieved here,
including documentation covering stubgen and monkeytype, stubtest, and
potentially more. However, the document is written such that it should
be possible to add a section on "Generating Annotations" as was
planned earlier.

[1]: python/mypy#11506
@sirosen sirosen mentioned this issue Feb 12, 2022
Merged
srittau pushed a commit to python/typing that referenced this issue Feb 21, 2022
@sirosen
Document how to test type annotations (#1071)
02d79c2 
This is a slightly broader refactor than just testing. It also
consolidates information about checking type coverage/completeness.

This originates from a thread on the mypy tracker [1]. In terms of
presentation, the goal is to present guidance and offer up several
options, many of which were proposed by contributors to that thread.

Several of the goals from that thread were not achieved here,
including documentation covering stubgen and monkeytype, stubtest, and
potentially more. However, the document is written such that it should
be possible to add a section on "Generating Annotations" as was
planned earlier.

[1]: python/mypy#11506
@sirosen
Copy link
Author

sirosen commented Feb 21, 2022

python/typing#1071 just merged, so there's now a doc for this! 🎉

@sirosen sirosen closed this as completed Feb 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Document how to test type annotations sirosen/typing
6 participants
@srittau @ikonst @sirosen @sobolevn @erictraut @hauntsaninja