From 86ee99c09a760b9901723a9b8ba7ffbe8ab54afe Mon Sep 17 00:00:00 2001 From: Stephen Rosen Date: Sat, 12 Feb 2022 18:03:16 +0000 Subject: [PATCH 1/2] Document how to test type annotations This is a slightly broader refactor than just testing. It also consolidates information about checking type coverage/completeness. This originates from a thread on the mypy tracker [1]. In terms of presentation, the goal is to present guidance and offer up several options, many of which were proposed by contributors to that thread. Several of the goals from that thread were not achieved here, including documentation covering stubgen and monkeytype, stubtest, and potentially more. However, the document is written such that it should be possible to add a section on "Generating Annotations" as was planned earlier. [1]: https://github.com/python/mypy/issues/11506 --- docs/source/libraries.rst | 27 ----- docs/source/reference.rst | 1 + docs/source/tools.rst | 207 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 208 insertions(+), 27 deletions(-) create mode 100644 docs/source/tools.rst diff --git a/docs/source/libraries.rst b/docs/source/libraries.rst index 371c1f74..96e5370e 100644 --- a/docs/source/libraries.rst +++ b/docs/source/libraries.rst @@ -282,33 +282,6 @@ Examples of known and unknown types class DictSubclass(dict): pass -Verifying Type Completeness -=========================== - -Some type checkers provide features that allows library authors to verify type -completeness for a “py.typed” package. E.g. Pyright has a special -`command line flag `_ for this. - -Improving Type Completeness -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Here are some tips for increasing the type completeness score for your -library: - -- If your package includes tests or sample code, consider removing them - from the distribution. If there is good reason to include them, - consider placing them in a directory that begins with an underscore - so they are not considered part of your library’s interface. -- If your package includes submodules that are meant to be - implementation details, rename those files to begin with an - underscore. -- If a symbol is not intended to be part of the library’s interface and - is considered an implementation detail, rename it such that it begins - with an underscore. It will then be considered private and excluded - from the type completeness check. -- If your package exposes types from other libraries, work with the - maintainers of these other libraries to achieve type completeness. - Best Practices for Inlined Types ================================ diff --git a/docs/source/reference.rst b/docs/source/reference.rst index d18df3aa..ba8ddab2 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -7,6 +7,7 @@ Type System Reference :caption: Contents: stubs + tools typing Module Documentation .. The following pages are desired in a new TOC which will cover multiple diff --git a/docs/source/tools.rst b/docs/source/tools.rst new file mode 100644 index 00000000..ca86947c --- /dev/null +++ b/docs/source/tools.rst @@ -0,0 +1,207 @@ +.. _tools: + +*************************************** +Tools for Working With Type Annotations +*************************************** + +Testing Annotation Accuracy +=========================== + +When creating a package with type annotations, authors may want to validate +that the annotations they publish meet their expectations. +This is especially important for library authors, for whom the published +annotations are part of the public interface to their package. + +There are several approaches to this problem, and this document will show +a few of them. + +.. note:: + + For simplicity, we will assume that type-checking is done with ``mypy``. + Many of these strategies can be applied to other type-checkers as well. + +Testing Using ``mypy --warn-unused-ignores`` +-------------------------------------------- + +Clever use of the ``--warn-unused-ignores`` can be used to check that certain +expressions are or are not well-typed. + +The idea is to write normal python files which contain valid expressions along +with invalid expressions annotated with ``type: ignore`` comments. When +``mypy --warn-unused-ignores`` is run on these files, it should pass. +A directory of test files, ``typing_tests/``, can be maintained. + +This strategy does not offer strong guarantees about the types under test, but +it requires no additional tooling. + +If the following file is under test + +.. code-block:: python + + # foo.py + def bar(x: int) -> str: + return str(x) + +Then the following file tests ``foo.py``: + +.. code-block:: python + + bar(42) + bar("42") # type: ignore [arg-type] + bar(y=42) # type: ignore [call-arg] + r1: str = bar(42) + r2: int = bar(42) # type: ignore [assignment] + +Checking ``reveal_type`` output from ``mypy.api.run`` +----------------------------------------------------- + +``mypy`` provides a subpackage named ``api`` for invoking ``mypy`` from a +python process. In combination with ``reveal_type``, this can be used to write +a function which gets the ``reveal_type`` output from an expression. Once +that's obtained, tests can assert strings and regular expression matches +against it. + +This approach requires writing a set of helpers to provide a good testing +experience, and it runs mypy once per test case (which can be slow). +However, it builds only on ``mypy`` and the test framework of your choice. + +The following example could be integrated into a testsuite written in +any framework: + +.. code-block:: python + + import re + from mypy import api + + def get_reveal_type_output(filename): + result = api.run([filename]) + stdout = result[0] + match = re.search(r'note: Revealed type is "([^"]+)"', stdout) + assert match is not None + return match.group(1) + + +For example, we can use the above to provide a ``run_reveal_type`` pytest +fixture which generates a temporary file and uses it as the input to +``get_reveal_type_output``: + +.. code-block:: python + + import os + import pytest + + @pytest.fixture + def _in_tmp_path(tmp_path): + cur = os.getcwd() + try: + os.chdir(tmp_path) + yield + finally: + os.chdir(cur) + + @pytest.fixture + def run_reveal_type(tmp_path, _in_tmp_path): + content_path = tmp_path / "reveal_type_test.py" + + def func(code_snippet, *, preamble = ""): + content_path.write_text(preamble + f"reveal_type({code_snippet})") + return get_reveal_type_output("reveal_type_test.py") + + return func + +pytest-mypy-plugins +------------------- + +``pytest-mypy-plugins`` [#pytest-mypy-plugins]_ is a plugin for ``pytest`` +which defines typing test cases as YAML data. +The test cases are run through ``mypy`` and the output of ``reveal_type`` can +be asserted. + +This project supports complex typing arrangements like ``pytest`` parametrized +tests and per-test ``mypy`` configuration. It requires that you are using +``pytest`` to run your tests, and runs ``mypy`` in a subprocess per test case. + +This is an example of a parametrized test with ``pytest-mypy-plugins``: + +.. code-block:: yaml + + - case: with_params + parametrized: + - val: 1 + rt: builtins.int + - val: 1.0 + rt: builtins.float + main: | + reveal_type({[ val }}) # N: Revealed type is '{{ rt }}' + +Improving Type Completeness +=========================== + +One of the goals of many libraries is to ensure that they are "fully type +annotated", meaning that they provide complete and accurate type annotations +for all functions, classes, and objects. Having full annotations is referred to +as "type completeness" or "type coverage". + +Here are some tips for increasing the type completeness score for your +library: + +- Make type completeness an output of your testing process. Several type + checkers have options for generating useful output, warnings, or even + reports. +- If your package includes tests or sample code, consider removing them + from the distribution. If there is good reason to include them, + consider placing them in a directory that begins with an underscore + so they are not considered part of your library’s interface. +- If your package includes submodules that are meant to be + implementation details, rename those files to begin with an + underscore. +- If a symbol is not intended to be part of the library’s interface and + is considered an implementation detail, rename it such that it begins + with an underscore. It will then be considered private and excluded + from the type completeness check. +- If your package exposes types from other libraries, work with the + maintainers of these other libraries to achieve type completeness. + +.. warning:: + + The ways in which different type checkers evaluate and help you achieve + better type coverage may differ. Some of the above recommendations may or + may not be helpful to you, depending on which type checking tools you use. + +``mypy`` disallow options +------------------------- + +``mypy`` offers several options which can detect untyped code. +More details can be found in the +mypy documentation on these options [#mypy-untyped-checks]_. + +Some basic usages which make ``mypy`` error on untyped data are:: + + mypy --disallow-untyped-defs + mypy --disallow-incomplete-defs + +``pyright`` type verification +----------------------------- + +pyright has a special command line flag, ``--verifytypes``, for verifying +type completeness. You can learn more about it from the +pyright documentation on verifying type completeness [#pyright-verifytypes]_. + +``mypy`` reports +---------------- + +``mypy`` offers several options options for generating reports on its analysis. +See the mypy documentation on report generation [#mypy-report]_ for details. + +References +========== + +.. [#mypy-report] mypy reporting options (https://mypy.readthedocs.io/en/stable/command_line.html#report-generation) + +.. [#mypy.api] mypy.api -- the python interface to mypy (https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application) + +.. [#mypy-untyped-checks] mypy CLI untyped checks (https://mypy.readthedocs.io/en/latest/command_line.html#untyped-definitions-and-calls) + +.. [#pyright-verifytypes] pyright --verifytypes (https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#verifying-type-completeness) + +.. [#pytest-mypy-plugins] pytest-mypy-plugins (https://github.com/typeddjango/pytest-mypy-plugins) From f237eef26c4940b9f4c91b482c12d3727d2a42d9 Mon Sep 17 00:00:00 2001 From: Stephen Rosen Date: Mon, 21 Feb 2022 18:18:33 +0000 Subject: [PATCH 2/2] Update the 'quality' doc per review - one minor typo fix - rename the doc to "quality" and re-title - switch to inline links and remove References --- docs/source/{tools.rst => quality.rst} | 41 +++++++++++--------------- docs/source/reference.rst | 2 +- 2 files changed, 18 insertions(+), 25 deletions(-) rename docs/source/{tools.rst => quality.rst} (82%) diff --git a/docs/source/tools.rst b/docs/source/quality.rst similarity index 82% rename from docs/source/tools.rst rename to docs/source/quality.rst index ca86947c..328550c9 100644 --- a/docs/source/tools.rst +++ b/docs/source/quality.rst @@ -1,8 +1,8 @@ .. _tools: -*************************************** -Tools for Working With Type Annotations -*************************************** +******************************************** +Testing and Ensuring Type Annotation Quality +******************************************** Testing Annotation Accuracy =========================== @@ -23,7 +23,7 @@ a few of them. Testing Using ``mypy --warn-unused-ignores`` -------------------------------------------- -Clever use of the ``--warn-unused-ignores`` can be used to check that certain +Clever use of ``--warn-unused-ignores`` can be used to check that certain expressions are or are not well-typed. The idea is to write normal python files which contain valid expressions along @@ -109,11 +109,15 @@ fixture which generates a temporary file and uses it as the input to return func + +For more details, see `the documentation on mypy.api +`_. + pytest-mypy-plugins ------------------- -``pytest-mypy-plugins`` [#pytest-mypy-plugins]_ is a plugin for ``pytest`` -which defines typing test cases as YAML data. +`pytest-mypy-plugins `_ is +a plugin for ``pytest`` which defines typing test cases as YAML data. The test cases are run through ``mypy`` and the output of ``reveal_type`` can be asserted. @@ -172,8 +176,8 @@ library: ------------------------- ``mypy`` offers several options which can detect untyped code. -More details can be found in the -mypy documentation on these options [#mypy-untyped-checks]_. +More details can be found in `the mypy documentation on these options +`_. Some basic usages which make ``mypy`` error on untyped data are:: @@ -184,24 +188,13 @@ Some basic usages which make ``mypy`` error on untyped data are:: ----------------------------- pyright has a special command line flag, ``--verifytypes``, for verifying -type completeness. You can learn more about it from the -pyright documentation on verifying type completeness [#pyright-verifytypes]_. +type completeness. You can learn more about it from +`the pyright documentation on verifying type completeness +`_. ``mypy`` reports ---------------- ``mypy`` offers several options options for generating reports on its analysis. -See the mypy documentation on report generation [#mypy-report]_ for details. - -References -========== - -.. [#mypy-report] mypy reporting options (https://mypy.readthedocs.io/en/stable/command_line.html#report-generation) - -.. [#mypy.api] mypy.api -- the python interface to mypy (https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application) - -.. [#mypy-untyped-checks] mypy CLI untyped checks (https://mypy.readthedocs.io/en/latest/command_line.html#untyped-definitions-and-calls) - -.. [#pyright-verifytypes] pyright --verifytypes (https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#verifying-type-completeness) - -.. [#pytest-mypy-plugins] pytest-mypy-plugins (https://github.com/typeddjango/pytest-mypy-plugins) +See `the mypy documentation on report generation +`_ for details. diff --git a/docs/source/reference.rst b/docs/source/reference.rst index ba8ddab2..c7fc57ee 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -7,7 +7,7 @@ Type System Reference :caption: Contents: stubs - tools + quality typing Module Documentation .. The following pages are desired in a new TOC which will cover multiple