Skip to content

Refactor _typing_extra module #10725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 8, 2024
Merged

Refactor _typing_extra module #10725

merged 5 commits into from
Nov 8, 2024

Conversation

Viicos
Copy link
Member

@Viicos Viicos commented Oct 27, 2024

Change Summary

  • Refactor the _typing_extra module to:
    • make use of https://typing-extensions.readthedocs.io/en/latest/#runtime-use-of-types. In a lot of places, we are only checking for some_obj is typing(_extensions).SomeType which can break at any time if typing_extensions backports some changes from Python, in which case typing.SomeType is not typing_extensions.SomeType.
    • Cleanup/simplify some is_* functions (e.g. is_newtype, is_classvar).
    • Add a new is_classvar_annotation function, that should be used in most cases. This function handles ClassVar being wrapped in Annotated, and also stringified annotations.
    • Rename all_literal_values to literal_values, replacing the unused literal_values function. When we drop support for Python 3.8, replace usage with a plain get_args.
    • Remove unecessary is_dataclass function, dataclasses.is_dataclass provides a type guard return annotation.
  • Adapt the codebase to use the new constructs. In some cases, this does fix some actual bugs

Related issue number

Checklist

  • The pull request title is a good summary of the changes - it will be used in the changelog
  • Unit tests for the changes exist
  • Tests pass on CI
  • Documentation reflects the changes where applicable
  • My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

@github-actions github-actions bot added the relnotes-fix Used for bugfixes. label Oct 27, 2024
Copy link

cloudflare-workers-and-pages bot commented Oct 27, 2024

Deploying pydantic-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 5a43b8d
Status: ✅  Deploy successful!
Preview URL: https://b118d254.pydantic-docs.pages.dev
Branch Preview URL: https://refactor-typing-extra.pydantic-docs.pages.dev

View logs

Copy link

codspeed-hq bot commented Oct 27, 2024

CodSpeed Performance Report

Merging #10725 will not alter performance

Comparing refactor-typing-extra (5a43b8d) with main (a25e028)

Summary

✅ 44 untouched benchmarks

Copy link
Contributor

github-actions bot commented Oct 27, 2024

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  pydantic
  dataclasses.py
  json_schema.py
  main.py
  pydantic/_internal
  _core_utils.py
  _dataclasses.py
  _fields.py
  _generate_schema.py
  _generics.py
  _model_construction.py
  _repr.py
  _std_types_schema.py
  _typing_extra.py 35, 199
Project Total  

This report was generated by python-coverage-comment-action

@Viicos Viicos force-pushed the refactor-typing-extra branch 3 times, most recently from 279daa8 to c68a904 Compare October 29, 2024 07:55
@Viicos Viicos force-pushed the refactor-typing-extra branch from c68a904 to 591dde0 Compare October 29, 2024 08:29
@Viicos Viicos force-pushed the refactor-typing-extra branch from 591dde0 to 0faf9b6 Compare October 29, 2024 08:34
@Viicos Viicos force-pushed the refactor-typing-extra branch 4 times, most recently from a5ef806 to 088b5c2 Compare October 30, 2024 08:44
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This display_as_type function needs to be refactored, as it is relatively expensive to recursively check for get_origin, get_args, etc. It is currently used:

  • In FieldInfo.__repr_args__, to make a string representation of the annotation attribute. This can be kept.
  • In get_type_ref, called for each arg of a parametrized type. We should find a simpler way to generate a core schema reference.

@Viicos Viicos force-pushed the refactor-typing-extra branch from 088b5c2 to 291dd9b Compare October 30, 2024 08:48
@Viicos Viicos marked this pull request as ready for review October 31, 2024 10:34
@sydney-runkle
Copy link
Contributor

Should we be concerned about some of the minor performance regressions we see for schema building here?

https://codspeed.io/pydantic/pydantic/branches/refactor-typing-extra

@Viicos
Copy link
Member Author

Viicos commented Oct 31, 2024

Should we be concerned about some of the minor performance regressions we see for schema building here?

My assumption is that the small regression comes from the fact that in hot functions (display_as_type, get_type_ref, has_instance_in_type) we now properly check for the two variants (typing and typing_extensions), e.g. for TypeAliasType.

I think the real problem is that we should find a way to avoid calling these functions so many times, this is what I'm working on currently as part of #10297

Copy link
Contributor

@sydney-runkle sydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, nice work @Viicos. Things feel much cleaner in the internal type management department with this PR. Left some general feedback 👍

@@ -184,7 +184,7 @@ def _get_caller_frame_info(depth: int = 2) -> tuple[str | None, bool]:
DictValues: type[Any] = {}.values().__class__


def iter_contained_typevars(v: Any) -> Iterator[TypeVarType]:
def iter_contained_typevars(v: Any) -> Iterator[TypeVar]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain this change, briefly? Thanks!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TypeVarType was defined as an alias to Any as a workaround, because mypy could not understand something like this:

TypeVarType: TypeAlias = TypeVar

But for pyright (and maybe mypy as of today), it is perfectly fine to use TypeVar as type annotations, it behaves as any other type

Copy link
Contributor

@sydney-runkle sydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for incorporating the feedback.

Happy to merge this now, given that we're going to see perf improvements bounce back up with #10769

@sydney-runkle sydney-runkle merged commit 5b7c290 into main Nov 8, 2024
55 checks passed
@sydney-runkle sydney-runkle deleted the refactor-typing-extra branch November 8, 2024 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
relnotes-fix Used for bugfixes.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants