Indirectly closes #9443.

What we're trying to achieve and how

get_type_ref is pretty expensive, and we'll try to reduce the amount of calls to this function.

Currently, for every call to GenerateSchema.generate_schema, we unconditionally end up with the following call chain: _generate_schema_from_property ->get_schema_or_ref -> get_type_ref.

The idea is thus to remove this part (L750-752):

So that any GenerateSchema method responsible for generating a core schema for a referenceable type handle it. For example, _model_schema, which already has this check:

Note that the if maybe_schema is not None branch is never reached (same for all the other referenceable types such as dataclasses, typeddict, etc) because the branch we're trying to remove handles it already.

So surely removing the branch from _generate_schema_from_property should work.

Challenges

However, this breaks for Pydantic models:

Let's say you have the following:

class Sub(BaseModel): pass

class Model(BaseModel):
    sub1: Sub
    sub2: Sub

When we generate a schema for sub2, we follow the call chain generate_schema -> _generate_schema_from_property. If we don't check for get_schema_or_ref, we would want it to be checked in _model_schema as shown above. However, the rest of the _generate_schema_from_property implementation will check for __get_pydantic_core_schema__(), and the __pydantic_core_schema__ property. In our example, Sub is already in the definitions thanks to sub1, so it would break by returning the complete core schema of Sub ¹ instead of a definition reference schema (produced by get_schema_or_ref).

So let's try to push things a bit further. Why not move the __get_pydantic_core_schema__()/__pydantic_core_schema__ logic to the _model_schema method? (same for dataclasses).

The issue is that while it would be fine for the __pydantic_core_schema__ property (only relevant for Pydantic models and dataclasses), the __get_pydantic_core_schema__() method can be set on pretty much anything.

Solution

That's why I had to remove the __get_pydantic_core_schema__ method from BaseModel. The current implementation is really only returning the cached __pydantic_core_schema__ property or delegating the generation to the GenerateSchema class.

During model construction, we were doing something even more confusing:

The following call chain happens: cls.__get_pydantic_core_schema__(cls, handler) -> handler(cls) ~ GenerateSchema().generate_schema(cls, from_dunder_get_core_schema=False).

I think having from_dunder_get_core_schema to avoid an infinite loop is a sign that the current implementation is quite convoluted.

Instead, by:

• removing the BaseModel.__get_pydantic_core_schema__ method, and moving the check for the __pydantic_core_schema__ attribute to GenerateSchema._model_schema
• Only checking for a __get_pydantic_core_schema__ method (and not the property) when calling GenerateSchema.generate_schema (which isn't an issue for Pydantic models as we removed it).

We:

• allow for a more consistent approach when it comes to schema generation of models. We can now call GenerateSchema().generate_schema(some_model_cls), and the call chain is way simpler: generate_schema -> ... -> _model_schema.
• move the logic to potentially call get_schema_or_ref (and check for the __pydantic_core_schema__ property) to the responsible methods: _model_schema, _dataclass_schema, etc.

Breaking changes concerns

We had a couple reports from people overriding BaseModel.__get_pydantic_core_schema__:

• the first one I could find: type casting does not work with multiple attributes of the same custom type #7102. The mentioned reason isn't really compelling, the user was doing things not the intended way. It also led to a scary and hacky fix that I would really like not having.
• Strange ValidationErrors with ≥2 nested instances of model with customized __get_pydantic_core_schema__ (embedded schemas differ, field ordering dependent) #10160, a user facing unexpected issues. I explained in length why it was the case (and relates to my explanation here).
• __get_pydantic_core_schema__ ignored in self-referencing models #10582, a user also facing unexpected issues (maybe the same ones).

Feels like doing so always led to unexpected behavior, and a hacky fix. That's why I'm in support of having this pattern for BaseModel removed. It might break existing code, but we are probably helping the users in avoiding unexpected issues in the future.

We could try to still support the existing BaseModel.__get_pydantic_core_schema__ method, but honestly it's been quite hard to implement this already, and it might be that it's impossible to have both this perf improvement and support for the existing behavior at the same time