Skip to content

Located a recursion in the the cache_objects logic in Django 1.7 that causes large flush lists #89

Open
@nsfyn55

Description

@nsfyn55

The bug is localized to invocations ofonly() and defer() on models that extend CachingMixin. It results in deep recursions that occasionally bust the stack, but otherwise create huge flush_lists

class MyModel(CachingMixin):
    id = models.CharField(max_length=50, blank=True)
    nickname = models.CharField(max_length=50, blank=True)
    favorite_color = models.CharField(max_length=50, blank=True)
    content_owner = models.ForeignKey(OtherModel)
m = MyModel.objects.only('id').all()

The bug occurs in the following lines(https://github.com/jbalogh/django-cache-machine/blob/f827f05b195ad3fc1b0111131669471d843d631f/caching/base.py#L253-L254). In this case self is a instance of MyModel with a mix of deferred and undeferred attributes:

fks = dict((f, getattr(self, f.attname)) for f in self._meta.fields
            if isinstance(f, models.ForeignKey))

The use of only() in the Django ORM does some meta programming magic that overrides the unfetched attributes with Django's DeferredAttribute implementation. Under normal circumstances an access to favorite_color would invoke DeferredAttribute.__get__ (https://github.com/django/django/blob/ad96254af98f961d6691481c27127613afeee840/django/db/models/query_utils.py#L88-L120) and fetch the attribute either from the result cache or the data source.

This is the problem when looping over the foreign keys in the Model and accessing their values, Cache Machine induces a recursion. getattr(self, f.attname) on an attribute that is deferred induces a fetch of a Model that has the CachingMixin applied and has deferred attributes. Seen in the line below.

 val = getattr(
                    non_deferred_model._base_manager.only(name).using(
                        instance._state.db).get(pk=instance.pk),
                    self.field_name
                )

Some testing in our environment shows that even in contrived scenarios flush lists can grow 2, 5, 10x the size of the same query executed without only() or defer()

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions