Skip to content

Located a recursion in the the cache_objects logic in Django 1.7 that causes large flush lists #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nsfyn55 opened this issue Jan 30, 2015 · 4 comments
Labels

Comments

@nsfyn55
Copy link

nsfyn55 commented Jan 30, 2015

The bug is localized to invocations ofonly() and defer() on models that extend CachingMixin. It results in deep recursions that occasionally bust the stack, but otherwise create huge flush_lists

class MyModel(CachingMixin):
    id = models.CharField(max_length=50, blank=True)
    nickname = models.CharField(max_length=50, blank=True)
    favorite_color = models.CharField(max_length=50, blank=True)
    content_owner = models.ForeignKey(OtherModel)
m = MyModel.objects.only('id').all()

The bug occurs in the following lines(https://github.com/jbalogh/django-cache-machine/blob/f827f05b195ad3fc1b0111131669471d843d631f/caching/base.py#L253-L254). In this case self is a instance of MyModel with a mix of deferred and undeferred attributes:

fks = dict((f, getattr(self, f.attname)) for f in self._meta.fields
            if isinstance(f, models.ForeignKey))

The use of only() in the Django ORM does some meta programming magic that overrides the unfetched attributes with Django's DeferredAttribute implementation. Under normal circumstances an access to favorite_color would invoke DeferredAttribute.__get__ (https://github.com/django/django/blob/ad96254af98f961d6691481c27127613afeee840/django/db/models/query_utils.py#L88-L120) and fetch the attribute either from the result cache or the data source.

This is the problem when looping over the foreign keys in the Model and accessing their values, Cache Machine induces a recursion. getattr(self, f.attname) on an attribute that is deferred induces a fetch of a Model that has the CachingMixin applied and has deferred attributes. Seen in the line below.

 val = getattr(
                    non_deferred_model._base_manager.only(name).using(
                        instance._state.db).get(pk=instance.pk),
                    self.field_name
                )

Some testing in our environment shows that even in contrived scenarios flush lists can grow 2, 5, 10x the size of the same query executed without only() or defer()

@nsfyn55 nsfyn55 changed the title Located a recursion in the the cache_objects logic in Django 1.7 Located a recursion in the the cache_objects logic in Django 1.7 that causes large flush lists Jan 30, 2015
@tobiasmcnulty
Copy link
Member

@nsfyn55 Thanks for the report! It sounds like you've done a fair bit of debugging around this already. Would you be up for submitting a PR with a test for the issue & potentially a fix if you find one?

@adamn
Copy link

adamn commented Sep 6, 2016

This ticket should be closed - Django 1.7 is no longer supported.

@tobiasmcnulty
Copy link
Member

@adamn Thanks for chiming in. Have you confirmed this is no longer an issue in 1.8?

@adamn
Copy link

adamn commented Sep 6, 2016

Was just looking at the title. It's probably an issue in 1.8+ so if the title can be fixed to "Recursion in cache_objects logic creates large flush lists" that would be good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants