Skip to content

Conversation

anudeepsamaiya
Copy link

@anudeepsamaiya anudeepsamaiya commented Oct 23, 2019

Issue Ref: #1425

Added indexes in the Meta of the TimeStampedModel.

Usage:

class Cheese(TimeStampedModel):
    name = models.CharField(max_length=50, null=True, blank=True)

or

class Sale(TimeStampedModel):
    quantity = models.IntegerField()

    class Meta(TimeStampedModel.Meta):
        db_table = "sale"

or

class Sale(TimeStampedModel):
    quantity = models.IntegerField()

    class Meta(TimeStampedModel.Meta):
        db_table = "sale"
         indexes = TimeStampedModel.Meta.indexes + [models.Index(fields=["quantity"])]

@trbs
Copy link
Member

trbs commented Oct 23, 2019

Could you please update docs/model_extensions.rst and put the usage example in the description as an example under *TimeStampedModel*.

@coveralls
Copy link

coveralls commented Oct 23, 2019

Coverage Status

Coverage increased (+0.04%) to 71.198% when pulling 70eb7a9 on anudeepsamaiya:add-indexes-for-timestamped-activator-models into 0f0ca99 on django-extensions:master.

anudeepsamaiya added a commit to anudeepsamaiya/django-extensions that referenced this pull request Oct 23, 2019
update docs with the usage examples of the Meta class.
update docs with the usage examples of the Meta class.
@anudeepsamaiya anudeepsamaiya force-pushed the add-indexes-for-timestamped-activator-models branch from a4a4017 to e2e2781 Compare October 23, 2019 15:24
@anudeepsamaiya
Copy link
Author

Could you please update docs/model_extensions.rst and put the usage example in the description as an example under *TimeStampedModel*.

I have added the docs. Thanks.

@@ -25,6 +25,7 @@ def save(self, **kwargs):
class Meta:
get_latest_by = 'modified'
ordering = ('-modified', '-created',)
indexes = [models.Index(fields=["created"]), models.Index(fields=["modified"])]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently these indexes are like adding db_index=True to each of the fields (created and modified). There is no need for Meta.indexes in this case.

If Meta.indexes should be defined I thing that is better to add index which is for both fields in descending order (to match the Meta.ordering). This way listing of the objects should be faster. Also individual indexes can also be in descending order (at least for modified) for latest() to use it.

Copy link
Author

@anudeepsamaiya anudeepsamaiya Oct 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with your points. So do you think it would be better if I modified the current version like this:

indexes = [ 
    models.Index(fields=["-created", "-modified"]),
    models.Index(fields=["-created"]), models.Index(fields=["-modified"])
]

@trbs
Copy link
Member

trbs commented Oct 24, 2019

Sorry guys, I was (mistakenly) under the impression that we already had indexes via db_index=True on the fields.

This change would introduce a schema migration for people so we have to think a little more careful about this imho.

Why not do a little differently... Add an IndexedTimeStampedModel (name is just a strawman) which is something like:

class IndexedTimeStampedModel(TimeStampedModel):
    class Meta(TimeStampedModel.Meta):
        indexes = [ 
            models.Index(fields=["-modified"])
            models.Index(fields=["-created"]),
        ]

(or models.Index(fields=["-modified", "-created"]) we have to decide on one multi column index or two separate ones. Do all supported django database offer multi column indexes ?)

This way we do not suddenly change the behavior of TimeStampedModel and create a schema migration after an upgrade of django-extensions for everybody.

@anudeepsamaiya
Copy link
Author

anudeepsamaiya commented Oct 24, 2019

Why not do a little differently... Add an IndexedTimeStampedModel (name is just a strawman) which is something like:

class IndexedTimeStampedModel(TimeStampedModel):
    class Meta(TimeStampedModel.Meta):
        indexes = [ 
            models.Index(fields=["-modified"])
            models.Index(fields=["-created"]),
        ]

I was thinking of implementing this way, but then most people will have to modify their code if they want to use indexes. On the other hand if we go with adding indexes to TimeStampedModel, then any new user will have indexes right from start and others will have database migrations. The latter one will surely introduce inconveniences but in favor of performance improvements.

Although, I am not in favor of going with latter option, we can totally avoid it by adding the document for IndexedTimeStampModel and properly explaining its pros and cons. I also think we can have indexed versions for other models. This will make using indexed versions a choice for the end user.

(or models.Index(fields=["-modified", "-created"]) we have to decide on one multi column index or two separate ones. Do all supported django database offer multi column indexes ?)

Yes MySQL, MariaDB, Postgres, Oracle, SQLServer all popular dbs support multi column indexes.

This way we do not suddenly change the behavior of TimeStampedModel and create a schema migration after an upgrade of django-extensions for everybody.

@trbs
Copy link
Member

trbs commented Oct 24, 2019

Cool lets do it that way and add index'd versions with some documentation.

This would also be a very nice example of how easy it can be to customize these things :-)

@YPCrumble
Copy link
Contributor

FWIW as a django-extensions user I think adding the index to my existing models, vs. having to make complex migrations to migrate my TimeStampedModel columns to IndexedTimeStampedModel columns, is the right way to go...per what @anudeepsamaiya said here:

On the other hand if we go with adding indexes to TimeStampedModel, then any new user will have indexes right from start and others will have database migrations.

This would be much easier for me than having to switch my existing TimeStampedModel's to IndexedTimeStampedModels.

@trbs I would think that for almost all use cases, having an index on created and modified fields that are used so often in sorting would be expected by default. See for instance this SO question/answers.

For large projects this would (I believe) significantly speed up queries, and for small projects this would make a negligible difference from the current implementation.

@trbs
Copy link
Member

trbs commented Nov 1, 2019

Unless you have TimeStampedModel on very large tables where you don't need the index and it will slow down the database, bloat the size (because of the additional indexes) and potentially mess a little with caching since you now evicting potentially hot pages :-)

Anyways so many potential cases and we really don't know what people are using it for.

I can be convinced to set the indexes by default, specially since I think having a complex migration path for people (I assume the majority) that do want to have the indexes by default.

So what I suggest to do is if there is consensus to make the indexes the default. Then we do so via:

  1. The class Meta indexes property
  2. Update the PR to fix the style of those as indicated in my comments/reviews
  3. Write more and better documentation of the change and how people that do NOT want the index should adjust their code (aka override Meta.indexes) to avoid getting them
  4. Write a short details explanation of the change for in the changes file with a link to documentation
  5. We bumb the major version number of Django-Extensions with the release of this change

(Please note that I will be traveling a lot this month, so appologies in advance for reviews and merging PR's to be a bit slow)

@YPCrumble
Copy link
Contributor

YPCrumble commented Nov 5, 2019

@trbs thanks for your thoughtful response! One thing to add.

I think the reason people would assume the indexes are in place is because TimeStampedModel sorts on the modified and created fields by default. I believe means that for most users the index will be of benefit any time they query a large number of models unless they've explicitly removed this Meta ordering on their TimeStampedModel.

Your suggestion for managing the change makes perfect sense. @anudeepsamaiya let me know if you want any help on implementing the changes? I have some time end of next week but not before then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants