Fixed #36426 -- Prefetch_related_objects to support all iterabe types. #19685

blingblin-g · 2025-07-30T14:38:57Z

Trac ticket number

Branch description

Support all iterable types (e.g., list, set, tuple) as input for prefetch_related_objects, making it more flexible and resolving #36426.

Checklist

This PR targets the main branch.
The commit message is written in past tense, mentions the ticket number, and ends with a period.
I have checked the "Has patch" ticket flag in the Trac system.
I have added or updated relevant tests.
I have added or updated relevant docs, including release notes if applicable.
I have attached screenshots in both light and dark modes for any UI changes.

sarahboyce

Thank you! I added initial comments

sarahboyce · 2025-07-31T12:04:29Z

tests/prefetch_related/test_prefetch_related_objects.py

+        # set
+        prefetch_related_objects({book}, "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )
+        # tuple
+        prefetch_related_objects((book,), "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )
+        # dict_values
+        d = {"a": book}
+        prefetch_related_objects(d.values(), "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )
+        # frozenset
+        prefetch_related_objects(frozenset([book]), "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )
+        # generator
+        prefetch_related_objects((x for x in [book]), "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )
+        # map
+        prefetch_related_objects(map(lambda x: x, [book]), "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )
+        # filter
+        prefetch_related_objects(filter(lambda x: True, [book]), "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )
+        # collections.deque
+        prefetch_related_objects(deque([book]), "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )
+
+        # custom iterable
+        class MyIterable:
+            def __iter__(self_inner):
+                yield book
+
+        prefetch_related_objects(MyIterable(), "authors")
+        self.assertCountEqual(
+            book.authors.all(), [self.author1, self.author2, self.author3]
+        )


I think it would be good to use subTest here and to assert that prefetch_related_objects is working as expected (we would need to assert the query count)

This could look like:

Suggested change

# set

prefetch_related_objects({book}, "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

# tuple

prefetch_related_objects((book,), "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

# dict_values

d = {"a": book}

prefetch_related_objects(d.values(), "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

# frozenset

prefetch_related_objects(frozenset([book]), "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

# generator

prefetch_related_objects((x for x in [book]), "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

# map

prefetch_related_objects(map(lambda x: x, [book]), "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

# filter

prefetch_related_objects(filter(lambda x: True, [book]), "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

# collections.deque

prefetch_related_objects(deque([book]), "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

# custom iterable

class MyIterable:

def __iter__(self_inner):

yield book

prefetch_related_objects(MyIterable(), "authors")

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

book = self.book1

class MyIterable:

def __iter__(self):

yield book

cases = {

"set": {book},

"tuple": (book,),

"dict_values": {"a": book}.values(),

"frozenset": frozenset([book]),

"generator": (x for x in [book]),

"map": map(lambda x: x, [book]),

"filter": filter(lambda x: True, [book]),

"deque": deque([book]),

"custom iterator": MyIterable(),

}

for case_type, case in cases.items():

with self.subTest(case=case_type):

# Clear the prefetch cache.

book._prefetched_objects_cache = {}

with self.assertNumQueries(1):

prefetch_related_objects(case, "authors")

with self.assertNumQueries(0):

self.assertCountEqual(

book.authors.all(), [self.author1, self.author2, self.author3]

)

However, the filter, map and generator cases are failing

Your approach definitely seems better. That said, as I mentioned before, I think it’s best to update the documentation instead, so this test will probably be removed. Thanks for the suggestion!

sarahboyce · 2025-07-31T12:05:27Z

tests/prefetch_related/test_prefetch_related_objects.py

+        # empty iterable
+        prefetch_related_objects([], "authors")
+        prefetch_related_objects(set(), "authors")
+        prefetch_related_objects((), "authors")
+        prefetch_related_objects(frozenset(), "authors")
+        prefetch_related_objects({}.values(), "authors")
+        prefetch_related_objects((x for x in []), "authors")
+        prefetch_related_objects(map(lambda x: x, []), "authors")
+        prefetch_related_objects(filter(lambda x: True, []), "authors")
+        prefetch_related_objects(deque([]), "authors")


I would separate out the empty cases to a new test

Suggested change

# empty iterable

prefetch_related_objects([], "authors")

prefetch_related_objects(set(), "authors")

prefetch_related_objects((), "authors")

prefetch_related_objects(frozenset(), "authors")

prefetch_related_objects({}.values(), "authors")

prefetch_related_objects((x for x in []), "authors")

prefetch_related_objects(map(lambda x: x, []), "authors")

prefetch_related_objects(filter(lambda x: True, []), "authors")

prefetch_related_objects(deque([]), "authors")

def test_prefetch_related_objects_with_empty_iterables(self):

empty_cases = {

"list": [],

"set": set(),

"tuple": (),

"dict_values": {}.values(),

"frozenset": frozenset(),

"generator": (x for x in []),

"map": map(lambda x: x, []),

"filter": filter(lambda x: True, []),

"deque": deque([]),

}

for case_type, case in empty_cases.items():

with self.subTest(case=case_type):

with self.assertNumQueries(0):

prefetch_related_objects(case, "authors")

sarahboyce · 2025-07-31T12:06:18Z

tests/prefetch_related/test_prefetch_related_objects.py

+        prefetch_related_objects(map(lambda x: x, []), "authors")
+        prefetch_related_objects(filter(lambda x: True, []), "authors")
+        prefetch_related_objects(deque([]), "authors")
+        prefetch_related_objects([book, book, book], "authors")


Note that this is under the " # empty iterable" comment but is not an empty iterable. There's no asserts, other than you showing that this doesn't error, not sure what is being tested here

sarahboyce · 2025-07-31T12:12:03Z

django/db/models/query.py

+            if isinstance(obj_list, Iterator):
+                obj_list, obj_list_for_next = tee(obj_list, 2)
+            else:
+                obj_list_for_next = obj_list
+
+            first_obj = next(iter(obj_list_for_next), None)
+
+            if first_obj is None:
+                continue


Note that the solution here is relatively complex and I would be more in favor of a docs change as Natalia suggested rather than try to support all cases
Note that Jacob suggested we update to first_obj = next(iter(obj_list)), which works in most cases (except filter, map and generator but not sure those cases are working currently in this solution)

Either of those would be preferable

Just chiming in to add that the cases similar to generator are not very realistic for prefetch_related_objects() because the premise is that you already have materialized instances in hand.

Thanks for the detailed review!
At first, I tried using first_obj = next(iter(obj_list)) as you suggested, but since the iterator gets consumed, it didn’t work as intended later on and the tests failed.
So I tried handling multiple cases separately, but like you said, I think it’s better to just update the docs rather than trying to support every case.
I’ll proceed that way!

I agree the generator/map/filter cases are not worth supporting, but we don't have to treat that as the reason to pass on the next(iter(... solution. What about just fleshing out the current caveat like this?

diff --git a/django/db/models/query.py b/django/db/models/query.py index cc4d9c1f22..d0788867f1 100644 --- a/django/db/models/query.py +++ b/django/db/models/query.py @@ -2390,7 +2390,7 @@ def prefetch_related_objects(model_instances, *related_lookups): # We assume that objects retrieved are homogeneous (which is the # premise of prefetch_related), so what applies to first object # applies to all. - first_obj = obj_list[0] + first_obj = next(iter(obj_list)) to_attr = lookup.get_current_to_attr(level)[0] prefetcher, descriptor, attr_found, is_fetched = get_prefetcher( first_obj, through_attr, to_attr diff --git a/docs/ref/models/querysets.txt b/docs/ref/models/querysets.txt index ef6ceb36ad..90dcaac9f2 100644 --- a/docs/ref/models/querysets.txt +++ b/docs/ref/models/querysets.txt @@ -4217,8 +4217,9 @@ Prefetches the given lookups on an iterable of model instances. This is useful in code that receives a list of model instances as opposed to a ``QuerySet``; for example, when fetching models from a cache or instantiating them manually. -Pass an iterable of model instances (must all be of the same class) and the -lookups or :class:`Prefetch` objects you want to prefetch for. For example: +Pass an iterable of model instances (must all be of the same class and able to be +iterated multiple times) and the lookups or :class:`Prefetch` objects you want to +prefetch for. For example:

blingblin-g force-pushed the ticket_36426 branch from 6f2c878 to ae8bdef Compare July 30, 2025 14:58

Fixed #36426 -- Prefetch_related_objects to support all iterabe types.

63a36ba

blingblin-g force-pushed the ticket_36426 branch from 3628ed6 to 63a36ba Compare July 30, 2025 15:03

sarahboyce reviewed Jul 31, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fixed #36426 -- Prefetch_related_objects to support all iterabe types. #19685

Fixed #36426 -- Prefetch_related_objects to support all iterabe types. #19685

Uh oh!

blingblin-g commented Jul 30, 2025 •

edited

Loading

Uh oh!

sarahboyce left a comment

Uh oh!

sarahboyce Jul 31, 2025

Uh oh!

blingblin-g Aug 1, 2025

Uh oh!

sarahboyce Jul 31, 2025

Uh oh!

sarahboyce Jul 31, 2025

Uh oh!

sarahboyce Jul 31, 2025

Uh oh!

jacobtylerwalls Jul 31, 2025

Uh oh!

blingblin-g Aug 1, 2025

Uh oh!

jacobtylerwalls Aug 1, 2025 •

edited

Loading

Uh oh!

Uh oh!

-        # empty iterable
-        prefetch_related_objects([], "authors")
-        prefetch_related_objects(set(), "authors")
-        prefetch_related_objects((), "authors")
-        prefetch_related_objects(frozenset(), "authors")
-        prefetch_related_objects({}.values(), "authors")
-        prefetch_related_objects((x for x in []), "authors")
-        prefetch_related_objects(map(lambda x: x, []), "authors")
-        prefetch_related_objects(filter(lambda x: True, []), "authors")
-        prefetch_related_objects(deque([]), "authors")
+    def test_prefetch_related_objects_with_empty_iterables(self):
+        empty_cases = {
+            "list": [],
+            "set": set(),
+            "tuple": (),
+            "dict_values": {}.values(),
+            "frozenset": frozenset(),
+            "generator": (x for x in []),
+            "map": map(lambda x: x, []),
+            "filter": filter(lambda x: True, []),
+            "deque": deque([]),
+        }
+        for case_type, case in empty_cases.items():
+            with self.subTest(case=case_type):
+                with self.assertNumQueries(0):
+                    prefetch_related_objects(case, "authors")

Uh oh!

Fixed #36426 -- Prefetch_related_objects to support all iterabe types. #19685

Are you sure you want to change the base?

Fixed #36426 -- Prefetch_related_objects to support all iterabe types. #19685

Uh oh!

Conversation

blingblin-g commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Trac ticket number

Branch description

Checklist

Uh oh!

sarahboyce left a comment

Choose a reason for hiding this comment

Uh oh!

sarahboyce Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

blingblin-g Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

sarahboyce Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

sarahboyce Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

sarahboyce Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

jacobtylerwalls Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

blingblin-g Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

jacobtylerwalls Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

blingblin-g commented Jul 30, 2025 •

edited

Loading

jacobtylerwalls Aug 1, 2025 •

edited

Loading