Skip to content

[3.14] GH-135171: Revert async generator expressions behavior #135352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: 3.14
Choose a base branch
from

Conversation

efimov-mikhail
Copy link
Contributor

@efimov-mikhail efimov-mikhail commented Jun 10, 2025

This PR could be a part of #135225.
Changes in behavior of async generator expressions are also reverted.

@serhiy-storchaka @markshannon @Yhg1s

@@ -4544,9 +4544,9 @@ codegen_async_comprehension_generator(compiler *c, location loc,
else {
/* Sub-iter - calculate on the fly */
VISIT(c, expr, gen->iter);
ADDOP(c, LOC(gen->iter), GET_AITER);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, is this change necessary? I think the test passes without it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so. There was a bug in previous commit. I've reverted related logic in codegen.c.

@efimov-mikhail efimov-mikhail marked this pull request as ready for review June 10, 2025 15:29
@efimov-mikhail efimov-mikhail marked this pull request as draft June 10, 2025 15:32
@efimov-mikhail efimov-mikhail marked this pull request as ready for review June 10, 2025 16:19
@efimov-mikhail
Copy link
Contributor Author

I've reverted all generator expressions related changes in codegen.c.
And for now I can see no problems with tests, included those which are added in related PR's.
Could you please review this again?
And backport for 3.13 is also neccessary.

Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@Yhg1s
Copy link
Member

Yhg1s commented Jun 10, 2025

@markshannon Do you want to take a look at this fix? Do you have time to do?

@markshannon
Copy link
Member

@efimov-mikhail are you sure that this isn't adding back too many GET_ITER/GET_AITER instructions?
Particularly, for inlined list comprehensions, this might add an extra GET_[A]ITER.

We should have tests for generator expressions and list comprehensions, both sync and async that __iter__ is called only once.
It seems that we only have test for sync gen expressions. The test is called test_genexpr_only_calls_dunder_iter_once, can you extend it to the other cases?

@markshannon
Copy link
Member

I've a had a quick look at the generated bytecode and looks like this PR is OK.

@Yhg1s if you need this in ASAP, I think it's good to merge.
We can add those tests to main in another PR and backport them later.

Copy link
Member

@markshannon markshannon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like tests to check that __iter__/__aiter__ is only called once in inlined comprehensions, but that can wait for another PR.

@serhiy-storchaka
Copy link
Member

Changes in this PR look similar to the full rollback in #135390, so I think it adds the right number of GET_ITER/GET_AITER at right place. The total effect of #135293 and this PR should restore the old bytecode generation.

@efimov-mikhail
Copy link
Contributor Author

Changes in this PR look similar to the full rollback in #135390, so I think it adds the right number of GET_ITER/GET_AITER at right place. The total effect of #135293 and this PR should restore the old bytecode generation.

Yes, changes in these two PRs are very similar. Although, there aren't exactly the same.
In fact, on 3.14 after this PR will be merged we will have issue #125038 fixed w/o additional GET_ITER instruction, but with null pointer checks in FOR_ITER.

But on 3.13 with #135390 being merged there is a different solution (from #125846).

So, I have a question why we don't revert #125846?
Do we want to preserve bytecode changes in 3.13.1 vs 3.13.0?

@serhiy-storchaka
Copy link
Member

Was not it reverted as the result of all these changes?

@efimov-mikhail
Copy link
Contributor Author

efimov-mikhail commented Jun 11, 2025

Was not it reverted as the result of all these changes?

It was reverted here.
But it wasn't on 3.13 even after #135390 is being merged.

@serhiy-storchaka
Copy link
Member

Thank you, this is a great observation.

@Yhg1s
Copy link
Member

Yhg1s commented Jun 11, 2025

Do we want to preserve bytecode changes in 3.13.1 vs 3.13.0?

I don't think I was aware of bytecode changes in 31.1. I tested the stdlib's compiled bytecode (including Lib/test) and saw no differences between 13.0, 13.3 and my rollback (and saw many changes between 13.3, 13.4 and 13.4 + the other fix-forward, but I didn't bother trying with this one applied).

In any case it's too late to fix a change in bytecode that slipped into 13.1, if 13.1-13.3 are all the same we should stick with that.

@serhiy-storchaka
Copy link
Member

It is not only change in bytecode generation. It is a behavior change in comparison with 3.12, 3.13.0 and 3.13.4.

@efimov-mikhail
Copy link
Contributor Author

efimov-mikhail commented Jun 11, 2025

To be precise, there is a bytecode change:

-> % ./python
Python 3.13.0 (tags/v3.13.0:60403a5409, Jun 11 2025, 19:23:44) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis('(x for x in range(10))')
  0           RESUME                   0

  1           LOAD_CONST               0 (<code object <genexpr> at 0x7fa6c309f450, file "<dis>", line 1>)
              MAKE_FUNCTION
              LOAD_NAME                0 (range)
              PUSH_NULL
              LOAD_CONST               1 (10)
              CALL                     1
              GET_ITER
              CALL                     0
              RETURN_VALUE

Disassembly of <code object <genexpr> at 0x7fa6c309f450, file "<dis>", line 1>:
   1           RETURN_GENERATOR
               POP_TOP
       L1:     RESUME                   0
               LOAD_FAST                0 (.0)
       L2:     FOR_ITER                 6 (to L3)
               STORE_FAST_LOAD_FAST    17 (x, x)
               YIELD_VALUE              0
               RESUME                   5
               POP_TOP
               JUMP_BACKWARD            8 (to L2)
       L3:     END_FOR
               POP_TOP
               RETURN_CONST             0 (None)

  --   L4:     CALL_INTRINSIC_1         3 (INTRINSIC_STOPITERATION_ERROR)
               RERAISE                  1
ExceptionTable:
  L1 to L4 -> L4 [0] lasti

and

-> % ./python                  
Python 3.13.1 (tags/v3.13.1:0671451779, Jun 11 2025, 19:25:28) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis('(x for x in range(10))')
  0           RESUME                   0

  1           LOAD_CONST               0 (<code object <genexpr> at 0x7f4332397bc0, file "<dis>", line 1>)
              MAKE_FUNCTION
              LOAD_NAME                0 (range)
              PUSH_NULL
              LOAD_CONST               1 (10)
              CALL                     1
              GET_ITER
              CALL                     0
              RETURN_VALUE

Disassembly of <code object <genexpr> at 0x7f4332397bc0, file "<dis>", line 1>:
   1           RETURN_GENERATOR
               POP_TOP
       L1:     RESUME                   0
               LOAD_FAST                0 (.0)
               GET_ITER
       L2:     FOR_ITER                 6 (to L3)
               STORE_FAST_LOAD_FAST    17 (x, x)
               YIELD_VALUE              0
               RESUME                   5
               POP_TOP
               JUMP_BACKWARD            8 (to L2)
       L3:     END_FOR
               POP_TOP
               RETURN_CONST             0 (None)

  --   L4:     CALL_INTRINSIC_1         3 (INTRINSIC_STOPITERATION_ERROR)
               RERAISE                  1
ExceptionTable:
  L1 to L4 -> L4 [0] lasti
>>> 

You can see additional GET_ITER instruction before FOR_ITER.
This change is made in #125846.

IMO, it's better to revert this change and fix regression in #127682.

@efimov-mikhail
Copy link
Contributor Author

I've created new PR for 3.13: #135403

@efimov-mikhail
Copy link
Contributor Author

I'd like tests to check that __iter__/__aiter__ is only called once in inlined comprehensions, but that can wait for another PR.

There were 3 tests related to this topic, and I've added the fourth.
Is it enough?
Or I miss something?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants