Handling exceptions within a `@contextmanager` function doesn't clear `sys.exception()` #111375

pR0Ps · 2023-10-27T01:38:56Z

Bug report

Bug description:

The issue

I would expect that handling exceptions within a contextlib.contextmanager-created function would work in the same way as other functions and clear the sys.exception() after an error is handled.

import contextlib
import sys

def p(msg):
    print(msg, repr(sys.exception()), sep=": ")

def ctx_gen():
    p("before yield")
    try:
        yield
    except:
        p("during handling")
    p("after handling")

ctx = contextlib.contextmanager(ctx_gen)

with ctx():
    1/0

The above example prints:

before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: ZeroDivisionError('division by zero')

Whereas since the error was handled by the except: block, my expectation was:

before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: None

Just working as designed?

From doing some digging, it seems like this is happening because the exception is still being handled by the _GeneratorContextManager.__exit__ function (added by the @contextlib.contextmanager decorator) that's driving the ctx_gen generator, even after the ctg_gen has handled it.

The following is a very rough approximation of how @contextlib.contextmanager drives ctx_gen:

c = ctx_gen()
next(c)  # __enter__()
try:
    # code inside the with block
    1/0
except Exception as e:  # __exit__(typ, exc, tb) for e
    # throw exception into generator and expect to run to end
    try:
        c.throw(e) 
    except StopIteration:
        pass
else:  # __exit__(None, None, None)
    # expect to run to end
    try:
        next(e)
    except StopIteration:
        pass

Running the above (including the definitions from the first code block) also prints:

before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: ZeroDivisionError('division by zero')

In the above code, it's more clear that the exception is still being handled by the except Exception as e: block until c.throw() returns/raises, which only happens after the generator exits. Therefore the exception is still being handled the entire time ctx_gen is running all the code after the first yield.

The fix?

Even though this behavior looks to be technically correct, it still seems unexpected and a bit of an abstraction leak.

Is this something that can be/should be fixed? Or should the behavior just be documented?

CPython versions tested on:

3.8, 3.9, 3.10, 3.11, CPython main branch

Operating systems tested on:

macOS

Linked PRs

The text was updated successfully, but these errors were encountered:

ericsnowcurrently · 2023-10-31T17:42:28Z

CC @ncoghlan

iritkatriel · 2023-11-01T16:13:55Z

This is a nice bug - everything is working as designed but the result is indeed unexpected.

I don't think it can be fixed in contextlib without changes to the interpreter, and if I'm right it's unlikely that this will use case will justify such changes.

Either way, for now at least we should mention this as a quirk of contextlib. @pR0Ps would you like to propose a doc patch?

pR0Ps · 2023-11-02T04:17:16Z

I was messing around with this and after I wired up a C API call to PyErr_SetHandledException(Py_None) just before the self.gen.throw line in the following code, I get the "expected" result.

Before:

cpython/Lib/contextlib.py

Lines 161 to 163 in 230e8e9

    
           try: 
        
               self.gen.throw(value) 
        
           except StopIteration as exc:

After:

try:
    __clear_current_exc()  # defined in C as: { PyErr_SetHandledException(Py_None); return Py_None; }
    self.gen.throw(value)
except StopIteration as exc:

This essentially amounts to running Python 2's sys.exc_clear() just before throwing the exception into the generator. While the generator still has to handle the exception that was thrown into it, it's doing it with no current exception context. This means that once it handles that thrown exception, there is no longer any exception being handled, making sys.exception() return None which matches what a user would expect.

With this change, the result of running the test case from above matches my expected result:

before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: None

At the moment, my implementation is just a proof of concept that I quickly hacked together, but I'd be willing to clean it up, fix the edge cases, write tests, etc. if this is something that would be considered acceptable to add in some fashion and not too hacky.

iritkatriel · 2023-11-02T05:28:13Z

Ah yes, that might work. You'd need to restore the exception after throw() returns. Make sure you have a test for when this whole thing is nested in another except block (and check the value of sys.exception() after the nested one).

…dicate that an exception was handled (#113302)

… to indicate that an exception was handled (python#113302)

pR0Ps added the type-bug An unexpected behavior, bug, or error label Oct 27, 2023

AlexWaygood added the stdlib Python modules in the Lib dir label Oct 27, 2023

pR0Ps mentioned this issue Nov 3, 2023

gh-111375: Fix handling of exceptions within @contextmanager-decorated functions #111676

Open

bedevere-app bot mentioned this issue Dec 20, 2023

gh-111375: Use NULL in the exception stack to indicate an exception was handled #113302

Merged

iritkatriel pushed a commit that referenced this issue Dec 21, 2023

gh-111375: Use NULL rather than None in the exception stack to in…

a2dd0e7

…dicate that an exception was handled (#113302)

ryan-duve pushed a commit to ryan-duve/cpython that referenced this issue Dec 26, 2023

pythongh-111375: Use NULL rather than None in the exception stack…

8ff005c

… to indicate that an exception was handled (python#113302)

kulikjak pushed a commit to kulikjak/cpython that referenced this issue Jan 22, 2024

pythongh-111375: Use NULL rather than None in the exception stack…

0a3a318

… to indicate that an exception was handled (python#113302)

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024

pythongh-111375: Use NULL rather than None in the exception stack…

1932de3

… to indicate that an exception was handled (python#113302)

Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024

pythongh-111375: Use NULL rather than None in the exception stack…

d7a4e43

… to indicate that an exception was handled (python#113302)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handling exceptions within a `@contextmanager` function doesn't clear `sys.exception()` #111375

Handling exceptions within a `@contextmanager` function doesn't clear `sys.exception()` #111375

pR0Ps commented Oct 27, 2023 •

edited by bedevere-app bot

Loading

ericsnowcurrently commented Oct 31, 2023

iritkatriel commented Nov 1, 2023

pR0Ps commented Nov 2, 2023 •

edited

Loading

iritkatriel commented Nov 2, 2023

Handling exceptions within a @contextmanager function doesn't clear sys.exception() #111375

Handling exceptions within a @contextmanager function doesn't clear sys.exception() #111375

Comments

pR0Ps commented Oct 27, 2023 • edited by bedevere-app bot Loading

Bug report

Bug description:

The issue

Just working as designed?

The fix?

CPython versions tested on:

Operating systems tested on:

Linked PRs

ericsnowcurrently commented Oct 31, 2023

iritkatriel commented Nov 1, 2023

pR0Ps commented Nov 2, 2023 • edited Loading

iritkatriel commented Nov 2, 2023

Handling exceptions within a `@contextmanager` function doesn't clear `sys.exception()` #111375

Handling exceptions within a `@contextmanager` function doesn't clear `sys.exception()` #111375

pR0Ps commented Oct 27, 2023 •

edited by bedevere-app bot

Loading

pR0Ps commented Nov 2, 2023 •

edited

Loading