Are you able to reproduce this without Cython? Or it only happens on C level tracing function? What about a minimal tracer with C-extension?

Are you able to reproduce this without Cython?

yes

What about a minimal tracer with C-extension?

I've never created a C extension before, so don't expect this to be pretty.

#include <Python.h>


static PyObject* lines(PyObject* self, PyObject* args) {
  PyObject* function = PyTuple_GetItem(args, 0);
  PyObject* code = PyFunction_GetCode(function);
  PyObject* co_code = PyCode_GetCode(code);
  printf("[");
  for (int byte_offset=0; byte_offset < PyObject_Length(co_code); byte_offset += 2) {
    printf("%i, ", PyCode_Addr2Line(code, byte_offset));
  }
  printf("]\n");
  Py_DECREF(co_code);
  return Py_None;
}


static int trace(PyObject* obj, PyFrameObject* frame, int what, PyObject* arg) {
  printf("%i %s\n", PyFrame_GetLineNumber(frame), what == PyTrace_CALL ? "call" : (what == PyTrace_LINE ? "line" : (what == PyTrace_RETURN ? "return" : "other")));
  return 0;
}


static PyObject* start(PyObject* self, PyObject* args) {
  PyEval_SetTrace(trace, NULL);
  return Py_None;
}


static PyObject* stop(PyObject* self, PyObject* args) {
  PyEval_SetTrace(NULL, NULL);
  return Py_None;
}


static PyMethodDef Methods[] = {
    {"lines", lines, METH_VARARGS, ""},
    {"start", start, METH_VARARGS, ""},
    {"stop", stop, METH_VARARGS, ""},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT,
    "cline",   /* name of module */
    NULL, /* module documentation, may be NULL */
    -1,       /* size of per-interpreter state of the module,
                 or -1 if the module keeps state in global variables. */
    Methods
};

PyMODINIT_FUNC
PyInit_cline(void)
{
    return PyModule_Create(&module);
}

Backtracked PyCode_Addr2Line to this line

3.11.6 doesn't reach it and no other path that might lead to -1 is taken in both versions. I'm nowhere close to understand why.

So there's a simple repro:

import dis

async def f():
    pass

print(list(f.__code__.co_lines()))
dis.dis(f)

3.11:

[(0, 2, 3), (2, 4, 3), (4, 6, 3), (6, 8, 4), (8, 10, 4)]
  3           0 RETURN_GENERATOR
              2 POP_TOP
              4 RESUME                   0

  4           6 LOAD_CONST               0 (None)
              8 RETURN_VALUE

3.12:

[(0, 2, 3), (2, 4, None), (4, 6, 3), (6, 8, 4), (8, 12, None)]
  3           0 RETURN_GENERATOR
              2 POP_TOP
              4 RESUME                   0

  4           6 RETURN_CONST             0 (None)
        >>    8 CALL_INTRINSIC_1         3 (INTRINSIC_STOPITERATION_ERROR)
             10 RERAISE                  1
ExceptionTable:
  4 to 6 -> 8 [0] lasti

3.13:

[(0, 6, 3), (6, 8, 4), (8, 12, None)]
   3           0 RETURN_GENERATOR
               2 POP_TOP
               4 RESUME                   0

   4           6 RETURN_CONST             0 (None)

None     >>    8 CALL_INTRINSIC_1         3 (INTRINSIC_STOPITERATION_ERROR)
              10 RERAISE                  1
ExceptionTable:
  4 to 6 -> 8 [0] lasti

As you can tell, the major difference between 3.12 and 3.11 is the exception table for the instrinsic call. For 3.13 (current main), the missing line number for 2 POP_TOP is filled, but the line number for the exception table is still missing. I'm not sure if this is intentional - I don't know if there is a good "number" that should be there.

I believe the root cause for PyFrame_GetLineNumber is the same. It's not hard to imagine that when you try to get the line number while executing the byte code in the excpetion table, the line number would be -1 (or converted to None at Python level).

I guess the question is to @iritkatriel or @markshannon : is this expected behavior? Should we have the line number assigned to the new-ish bytecode in the exception region?

Also found #107932 (comment)

Assuming these gaps are here to stay, how should line_profiler deal with them? Fill forward maybe?

These instructions should have line numbers. I think we should fix this.

On second thought maybe not - it's a "RERAISE 1" so it propagates the line number of the original RAISE, and we don't want it to create another line event.

Picking this up again - what is the actual issue here? Is it that a line number is -1?
I think this should be expected as possible.

I see. line_profiler reuses the last known line number and this didn't cause any trouble so far. Can be closed I guess.

Thanks.