Thanks for the report. I can reproduce the issue.

Closures are almost working in viper. But it's storing a raw integer value instead of an integer object when assigning to the closed-over variable.

As a workaround you can explicitly create the integer-object (a small int) via:

def foo():
    x : int = 0
    @micropython.viper
    def inner() -> int:
        nonlocal x
        q : int = int(x)
        q += 1
        x = q << 1 | 1  # <- workaround!
        return int(x)
    return inner

bar = foo()
bar()  # 1
bar()  # 2

As I see it, this would make it possible to program generators with @micropython.viper.
Could someone please give a very short explanation, why/how this workaround works?

Could someone please give a very short explanation, why/how this workaround works?

In Python everything is an object, and the system needs to know how to extract the object type. For integer objects the low bit is set. So, the integer value x is packed into an object via the formula x << 1 | 1.

Viper works with raw integer values, not integer objects, so to communicate with Python code it must convert between objects and raw values. The bug here is that it's not done automatically.

It's perhaps worth noting that if the bug is fixed the workround will produce unexpected results. I opted for the integer array approach to avoid this hazard. Array indexing is fast and there doesn't seem to be a performance penalty.

I know this is an old issue, but there is still another solution which I think is a bit more readable:

import builtins
def foo():
    x = 0
    @micropython.viper
    def inner() -> int:
        nonlocal x
        x = builtins.int( int(x)+1 )
        return int(x)
    return inner

Tested on MicroPython v1.23.0-preview.47.g16c6bc47c on 2024-01-17; Generic ESP32S3 module with Octal-SPIRAM with ESP32S3.

Another idea does not work but yields a new error:
The idea was: make the enclosing function @micropython.viper
too, then its variables, used for the closure already have the C format.

@micropython.viper
def foo():
    x : int = 0         # <-- ViperTypeError: local 'x' used before type known
    @micropython.viper
    def inner() -> int:
        nonlocal x
        q : int = x
        q += 1
        x = q  # x never increments
        return int(x)
    return inner

bar = foo()
print(bar())  # 0
print(bar())  # 0

@bixb922 Please can you explain how this works? :)

I'm still in the process of finding out how this all works. I'll ramble a bit about this topic here :-), please bear with me for the long post.

Also: viper has improved since the OP. I could not reproduce all examples given.

The viper int data type and the MicroPython int are two very different beasts. The viper data type refers to a raw memory location on the stack and is not an object, just a raw variable (much like a C stack variable). The MicroPython int data type has an object descriptor, and may be (AFAIK) allocated on the stack for local variables and on the heap for global variables and instance variables (is this last statement correct?).

Inside a viper function, int always means viper int. If you want to access the MicroPython int, you can import builtins and use builtins.int. So I think it's best to refer to these two data types as "viper int" and "builtins.int".

The viper code emitter (which is the same as the native code emitter plus the viper data types) detects at compile-time which local variables are of one of the viper types. It does this by taking the first statement where the variable is created (assigned). If it gets assigned a viper int, then it is flagged as such, if not, it's an "object". This is why viper ints only can local variables (edit: nonlocal being a special case).

Once a variable is detected as "viper int", the type cannot be changed and trying to do this will cause the compile-time error "ViperTypeError: local 'x' has type 'int' but source is 'object'".

Due to these differences, interoperability between these two int data types cannot be entirely transparent.

These are cases where the transition has to be explict, with a type cast or explicit conversion:

• Expressions having the two data types cause compile-time errors like "ViperTypeError: can't do binary op between 'int' and 'object'". Use the viper int() cast (different from the MicroPython int() function, which is builtins.int()).
• Expressions combining of float and viper int (or any other object)
• Initial assignment with expressions outside some range, example x = 0xffffffff (example taken from post 32 bit integer logic in Viper? #11259, which you all surely know). Note that 0x3fffffff is the maximum value and not 0x7fffffff. Is this the "small integer" range of MicroPython (ESP32-S3)? Initialization with integer expressions outside this range will create a builtins.int object.
• Viper function parameters and return value: type hints needed for casting/converting
• Assign to a builtins.int variable which does not reside on the heap. This is the case of nonlocal of the OP (edited, could not reproduce this).
• There are probably more cases I'm not aware of, any ideas what else to test?

These are cases where the transition seems to be automatic:

• Pass a parameter to a function. Viper ints are converted automatically a builtins.int. (For example, this is why type(viper_int_variable) returns builtin class int)
• Assign to a global variable, or to self.x (instance variable). Unlike nonlocal, these variables are on the heap, and this kind of assignment converts to builtins.int on the fly. There must be a reason why this is handled differently from the nonlocal case, I don't know.
• There are probably more cases I'm not aware of, any ideas what else to test?

For the closure example, the simplest code would be:

def nonlocal_test_inner_viper():
    x = 1
    @micropython.viper
    def inner_function():
        nonlocal x
        # Next line gives "ViperTypeError: can't do binary op between 'object' and 'int'"
        x = x + 1
    inner_function()
    return inner_function

The message means: "x is an object and 1 is a viper int, can't add them" (At first I read this the other way around and got stumped).

When you convert the 1 to a builtins.int, everything works well. (edit) I think there is no problem with viper here.

I am writing all this stuff down to a document to be put soon somewhere on github. I'd love to get some feedback.

Thanks for that, very informative.

Note that 0x3fffffff is the maximum value

This is the MicroPython object model - I suggest you read this.

This is the MicroPython object model - I suggest you read this.

Thanks!!

Here is still another twist. And yes, there is a problem somewhere, but a bit different from the OP:

def nonlocal_test():
    x = 1
    y = None
    z = None
    @micropython.viper
    def internal_viper():
        nonlocal x, y, z
        # Operation of viper int with nonlocal builtins.int needs 
        # converting all terms to builtins.int, if not we get a ViperTypeError
        x = x + builtins.int(1)
        viperx:int = 111
        # The next statement does *not* convert from viper int to builtins.int
        y = viperx
        # This works:
        z = builtins.int(viperx*2)
    internal_viper()
    print(f"nonlocal test {x=}, expected 2")
    print(f"nonlocal test {y=}, expected 222")
    print(f"nonlocal test {z=}, expected 111")

nonlocal_test()

The output is:

nonlocal test x=2, expected 2
nonlocal test y=55, expected 222
nonlocal test z=222, expected 111

With the way objects are stored now in mind, it seems that y gets the viper format int, which is interpreted as half the value.

So nonlocal int operations only work if the nonlocal was previously created as int. Other data types seem not to be affected.

As this is very interesting, moving back to discussions probably is best here, as new posts in issues are mailed to all core developers and thus make a lot of noise. I appreciate the discussion and think it would be great if you/we could copy your/our contribution over to the original discussion too, for others searching viper. No offence meant, just my feeling:-)