Skip to content

uasyncio: core.py frozen results in OverflowError: overflow converting long int to machine word #177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
riegaz opened this issue May 14, 2017 · 19 comments

Comments

@riegaz
Copy link

riegaz commented May 14, 2017

I just created the following files within the modules folder:
uasyncio / init.py
core.py
queues.py
collections / deque.py

When I then run my code I get the following error:

Traceback (most recent call last):
File "main.py", line 226, in
File "main1.py", line 92, in main
File "uasyncio/core.py", line 133, in run_forever
File "uasyncio/core.py", line 38, in call_later_ms_
OverflowError: overflow converting long int to machine word

This is the function:

    def call_later_ms_(self, delay, callback, args=()):
        self.call_at_(time.ticks_add(self.time(), delay), callback, args)
@riegaz riegaz changed the title uasyncio: core.py frozen results in overflowerror uasyncio: core.py frozen results in OverflowError: overflow converting long int to machine word May 14, 2017
@pfalcon
Copy link
Contributor

pfalcon commented May 14, 2017

If you want to have this resolved, you will need to provide a minimal testcase to reproduce this.

@riegaz
Copy link
Author

riegaz commented May 14, 2017

For my case, the delay variable becomes 1045220558000 which is far beyond the sys.maxint.

Testcase:
Make this file frozen:

import utime as time
def overfloww(val):
    return time.ticks_add(time.ticks_ms(), val)

and call it with the value from above.

I guess this is a bug within the uasyncio core.py?

@riegaz
Copy link
Author

riegaz commented May 14, 2017

Why actually

int(2147483647 + 1) does give me 2147483648? shouldn't it give some overflowed value?

for now I assume that 2147483647 is the sys.maxint for micropython, it is at least for python2.7 on windows.

@dpgeorge
Copy link
Member

For my case, the delay variable becomes 1045220558000

In ms that's about 33 years. Why is your code scheduling something so far in the future?

int(2147483647 + 1) does give me 2147483648? shouldn't it give some overflowed value?

No, Python has arbitrary sized integers so there is no overflow in Python scripts.

@riegaz
Copy link
Author

riegaz commented May 14, 2017

In ms that's about 33 years. Why is your code scheduling something so far in the future?

It is a reminder for my 65. birthday 🍰 No, I guess there is a problem with the uasyncio after make it frozen. I just created two tasks and call open_connection(). How to fix this? Am I the first one freezing uasyncio?

No, Python has arbitrary sized integers so there is no overflow in Python scripts.

Then the time.ticks_add function needs to be changed? Or is 33 adding 33 years as ticks a minor usecase?

@dpgeorge
Copy link
Member

No, I guess there is a problem with the uasyncio after make it frozen. I just created two tasks and call open_connection(). How to fix this? Am I the first one freezing uasyncio?

Please provide your full test case that breaks, ideally stripped down to a minimal script that still exibits the same problem.

Then the time.ticks_add function needs to be changed? Or is 33 adding 33 years as ticks a minor usecase?

While there's no overflow in scripts, there can still be overflow in the C code for certain operations. In this case 33 years in milliseconds is a number that ticks_diff is not designed to handle. Note that having such a large dynamic range, from ms to years, is difficult to achieve within fixed memory requirements.

@pfalcon
Copy link
Contributor

pfalcon commented May 14, 2017

Am I the first one freezing uasyncio?

No, I froze not just uasyncio, but full web stack and an application built on it, see
micropython/micropython#1346 (comment)

@pfalcon
Copy link
Contributor

pfalcon commented May 14, 2017

Then the time.ticks_add function needs to be changed?

No. But you may need to change the way you use it.

Or is 33 adding 33 years as ticks a minor usecase?

uasyncio doesn't work with years. It's a real-time scheduling system. Read the docs for how ticks_add() works. If you want to create an alarm in years, you will need to create an alarm which will work like that, uasyncio doesn't support creation of alarms, just scheduling of realtime tasks.

@riegaz
Copy link
Author

riegaz commented May 14, 2017

Main function:

import uasyncio as asyncio
from uasyncio.queues import Queue
import gc

@asyncio.coroutine
def Tcp(url):
    print('tcpier started '+str(gc.mem_free()))
    while True:
        success, reader, writer = yield from asyncio.open_connection(url, 3674)
        if not success:
            print('unable to connect to '+str(url)+' '+str(gc.mem_free()))
            await asyncio.sleep(1)
            continue
        else:
            #print(reader, writer)
            print('connected to '+str(url))
            while success:
                d = yield from reader.read()
                if not d:
                    print('connection lost..trying to connect again')
                    break
                #print('data '+str(gc.mem_free()))
                q.put_nowait(d)
                d=None
                gc.collect()

@asyncio.coroutine
async def Data():
    print('processer started '+str(gc.mem_free()))
    while True:
        data = await(q.get())
        print('GET DATA') 

def main():
    loop = asyncio.get_event_loop()
    loop.create_task(Data())
    loop.create_task(Tcp('192.168.178.27'))
    loop.run_forever()
    print('some return')

q = Queue()

Here are my modules. I changed the following:

core.py added a print of the delay value

    def call_later_ms_(self, delay, callback, args=()):
	print('delay '+str(delay))
        self.call_at_(time.ticks_add(self.time(), delay), callback, args)

init.py changed the open_connection function to allow for tcp connection

def open_connection(host, port):
    if DEBUG and __debug__:
        log.debug("open_connection(%s, %s)", host, port)
    s = _socket.socket()
    #s.setblocking(False)
    ai = _socket.getaddrinfo(host, port)
    addr = ai[0][4]
    success = False
    try:
        s.settimeout(1)
        s.connect(addr)
        s.setblocking(False)
        success = True
    except OSError as e:
        #if e.args[0] != uerrno.EINPROGRESS:
        #    raise
        return success, None, None
    if DEBUG and __debug__:
        log.debug("open_connection: After connect")
    yield IOWrite(s)
#    if __debug__:
#        assert s2.fileno() == s.fileno()
    if DEBUG and __debug__:
        log.debug("open_connection: After iowait: %s", s)
    return success, StreamReader(s), StreamWriter(s, {})

uasyncio.zip
logging.zip

Freeze this files, and run this tcp server in a python notebook:

#%%
import socket, time, sys
import threading
import pprint
from datetime import datetime
from random import randint
import pickle
#%%
TCP_IP = ''
TCP_PORT = 3674
BUFFER_SIZE = 1024
s = socket.socket()
s.bind((TCP_IP,TCP_PORT))
s.listen(10)

#%%
client_socket, addr = s.accept()
client_socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
print ('Connected with ' + addr[0] + ':' + str(addr[1]))

#%%
data = 'testmeifyoucan'.encode()
           
#%%
print(addr)
s.close()
#%%
for e in range(0,100):
    for i in range(0,100):
        client_socket.send(data)
        time.sleep(0.05)
    print(e)
    time.sleep(0.5)
print(i)
#%%

client_socket.send(data)

@peterhinch
Copy link
Contributor

@riegaz Are you confusing integers with ticks values? Integers have arbitrary precision: arithmetic operators produce expected values regardless of magnitude. Ticks values use modular arithmetic to constrain them to a single machine word. This is done for reasons of efficiency but necessarily limits their dynamic range. Long duration delays may readily be implemented e.g. by using utime.time().

As a personal opinion I think a test case is more likely to command the attention of the maintainers if raised against the standard official version of the library.

@riegaz
Copy link
Author

riegaz commented May 15, 2017

@riegaz Are you confusing integers with ticks values? Integers have arbitrary precision: arithmetic operators produce expected values regardless of magnitude. Ticks values use modular arithmetic to constrain them to a single machine word. This is done for reasons of efficiency but necessarily limits their dynamic range. Long duration delays may readily be implemented e.g. by using utime.time().

I was just joking about the 33 years. For me I just found a problem arising after creating a frozen module out of uasyncio. I do not want to abuse the uasyncio for an alarm clock.

As a personal opinion I think a test case is more likely to command the attention of the maintainers if raised against the standard official version of the library.

I only moved the socket.setblocking(False) after the connect() because it prevents to use the code for my TCP server. But you are right. I will use the original uasyncio and try to reproduce this. Since my code works without beeing frozen I thought I could just show my example.

@riegaz
Copy link
Author

riegaz commented May 15, 2017

@peterhinch So I just downloaded the fresh uasyncio and moved it to modules.

uasyncio / init.py
core.py
queues.py

I ran the same code I provided above. The result is as posted in my first post:

Traceback (most recent call last):
File "main.py", line 226, in
File "main1.py", line 92, in main
File "uasyncio/core.py", line 133, in run_forever
File "uasyncio/core.py", line 38, in call_later_ms_
OverflowError: overflow converting long int to machine word

@dpgeorge
Copy link
Member

@riegaz you need to provide more information, and a proper reproducible test case. What is main.py and main1.py? uasyncio.open_connection() only returns 2 values so your "main" code posted abouv does not work. Please use unmodified library code for uasyncio. Please state the board/port/hardware you are running on. Please provide a full, minimal case that breaks.

@riegaz
Copy link
Author

riegaz commented May 16, 2017

Here is my minimal testcase:

board: esp8266 (ESP-12S)

I set up building environment like here:

http://akshaim.github.io/IoT/MicroPython_ESP8266/MP_ESP_101.html

Create the following folder within esp-open-sdk/micropython/esp8266/modules
uasyncio / init.py core.py queues.py

I added logging.py to the modules folder as well

Here is the firmware I build:
firmware-combined.zip

main.py

import uasyncio as asyncio
from uasyncio.queues import Queue
import gc
import socket as _socket
from uasyncio.core import *

async def ProcessData1():
    print('processer started '+str(gc.mem_free()))
    while True:
        data = await(q.get())
        
def main():  
    loop = asyncio.get_event_loop()
    loop.create_task(ProcessData1())
    loop.run_forever()
    print('some return')

print('asd3')
q = Queue()

main()

which fails with this message:

PYB: soft reboot
#12 ets_task(40100164, 3, 3fff829c, 4)
asd3
processer started 22016
Traceback (most recent call last):
File "main.py", line 22, in
File "main.py", line 16, in main
File "uasyncio/core.py", line 129, in run_forever
File "uasyncio/core.py", line 38, in call_later_ms_
OverflowError: overflow converting long int to machine word
MicroPython v1.8.7-751-g0c57979c on 2017-05-15; ESP module with ESP8266
Type "help()" for more information.

Another testcase:

https://github.com/peterhinch/micropython-async/blob/master/aqtest.py

@dpgeorge
Copy link
Member

What implementation of uasyncio/queues.py And what about collections.deque, since that's needed be the default queues.py? Running your main() code above works ok for me using a dummy Queue class.

@riegaz
Copy link
Author

riegaz commented May 16, 2017

Sorry, I forgot to mention collections.deque. I used everything as is from micropython-lib. I changed nothing.

@dpgeorge
Copy link
Member

@riegaz thanks for providing that info, I think I found the problem: the freezing of floats was recently broken by micropython/micropython@ec53460 and I've now fixed that in micropython/micropython@88c51c3 . It's good that you picked this problem up otherwise it may have made it into the upcoming v1.9 release, so thanks for submitting this issue!

Please can you try the latest MicroPython code to see if your code now works?

@riegaz
Copy link
Author

riegaz commented May 16, 2017

@dpgeorge I can confirm that it works. Thanks a lot!

@pfalcon
Copy link
Contributor

pfalcon commented May 16, 2017

Thanks for figuring it out everyone.

@pfalcon pfalcon closed this as completed May 16, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants