Add precise_time module #839

margaret · 2018-05-14T21:27:54Z

Changes

This adds the precise_time module which exposes system ticks as an int.

I'm pretty much just going off of shared-bindings/time/ and https://github.com/adafruit/circuitpython/blob/master/docs/common_hal.md

I basically copy-pasted the common-hal/time/__init__.* to common-hal/precise_time/__init__.*. Not sure if there's a way to get the shared_bindings/precise_time files to use the ones from common-hal/time/ instead.

Testing

See comments. Manually tested only on gemma m0 board (that is the board I have).

Status 👀

Addressing review comments.

Edit 2018-06-01: planning to finish this this weekend 🐌

Issue

#519 – More accurate time reporting

tannewt

Thank you for the early PR! I've added a couple notes

tannewt · 2018-05-15T13:54:51Z

shared-bindings/precise_time/__init__.c

+//|   Returns an always increasing value of time with an unknown reference
+//|   point. Only use it to compare against other values from `monotonic`.
+//|   Unlike time.monotonic, which is a float, the precision of this does not
+//|   degrade over time.


What units is this?

tannewt · 2018-05-15T13:56:00Z

shared-bindings/precise_time/__init__.h

+#include <stdint.h>
+
+extern uint63_t common_hal_time_monotonic(void)
+extern void common_hal_time_delay_ms(uint32_t)


You'll need semicolons here.

margaret · 2018-05-15T17:31:01Z

Adafruit CircuitPython 3.0.0-alpha.6-86-g1b4e064-dirty on 2018-05-15; Adafruit Gemma M0 with samd21e18
>>> 
>>> import precise_time
>>> import time
>>> time.monotonic();precise_time.monotonic()
49.099
49100
>>> time.monotonic();precise_time.monotonic()
52.188
52189
>>> time.monotonic();precise_time.monotonic()
67.481
67482
>>> time.monotonic();precise_time.monotonic()
100.681
100682
>>> time.monotonic();precise_time.monotonic()
105.638
105639
>>> time.monotonic();precise_time.monotonic()
190.729
190730

margaret · 2018-05-15T18:26:55Z

Manually tested build at f1ea5d3 precise_time.sleep on gemma_m0 board ✅

margaret · 2018-05-15T18:36:28Z

Some of the Travis builds are failing with

E: Package 'gcc-multilib' has no installation candidate
The command "sudo apt-get install -y python3 gcc-multilib pkg-config libffi-dev libffi-dev:i386 qemu-system" failed and exited with 100 during .

~~Seems like a flake? Can someone 🔄 those?~~ nvm

dhalbert · 2018-05-15T19:31:41Z

The new common-hal routines need to be implemented (or dummied up) for the esp8266 and nrf52 builds as well.

dhalbert · 2018-05-15T19:41:36Z

This is going to be great!

tannewt · 2018-05-17T17:34:05Z

@margaret please mention us when this is ready for another review.

margaret · 2018-05-21T16:39:09Z

@tannewt @dhalbert ready for review again. Should I rebase and/or squash?

The two failures were on not nrf builds that passed on HEAD~1. Error is The command "sudo apt-get install realpath" failed and exited with 100 during . which seems like a flake.

dhalbert

see comments

dhalbert · 2018-05-15T19:27:35Z

shared-bindings/precise_time/__init__.c

+//|   :rtype: int
+//|
+STATIC mp_obj_t precise_time_monotonic(void) {
+    return mp_obj_new_int_from_uint(common_hal_precise_time_monotonic());


(I have not been in the loop on this, so @tannewt and @margaret please let me know if you've discussed this already).

mp_obj_new_int_from_uint() will silently truncate the common_hal_precise_time_monotonic() value when it overflows 2^32. 2^32 msecs is about 49.7 days. I could imagine a long-lived sensing program that hits that limit. If the incoming common_hal_precise_time_monotonic() is > 2^32 you could call mp_obj_new_int_from_ull(), which takes a long long (which is 64 bits).

Note that on non-longint builds, py/objint.c is used, and mp_obj_new_int_from_ull() throws OverflowError no matter what it's passed (even the arg would fit in a small int). On longint impls, py/objint_mpz.c is used. So the value needs to be tested and one or the other needs to be called depending on the size.

I think the behavior on non-longint impls should probably be documented: precise_time.time_monotonic() will raise OverflowError when the board has been running for 2^30 msecs or more (about 12.4 days)

We hadn't discussed how to deal with overflow, but these suggestions sound good. I'll work on these items later today.

dhalbert · 2018-05-15T19:28:38Z

shared-bindings/precise_time/__init__.c

+//|
+//|   :param int seconds: the time to sleep in milliseconds
+//|
+STATIC mp_obj_t precise_time_sleep(mp_obj_t milliseconds_o) {


@tannewt @margaret Maybe this should be called precise_time.sleep_ms, since the units are different than time.sleep? If we ever implement microsecond sleeps then we can have a precise_time.sleep_us as well.

That makes sense. I'll rename it to sleep_ms.

dhalbert · 2018-05-19T17:00:54Z

ports/nrf/mpconfigport.h

@@ -221,6 +221,7 @@ extern const struct _mp_obj_module_t pulseio_module;
 extern const struct _mp_obj_module_t busio_module;
 extern const struct _mp_obj_module_t board_module;
 extern const struct _mp_obj_module_t os_module;
+extern const struct _mp_obj_module_t precise_time;


This gave an error on the nrf build. I think you meant precise_time_module on line 224 here.

./mpconfigport.h:275:62: error: 'precise_time_module' undeclared here (not in a function); did you mean 'precise_time'? { MP_OBJ_NEW_QSTR (MP_QSTR_precise_time ), (mp_obj_t)&precise_time_module }, \ ^ ../../py/objmodule.c:233:5: note: in expansion of macro 'MICROPY_PORT_BUILTIN_MODULES' MICROPY_PORT_BUILTIN_MODULES ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ make: *** [build-feather52-s132/py/objmodule.o] Error 1

dhalbert · 2018-05-21T17:09:21Z

I restarted the failed jobs and they ran fine.
I made some inline comments a while ago but I think GitHub didnt show you them because I hadn't started a review. Now I did, see what you think.

jepler · 2018-05-21T18:07:51Z

I'd hate to do all this work and end up with something that can't keep precise time indefinitely without needing to use longints. 12.4 days to OverflowError on a non-express board is not a great outcome. 12.4 days until your carefully crafted code becomes markedly slower because it switched to 'long's isn't that hot either.

To that end, I have two general ideas. Option 1, return a preallocated list of two ints, for example [seconds,microseconds] or [seconds,milliseconds]. Now you're good for 2^30 seconds of runtime, but arithmetic on time values becomes more complicated and needs to allocate lists to do it too.

Option 2, operate in modulo mode, such as modulo 2^30, so that the the monotonic time simply wraps after that many milliseconds. You need some helper functions for comparing times (e.g., time 0 has to be treated as less than 2^30-1) and adding times (e.g., adding 30 seconds to 2^30-1 seconds has to wrap), but at least it's all on the efficient-to-store integer type. But you can't say anything about time differences greater than 2^30 milliseconds.

These options are pretty half-baked, but maybe there's a good idea in here waiting to be extracted and implemented.

margaret · 2018-05-21T18:48:58Z

@jepler My initial thought is that something like option 1 would be a nicer interface from the user's POV since they won't have to write their own logic to handle the switch every 12 days. @tannewt any thoughts?

tannewt · 2018-05-22T17:13:21Z

@margaret I think the simplest thing is to not support precise_time on non-express builds where long ints aren't supported.

This macro can test for it:

#if MICROPY_LONGINT_IMPL == MICROPY_LONGINT_IMPL_NONE

dhalbert · 2018-06-12T13:17:45Z

I revisited this again, and found this interesting new accepted PEP, to be implemented in Python 3.7: https://www.python.org/dev/peps/pep-0564/. See below. The new functions defined in this PEP return integer nanosecond values. We could implement some of these: we still have a wraparound or overflow problem, but using these names allows compatibility with regular Python.

We could still have a single precise_time.time_monotonic() that could return a tuple of (seconds, nanoseconds), or maybe call it integer_time.time_monotonic().

Quoting from PEP 564:

This PEP adds six new functions to the time module:

time.clock_gettime_ns(clock_id)

time.clock_settime_ns(clock_id, time: int)

time.monotonic_ns()

time.perf_counter_ns()

time.process_time_ns()

time.time_ns()

These functions are similar to the version without the _ns suffix, but return a number of nanoseconds as a Python int.

For example, time.monotonic_ns() == int(time.monotonic() * 1e9) if monotonic() value is small enough to not lose precision.

These functions are needed because they may return "large" timestamps, like time.time() which uses the UNIX epoch as reference, and so their float-returning variants are likely to lose precision at the nanosecond resolution.

dhalbert · 2018-06-12T14:39:54Z

Note that 2^30 nanoseconds is about 1 second, so on a non-longint built, the new PEP 564 functions aren't really helpful at all.

PaulKierstead · 2018-09-01T04:17:28Z

It would be really nice to have a function which efficiently returned an int with ms or ns resolution. PEP 564 looks like it would be nice as it is CPython compatible. My only real concern is how efficient is it going to be. Ultimately there is a bunch of cases were basically you want to keep comparing the current time to some known point in time; i.e. have you exceeded the timestamp of your next event time. This is a really common use case. When the events are coming fast and furious, it would be nice if it was highly efficient. I'm inclined to think in addition to the functions which go ahead and allocate at int object (at considerable expense) it might be nice to have one which just compares your previously allocated int to the current tick count (in some useful unit). I'm just musing there. If that is too painful, then the PEP 564 looks best to me, as it is standard-y. It requires some kind of big int support, but unless you want the handstands that utime when through to handle rollover (and things like utimeq), you need big int support anyway. (or at least 60-odd bit support). So the real decisions, IMO, would be PEP 564 + big int vs some kind of utime system where we get time which rolls-over (and all that entails).

dhalbert · 2018-10-06T23:25:31Z

Interest in this topic has revived recently. I did a little more looking and thinking.

I think people rarely need msec resolution over weeks, but they need time keeping that doesn't wrap around or that is precise enough to measure relatively short intervals. There are no builtin timing mechanisms that start counting at zero when they're instantiated.

We could implement an elapsed time interval counter that counts, in, say, msec or usec. It would start at zero on __init__(), could be started and stopped, and could be reset to zero. This resembles what @jepler and @PaulKierstead are proposing.
We could implement a subset of PEP564: time.time_ns() and time.monotonic_ns(), but only on longint builds. On non-longint builds, we would throw NotImplementedError.
This library: https://github.com/flit/elapsedtimer has several useful features and a simple interface, and could be usable. It could be implemented as a builtin module or reimplemented on top of time.monotonic(), maybe hiding wraparound. It handles case 1 above, I think.

tannewt · 2018-10-23T00:42:32Z

@margaret Thanks again for starting this work. I'm closing this PR because we just merged in time.monotonic_ns support which is in CPython 3.7.

add precise_time module

1a1aab8

tannewt reviewed May 15, 2018

View reviewed changes

margaret added 3 commits May 15, 2018 10:11

add missing semicolons

45d42c3

more detail in module docs

1b4e064

compiles on gemma_m0

6ed180d

margaret added 3 commits May 15, 2018 13:49

fix rst spacing

411140a

add precise_time.monotonic to tick count

0490d5b

add precise_time.sleep

f1ea5d3

margaret changed the title ~~[WIP] add precise_time module~~ Add precise_time module May 15, 2018

fix rst

419d2ee

margaret added 3 commits May 19, 2018 08:50

add precise_time to esp8266 common-hal

2bbda80

add precise_time to nrf common-hal

cd717e0

fix qstr def for precise_time_module in nrf

ab3d2a6

dhalbert requested changes May 21, 2018

View reviewed changes

dhalbert added this to the Long term milestone Jun 15, 2018

tannewt closed this Oct 23, 2018

dhalbert mentioned this pull request May 8, 2020

time is messed-up (RTC and utime) micropython/micropython#5969

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add precise_time module #839

Add precise_time module #839

margaret commented May 14, 2018 •

edited

Loading

tannewt left a comment

tannewt May 15, 2018

tannewt May 15, 2018

margaret commented May 15, 2018 •

edited

Loading

margaret commented May 15, 2018 •

edited

Loading

margaret commented May 15, 2018 •

edited

Loading

dhalbert commented May 15, 2018

dhalbert commented May 15, 2018

tannewt commented May 17, 2018

margaret commented May 21, 2018

dhalbert left a comment

dhalbert May 15, 2018

margaret May 21, 2018

dhalbert May 15, 2018

margaret May 21, 2018

dhalbert May 19, 2018

dhalbert commented May 21, 2018

jepler commented May 21, 2018

margaret commented May 21, 2018 •

edited

Loading

tannewt commented May 22, 2018

dhalbert commented Jun 12, 2018

dhalbert commented Jun 12, 2018

PaulKierstead commented Sep 1, 2018

dhalbert commented Oct 6, 2018

tannewt commented Oct 23, 2018

Add precise_time module #839

Add precise_time module #839

Conversation

margaret commented May 14, 2018 • edited Loading

Changes

Testing

Status 👀

Issue

tannewt left a comment

Choose a reason for hiding this comment

tannewt May 15, 2018

Choose a reason for hiding this comment

tannewt May 15, 2018

Choose a reason for hiding this comment

margaret commented May 15, 2018 • edited Loading

margaret commented May 15, 2018 • edited Loading

margaret commented May 15, 2018 • edited Loading

dhalbert commented May 15, 2018

dhalbert commented May 15, 2018

tannewt commented May 17, 2018

margaret commented May 21, 2018

dhalbert left a comment

Choose a reason for hiding this comment

dhalbert May 15, 2018

Choose a reason for hiding this comment

margaret May 21, 2018

Choose a reason for hiding this comment

dhalbert May 15, 2018

Choose a reason for hiding this comment

margaret May 21, 2018

Choose a reason for hiding this comment

dhalbert May 19, 2018

Choose a reason for hiding this comment

dhalbert commented May 21, 2018

jepler commented May 21, 2018

margaret commented May 21, 2018 • edited Loading

tannewt commented May 22, 2018

dhalbert commented Jun 12, 2018

dhalbert commented Jun 12, 2018

PaulKierstead commented Sep 1, 2018

dhalbert commented Oct 6, 2018

tannewt commented Oct 23, 2018

margaret commented May 14, 2018 •

edited

Loading

margaret commented May 15, 2018 •

edited

Loading

margaret commented May 15, 2018 •

edited

Loading

margaret commented May 15, 2018 •

edited

Loading

margaret commented May 21, 2018 •

edited

Loading