Skip to content

memory_reduction #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Jan 7, 2022
Merged

memory_reduction #15

merged 18 commits into from
Jan 7, 2022

Conversation

jposada202020
Copy link
Contributor

No description provided.

@jposada202020
Copy link
Contributor Author

jposada202020 commented May 18, 2021

Please feel free to suggest name changes, and comments @tannewt

Notes:

I have it working alone for the Qt-Py with a py extension

Adafruit CircuitPython 6.1.0 on 2021-01-21; Adafruit QT Py M0 with samd21e18
>>> import dps310_simpletest
Temperature = 25.01 *C
Pressure = 1020.45 hPa

However when I added the code for the Full Featured class in the same file, I got a memory error.

So I decided to separate the files and include the light class in the __init__.py however also have a memory error.

So two things here:

  1. is the file size a factor here?
  2. I know python while importing will parse the files, so even with putting the lighter version in the __init__ it will parse the other files in the directory, and maybe the qt-py runs out of memory while parsing?

Conclusion

The present change will have memory allocation error, if loaded with th py file, however it tested ok with the mpy files.

Adafruit CircuitPython 6.1.0 on 2021-01-21; Adafruit QT Py M0 with samd21e18
>>> import dps310_simpletest
Temperature = 31.04 *C
Pressure = 1021.47 hPa

@jposada202020 jposada202020 marked this pull request as ready for review May 18, 2021 10:50
@tannewt
Copy link
Member

tannewt commented May 18, 2021

  • is the file size a factor here?

If you get a memory error on import then use an mpy. It'll strip the white space and comments. Comments take space when parsing I believe. I think the resulting imported size is still the same.

To measure the memory footprint after import use gc.mem_free().

  • I know python while importing will parse the files, so even with putting the lighter version in the __init__ it will parse the other files in the directory, and maybe the qt-py runs out of memory while parsing?

It'll only parse the file that matches the import statement. That file may also import which would cause more files to be parsed.

@jposada202020
Copy link
Contributor Author

Just to be clear, in our options here. Fail means= Memory Error - allocation Error

Method Extension Test
Original Library mpy Fail
Customized File for QtPy only py Pass
File with two classes, one for QTPy + DPS310_Featured mpy Fail
library split in two __init__ + DPS310_Featured py Fail
library split in two __init__ + DPS310_Featured mpy Pass

So according to this, we have two options. Two differents files, one for the Qtpy, or the library with the general methods in the _init_.py and only to be used for the qtpy as mpy

@tannewt
Copy link
Member

tannewt commented May 19, 2021

@jposada202020 what is the difference between the qtpy version that works with py and the __init__ that works as mpy? Does the __init__ alone work as py too?

@jposada202020
Copy link
Contributor Author

jposada202020 commented May 19, 2021

@jposada202020 what is the difference between the qtpy version that works with py ?

I tailored the class, to have only basic configuration for the QTpy and nothing else. the __init__ version has, basic configuration in the _init_ and the full configuration class in a diferent py file. So basically the __init__ version is the version included in the current PR. You could use the initi version apart and work with the QTPy alone, however when they are packaged like this they do not work, see explanation below.

@jposada202020 Does the __init__ alone work as py too?

No it does not, I tried but it does not, do not understand why, that is why I asking about the parsing in the import, I do not have an explanation on why it does not work. I am not importing the Class CV and other properties if I am only using the class in the init, however it does not work.

@jposada202020
Copy link
Contributor Author

jposada202020 commented May 20, 2021

@tannewt it worked!!! 🛩️ works with py files. Tested in QTPY and Feather RP2040. Could you please propose a better name for the Featured class, I used Configurable, but I do not like it. Documentation is good, I add another example, so once you propose a better name I could polish and do the little details

@tannewt
Copy link
Member

tannewt commented May 20, 2021

@tannewt it worked!!! small_airplane works with py files. Tested in QTPY and Feather RP2040. Could you please propose a better name for the Featured class, I used Configurable, but I do not like it. Documentation is good, I add another example, so once you propose a better name I could polish and do the little details

How about basic and advanced? I don't think you need to call it dps310_basic either. basic is just fine because the package has dps310 in it.

@jposada202020
Copy link
Contributor Author

Good Idea, will do changes as suggested. I like the idea of basic and advanced... and have the basic file without a name

@jposada202020
Copy link
Contributor Author

Test were done using different boards, and different versions of CircuitPython

Tests done for the RP2040

  1. Usage code inside the class for readthedocs --> OK
  2. Code in Readme.rst --> OK
  3. Three examples
Adafruit CircuitPython 6.2.0 on 2021-04-05; Adafruit Feather RP2040 with rp2040
>>> import dps310_simpletest_advanced
Temperature = 26.55 *C
Pressure = 1025.00 hPa
Adafruit CircuitPython 6.2.0 on 2021-04-05; Adafruit Feather RP2040 with rp2040
>>> import dps310_low_power_weather_station
Temperature = 26.46 *C
Pressure = 1025.00 hPa
Adafruit CircuitPython 6.2.0 on 2021-04-05; Adafruit Feather RP2040 with rp2040
>>> import dps310_simpletest
Temperature = 26.37 *C
Pressure = 1025.02 hPa

Tests done for the QTPY MO

Only the simpletest was done here as other class will not load because memory restrictions

Adafruit CircuitPython 6.1.0 on 2021-01-21; Adafruit QT Py M0 with samd21e18
>>> import dps310_simpletest
Temperature = 26.55 *C
Pressure = 1025.01 hPa

NOTE

For I do not what reason I had to change the import order for the QTPY in the example code to avoid the memory Error. I let the designers explain that to me. Thanks

Tests done for the Metro ESP32S2

Adafruit CircuitPython 7.0.0-alpha.1-853-gcbe1aa74f on 2021-04-26; Adafruit Metro ESP32S2 with ESP32S2
>>> import dps310_simpletest
Temperature = 26.21 *C
Pressure = 1025.00 hPa
Adafruit CircuitPython 7.0.0-alpha.1-853-gcbe1aa74f on 2021-04-26; Adafruit Metro ESP32S2 with ESP32S2
>>> import dps310_simpletest_advanced
Temperature = 26.28 *C
Pressure = 1024.97 hPa
Adafruit CircuitPython 7.0.0-alpha.1-853-gcbe1aa74f on 2021-04-26; Adafruit Metro ESP32S2 with ESP32S2
>>> import dps310_low_power_weather_station
Temperature = 26.34 *C
Pressure = 1024.93 hPa

Tests done for the FeatherS2

Adafruit CircuitPython 6.1.0-beta.2 on 2020-12-03; FeatherS2 with ESP32S2
>>> import dps310_simpletest
Temperature = 26.20 *C
Pressure = 1024.91 hPa
Adafruit CircuitPython 6.1.0-beta.2 on 2020-12-03; FeatherS2 with ESP32S2
>>> import dps310_simpletest_advanced
Temperature = 26.22 *C
Pressure = 1024.89 hPa
Adafruit CircuitPython 6.1.0-beta.2 on 2020-12-03; FeatherS2 with ESP32S2
>>> import dps310_low_power_weather_station
Temperature = 26.18 *C
Pressure = 1024.89 hPa

NOTE: FOR RELEASING PURPOSES THIS IS A MAJOR CHANGE, IT WILL REQUIRE VERSION CHANGE

@tannewt
Copy link
Member

tannewt commented May 21, 2021

Good Idea, will do changes as suggested. I like the idea of basic and advanced... and have the basic file without a name

I think you should have adafruit_dps310/basic.py and adafruit_dps310/advanced.py. No need to have dps310 in the name twice.

@jposada202020
Copy link
Contributor Author

Will do :) Thanks

@jposada202020
Copy link
Contributor Author

Final test was done with RP2040 and Qtpy M0. the simpletest did not load with a py, however with the mpy files was ok

Adafruit CircuitPython 6.2.0 on 2021-04-05; Adafruit QT Py M0 with samd21e18
>>> import dps310_simpletest
Temperature = 24.78 *C
Pressure = 1016.48 hPa

Temperature = 24.79 *C
Pressure = 1016.48 hPa

When extracting the basic file from the directory the memory error go away, and before changing the import order also correct the memory error, but when testing today this did not happened. To discuss in Discor. (was my two questions las friday :) )

@tannewt
Copy link
Member

tannewt commented May 24, 2021

@jposada202020 want to do anything else on this?

@jposada202020
Copy link
Contributor Author

@tannewt I think we are good to merge,however if you agree, could we wait until I do the BME280?, I want to see if I could learn/have some other elements that maybe could help here also.

@tannewt
Copy link
Member

tannewt commented May 24, 2021

Up to you. :-)

Copy link
Contributor

@lesamouraipourpre lesamouraipourpre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, please consider adding sea_level_pressure as a property to basic.py. It is referenced in multiple places but there is no getter/setter for it.

For the elevation property, please add some documentation stating that the sea_level_pressure needs to be set accurately and daily (or more frequently if the pressure is changing quickly) to give accurate results. If they are only using the pressure and not the elevation then this does not need set. Unfortunately, I do not know of a good source of sea-level pressure values to recommend.
Note: My Garmin feinx 6 can drift by almost 100m of elevation (usually less than 50m) if I have not used GPS to record an activity or manually calibrate it in a while.

Circup

There should also be instructions explaining that circup may fail on a small board (as it will try to install everything) and explain how to download the library from github/PyPI and only install the basic.py.

jposada202020 and others added 7 commits May 25, 2021 07:48
Co-authored-by: James Carr <70200140+lesamouraipourpre@users.noreply.github.com>
Co-authored-by: James Carr <70200140+lesamouraipourpre@users.noreply.github.com>
Co-authored-by: James Carr <70200140+lesamouraipourpre@users.noreply.github.com>
Co-authored-by: James Carr <70200140+lesamouraipourpre@users.noreply.github.com>
Co-authored-by: James Carr <70200140+lesamouraipourpre@users.noreply.github.com>
Co-authored-by: James Carr <70200140+lesamouraipourpre@users.noreply.github.com>
Co-authored-by: James Carr <70200140+lesamouraipourpre@users.noreply.github.com>
@jposada202020
Copy link
Contributor Author

I see, let see if it fits in the basic running in a QtPy, but I like it very much, hopefully it will fit :)

@jposada202020
Copy link
Contributor Author

@lesamouraipourpre the sea_level_pressure property fits, so I added. Could you help me with a fgood explanation on the Readme regarding how to use the library.
We can install the library, and use it with Circup or Pypi, however for small boards we must use the mpy version of the basic.py file, the other will raise a memory error.
But you could have them install it in the board no problem.

Adafruit CircuitPython 6.2.0 on 2021-04-05; Adafruit QT Py M0 with samd21e18
>>> import dps310_simpletest
Temperature = 27.15 *C
Pressure = 1015.92 hPa
Sea Level Pressure = 1013.25 

@jposada202020 jposada202020 marked this pull request as ready for review May 27, 2021 13:45
Copy link
Contributor

@lesamouraipourpre lesamouraipourpre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested the changes and they work fine on a QT-PY (with only the basic installed).

Attached is diff against README.rst with a possible explanation of how to install on memory-limited boards.
dps310.diff.txt

@jposada202020
Copy link
Contributor Author

Oh..... I love the comments and the explanation, make me laugh, it was very good Thanks!!
I think we will need to get @kattni approval.
For me is just perfect :)

Copy link
Contributor

@kattni kattni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks everyone for contributing to this! For the README update, I viewed the diff, and there are some minor issues with it that can be addressed once it's included in this or a future PR. Please decide how you want to handle that, and get it added here if that's the plan so I can request changes.

I also want to make sure folks are aware that these changes affect at least one guide. @jposada202020 are you prepared to update the guide, or did you have a plan for that? I can't remember if we got you added to the Learn system or not.

@jposada202020
Copy link
Contributor Author

@kattni Yes I have been added. more details on that..
I propose that we made the changes on the readme after this PR is merged.
I also propose to merge this PR when I am being able 100% to do the learning guide.
There is no rush and I prefer that both things are in sync, hope you do not mind @lesamouraipourpre . This is an old issue, and I consider this more as an improvement, so no need to rush on my part.

@jposada202020
Copy link
Contributor Author

@lesamouraipourpre While looking at the learning guide to make the changes, something came up I would like to test this with Blinka, (both in the RP and the Pico with Micropython). what do you think?
just to check those boxes too :)

Copy link
Contributor

@kattni kattni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as the guide is updated in tandem, I'm prepared to approve this. Please try to remember to tag me on the PR for the README update.

Thanks!

@jposada202020
Copy link
Contributor Author

Will do thanks

@lesamouraipourpre
Copy link
Contributor

@lesamouraipourpre While looking at the learning guide to make the changes, something came up I would like to test this with Blinka, (both in the RP and the Pico with Micropython). what do you think?
just to check those boxes too :)

If I remember tomorrow I can try it on my RPi3 with the new library, seeing as that's its normal home for logging to adafruit.io.
I also have some Picos about but haven't tried them with Blinka/Micropython yet.

@lesamouraipourpre
Copy link
Contributor

RaspberryPi 3

All examples run correctly.
I've modified and simplified my monitoring code to the following which transmits to Adafruit.IO correctly as well:

import board
import busio
import time

# pip install adafruit-io
from Adafruit_IO import *

from adafruit_dps310.advanced import DPS310_Advanced as DPS310

def configure_dps310_sensor(i2c):
    dps310 = DPS310(i2c)

    dps310.wait_temperature_ready()
    dps310.wait_pressure_ready()

    # Use the figure from your local forecast for today.
    # hPa (hectoPascals) == millibars
    dps310.sea_level_pressure = 1015.0
    print("Altitude: {:.0f}m".format(dps310.altitude))

    return dps310

def configure_adafruit_io():
    try:
        from secrets import secrets
    except ImportError:
        print ("secrets are kept in secrets.py, please add them there!")
        raise

    return Client(secrets["aio_username"], secrets["aio_key"])

def main():
    i2c = board.I2C()
    # i2c = busio.I2C(board.SCL, board.SDA)

    room_sensor = configure_dps310_sensor(i2c)
    io = configure_adafruit_io()

    while True:
        temperature = room_sensor.temperature
        pressure = room_sensor.pressure
        print ("{:.1f}° {:.1f}".format(temperature, pressure))

        try:
            io.send_data("room-temperature", temperature)
            io.send_data("room-pressure", pressure)
        except Exception:
            # IOError, so wait longer
            time.sleep(300)

        time.sleep(30)

if __name__ == "__main__":
    main()

@jposada202020
Copy link
Contributor Author

That is a nice example, would you mind if we use it? for the library?

@lesamouraipourpre
Copy link
Contributor

That is a nice example, would you mind if we use it? for the library?

Go for it. License is Unlicense to keep reuse happy. And maybe add a link to an Adafruit.IO learn guide.
I've not tried this on a microcontroller, I've got a nagging memory that it might be Blinka only but I can't remember why.

example secrets.py:

secrets = {
    "aio_username" : "YOUR_USERNAME",
    "aio_key" : "YOUR_AIO_TOKEN",
}

@jposada202020
Copy link
Contributor Author

Will do in a different PR, thanks

@lesamouraipourpre
Copy link
Contributor

Micropython on RP Pico is a no-go. There seem's to be an issue with Adafruit_CircuitPython_Register getting the wrong value; if I bypass Register and manually read the I2C bus I get the correct value.

@tannewt Is there any reason that you can think of that Register wouldn't work on MicroPython. If you think it should work, I'll work up minimal example code to file

@jposada202020
Copy link
Contributor Author

I found the same problem, Did not go as deep as yourself in the troubleshooting. See issue here adafruit/Adafruit_Blinka#482.
But I think this new info is important for @makermelissa

@makermelissa
Copy link
Collaborator

Thanks for finding that @lesamouraipourpre and @jposada202020. I'm working on finishing up a guide, but am curious to look more into that right after. I suspect register is using some feature of Blinka that's perhaps implemented differently. If either of you want to take a deeper look, that would be awesome.

@lesamouraipourpre
Copy link
Contributor

Using the pending PR on Blinka (adafruit/Adafruit_Blinka#483) this library works on Blinka/MicroPython on RP2040.

@jposada202020
Copy link
Contributor Author

That is good news, I think I will merge over the weekend, that way when releasing next week I could correct the learning guide.
Thanks for checking that out

@evaherrada evaherrada changed the base branch from master to main June 7, 2021 18:52
@FoamyGuy
Copy link
Contributor

FoamyGuy commented Jan 7, 2022

I merged main and resolved conflicts to get this passing CI. Successfully tested the basic and advanced simpletest scripts with a Feather RP2040 and DPS310.

I'm going to merge this and update the relevant parts of the guide page now

@FoamyGuy FoamyGuy merged commit 6398fd4 into adafruit:main Jan 7, 2022
@ladyada
Copy link
Member

ladyada commented Jan 7, 2022

thanks!

adafruit-adabot added a commit to adafruit/Adafruit_CircuitPython_Bundle that referenced this pull request Jan 7, 2022
Updating https://github.com/adafruit/Adafruit_CircuitPython_DPS310 to 2.0.0 from 1.2.7:
  > Merge pull request adafruit/Adafruit_CircuitPython_DPS310#15 from jposada202020/light_class
  > update rtd py version
Neradoc added a commit to Neradoc/Adafruit_CircuitPython_FunHouse that referenced this pull request Jan 9, 2022
This fixes this after the latest DPS310 release. I assumed we want the advanced by default on the Fun House.
```py
Traceback (most recent call last):
  File "code.py", line 13, in <module>
  File "adafruit_funhouse/__init__.py", line 110, in __init__
  File "adafruit_funhouse/peripherals.py", line 86, in __init__
AttributeError: 'module' object has no attribute 'DPS310'
```
adafruit/Adafruit_CircuitPython_DPS310#15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants