Skip to content

Deceiving exception handling messages when available memory is low (below ~41k) #138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jake1164 opened this issue Jul 7, 2023 · 3 comments
Open

Deceiving exception handling messages when available memory is low (below ~41k) #138

jake1164 opened this issue Jul 7, 2023 · 3 comments

Comments

@jake1164
Copy link

jake1164 commented Jul 7, 2023

I have run into issues with using the Requests library on a Pico W while getting JSON responses from an https url. The exception messages will vary based on the amount of free memory. I have created a quick sample to test and show the behavior.

The sample just loads images into a list and prints out the JSON returned until the board runs low on memory.

Below ~41k the exceptions make it look like a certification error and has the message 'Sending request failed'
Below ~27k it just fails and gives no exception message at all.

The same result happens with requests.get(url) and requests.get(url, stream=True).

Crudely tested on:
CircuitPython 8.2.0, 8.1.0, 8.0.5, 8.0.2
Requests 1.14.1, 1.13.4, 1.12.9

before 13488
appended images 44160
appended image 44000
Getting json
getting url: https://api.weather.gov/alerts/active?area={state}
free memory 44000
free memory after 6224
{'correlationId': '606fcf6', 'status': 400, 'instance': 'https://api.weather.gov/requests/606fcf6', 'detail': 'Bad Request', 'type': 'https://api.weather.gov/problems/BadRequest', 'title': 'Bad Request', 'parameterErrors': [{'parameter': 'query.area[0]', 'message': 'Does not have a value in the enumeration ["AM","AN","GM","LC","LE","LH","LM","LO","LS","PH","PK","PM","PS","PZ","SL"]'}, {'parameter': 'query.area[0]', 'message': 'Does not have a value in the enumeration ["AL","AK","AS","AR","AZ","CA","CO","CT","DE","DC","FL","GA","GU","HI","ID","IL","IN","IA","KS","KY","LA","ME","MD","MA","MI","MN","MS","MO","MT","NE","NV","NH","NJ","NM","NY","NC","ND","OH","OK","OR","PA","PR","RI","SC","SD","TN","TX","UT","VT","VI","VA","WA","WV","WI","WY","MP","PW","FM","MH"]'}, {'parameter': 'query.area[0]', 'message': 'Failed to match exactly one schema'}]}
before 11184
appended images 41856
appended image 41696
Getting json
getting url: https://api.weather.gov/alerts/active?area={state}
free memory 41696
x509-crt-bundle:PK verify failed with error FFFFBD70
x509-crt-bundle:Failed to verify certificate
requests.get(url): Sending request failed
{}
...
before 32640
appended images 30336
appended image 30176
Getting json
getting url: https://api.weather.gov/alerts/active?area={state}
free memory 30176
requests.get(url): Sending request failed
{}
before 30336
appended images 28032
appended image 27872
Getting json
getting url: https://api.weather.gov/alerts/active?area={state}
free memory 27872
requests.get(url): 
{}

Any secrets to getting this to work on low memory situations (like a pico driving a Matrix LED)?
Can we get better exception messages to indicate what is happening?
@jake1164 jake1164 changed the title Poor exception handling (lack of meaningful messages) for low available memory (below ~41k) Deceiving exception handling messages when available memory is low (below ~41k) Jul 7, 2023
@jake1164
Copy link
Author

jake1164 commented Jul 9, 2023

Is 41k really the minimum memory requirement for SSL to work?

@tannewt
Copy link
Member

tannewt commented Jul 10, 2023

SSL does take a good amount of memory and network requests generally do. You may be able to save memory with the json_stream library: https://github.com/adafruit/Adafruit_CircuitPython_JSON_Stream/ It'll pick values out in a stream rather than creating a bunch of objects holding all of the json data.

@jake1164
Copy link
Author

SSL does take a good amount of memory and network requests generally do. You may be able to save memory with the json_stream library: https://github.com/adafruit/Adafruit_CircuitPython_JSON_Stream/ It'll pick values out in a stream rather than creating a bunch of objects holding all of the json data.

This is interesting, but does not really address the problem since the exception is thrown at the requests.get(url) which is the equivilant of your resp = session.get(SCORE_URL) which is before the JSON is read into memory.

On a side note, your library crashes at resp.close() with a ValueError: invalid syntax for integer with base 16 exception thrown from the requests library. Specifically at line 339 of requests.close()

                    chunk_header = bytes(self._readto(b"\r\n")).split(b";", 1)[0]
                    ```

@bablokb bablokb mentioned this issue Feb 28, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tannewt @jake1164