Skip to content

atmel-samd: Brownout / low battery causes SPI flash erase on Circuit Playground Express #280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tdicola opened this issue Sep 24, 2017 · 22 comments
Assignees
Labels
Milestone

Comments

@tdicola
Copy link

tdicola commented Sep 24, 2017

I've noticed an odd but somewhat nasty issue that appears to be reproducible. With Circuit Playground Express (but perhaps other M0 express and maybe M0 basic boards) if my project uses a lipo battery that runs low enough to brownout and turn off, after I swap in a new battery the entire SPI flash is erased and my old program is gone. It's happened twice in a row with my arc reactor project, and I have a feeling it always happens when power goes low like this (I've only ever run the battery completely dead twice and both times the FS was destroyed on next power up). My suspicion is in the low voltage or brownout state perhaps the SPI flash chip isn't readable and the board brownout resets, fails to read the flash, and assumes the FS is bad and erases it all (or maybe the FS was actually corrupted during the brownout).

Opening this as an issue for now to investigate further. I think we want to setup a board with a bench supply to VBAT and start it at 3.3 volts, then lower it bit by bit into brownout mode (probably around 2.5 volts or so) and see if it repros for easier investigation. I'm not sure there's a fix but if we can understand what's happening better perhaps there are workarounds or ways to prevent the FS from being destroyed.

@tannewt tannewt added the bug label Sep 25, 2017
@tannewt tannewt added this to the 2.0 milestone Sep 25, 2017
@dhalbert
Copy link
Collaborator

dhalbert commented Oct 4, 2017

I have a simple fix that won't reset the filesystem if it reboots into safe mode. I did try to reproduce this with a bench supply, and only managed to make it happen a couple of times, erratically. I also wrote a high-duty-cycle blinky program on CPX that lit all the Neopixels, and ran a LiPo down, but could not get it to erase. @tdicola when you're back I could pass you a .uf2 with the fix and you can test it with the arc reactor project.

@dhalbert
Copy link
Collaborator

dhalbert commented Oct 9, 2017

@tdicola Patch is in branch 2.x_issue_280_brownout in my repo https://github.com/dhalbert/circuitpython/. I have been fiddling with the voltage on a CPX but have only gotten it to wipe the flash once, and I can't replicate reliably.

@dhalbert
Copy link
Collaborator

@tdicola Pull request #319 would fix this, I think. I was able to break the filesystem a few times with the bench supply, and then after #319, I couldn't, but it wasn't really a good test. If you see this problem again after building the latest 2.x or using the upcoming 2.0.1 release, please reopen.

@BillyDonahue
Copy link

Using cpx 2.1.0, I have the same problem very reproducible with multiple units. In my case the fs is wiped reliably by loosening and wiggling the Lipo battery connector to produce an intermittent contact.

@deshipu
Copy link

deshipu commented Dec 4, 2017

I had the same problem with a feather m0 express once, when I didn't remove the USB plug all the way.
I wonder if it could be a quicker and more convenient way of erasing flash than having to flash a special firmware for it.

@dhalbert
Copy link
Collaborator

jafine on discord reports a CIRCUITPY erase with 2.2.0, perhaps due to a low battery.

@dhalbert dhalbert reopened this Jan 22, 2018
@jafine
Copy link

jafine commented Jan 23, 2018

I observed this occurring multiple times while we were doing demoing the TR808 drum (with fruit attached). In order to recalibrate, we were un-plugging and re-plugging in a 3.7V lipo battery.One of our kid helpers was leaving the lipo plug loosely in the socket, so that she could quickly disconnect and reconnect. I suspect that it was the quick disconnection and reconnection of power that may have contributed to this. I can re-create the issue by replicating the the loose socket quick switching of power. Later I connected a JST push-button switch to prevent this, and noticed that it eventually wiped again. This follows with what @BillyDonahue mentioned previously with 2.1.0.

@croadfeldt
Copy link

croadfeldt commented Apr 30, 2018

Not sure if it's known or not, but this appears to be an issue still with 2.2.4 as well.

My setup is the development kit, with the 3 AAA batteries, battery case and CPX. Often enough if the battery switch is shut off, the code is lost and filesystem is reformatted when the battery switch is turned back on.

@tannewt
Copy link
Member

tannewt commented May 1, 2018

Huh, I wonder if its reproducible with a AAA switch because there is bounce on the switch.

@croadfeldt Could you save file, load an arduino sketch and then switch it on and off. Its still not clear to us if this is a CircuitPython issue or a hardware/SPI flash issue.

@croadfeldt
Copy link

@tannewt Will do later this week when I have some time again.

@tannewt
Copy link
Member

tannewt commented May 2, 2018

Thanks @croadfeldt! @dhalbert is looking into the brownout detector settings now too.

@dhalbert
Copy link
Collaborator

dhalbert commented May 3, 2018

Raising brownout voltage limit to 2.77V did not fix the problem. After testing by turning various things off, adding delays, etc., I think that the usual reason for this problem is the boot_out.txt write that happens on every hard reset. If the write gets interrupted by a power fail (or reset?), the filesystem is damaged and will be re-created on the next startup. If I turn off the boot_out.txt write, I cannot reproduce the problem. If it's on, bobbling the power will cause it to happen in short order.

Usually what's written to boot_out.txt is the same every time. It contains mostly version information. It only changes if there's actually a boot.py that prints out something different, or the version of CircuitPython changes.

So we could check to see whether the file contents is identical what we want to write, and skip the write in that case, which would be most of the time. I'll try this out.

We can't really delay writing the contents of boot_out.txt, because when USB comes up, the filesystem is then usually set to be read-only to CircuitPython, and it's too late to write the file.

(tagging @ladyada)

@dhalbert
Copy link
Collaborator

dhalbert commented May 3, 2018

This will also save wear and tear on the flash.

@croadfeldt
Copy link

croadfeldt commented May 3, 2018

@dhalbert @tannewt
Results

started w/ circuitpython 2.2.4 - running code.py - no usb connection, flop power switch on battery pack quickly results in wiped filesystem with just boot_txt

compiled code running raw - no usb connection, flop power switch has no effect on code, operation

redploy of circuitpython 2.2.4 - reinstall of code.py on filesystem, FWIW this persists through compiled code push from arduino IDE, this was a neat surprise when I put the 2.2.4 CP back on and my files re-appeared.

@dhalbert
Copy link
Collaborator

dhalbert commented May 3, 2018

@croadfeldt Could you try a couple of things:

  1. Here is a test version of CircuitPython: cpx-2v7-no-boot-out.uf2.zip. The .uf2 file is in a .zip file because GItHub requires it.Could you unzip it and try this version? It will refuse to work if the battery voltage is <2.8v (instead of 1.7v). It also does not write boot_out.txt when it starts up. If you still see a boot_out.txt, it is old. If the filesystem is erased, CIRCUITPY will be completely empty on startup. Try this test version with the battery switch as you have before, and see if your own files get erased or not.

  2. If you have a multimeter, could you measure the voltage (under load) of the current set of batteries in the battery holder? The easiest way to do this is to measure the voltage across the VOUT and GND pins on the CPX (with USB not attached). Those pins are on either side of the battery socket. I want to know if the current batteries you're using are kind of weak. I have the same battery holder, and with a brand new set of batteries I measure 4.48V. With some good batteries I am getting 4.35V. It's a little less than 1.5V*3 or so because there's a protection diode that drops the voltage about 0.2 or 0.3V. It's not bad if the batteries are a bit run down -- we need to handle that case.

@croadfeldt
Copy link

@dhalbert

Quick update.

  1. Was able to test this quick this morning, in testing 2v7, I was not able to reproduce the filesystem wipe. Put 2.2.4 back on, wiped the filesystem within 30 seconds.

  2. Will be able ot test with a multimeter later tonight.

Chris

@tannewt
Copy link
Member

tannewt commented May 3, 2018

Thank you for the help @croadfeldt !

@dhalbert
Copy link
Collaborator

dhalbert commented May 3, 2018

@croadfeldt Here is a more sophisticated update:
cpx-2v7-smart-boot_out.uf2.zip. This file will write boot_out.txt only if the version information has changed or there is a boot.py file or equivalent, and it waits 1.5 seconds before writing the file. So the first time you use this, it will write the file (after 1.5 seconds), and then after that won't write the file, because it won't be different. Thanks for all the testing!

@croadfeldt
Copy link

croadfeldt commented May 4, 2018

@tannewt @dhalbert - you're welcome! Thanks for your work to make this all work so well. :)

More results:

  1. cpu-2v7-smart-boot_out does NOT wipe the filesystem. boot_out populates if the file is missing on startup as well. So both seem to be as expected, no wipe and boot_out populates.

  2. Voltage is 4.678 under moderate load, moderate being all 10 LEDs up and running. No LEDS is 4.8v.

  3. Putting 2.2.4 back on and I can now cause a FS wipe in under 5 seconds, lots of practice. 👍

@dhalbert
Copy link
Collaborator

dhalbert commented May 4, 2018

@croadfeldt ok, thanks! This is going into the next release. Your testing and your ability to break it have been very helpful 😃

Your batteries seem better than mine!

@croadfeldt
Copy link

@dhalbert You're welcome, glad to be of help. I saw the note on PR #802, I see others are working as well. Awesome, thank you sir!

@tannewt
Copy link
Member

tannewt commented May 4, 2018

Thanks again everyone! Glad to see this much improved!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants