Skip to content

esp32: Add esp32.Partition class to interface to OTA functions. #4910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from

Conversation

dpgeorge
Copy link
Member

@dpgeorge dpgeorge commented Jul 9, 2019

Following on from the comment #3576 (comment) here is a proposal for a wrapper class for the ESP32 OTA/partition functions.

The API is based around a single Partition class. The constructor is used to create new Partition objects, and methods manipulate such objects.

from esp32 import Partition

# Constructors
Partition(Partition.FIRST, type=Partition.TYPE_APP)
Partition(Partition.FIRST, type=Partition.TYPE_DATA)
Partition(Partition.FIRST, type=Partition.TYPE_DATA, subtype=2, label='nvs')
Partition(Partition.BOOT) # get boot partition
Partition(Partition.RUNNING) # get currently running partition

# Methods
part.info() # returns a 6-tuple of (type, subtype, addr, size, label, encr)
part.readblocks(block_num, buf)
part.writeblocks(block_num, buf)
part.eraseblocks(block_num, size)
part.set_boot() # sets current as bootable, for next reset
part.get_next_update() # returns new Partition

I didn't yet test the OTA functionality

Edit: updated signatures for partition methods.

@Lisa999
Copy link

Lisa999 commented Jul 19, 2019

I'll upgrade to v1.11 and pull this PR to replace my own partition code and see if it works. I'll post the results here in a few days...

@Lisa999
Copy link

Lisa999 commented Jul 20, 2019

AttributeError: 'Partition' object has no attribute 'metadata'

part.info() is working!

>>> part.info()
(0, 16, 65536, 2621440, 'test', False)

0?
Searching... Found it:
`#define PART_TYPE_APP 0x00
#define PART_SUBTYPE_FACTORY 0x00
#define PART_SUBTYPE_OTA_FLAG 0x10
#define PART_SUBTYPE_OTA_MASK 0x0f
#define PART_SUBTYPE_TEST 0x20

#define PART_TYPE_DATA 0x01
#define PART_SUBTYPE_DATA_OTA 0x00
#define PART_SUBTYPE_DATA_RF 0x01
#define PART_SUBTYPE_DATA_WIFI 0x02
#define PART_SUBTYPE_DATA_NVS_KEYS 0x04
#define PART_SUBTYPE_DATA_EFUSE_EM 0x05

#define PART_TYPE_END 0xff
#define PART_SUBTYPE_END 0xff
`
16 is the subtype, in decimal, so it's 0x10 in hex meaning it has an OTA_FLAG which is correct!

65535 is the offset, correct. 2621440 is the size of the partition, correct again.

So far, it's working.

But how to get a list of all partitions?

@dpgeorge
Copy link
Member Author

AttributeError: 'Partition' object has no attribute 'metadata'

part.info() is working!

Yes, sorry, I updated the API without updating the comment (which is now updated).

But how to get a list of all partitions?

I'm don't think #3576 had a way to list all partitions, so that's which I didn't implement it. I thought that the user would anyway know all partitions (eg by name) so could use Partition(FIRST, ...) to retrieve them all.

But if listing all partitions is necessary there's an API in the IDF which could be exposed, eg to get an iterator which iterates over the partitions, like:

for part in Partition.ifind(type=..., subtype=..., label=...):
    print(part)

ifind() would be a static method of Partition and would replace the Partition(FIRST, ...) constructor.

@dpgeorge
Copy link
Member Author

dpgeorge commented Jul 22, 2019

I just pushed a commit to add Partition.find(...) which returns a list of all partitions of the given type/subtype/label. (Making it an iterator was not easy because there would be a memory leak, not releasing the IDF internal iterator.)

To use:

app_parts = Partition.find(Partition.TYPE_APP)
data_parts = Partition.find(Partition.TYPE_DATA)
nvs_parts = Partition.find(Partition.TYPE_DATA, label='nvs')

@Lisa999
Copy link

Lisa999 commented Aug 16, 2019

@dpgeorge: I have also reworked the flashdbev.py file so that it's using the newly created Partition class and the flashdrive is mounted on a named Partition that can be put anywhere:

**
import esp
from esp32 import Partition

class FlashBdev:

SEC_SIZE = 4096

def __init__(self, blocks, offset=None):
    self.blocks = blocks
    if not offset:
        self.START_SEC = esp.flash_user_start() // self.SEC_SIZE
    else:
        self.START_SEC = offset // self.SEC_SIZE

def readblocks(self, n, buf):
    #print("readblocks(%s, %x(%d))" % (n, id(buf), len(buf)))
    esp.flash_read((n + self.START_SEC) * self.SEC_SIZE, buf)

def writeblocks(self, n, buf):
    #print("writeblocks(%s, %x(%d))" % (n, id(buf), len(buf)))
    #assert len(buf) <= self.SEC_SIZE, len(buf)
    esp.flash_erase(n + self.START_SEC)
    esp.flash_write((n + self.START_SEC) * self.SEC_SIZE, buf)

def ioctl(self, op, arg):
    #print("ioctl(%d, %r)" % (op, arg))
    if op == 4:  # BP_IOCTL_SEC_COUNT
        return self.blocks
    if op == 5:  # BP_IOCTL_SEC_SIZE
        return self.SEC_SIZE

size = esp.flash_size()
vfspart = Partition(Partition.FIRST, type=Partition.TYPE_DATA, label='fatvfs').info()
if size < 1024*1024:
# flash too small for a filesystem
bdev = None
elif vfspart:
# if we have a partition for the filesystem, use it
bdev = FlashBdev(vfspart[3] // FlashBdev.SEC_SIZE, vfspart[2])
#print("FAT filesystem found on partition '{}'".format(vfspart[4].strip()))
else:
# for now we use a fixed size for the filesystem
bdev = FlashBdev(2048 * 1024 // FlashBdev.SEC_SIZE)
**

Using this, PR #4987 isn't really necessary anymore.

API is:

    from esp32 import Partition

    # Constructors
    Partition(Partition.FIRST, type=Partition.TYPE_APP)
    Partition(Partition.FIRST, type=Partition.TYPE_DATA)
    Partition(Partition.FIRST, type=Partition.TYPE_DATA, subtype=2, label='nvs')
    Partition(Partition.BOOT)
    Partition(Partition.RUNNING)

    # Methods
    part.metadata() # returns a 6-tuple of (type, subtype, addr, size, label, encr)
    part.readinto(offset, buf)
    part.write(offset, buf)
    part.erase(addr, size)
    part.set_boot()
    part.get_next_update() # returns new Partition
The argument is either an interger (BOOT/RUNNING) or a string label.
@dpgeorge dpgeorge force-pushed the esp32-partition-class branch from fba304c to 058c74d Compare August 17, 2019 04:34
@dpgeorge
Copy link
Member Author

Thanks @Lisa999 for testing. I agree it'd be good to replace the existing flashbdev code to use this new Partition class.

I've just pushed some small changes to the Partition constructor to simplify it now that the Partition.find(...) static method exists. To construct a Partition you now either pass in a label, or a constant, like so:

from esp32 import Partition
bootpart = Partition(Partition.BOOT)
runningpart = Partition(Partition.RUNNING)
fatfs = Partition('fatfs')

For any more complex Partition lookup use Partition.find(type, subtype, label)

I'm happy to merge this PR as it is now, then proceed to update flashbdev.py and partitions.csv in a separate PR.

@dpgeorge
Copy link
Member Author

Ok, this was merged in 05eb897, with docs added, and also writeblocks modified to erase before write, so the Partition class acts like a proper block device.

@dpgeorge dpgeorge closed this Aug 20, 2019
@dpgeorge dpgeorge deleted the esp32-partition-class branch August 20, 2019 06:53
@dpgeorge
Copy link
Member Author

See #5027 for follow up.

@tve
Copy link
Contributor

tve commented Mar 31, 2020

How were the OTA-related methods tested, e.g. esp32_partition_set_boot? As far as I know they require an ota_boot partition and none of the provided partition tables includes one, so I wonder...

@dpgeorge
Copy link
Member Author

dpgeorge commented Apr 2, 2020

How were the OTA-related methods tested,

I did not test set_boot() or get_next_update() methods.

@Lisa999
Copy link

Lisa999 commented Apr 2, 2020

How were the OTA-related methods tested,

I did not test set_boot() or get_next_update() methods.

I did test 'set_boot()' and it's working just fine in my project...

@chtinnes
Copy link

chtinnes commented Apr 7, 2020

For me it is not working: #5471

Partition table and partitions look good but calling set_boot() gives me the error ESP_ERR_INVALID_ARG.

EDIT: Working now =) I used ota_1 as first ota partition slot instead of ota_0 =/

tannewt added a commit to tannewt/circuitpython that referenced this pull request Jun 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants