WIP: Python composable q/spi flash filesystems #5249

andrewleech · 2019-10-22T06:46:51Z

This PR aims to simplify the definition and use of filesystems in general, but most specifically making it easier to create filesystems on spi and qspi flash chips.

The interfaces to these filesystems has been extended both for access from micropython at runtime, and in C for board definitions.

For example:

>>> import machine, uos
>>> vfs = uos.VfsFat(machine.SPIFlash(machine.QSPI(), length=16*1024*1024))
>>> uos.mount(vfs, "/mine")
>>> uos.listdir("/mine")
['device_test', 'assays', 'results', 'System Volume Information']

The machine.SPIFlash() constructor also takes a start argument which lets you offset the start of a filesystem, making it easy to have multiple separate filesystems present on a single flash chip.

Example stm32 board definition
bdev.c

#include "drivers/bus/qspi.h"
#include "drivers/memory/spiflash.h"

extern const mp_qspi_proto_t qspi_proto;

// External SPI flash uses standard QSPI interface
STATIC mp_spiflash_cache_t spi_bdev_cache;

mp_spiflash_t spiflash_instance = {
    .qspi_proto = &qspi_proto,
    .cache = &spi_bdev_cache,
};

mpconfigboard.h

#define MICROPY_VFS_ROOT_FS_MOUNT MP_QSTR__slash_flash
#define MICROPY_VFS_ROOT_FS_FORMAT &mp_fat_vfs_type
#define MICROPY_VFS_ROOT_FS_BDEV mp_machine_spiflash_make_bdev( \
    /* spi */ &spiflash_instance,  \
    /* start_block */ 0,  \
    /* block_count */ (16*1024*1024)/MP_SPIFLASH_DEFAULT_BLOCK_SIZE,  \
    /* block_size */ MP_SPIFLASH_DEFAULT_BLOCK_SIZE \
)

In conjunction with #5228 changing the format of the boot or any runtime filesystem to LittleFS should be as simple as

mpconfigboard.h

// #define MICROPY_VFS_ROOT_FS_FORMAT &mp_fat_vfs_type
#define MICROPY_VFS_ROOT_FS_FORMAT &mp_type_vfs_littlefs2

or

>>> vfs = uos.VfsLittleFS1(machine.SPIFlash(machine.QSPI(), length=16*1024*1024))

TODO:

Test / cleanup stm32 main with regard to internal flash usage
Update other stm32 boards which have spiflash chips defined
Finish updating ports other than stm32
update documentation

…aces. Consolidate interfaces to machine.Spi Simplify configuration of drivers/spiflash device.

…_spi_obj_t.

…ne.SPIFlash interfaces.

…s from mountpoint, filesystem type and block device.

dpgeorge · 2019-10-30T03:44:54Z

Thanks for this. I had a look through the commits and some of it is good but some of it is overkill and what seems to be unnecessary refactoring.

From a high-level view, I agree it'd be good to provide a Python-level interface to construct/configure SPI-based block devices. But it doesn't necessarily need to take a SPI/QSPI object, it can just take an integer specifying which internal bus/flash to use. Eg machine.SPIFlash(0) accesses the first built-in flash device. And it can be partitioned like machine.SPIFlash(0, start=1024, length=10*1024). And maybe the class can simply be machine.Flash() to also cover internal flash like on PYBv1.x.

One problem is how the user can specify the layout (partitioning) of the flash storage, if they are not building custom firmware. It's not really possible to do this in a (non frozen) script because that script must come from the filesystem, which doesn't yet exist.

On esp32 there's a partition table which fills this role of laying out where the filesystem is, but even there it still needs a frozen _boot.py to specify how to mount the filesystem.

Maybe we need a small "frozen boot sector" on each port which contains a (small) frozen Python script that brings up the initial filesystem (or whatever it wants to do) and which can be dynamically frozen via the user without rebuilding firmware. It can be as simple as storing a "bare" .mpy file in the first block of the flash, or even somewhere in internal flash.

Well, this relates to the long-requested feature of dynamically frozen scripts. So perhaps that feature is needed first, before the filesystem layout can be configured via the user.

That's a big digression, so in the meantime to make progress implementing machine.Flash() (or Storage() or SPIFlash()) would be a good start.

dpgeorge · 2019-10-30T03:46:17Z

drivers/memory/spiflash.h

 typedef struct _mp_spiflash_t {
-    const mp_spiflash_config_t *config;


The reason this config struct was separated out was so that it could be put in ROM, while the mp_spiflash_t struct was kept small and in RAM. What was your reason to combine them?

I did this early-on as I was still understanding the overall structure of spiflash subsystem, thinking it simplified things. For boards where the interfaces are hard-coded I can see how this makes sense though, probably better if I revert this change.

dpgeorge · 2019-10-30T03:48:10Z

extmod/machine_qspi.h

+    uint32_t (*read_cmd)(mp_obj_base_t *obj, uint8_t cmd, size_t len);
+    void (*read_cmd_qaddr_qdata)(mp_obj_base_t *obj, uint8_t cmd, uint32_t addr, size_t len, uint8_t *dest);
+
+} mp_machine_qspi_p_t;


Wouldn't it be possible to just reuse mp_qspi_proto_t (in drivers/bus/qspi.h) for this purpose?

andrewleech · 2019-11-06T00:26:28Z

But it doesn't necessarily need to take a SPI/QSPI object, it can just take an integer specifying which internal bus/flash to use. Eg machine.SPIFlash(0) accesses the first built-in flash device.

Passing in python objects was done to allow defining a spiflash interface on any one of hardware spi, soft spi, hardware qspi or soft qspi.
While the hardware variants of these could be known and indexed at compilation, the software interfaces can be added at runtime, so I don't think a simple integer indexing scheme is going to allow for this.
Perhaps it could accept a list of qstr's for hard coded types (SPI_1, SPI_9, QSPI_1, INTERNAL) or objects for software/runtime generated interfaces, it would just require either a dynamic or maintained lookup table of qstrs to proto's for each port.

it can be partitioned like machine.SPIFlash(0, start=1024, length=10*1024).

Yes, this is exactly what's intended with this change, the start argument is already provided (just not shown in my python examples above).

maybe the class can simply be machine.Flash() to also cover internal flash like on PYBv1.x.

I certainly intended to expose the internal flash (in a separate pr). A block interface for it is already written, perhaps it could be exposed behind the same python interface. It would just mean a bit of branching in C to point the interface to either the spiflash block interface or the pybflash one.

osresearch · 2020-11-24T19:12:03Z

Having the SPIFlash device take a python SPI object makes it easier to add different SPI devices, especially for boards that have a variety of interfaces. Hardcoding integers seems like very limiting and would require recompilation to add support for new devices.

There is also unification that could happen in the spi device protocol structs - mp_machine_spi_p_t and mp_spi_proto_t are not compatible, despite having almost the same interface.

Fix usb_hid_gc_collect()

andrewleech · 2022-06-27T02:02:57Z

This PR has been a useful talking point towards planning a new / consolidated storage interface for micropython, but is not very helpful in its own right. As such it's better to close this and work as a team on an improved storage layer.

pi-anl added 3 commits October 22, 2019 17:49

extmod/spiflash: Add machine.SPIFlash machine.QSPI micropython interf…

3cfb691

…aces. Consolidate interfaces to machine.Spi Simplify configuration of drivers/spiflash device.

machine/spi: Update SPI classes to use new base class type mp_machine…

335b5b4

…_spi_obj_t.

stm32/spi: Align SPI type with extmod/spi, add machine.QSpi and machi…

d529fe2

…ne.SPIFlash interfaces.

pi-anl force-pushed the extmod_spiflash branch from c534955 to 8cb494e Compare October 22, 2019 06:49

pi-anl added 5 commits October 23, 2019 09:50

stm32/flash: Allow system to run without flash/storage defined.

016ccc1

stm32/storage: Align with extmod/vfs block protocol.

d8c7cff

extmod/vfs: add function mp_vfs_load_filesystem to create and mount f…

d8e43e9

…s from mountpoint, filesystem type and block device.

stm32/main: Simplify definition of root filesystem.

1c04d8a

stm32/spiflash: Update definitions for mboot and boards using spi flash.

f68f7b2

andrewleech force-pushed the extmod_spiflash branch from 8cb494e to f68f7b2 Compare October 22, 2019 22:51

dpgeorge reviewed Oct 30, 2019

View reviewed changes

dpgeorge mentioned this pull request Nov 14, 2019

stm32: littlefs support #5330

Merged

tannewt pushed a commit to tannewt/circuitpython that referenced this pull request Aug 28, 2021

Merge pull request micropython#5249 from dhalbert/usb_hid-gc-fix

dd89ff2

Fix usb_hid_gc_collect()

dpgeorge added the drivers label Nov 30, 2021

andrewleech closed this Jun 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

WIP: Python composable q/spi flash filesystems #5249

WIP: Python composable q/spi flash filesystems #5249

Uh oh!

andrewleech commented Oct 22, 2019 •

edited

Loading

Uh oh!

dpgeorge commented Oct 30, 2019

Uh oh!

dpgeorge Oct 30, 2019

Uh oh!

andrewleech Nov 6, 2019 •

edited

Loading

Uh oh!

dpgeorge Oct 30, 2019

Uh oh!

andrewleech commented Nov 6, 2019 •

edited

Loading

Uh oh!

osresearch commented Nov 24, 2020

Uh oh!

andrewleech commented Jun 27, 2022

Uh oh!

Uh oh!

		typedef struct _mp_spiflash_t {
		const mp_spiflash_config_t *config;

Uh oh!

WIP: Python composable q/spi flash filesystems #5249

WIP: Python composable q/spi flash filesystems #5249

Uh oh!

Conversation

andrewleech commented Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpgeorge commented Oct 30, 2019

Uh oh!

dpgeorge Oct 30, 2019

Choose a reason for hiding this comment

Uh oh!

andrewleech Nov 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dpgeorge Oct 30, 2019

Choose a reason for hiding this comment

Uh oh!

andrewleech commented Nov 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

osresearch commented Nov 24, 2020

Uh oh!

andrewleech commented Jun 27, 2022

Uh oh!

Uh oh!

andrewleech commented Oct 22, 2019 •

edited

Loading

andrewleech Nov 6, 2019 •

edited

Loading

andrewleech commented Nov 6, 2019 •

edited

Loading