Skip to content

nrf: Add block device for VFS (v2) #7619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Aug 8, 2021

Conversation

dpgeorge
Copy link
Member

@dpgeorge dpgeorge commented Aug 5, 2021

This is a rebase of #5472 on latest master.

@codecov-commenter
Copy link

codecov-commenter commented Aug 5, 2021

Codecov Report

Merging #7619 (e5e0553) into master (f834fef) will decrease coverage by 0.01%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7619      +/-   ##
==========================================
- Coverage   98.29%   98.28%   -0.02%     
==========================================
  Files         154      154              
  Lines       20015    20014       -1     
==========================================
- Hits        19674    19671       -3     
- Misses        341      343       +2     
Impacted Files Coverage Δ
py/obj.c 96.82% <0.00%> (-0.80%) ⬇️
py/runtime.c 99.39% <0.00%> (ø)
py/objenumerate.c 100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f834fef...e5e0553. Read the comment docs.

@mzdaniel
Copy link
Contributor

mzdaniel commented Aug 6, 2021

Thank you for looking into this @dpgeorge.

Compilation works straight now. Here are some of the issues I found:

  1. extmod/vfs_basic.py test is failing (regresion since nrf: Add flash block device for VFS filesystems. #5472)

./run-tests.py --clean-failures
./run-tests.py --target nrf extmod/vfs_basic.py
./run-tests.py --print-failures| less

stat /x OSError
1 mount False False
['test_mnt']
-('test_mnt', 16384, 0)
-StopIteration
-StopIteration
...
+Traceback (most recent call last):

  • File "", in
    +AttributeError: 'module' object has no attribute 'ilistdir'
    +CRASH
    \ No newline at end of file

FAILURE /data/micropython/repo/micropython/tests/results/extmod_vfs_basic.py

BTW may I ask for a review of #6395 ? tail on the previous output would look as follow

-5 chdir /subdir
-OSError
-/
+Traceback (most recent call last):

  • File "", line 90, in
    +AttributeError: 'module' object has no attribute 'ilistdir'
    +CRASH
    \ No newline at end of file

FAILURE /data/micropython/repo/micropython/tests/results/extmod_vfs_basic.py

  1. The following code seems to freeze micropython.

echo "import os; print(os.listdir()); os.mkdir('lib')" > mkdir_test
mpremote run mkdir_test
[]
(... and at this point micropython freezes)

@dpgeorge
Copy link
Member Author

dpgeorge commented Aug 7, 2021

Thanks @mzdaniel for testing.

I have added uos.ilistdir to fix point 1.

Point 2 was due to the flash size ioctl reporting bytes rather than blocks, and so littlefs thought the flash was much larger than it really is and so it attempted to read/write out-of-bounds flash. This is now fixed.

I also tidied up a few other minor things. So this should now be ready to go in.

@mzdaniel
Copy link
Contributor

mzdaniel commented Aug 8, 2021

Sound great! And thank you for your changes Damien.
I'll propose in a new PR changing the FLASH_PAGESIZE default and increasing fs size wherever appropriate.

glennrub and others added 21 commits August 8, 2021 23:09
Calculate the unused flash area on the target device.  The values will be
exposed by _unused_flash_start and _unused_flash_length.  The start address
and the length are not aligned to either word or pages.
This commit adds the "nrf" module for port specific modules and objects.
Included in it is the "Flash" object which exposes a block device
implementation to access internal SoC flash.

Thanks to @aykevl aka Ayke van Laethem for the initial implementation.
Update flash.c to also be compiled in when
MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE is enabled and SoftDevice is
present.

Update bluetooth/ble_drv.c to forward flash events to flash.c when
MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE is enabled.
This documents parameters that can be passed to make to enable a specific
file system to included in the build.  Also, document the Makefile override
parameter "FS_SIZE" that can be used to tune the size of the flash region
to use as internal flash file system.
Disable MICROPY_FATFS_MULTI_PARTITION configuration because there is no
partition table in the flash for FATFS to read.

Also, set MICROPY_FATFS_MAX_SS to the size of a flash page.  For nrf51 the
value 1024 is set.  For nrf52/nrf91 the value 4096 is set.
Add posibility to override linker script "_fs_size" from make by adding the
FS_SIZE parameter.  The syntax of value is linker script syntax.  For
example, the value of 131072 bytes can be written as 128K like this:
FS_SIZE=128K.

If not set, default value for "_fs_size" from linker script will be used.
Enable MICROPY_PY_IO_FILEIO if MICROPY_VFS_FAT, MICROPY_VFS_LFS1
or MICROPY_VFS2 has been enabled.
Enable the following features for all boards except
nrf51 boards with SoftDevice present:

- MICROPY_VFS
- MICROPY_PY_NRF
- MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE
Update the Makefile to handle FROZEN_MANIFEST, and the README with some
small samples on how to use freeze manifests.  And add BOARD_DIR to the
Makefile which can be referenced in boards/<board>/mpconfigboard.mk to
include a board specific manifest.
Add a helper script _mkfs.py which automatically formats the file system if
nrf.Flash() is located and a VFS file system has been included in the
compilation.

The precedence is: first LFS1, LFS2 then FAT.
Set the default manifest to "modules/manifest.py".  This includes files
from the folder "modules/scripts".  The manifest default value is overriden
by all nrf51 boards that have SoftDevice present (SD=s110) to save flash.

Also add "modules/manifest.py" which is set to freeze
"modules/scripts/_mkfs.py".
extmod/vfs_lfs.c needs to resolve `mp_hal_time_ns()` in order to calculate
a timestamp from 1970 epoch.  A wall clock is not available in the nrf
port, hence the function is implemented to resolve compilation linkage
error.  The function always return 0.
Enable LittleFS v2 for all targets, except nrf51 targets when SoftDevice is
present.
This was missed in the initial implementation of the uos module.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge dpgeorge merged commit e5e0553 into micropython:master Aug 8, 2021
@dpgeorge dpgeorge deleted the nrf-add-blockdev branch August 8, 2021 13:40
tannewt pushed a commit to tannewt/circuitpython that referenced this pull request Feb 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants