Quite a few of the questions in the forum and in the issue tracker relates to using rshell and fatfs. I decided to PR the working branch which has FatFS working for pca10056. I'll continue updating the branch a bit further to also add LittleFS hooks. In the meanwhile it can be used as a base to get rshell up running for those that wish for editing files through that tool.

I'm having some issues formatting the FAT partition using native block device functions. (When multipartition is enabled). It looks like f_mkfs throws an ABORT as i do not have an MBR (0xAA55 in offset 510). I do see STM port returns a manually construction MBR (block 0). Is there a way to also allow f_mkfs to construct the MBR in this case, or is there a different way of setting up the MBR?

Thank you glennrub for this PR.

When adding MP_QSTR_ilistdir and MICROPY_PY_UERRNO (and of course MICROPY_VFS_VFAT, MICROPY_VFS_LFS1, MICROPY_VFS_LFS2 plus minor changes on moduos.c) all Fat and LittleFS tests are passing (with vfs_fat_ramdisklarge.py (way too large for Particle Xenon boards) and vfs_userfs (uio not available) skipping), which is fantastic.

Using vfat seems to work pretty well, formatting and using the fs

>>> import nrf, os
>>> os.umount('/flash')
>>> os.VfsFat.mkfs(nrf.flashbdev)
>>> os.mount(nrf.flashbdev, '/')
>>> os.mkdir('/lib')
>>> with open('/lib/test.py', 'w') as f:
...     f.write("print('Hello world\\n')")
...
22
>>> print(open('/lib/test.py').read())
print('Hello world\n')

Shouldnt modules under /lib be able to import?

>>> import test
Traceback (most recent call last):
  File "<stdin>", in <module>
ImportError: no module named 'test'

Writing on /test.py will import correctly

>>> os.rename('/lib/test.py', '/test.py')
>>> import test
Hello world

The extended interface needed for littlefs seems to be missing so far, but maybe can be worked out in a future PR? I am happy to help.

>>> os.VfsLfs2.mkfs(nrf.flashbdev)
Traceback (most recent call last):
  File "<stdin>", in <module>
TypeError: function takes 3 positional arguments but 4 were given

In any case thank you again for this great addition (including the ability to use rshell ;) and happy New Year!

I'm having some issues formatting the FAT partition using native block device functions. (When multipartition is enabled). It looks like f_mkfs throws an ABORT as i do not have an MBR (0xAA55 in offset 510). I do see STM port returns a manually construction MBR (block 0). Is there a way to also allow f_mkfs to construct the MBR in this case, or is there a different way of setting up the MBR?

You're using the FM_SFD flag so it shouldn't need a partition table... I'd just disable multipartition support because it's not really needed (you can partition a block device yourself and construct an independent VfsFat object for each partition).

Also, what is the block size and how many blocks are there? It may be that FATFS cannot format what you give it, eg too small.

OK, I have add this pull to my local repo:

git fetch origin pull/5472/head:NRFFAT
git checkout NRFFAT

but after building I don't seem to see the included PR, do I need to do anything else other than the above to include a PR ?

Do I need to change my make command ??

For my custom board, I added the following to mpconfigboard.h

// STORAGE
#define MICROPY_VFS                   (1)
#define MICROPY_HW_ENABLE_STORAGE     (1)
#define MICROPY_MBFS                  (0)

and my make is:

make -j8 BOARD=holyiot_18010 MICROPY_VFS_VFAT=1 -f /nfs/qnap/dev/uPython/build/ports/nrf/custom.mk

And the build fails:

/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/ccjexAfe.ltrans3.ltrans.o: in function `os_sync.lto_priv.388':
/nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/modules/uos/moduos.c:92: undefined reference to `disk_ioctl'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/ccjexAfe.ltrans2.ltrans.o: in function `factory_reset_filesystem':
/nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:134: undefined reference to `f_mkfs'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:141: undefined reference to `f_setlabel'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:142: undefined reference to `mp_fat_vfs_type'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/ccjexAfe.ltrans2.ltrans.o:(.rodata+0x16e4): undefined reference to `mp_fat_vfs_type'
collect2: error: ld returned 1 exit status
make: *** [Makefile:426: build-holyiot_18010/firmware.elf] Error 1

Similar thing happens with pca10056:

CC build-pca10056-s140/pins_gen.c
LINK build-pca10056-s140/firmware.elf
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/ccTw2LFD.ltrans2.ltrans.o:(.rodata+0xbe0): undefined reference to `mp_fat_vfs_type'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/ccTw2LFD.ltrans3.ltrans.o: in function `os_sync.lto_priv.578':
/nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/modules/uos/moduos.c:92: undefined reference to `disk_ioctl'
collect2: error: ld returned 1 exit status
make: *** [Makefile:425: build-pca10056-s140/firmware.elf] Error 1

Would it be possible to provide some notes on how to implement this PR :-)

Not sure what is the issue. The tests I mentioned above were done introducing the exact same changes you did in mpconfigboard.h into pca10059 and simply make BOARD=pca10059 MICROPY_VFS_FAT=1 as mention in https://github.com/micropython/micropython/tree/master/ports/nrf#enable-micropy_vfs_fat

Hope it helps

Thanks, I tried a fresh clone of micropython, but as soon as I un-comment:

//#define MICROPY_VFS (1)
//#define MICROPY_HW_ENABLE_STORAGE (1)
//#define MICROPY_MBFS (0)

The build fails, when commented the build is successful.

I have done this a few times to be sure, did you also pull in PR5472 or ar you using the master branch ?

I don't understand. I thought your repo was in this state
#5472 (comment)

From there make BOARD=pca10056 MICROPY_VFS_FAT=1 does compile and glennrub said its working there.

@mzdaniel -

don't understand. I thought your repo was in this state

Yes, but I don't know if this is the complete / only command to run to pull in the PR as I have never attempted that before.

If i revert to the master, then the build is successful, however os shows no new FS functions:

MicroPython v1.12-45-gbfbd94401 on 2020-01-11; PCA10056 with NRF52840
Type "help()" for more information.
>>> import os
>>> dir(os)
['__class__', '__name__', 'remove', 'sep', 'dupterm', 'ilistdir', 'listdir', 'stat', 'uname']
>>> 

OK, sunday morning and si I thought I would try this again:

1. Clone fresh micropython
2. Apply PR as above.
3. Add 3 lines to mpconfigboard.h
4. Add "MICROPY_VFS_FAT=1" to make command

and:

make -j8 BOARD=holyiot_18010 SD=s140 MICROPY_VFS_VFAT=1 -f /nfs/qnap/dev/uPython/build/ports/nrf/custom.mk
CC build-holyiot_18010-s140/pins_gen.c
LINK build-holyiot_18010-s140/firmware.elf
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/ccxqr8jq.ltrans3.ltrans.o: in function `factory_reset_filesystem':
/nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:134: undefined reference to `f_mkfs'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:141: undefined reference to `f_setlabel'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:142: undefined reference to `mp_fat_vfs_type'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/ccxqr8jq.ltrans1.ltrans.o:(.rodata+0x1254): undefined reference to `mp_fat_vfs_type'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/ccxqr8jq.ltrans4.ltrans.o: in function `os_sync.lto_priv.493':
/nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/modules/uos/moduos.c:92: undefined reference to `disk_ioctl'
collect2: error: ld returned 1 exit status

There just has to be some step missing, I have spent 2 days on this and made no progress whatsoever.

Disable the 3 lines in mpconfigboard.h and it builds, enable them and it does not.

How / where can I check that the following are correctly defined:

f_mkfs
f_setlabel
mp_fat_vfs_type
disk_ioctl

As these appear to be the vars / functions that is causing the errors ??

OK, finally found the cause, it was due to a typo in my config but not in my posts here:
MICROPY_VFS_VFAT=1

Which should have been:
MICROPY_VFS_FAT=1

So the build now works and I can access the file system after:

import nrf, os
os.VfsFat.mkfs(nrf.flashbdev)
os.mount(nrf.flashbdev, '/')

But still unable to use rshell on the PCA10056:

Using buffer-size of 32
Connecting to /dev/tty.usbmodem0006835105191 (buffer-size 32)...
Trying to connect to REPL  connected
Testing if ubinascii.unhexlify exists ... Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/rshell/main.py", line 1282, in connect
    ip_address = socket.gethostbyname(port)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/rshell", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/rshell/command_line.py", line 4, in main
    rshell.main.main()
  File "/usr/local/lib/python3.7/site-packages/rshell/main.py", line 2913, in main
    real_main()
  File "/usr/local/lib/python3.7/site-packages/rshell/main.py", line 2875, in real_main
    connect(args.port, baud=args.baud, wait=args.wait, user=args.user, password=args.password)
  File "/usr/local/lib/python3.7/site-packages/rshell/main.py", line 1288, in connect
    connect_serial(port, baud=baud, wait=wait)
  File "/usr/local/lib/python3.7/site-packages/rshell/main.py", line 1312, in connect_serial
    dev = DeviceSerial(port, baud, wait)
  File "/usr/local/lib/python3.7/site-packages/rshell/main.py", line 1602, in __init__
    Device.__init__(self, pyb)
  File "/usr/local/lib/python3.7/site-packages/rshell/main.py", line 1366, in __init__
    unhexlify_exists = self.remote_eval(test_unhexlify)
  File "/usr/local/lib/python3.7/site-packages/rshell/main.py", line 1490, in remote_eval
    return eval(self.remote(func, *args, **kwargs))
  File "<string>", line 0
    
    ^
SyntaxError: unexpected EOF while parsing

on my custom board, everything works including rshell.

OK, a new finding.

If you set: MICROPY_VFS_FAT=0

Then you get the error I was seeing before:

LINK build-holyiot_18010-s140/firmware.elf
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/cc4cqcql.ltrans3.ltrans.o: in function `factory_reset_filesystem':
/nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:134: undefined reference to `f_mkfs'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:141: undefined reference to `f_setlabel'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/main.c:142: undefined reference to `mp_fat_vfs_type'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/cc4cqcql.ltrans1.ltrans.o:(.rodata+0x1268): undefined reference to `mp_fat_vfs_type'
/usr/lib/gcc/arm-none-eabi/7.3.1/../../../arm-none-eabi/bin/ld: /tmp/cc4cqcql.ltrans4.ltrans.o: in function `os_sync.lto_priv.496':
/nfs/qnap/dev/uPython/build/micropython/micropython/ports/nrf/modules/uos/moduos.c:92: undefined reference to `disk_ioctl'
collect2: error: ld returned 1 exit status
make: *** [Makefile:426: build-holyiot_18010-s140/firmware.elf] Error 1

Thanks a lot @mzdaniel and @pacmac for the great feedback.

I've updated the branch with a second commit.
I'll sort out a few more things before I squash/split and amend the branch. I'll give a heads up when i start shuffling the commits around.

@pacmac, i have added support for MICROPY_VFS_FAT=0. Good catch!

Pending work i'm looking at is to allow for MICROPY_VFS_FAT on the side of MICROPY_VFS_LFSx, so that FatFS can be used on SD cards, while internal flash is using LittleFS. Right now, i believe they will fight a bit formatting.

I'm considering that none of the compiled in FS should auto format the flash partition, but only mount it if it is possible. In other words, this should should be an explicit documented action how to format the various VFS file systems.

I'm also looking into splitting the mpconfigport.h into device specific configurations instead of ifdef-ing to much in the common file, meaning that the devices will have a section of default configuration each and one common. As VFS + file system will never fit on nrf51 with SoftDevice i need to set up some rules.

Some stats on RAM/ROM usage collected:

make BOARD=pca10056 LTO=0 - MBFS (MICROPY_VFS=0, MICROPY_HW_ENABLE_STORAGE=0, MICROPY_HW_ENABLE_STORAGE=0)
   text    data     bss     dec     hex filename
 161100      36    2448  163584   27f00 build-pca10056/firmware.elf

make BOARD=pca10056 LTO=0
   text    data     bss     dec     hex filename
 162188      36    2444  164668   2833c build-pca10056/firmware.elf

make BOARD=pca10056 LTO=0 MICROPY_VFS_FAT=1
   text    data     bss     dec     hex filename
 176980      36    2960  179976   2bf08 build-pca10056/firmware.elf

make BOARD=pca10056 LTO=0 MICROPY_VFS_LFS1=1
   text    data     bss     dec     hex filename
 172732      36    2444  175212   2ac6c build-pca10056/firmware.elf

make BOARD=pca10056 LTO=0 MICROPY_VFS_LFS2=1
   text    data     bss     dec     hex filename
 177372      36    2444  179852   2be8c build-pca10056/firmware.elf

make BOARD=pca10056 LTO=0 MICROPY_VFS_FAT=1 MICROPY_VFS_LFS1=1
   text    data     bss     dec     hex filename
 187572      36    2960  190568   2e868 build-pca10056/firmware.elf

make BOARD=pca10056 LTO=0 MICROPY_VFS_FAT=1 MICROPY_VFS_LFS2=1
   text    data     bss     dec     hex filename
 192212      36    2960  195208   2fa88 build-pca10056/firmware.elf

Also, what is the block size and how many blocks are there? It may be that FATFS cannot format what you give it, eg too small.

@dpgeorge 256K flash, 4096 block size (which is also my page size on the nrf52840). I'll postpone this a bit, as it is always working with uos.VfsFat.mkfs(), and with FM_SFD without native functions. I'll remove the native functions from the branch for now and look at this later. And, as i wrote in previous comment, i'm considering to remove the main.c auto-format in general to make it an explicit user action.

Fantastic @glennrub! According to a very quick test, with this PR, littlefs works natively on xenon's flash and rshell with it. I'll do more extensive tests, including migrating a production device to littlefs.

Re splitting mpconfigport.h: +1. This should make the code cleaner and improve reliability.

Regarding VFS block size, 4K for a tiny FS of 256K seems really big. I didn't read littlefs specs (yet), but if a file takes a full block, this means the fs cannot host many files.

I'll sort out a few more things before I squash/split and amend the branch. I'll give a heads up when i start shuffling the commits around.

@mzdaniel , @pacmac, So here is the heads up! :) i'll start shuffling the commits now (squash/split).

Awesome @glennrub! Look forward to test them. BTW, I have a device continuously running on a lipo battery and using littleFS for a few months now with no issues.

Updated the PR with a split of the earlier commits. In the process i also moved out the nrf module that was placed under modules/uos into a separate folder as it does not logically belong in uos. I did not have pyb module as other ports exposing their block device and it did not make sense to put a block device into the board module as it is a SoC feature. So i kept the nrf module adding the block device underneath that one.

All target boards now have LittleFS2 activated (except nrf51 with s110) and 64K file system size as default. This is too small for FAT in order to format it, which i expect no-one will use on the internal flash file system.

Also cherry-picked the commits from #5379 to freeze a manifest containing the file system formatting script.

What's left:

• Get the new files into codeformat.py. @dpgeorge, how do i do this? should i just add a ports/*/*/*/*.[ch] to get a forth level included (ports/nrf/modules/nrf/*)? So far i manually run codeformat on the files.
• Fine tune commit messages, unless we want to squash a bit.
• Swap to use MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE instead of MICROPY_HW_ENABLE_STORAGE as enabler for flash block device. @dpgeorge, I had a hard time figuring out which of these to use, so i picked the shortest one. Was that a wrong decision ?

Get the new files into codeformat.py

Just add another entry in PATHS, like ports/nrf/modules/uos/*.[ch]

Swap to use MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE instead of MICROPY_HW_ENABLE_STORAGE

stm32 supports both internal and external flash storage (the latter via an external SPI flash). MICROPY_HW_ENABLE_STORAGE is an internal setting that boards do not configure directly, but rather it gets set if-and-only-if there is either internal or external storage (ie at least one block device defined). MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE is for a board to configure if it wants to disable internal flash storage (which is enabled by default).

It might make sense to follow that logic here, but not 100% sure.

stm32 supports both internal and external flash storage (the latter via an external SPI flash). MICROPY_HW_ENABLE_STORAGE is an internal setting that boards do not configure directly, but rather it gets set if-and-only-if there is either internal or external storage (ie at least one block device defined).

Is my interpretation of MICROPY_HW_ENABLE_STORAGE correctly, that it is used to enable the pyb.Flash class, and it also acts as a grouping config in order to enable mounting and formatting in main.c if any of the two blockdevs have been selected?

MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE is for a board to configure if it wants to disable internal flash storage (which is enabled by default).

I guess this should be similar in the nrf-port. In addition I'm thinking that it also makes sense to change main.c in the nrf-port to also test against MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE instead of MICROPY_HW_ENABLE_STORAGE for auto mounting and formatting as only the const object from nrf/flashbdev.c is used directly in main.c for mounting, not going through a Flash object (stm32/storage.c).

I succeeded in compiling the glennrub:nrf_blockdev branch from scratch. However, enabling MICROPY_PY_IO in mpconfigport.h results in following compilation error:
build-feather52840-s140/py/modio.o:(.rodata.mp_module_io_globals_table+0x14): undefined reference to 'fatfs_type_fileio'
build-feather52840-s140/py/modio.o:(.rodata.mp_module_io_globals_table+0x1c): undefined reference to 'fatfs_type_textio'

Not sure how to fix this though. It seems that both the defines for mp_type_fileio and mp_type_textio in mpconfigport.h need to be adapted.

@glennrub, @dpgeorge. Could it be possible to merge this PR? It used to be in great shape for significant periods of time, but of course plenty of things keep changing underneath. Currently the PR rebased on top of master compiles (after solving trivial conflicts), formats the flash, and writing simple files with open seem to work, but trying to create directories or writiting files with mpremote freezes the microcontroller.

Would it be easier to split the PR at this point? (eg: adding the flash device in itself would be a great incremental addition)

And of course let me know if we need more testing, I have several nrf boards we can use.

Thank you for keeping to push forward Micropython!

Could it be possible to merge this PR? It used to be in great shape for significant periods of time, but of course plenty of things keep changing underneath.

I've rebased this PR on the latest master and submitted it as #7619. Would be good to get it merged.

Fantastic. I'll add my findings there

Merged in 3b594f7 through 85f0ce2 (via #7619).