Skip to content

SPIFFS + ESP32 Sketch Data Upload Problem (mkspiffs) #1022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
smacyas opened this issue Jan 22, 2018 · 28 comments
Closed

SPIFFS + ESP32 Sketch Data Upload Problem (mkspiffs) #1022

smacyas opened this issue Jan 22, 2018 · 28 comments
Labels
Status: Stale Issue is stale stage (outdated/stuck) Type: Bug 🐛 All bugs

Comments

@smacyas
Copy link

smacyas commented Jan 22, 2018

Hardware:

Board: ?ESP32 Dev Module?
Core Installation/update date: latest
IDE name: Arduino IDE
Flash Frequency: 40Mhz?
Upload Speed: 115200

Description:

Hello.
I need help.
I have WROOM-32 ESP32 module.
In folder of my ino-project I created folder ‘data’ and put into it some files: css, html, txt by ESP32 Sketch Data Upload (https://github.com/me-no-dev/arduino-esp32fs-plugin), I have uploaded them successfully to my module.
When I try to read file from SPIFFS (previously mounted SPIFFS, of course) (latest arduino-esp32, https://github.com/espressif/arduino-esp32), listDir(SPIFFS, "/", 0) - OK; but while I use readFile(SPIFFS, "/style.css"); there is some trash, but no correct data. When I use write and read file, which was NOT uploaded by ESP32 Sketch Data Upload, it is OK.

By the way, I noticed string: flash read err, 1000 ets_main.c 371

Any ideas?
Thanks.

Sketch:

Debug Messages:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
ets_main.c 371
ets Jun 8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:956
load:0x40078000,len:0
load:0x40078000,len:13076
entry 0x40078a58

Starting device...
ESP32 Chip ID: 9238022
FreeHeap: 126324
ChipRevision: 1
CpuFreqMHz: 240
SdkVersion: v3.1-dev-239-g1c3dd23f-dirty
FlashChipSize: 4194304
FlashChipSpeed: 40000000
FlashChipMode: 2

SPIFFS Mount succesfull:
Listing directory: /
FILE: /door.png SIZE: 846
FILE: /favicon.ico SIZE: 239
FILE: /index.html SIZE: 4875
FILE: /logo.png SIZE: 5075
FILE: /profile.png SIZE: 898
FILE: /style.css SIZE: 5844
FILE: /style.txt SIZE: 101
FILE: /variables.txt SIZE: 41

Started WiFi Station Terminal Access (STA) regim
Current SSID: myssid

ESP32 MAC STA: [D][WiFiGeneric.cpp:293] _eventCallback(): Event: 2 - STA_START
30:AE:A4:06:F6:8C
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 1 - SCAN_DONE
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 4 - STA_CONNECTED
[D][WiFiGeneric.cpp:293] _eventCallback(): Event: 7 - STA_GOT_IP

WiFi connected
WiFi channel: 1
WiFi RSSI: -56
192.168.200.162
Reading file: /style.css
Read from file: яяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяяя

@marhul
Copy link

marhul commented Jan 28, 2018

Bug in Arduino-esp32 after 20/12/2017 ... duplicate issue: me-no-dev/arduino-esp32fs-plugin#6 (comment) and original source of solution(tips): https://www.mgo-tec.com/blog-entry-esp32-spiffs-easy-websocket-library.html

@copercini
Copy link
Contributor

@me-no-dev Here it's working after change size of per-file metadata field from 4 to 0 on SPIFFS configs of IDF =)

@smacyas
Copy link
Author

smacyas commented Jan 31, 2018

Where is it in Arduino IDE?

@copercini
Copy link
Contributor

@smacyas this option is only available using arduino as IDF component... For Arduino IDE it depends of @me-no-dev recompiles the core

@smacyas
Copy link
Author

smacyas commented Jan 31, 2018

@copercini thanks. Will be waiting for @me-no-dev recompile

@shadowofsoul
Copy link

@copercini from your message i understood that was from the configs of the components itself, but actually, you must change this value from make menuconfig and works ! Many thanks for your tip!

@smacyas
Copy link
Author

smacyas commented Feb 1, 2018

@copercini There is a file Arduino\hardware\espressif\esp32\tools\sdk\include\spiffs\spiffs_config.h, may be this option (size of per-file metadata field) is there can be defined?

@smacyas
Copy link
Author

smacyas commented Feb 1, 2018

okey, if it is so long to recompile core with newest IDF with this bug fix, may be someone will tell how to do it myself to make arduino-esp32 from IDF?

@copercini
Copy link
Contributor

@tferrin
Copy link
Contributor

tferrin commented Feb 4, 2018

Pretty sure this problem is because of the change to the definition for SPIFFS_OBJ_META_LEN in spiffs_config.h that occurred a while back, but @me-no-dev will need to confirm. There was no corresponding change to the data upload plugin https://github.com/me-no-dev/arduino-esp32fs-plugin and that means file objects created by the plugin are no longer compatible with the SPIFFS library. As a temporary workaround I defined SPIFFS_OBJ_META_LEN to be 0 in Arduino/hardware/espressif/esp32/tools/sdk/include/spiffs/spiffs_config.h, then restarted the IDE. Reading uploaded data works fine for me now.

@bospre
Copy link

bospre commented Feb 4, 2018

Hello,
I just tried your solution, but it did not work for me.
Is there something else which has to be configured?
I changed the line #define SPIFFS_OBJ_META_LEN (CONFIG_SPIFFS_META_LENGTH)
to #define SPIFFS_OBJ_META_LEN (0)
but I still cannot read the data...

@tferrin
Copy link
Contributor

tferrin commented Feb 5, 2018

@bospre, that is the only change I made, along with restarting the Arduino IDE. Before the change file.read() returned bogus data. After the change all is good. EDIT: Actually there was more thing I did that is what made the real difference -- see next sentence below:

EDIT: Turns out the real workaround is just to run the SPIFFS_Test.ino program after uploading your data file. As part of that test, the program writes a 1MB test file and writing that file somehow makes the uploaded data file subsequently readable.

@baggior
Copy link
Contributor

baggior commented Feb 6, 2018

@tferrin I think is rather impossible that you changed only the define in the 'spiffs_config.h' include file to
#define SPIFFS_OBJ_META_LEN (0)

the binary "libspiffs.a" is already compiled with the other setting and your define doesn't affect the code inside

you'd have to recompile or get another "libspiffs.a" already compiled with the header modified setting

@baggior
Copy link
Contributor

baggior commented Feb 6, 2018

So to fix this SPIFF issue then
could any Collaborator of this repository recompile the SDK then?
an alternative is to modify and recompile the dependent mkspiffs tool

(one of both)

@smacyas
Copy link
Author

smacyas commented Feb 6, 2018

You have to replace file from tools/sdk/lib/libspiffs.a to the same file from 70d0d46 commit. Thanks for this problem solving to @vindolin

@smacyas
Copy link
Author

smacyas commented Feb 6, 2018

I do not know why official person (@me-no-dev ) does not recompile the core

@baggior
Copy link
Contributor

baggior commented Feb 6, 2018

@smacyas thank you
but a better alternative is to use the new mkspiffs tool that contains the new sttings SPIFFS
is in the PR igrr/mkspiffs#36

@lonerzzz lonerzzz changed the title SPIFFS + ESP32 Sketch Data Upload Problem SPIFFS + ESP32 Sketch Data Upload Problem (mkspiffs) Feb 12, 2018
@lonerzzz
Copy link
Contributor

lonerzzz commented Feb 12, 2018

@smacyas I suspect @me-no-dev is enjoying the Spring Festival or otherwise busy. He has been pulled into a number of things.
@igrr @copercini Can either of you provide a version of your built binary on a branch and we can reference it here for those who need it?

@lonerzzz
Copy link
Contributor

As per (espressif/esp-idf#1610 (comment)), @igrr has provided a 0.2.2 version () of mkspiffs that can be downloaded on its own (https://github.com/igrr/mkspiffs/releases/tag/0.2.2) until @me-no-dev spins a new version.

@baggior
Copy link
Contributor

baggior commented Feb 17, 2018

adopting the @igrr solution on espressif/esp-idf#1610

calling
mkspiffs.exe -c ./data -p 256 -b 4096 -s 1503232 -d 5 my.spiffs.bin
'SPIFFS mount failed with error: -10025'

anyway now seems to work

@rfestag
Copy link
Contributor

rfestag commented Feb 27, 2018

I'm honestly a bit confused still, so hopefully somebody can explain this. What is the correct way to fix this? It is a bit of a hack to manually download the new mkspiffs and drop it in the tools post install.

After switching to the head of master and updating my git submodules, it looks like tools/mkspiffs/ is no longer part of the git repo? Does a submodule need to be added/modified? What needs to be done so that people who install arduino-esp32 actually get a working mkpsiffs?

@tferrin
Copy link
Contributor

tferrin commented Feb 27, 2018

As @lonerzzz pointed out above, you need download and install version 0.2.2 of mkspiffs. It gets installed in hardware/espressif/esp32/tools/mkspiffs.

@rfestag
Copy link
Contributor

rfestag commented Feb 28, 2018

Sorry I wasn't clearer. I would consider @lonerzzz's solution a workaround, not a fix. Previously, arduino-esp32 provided the tool (I'm not sure if it was a submodule, or if it was included directly, but either way when I followed the install procedures, it was there). It looks like that is no longer the case. So, I could see one of the following being done:

  1. Include mkpsiffs in tools/mkspiffs again.
  2. tools/mkspiffs should be a submodule that pulls in the correct version of the tool
  3. The install instructions should be updated to tell people to download/install the tool

As a user of this library, I would prefer to see either option 1 or option 2, since the tool is located within the arduino-esp32 tree. If it is no longer appropriate for mkspiffs to be an esp32 tool, then the arduino-esp32fs-plugin should be updated to include its own copy of mkspiffs (and people probably shouldn't be directed to include it in the tools directory).

@tferrin
Copy link
Contributor

tferrin commented Feb 28, 2018

I'm sure this will be fixed in due time. There has been some lag in commits during the past month due to work on other projects by key contributors. And please note that there has yet to be an actual release of the arduino-esp32 core, so officially it's all still experimental code. If you want a code base that is better supported you should consider the official Espressif IoT Development Framework -- https://github.com/espressif/esp-idf. It sees more active development and commits to the code base and is at the release 3.0-rc1 stage. (Not trying to be a nerd or make excuses. Just observations by one of many code contributors.)

@rfestag
Copy link
Contributor

rfestag commented Feb 28, 2018

Sure, and I get that this is still experimental/in development. I dug a bit more into the get.py script that is part of the install, and it looks like with a few minor changes there it is easy enough to pull in the correct mkpsiffs. I'll submit a pull request with those changes.

@g01d10x
Copy link

g01d10x commented Jun 13, 2018

Thanks @tferrin for the explanation :) Works like a charm for me on MacOS with Arduino 1.8.5 !

@stale
Copy link

stale bot commented Aug 1, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Status: Stale Issue is stale stage (outdated/stuck) label Aug 1, 2019
@stale
Copy link

stale bot commented Aug 15, 2019

This stale issue has been automatically closed. Thank you for your contributions.

@stale stale bot closed this as completed Aug 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Stale Issue is stale stage (outdated/stuck) Type: Bug 🐛 All bugs
Projects
None yet
Development

No branches or pull requests

10 participants