Skip to content

Sketch too big, but identical sources are compiling and flashing under esp-idf environment #339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
croisez opened this issue Apr 29, 2017 · 30 comments

Comments

@croisez
Copy link

croisez commented Apr 29, 2017

I am writing a program which uses following capabilities of the ESP32: Blufi protocol, reception of BLE advertisings, HTTPS post (WiFiClientSecure), and WiFi.

I have written such program using Eclipse/esp-idf and it is compiling/flashing fine.
I wanted to try the Arduino way, and I am stuck with size of sketch too big error.

Multiple libraries were found for "WiFi.h"
 Used: /Users/lmc/Dropbox/arduino/hardware/espressif/esp32/libraries/WiFi
 Not used: /Applications/Arduino.app/Contents/Java/libraries/WiFi
Using library Blufi at version 1.0 in folder: /Users/lmc/Dropbox/arduino/hardware/espressif/esp32/libraries/Blufi 
Using library WiFiClientSecure at version 1.0 in folder: /Users/lmc/Dropbox/arduino/hardware/espressif/esp32/libraries/WiFiClientSecure 
Using library WiFi at version 1.0 in folder: /Users/lmc/Dropbox/arduino/hardware/espressif/esp32/libraries/WiFi 

Sketch uses 1,095,399 bytes (104%) of program storage space. Maximum is 1,044,464 bytes.
Global variables use 70,684 bytes (23%) of dynamic memory, leaving 224,228 bytes for local variables. Maximum is 294,912 bytes.
processing.app.debug.RunnerException: Sketch too big; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing it.

The functionalities are very identical, I don't include unused #includes.
As concern Blufi, I have written a lib for ESP32-Arduino from the Blufi example found in esp-idf.

How could I workaround this error and make my sketch flashable?

@me-no-dev
Copy link
Member

bluetooth and wifi use a whole lot of flash and in Arduino's partition config there are 2 sketch partitions 1MB each. I guess you went over that, which means that it will step into the next partition. I've seen report before and I have hit the limit myself. Taking more space for sketch means that it will be twice that much less available for SPIFFS when it comes. I might try to 1.25MB and have 0.5MB less SPIFFS

@torntrousers
Copy link
Contributor

I have the same problem when using BLE and WiFi. Is tweaking the partition size something simple to do which I can try locally?

@me-no-dev
Copy link
Member

Here is the partition definition: https://github.com/espressif/arduino-esp32/blob/master/tools/partitions/default.csv
you can try to adjust the size from the first ota partition onward

@antelder
Copy link

I can't work out the partition definitions! Changing app0 to 2M - "app0, app, ota_0, 0x10000, 2M" I still get the Sketch to big error and "Sketch uses 1068755 bytes (102%) of program storage space. Maximum is 1044464 bytes.". Could you show me an example default.csv for if I don't use OTA or SPIFFS and just want a big enough 1 sketch partition?

@me-no-dev
Copy link
Member

stupidity on my end... you need to change the maximum size in the board definitions also
https://github.com/espressif/arduino-esp32/blob/master/boards.txt#L10

@antelder
Copy link

Ok yes changing that too does get the compile working again. Thanks!
This was working ok with the Arduino/ESP32 code from 17Apr and earlier so somethings happened since then to increase this sketch from using 82% to 102% of the 1M partition size. Is that some transient thing or whats the long term solution here?

@me-no-dev
Copy link
Member

Long term is to increase the partition size :) BT stack is huge

@antelder
Copy link

Its a bit of a pain to have to manually fiddle about updating system files so I hope you mean increase the default? That would be good, even the small change you mention above - 1.25MB and have 0.5MB less SPIFFS - would be enough to get this sketch working.

@me-no-dev
Copy link
Member

Yes i will increase it :)

@me-no-dev me-no-dev added the Status: To be implemented Selected for Development label May 19, 2017
@Nakul93
Copy link

Nakul93 commented Nov 13, 2017

How to increase the partition size? I have tried changing the csv file but it didn't work. Please give me an example.

@me-no-dev
Copy link
Member

what did you change it to? what was the error that you got?

@shahabmusic
Copy link

how did you change those numbers? please help

@beegee-tokyo
Copy link
Contributor

@shahabmusic @Nakul93 Small tutorial: Change partition size

@shahabmusic
Copy link

I followed the tutorial and chanted the default.csv and board. txt. Since then none of my sketches. I tried to revert back all the changes to the original. However I no longer can run any sketch. so I stuck.
When I reset ESP32, I get:
`ets Jun 8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:956
load:0x40078000,len:0
load:0x40078000,len:13076
entry 0x40078ad0
`
What do you think I should do?

@beegee-tokyo
Copy link
Contributor

@shahabmusic

  • what IDE are you using? Arduino IDE or PlatformIO or Visual Studio Code?
  • how are you flashing the device? through USB or OTA (WiFI)?
  • what changes did you do to the default.csv?
  • if you are on PlatformIO or Visual Studio Code, delete the .pioenvs folder in your project before flashing (do a CLEAN before UPLOAD)

@shahabmusic
Copy link

shahabmusic commented Jan 18, 2018

I am using Arduino. I have esp32_core_board_v2 that has a wroom-32 module soldered on it. so I assume it has 4MB flash.
So here is my my CSV file:

# Name, Type, SubType, Offset, Size, Flags nvs, data, nvs, 0x9000, 0x5000, otadata, data, ota, 0xe000, 0x2000, app0, app, ota_0, 0x10000, 0x140000, app1, app, ota_1, 0x150000,0x140000, eeprom, data, 0x99, 0x290000,0x1000, spiffs, data, spiffs, 0x291000,0x16F000,

I also changed these two lines of board.txt:

esp32.upload.maximum_size=2031616 esp32.upload.maximum_data_size=294912

I also tried to erase flash using:
python esptool.py --port COM3 erase_flash
Still having issues.

@beegee-tokyo
Copy link
Contributor

Did you loose the format of the CSV file when you copied it or does it look like above? It needs newline at the end of each entry, or all would be a comment and no partition created.
Here is what the file should look like:

# Name, Type, SubType, Offset, Size, Flags 
nvs, data, nvs, 0x9000, 0x5000, 
otadata, data, ota, 0xe000, 0x2000, 
app0, app, ota_0, 0x10000, 0x140000, 
app1, app, ota_1, 0x150000,0x140000, 
eeprom, data, 0x99, 0x290000,0x1000, 
spiffs, data, spiffs, 0x291000,0x16F000,

make sure in your default.csv there are newline (CR or CR/NL) at the end of each entry.

@shahabmusic
Copy link

shahabmusic commented Jan 18, 2018

I made a copy of the csv file before editing it. I just copied it back to partitions folder so no I did not loose the format.
I also have esp IDF. I tried to compile and flash one of the examples but I was not successful. same result as arduino.

@shahabmusic
Copy link

So I just downloaded boards.txt and default.csv from git repository and replaced the old files and I am still having issue. However I could successfully upload an example using esp-idf.
What should I do with the arduino ide now to make it work again?

@beegee-tokyo
Copy link
Contributor

Not sure about your problem. I tested all 3 possibilities to change partition sizes and they worked for me.

Maybe you need to reinstall the complete arduino-esp32 to get back to normal.

@shahabmusic
Copy link

I managed to change it back to default by reinstalling the arduino-rsp32. However it stops working as soon as I make even a minor change in default.csv
Could this be because of the type or size of the memory IC?

@beegee-tokyo
Copy link
Contributor

@shahabmusic can you post here an example of a changed default.csv that doesn't work.

@erikwilson
Copy link
Contributor

erikwilson commented Jan 30, 2018

I have had success using the following default.csv based off of minimal.csv:

# Name,   Type, SubType, Offset,   Size, Flags
nvs,      data, nvs,     0x9000,   0x5000,
otadata,  data, ota,     0xe000,   0x2000,
app0,     app,  ota_0,   0x10000,  0x340000,
eeprom,   data, 0x99,    0x350000, 0x1000,
spiffs,   data, spiffs,  0x351000, 0xAF000,

and boards.txt to have modified values of .upload.maximum_size=3407872

Wasn't sure if we can omit the otadata or spiffs partitions altogether if we aren't directly using those features.

It appears that some of the maximum_sizes are currently incorrect, like Heltec devices 1044464.

Personally I wouldn't partition for ota updates as default, peripheral libraries add a lot to the compiled size, at least would be nice to have more dynamic partitioning.

@shahabmusic
Copy link

So I finally managed to fix it. Thank you guys for helping me.
@beegee-tokyo problem was not from default.csv file. as @erikwilson has mentioned, upload.maximum_size was the problem. Apparently most of the calculation on the Internet are wrong.
Only the default values or those mentioned in the tutorial mentioned by @beegee-tokyo work without any problems. Any other size or custom default.csv enteries won't work properly.

@torntrousers
Copy link
Contributor

Wow, some spare time so I had a play with ESP32's this afternoon to see how things have progressed ... and still, after almost a year, using BLE and Wifi together fail with Sketch too big and I have to fiddle about changing with the partition sizes. Surely BLE and Wifi is the Big Thing for the ESP32 so this should just work out-of-the-box?

@trademark18
Copy link

Just digging into ESP 32 and my use case demands both BT and Wi-Fi (of course!). Has any progress been made toward fitting both stacks without resizing partitions? I suppose I could go through the Bluetooth stack and purge everything I don't need from it, but that's a lot of work! Perhaps there's a lighter Bluetooth implementation out there that just dumps raw signal data? I'd be ok with that for my project.

@Netoperz
Copy link

Netoperz commented Sep 5, 2018

Same here, we have developed a product on ESP32, we have put over a year of development, and yesterday, the last part has been added.
The BLE UART , and i cannot flash the sketch, it is 130% too big.
So why de fu*%$%^$* on the ESP32 main website on espressif there is no "BIG WRITING" "WE DO HAVE WI-FI AND BLUETOOTH BUT YOU WONT BE ABLE TO USE IT AT ONES"
The second thing is that, it is useless in battery powered scenario, ble can draw 115 mA ! that's freaking lot. So for battery powered solutions i rather suggest using some ARM core/ble chip with gpios , that can run 4 years on two AA batery ;)

@Netoperz
Copy link

Netoperz commented Sep 5, 2018

Ok, so i have solved the problem in some way, hope this help others.
Step by step on windows 10 , and Arduino IDE 1.8.5
ESPboards installed using board manager and json config (not cloning using GIT as it was before )

To get more space for your sketch, yo have to go:

C:\Users[YOUR_USER_NAME]\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.0\tools\partitions

[YOUR_USER_NAME] - is your home catalog name
And You will see something like this.
Those are ready to use partition tables, just check if there is one that fits your needs, one has no ota one has ota and small spiffs and so on.
In my case the best is "min_spiffs" that gives 2mb for sketch and you can still use OTA, in my case with ble wifi, mqtt, string, and a few more things that ineed it's fine.

zrzut ekranu 18

You should make backup of orginal defaiult.csv , as you see i have zipped it.
Than you need close arduino, and for me i had to reboot the system and do not open arduino at all to be able to delate the orginal default.csv and duplicate the min_spiffs.csv and rename it to the "default.csv"

That is not the end. As you may see in the partition table You have sizez in hex, so in my case it is 1E0000.
You need to put that value to the boards configuration file that is under:

C:\Users[YOUR_USER_NAME]\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.0\

You need to find Your board, and change the
upload.maximum_size
from the default to the new amount, in my case it is 1E0000, but you cannot put it there in hex, You need to change it to dec (calculate on one of online calculators)

for 1E0000 you will get 1966080

So for my board dOIT DEVKIT V1 it should look like this:

esp32doit-devkit-v1.upload.maximum_size=1966080

save and start arduino, compile the skect and you see that you have plenty of space now :)

if You publish the code on Github, or share it to someone remember to tell how to solve that stupid size problem, and even provide the config info for the boards file and partition sizes. That will help a lot.

Hope this helps

@sabsik
Copy link

sabsik commented Nov 8, 2018

The easiest way to increase size of the sketch is within Arduino IDE go to "Tools->Partition Scheme" and then instead of "Default" select either "Minimal SPIFFS (Large APPS with OTA)" or "No OTA (Large APP)"

In this way you can have sketch size up to 1966080 bytes or 2097152 bytes respectively (the sizes actually depend on the board you use). If this is not enough, do what Netoperz recommended.

Of course, doing so you sacrifice size of SPIFFS or OTA. For the actual sizes of sketches, SPIFFS, etc. if you select those Schemes, look in the csv files mentioned by Netoperz.

image

Disclaimer: I tested it on vMicro within Visual Studio (which has similar feature), not in Arduino IDE, but I assume it must work the same way there.

@fmalekpour
Copy link

Just a note for anyone else who is looking for do this on platformio, you can change the partition with a simple line in platformio.ini:

To use Minimal SPIFFS
board_build.partitions = min_spiffs.csv

Huge app:
board_build.partitions = huge_app.csv

Complete list of pre-defined partition tables can be found at:
https://github.com/espressif/arduino-esp32/tree/master/tools/partitions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests