Occasional OTA crash #2212
Comments
|
which version of the core are you running? |
|
How can I check that? It looks like I have 1.0.0 |
|
I got a similar error But it happens to me all the time, and I code is just the example code introduccing my SSID, PASSWORD and URL where the .bin file is. Thanks! |
|
https://github.com/espressif/arduino-esp32/tree/master/libraries/Update/examples/AWS_S3_OTA_Update Guru Meditation Error: Core 1 panic'ed (LoadStoreError). Exception was unhandled.
Core 1 register dump:
PC : 0x400d25c4 PS : 0x00060a30 A0 : 0x800d2667 A1 : 0x3ffb1df0
A2 : 0x3ffbb2fc A3 : 0x3ffb1e3f A4 : 0x00000001 A5 : 0x40000000
A6 : 0x00044b60 A7 : 0xff000000 A8 : 0x00000000 A9 : 0xffffffff
A10 : 0x00548f70 A11 : 0x3ffbb4a0 A12 : 0x8008eb2c A13 : 0x3ffb1d00
A14 : 0x00000000 A15 : 0x3ffb0060 SAR : 0x0000000b EXCCAUSE: 0x00000003
EXCVADDR: 0x40000000 LBEG : 0x400014fd LEND : 0x4000150d LCOUNT : 0xffffffff
Backtrace: 0x400d25c4:0x3ffb1df0 0x400d2664:0x3ffb1e10 0x40142c6e:0x3ffb1e30 0x400d4009:0x3ffb1e60 0x400d4035:0x3ffb1e80 0x400d39fa:0x3ffb1ea0 0x400d205d:0x3ffb1ec0 0x400d232f:0x3ffb1f90 0x400d4e21:0x3ffb1fb0 0x4008e919:0x3ffb1fd0
0x400d25c4: WiFiClientRxBuffer::read(unsigned char*, unsigned int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/WiFi/src/WiFiClient.cpp:521
0x400d2664: WiFiClient::read(unsigned char*, unsigned int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/WiFi/src/WiFiClient.cpp:521
0x40142c6e: WiFiClient::read() at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/WiFi/src/WiFiClient.cpp:521
0x400d4009: Stream::timedRead() at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/cores/esp32/Stream.cpp:279
0x400d4035: Stream::readBytes(char*, unsigned int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/cores/esp32/Stream.cpp:279
0x400d39fa: Stream::readBytes(unsigned char*, unsigned int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/cores/esp32/Stream.h:103
(inlined by) UpdateClass::writeStream(Stream&) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/Update/src/Updater.cpp:342
0x400d205d: execOTA() at /Users/nikolay/Documents/Arduino/AWS_S3_OTA_Update/AWS_S3_OTA_Update.ino:158
0x400d232f: loop() at /Users/nikolay/Documents/Arduino/AWS_S3_OTA_Update/AWS_S3_OTA_Update.ino:221
0x400d4e21: loopTask(void*) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/cores/esp32/main.cpp:25
0x4008e919: vPortTaskWrapper at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/port.c:355 (discriminator 1)Guru Meditation Error: Core 1 panic'ed (LoadStoreError). Exception was unhandled.
Core 1 register dump:
PC : 0x400d2024 PS : 0x00060e30 A0 : 0x800d20c7 A1 : 0x3ffb1ca0
A2 : 0x3ffbb684 A3 : 0x3ffb1cef A4 : 0x00000001 A5 : 0x40000000
A6 : 0x000390f4 A7 : 0xff000000 A8 : 0x00000000 A9 : 0xffffffff
A10 : 0x00e118af A11 : 0x3ffc6f0c A12 : 0x8008eb2c A13 : 0x3ffb1ba0
A14 : 0x00000000 A15 : 0x3ffb0060 SAR : 0x0000000b EXCCAUSE: 0x00000003
EXCVADDR: 0x40000000 LBEG : 0x400014fd LEND : 0x4000150d LCOUNT : 0xfffffffe
Backtrace: 0x400d2024:0x3ffb1ca0 0x400d20c4:0x3ffb1cc0 0x40145992:0x3ffb1ce0 0x400d5ffd:0x3ffb1d10 0x400d6029:0x3ffb1d30 0x400d570a:0x3ffb1d50 0x400d4956:0x3ffb1d70 0x400d4f4a:0x3ffb1db0 0x400d51a9:0x3ffb1ea0 0x400d1c9e:0x3ffb1f40 0x400d7359:0x3ffb1fb0 0x4008e919:0x3ffb1fd0
0x400d2024: WiFiClientRxBuffer::read(unsigned char*, unsigned int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/WiFi/src/WiFiClient.cpp:521
0x400d20c4: WiFiClient::read(unsigned char*, unsigned int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/WiFi/src/WiFiClient.cpp:521
0x40145992: WiFiClient::read() at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/WiFi/src/WiFiClient.cpp:521
0x400d5ffd: Stream::timedRead() at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/cores/esp32/Stream.cpp:76
0x400d6029: Stream::readBytes(char*, unsigned int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/cores/esp32/Stream.cpp:76
0x400d570a: Stream::readBytes(unsigned char*, unsigned int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/cores/esp32/Stream.h:103
(inlined by) UpdateClass::writeStream(Stream&) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/Update/src/Updater.cpp:342
0x400d4956: HTTPUpdate::runUpdate(Stream&, unsigned int, String, int) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/HTTPUpdate/src/HTTPUpdate.cpp:45
0x400d4f4a: HTTPUpdate::handleUpdate(HTTPClient&, String const&, bool) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/HTTPUpdate/src/HTTPUpdate.cpp:45
0x400d51a9: HTTPUpdate::update(WiFiClient&, String const&, String const&) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/HTTPUpdate/src/HTTPUpdate.cpp:45
0x400d1c9e: loop() at /var/folders/wt/t_3rnhg91bs7_m_l448f7fp80000gn/T/arduino_modified_sketch_699172/httpUpdate.ino:53 (discriminator 3)
0x400d7359: loopTask(void*) at /Users/nikolay/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/cores/esp32/main.cpp:25
0x4008e919: vPortTaskWrapper at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/port.c:355 (discriminator 1)Looks like it always related to the WiFiClientRxBuffer::read. |
|
LoadStoreFailure is a Buffer location error, the referenced data location is not within the Cpu's RAM. Set Core Debug Level to ERROR or WARNING and rerun your test. See if the RAM allocation error is listed in the console log. Chuck. |
@stickbreaker I have tried it already. There were nothing in terminal output. I have tried those both debug levels. |
|
@nikuz Reading through HttpClient. it looks like it is a heap fragmentation error. Chuck. |
|
String and Stream both had a few bugs in them when exactly 64 bytes was printed, these were fixed on master a week or so ago. There may be other heap leaks in there still though. |
|
@stickbreaker same behaviour even with the heapfree call. I have added the additional log for progress monitoring. Looks like LoadStoreError always happens on the 237568 bytes of stream download. I have tried to update to two different *.bin sketches, which has a not the same size.. |
|
One time I got this error with debug log ERROR enabled, right before LoadStoreError: (34214) esp_image: invalid segment length 0xfefefefe
[E][HTTPUpdate.cpp:408] runUpdate(): Update.end failed! (Could Not Activate The Firmware) |
|
Sometimes updates happens. I'll check, maybe something wrong in my environment. Maybe I'll move the update file somewhere closer to the esp itself. |
|
I am able to reproduce with a minimal sketch. I'll dig in and take a look. Monitor: |
|
Looks like @Jeroen88 has been working on HTTPUpdate. Looping him in. |
|
HTTPUpdate works just fine if I'm doing update from my local network. |
|
Here is the full code if someone wants to try to reproduce. |
Still not stable. |
|
@poengAlex, @joseitor, @nikuz and @lbernstone, could you please test PR #2259? I could reproduce the error with the sketch posted by @lbernstone and it ran without error after this fix. I think it will solve your issues (and possibly a lot more!) |
|
@Jeroen88 for me it works very well! Thank you very much! I have tested on three different devices. And update successfully passed every time. |
|
@nikuz that's nice, thnx for the quick testing! |
|
I have similar error with 1.0.0 |
|
This issue was almost certainly fixed with #2263. Please pull the new version of WiFiClient.cpp and see if that fixes your problem. Otherwise, you will need to open a new issue. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
This stale issue has been automatically closed. Thank you for your contributions. |
|
I can see that the issue is closed. It didn't help me, but I helped myself and want to share the resolve here. In my case, I used ESP32 core downloaded in Dec 2019. It gave me random issues while OTA, mostly, when calling Upload.end, sometimes, it gave me MD5 mismatch error, sometime, it said it couldn't read from the flash, and most of the times it said that it couldn't activate the boot partition. Since I was observing brown out detection during several high power tasks in the code, I first tried disabling brown out detection in the startup code using WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 0); //disable brownout detector. This solved my code issues but not the OTA issues. On my ESP32-Cam, the brown out was occurring most probably since I was using an old Arduino mega to power it up which used a over current protection resistor in series to the main USB supply. I shortened it and guess what! problem solved.... Disabling brown out isn't good, in my opinion. It should never occur. The cam and the WiFi requires quite a lot of power that needs to be provided otherwise it even reduces WiFi transfer speeds. |
|
I can confirm that this issue is not fixed yet. I'm using a version with this patch applied and see the following occassional stacktrace: |
|
I too am having the same issue. It happens when I start streaming video from the ov2640 (using the example code). If I disable the OTA handle function, everything works OK. I'm not trying to update code and stream at the same time. Processor: ESP32-D0WD V1 Before the call there is about 44K of heap available. Stack trace: |
|
I would guess that the heap could be so fragmented that it can not allocate a continuous buffer for 1460 bytes. What I did not expect is int WiFiUDP::parsePacket(){
if(rx_buffer)
return 0;
struct sockaddr_in si_other;
int slen = sizeof(si_other) , len;
char * buf = (char *)malloc(1460);
if(!buf){
return 0;
}
if ((len = recvfrom(udp_server, buf, 1460, MSG_DONTWAIT, (struct sockaddr *) &si_other, (socklen_t *)&slen)) == -1){
free(buf);
if(errno == EWOULDBLOCK){
return 0;
}
log_e("could not receive data: %d", errno);
return 0;
}
remote_ip = IPAddress(si_other.sin_addr.s_addr);
remote_port = ntohs(si_other.sin_port);
if (len > 0) {
rx_buffer = new cbuf(len);
rx_buffer->write(buf, len);
}
free(buf);
return len;
} |
|
Oddly it took me a while to re-create the bug this morning after having changed the way the camera worked to utilitize the RTOS approach from here (which seemed to fix everything and increase stability): https://www.hackster.io/anatoli-arkhipenko/multi-client-mjpeg-streaming-from-esp32-47768f In both tests I am using the latest version of the camera code directly integrated into my source code in PlatformIO. After reverting my code back and flashing a few times, I was able to get the bug again after launching the camera stream like before. The fix seems to have prevented the hard crash. Now I sometimes get the following error in the serial debug console: I will keep this branch open in my code if you want me to play with some other things although I'm going to continue my dev using the technique listed above. |
|
The issue started to happen again, even with the new freeRTOS code. After troubleshooting, I realized that updating to the espressif32 3.1 release broke the code. After reverting back to 3.0, the "No more processes" and/or hard crashes (depending on if I caught a failed malloc rather than new char). It seems like it is somehow running out of ram despite the available heap size being greater than the desired allocation. Still not sure what is actually causing it, only how to avoid pissing it off. |
|
@me-no-dev I tried upgrading to the new espressif32 3.2 release and the same bug appears to be there too :( |
Hardware:
Board: ESP32-wroom-32D (custom board)
Core Installation/update date: 1.0.0?
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: no
Upload Speed: 115200
Computer OS: Windows 10
Description:
Hi
I have build a simple test program to check the OTA functionality on ESP32. There are 2 versions of a bin fil I download with on integer value different in the two. Both programs start OTA and download the other bin file. If the same bin fil starts after a reset the OTA must have failed somehow. I also have a timer that restarts the MCU after 5 minutes.
After running the program over the weekend I have 6722 successful OTA and 162 fails. 2.4% fail rate. There is a few instances (<5) where the MCU simply freeze and do the restart failsafe kicks in. But the most of the failures there is a LoadStoreError. The full exception you can find bellow.
Do anyone else have this problem? Any solutions?
Sketch:
Debug Messages:
The text was updated successfully, but these errors were encountered: