HTTPS OTA-update fails because esp-tls-mbedtls reports error `-0x7100` (IDFGH-16729)

[DCSBL]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.2.1 (this is the version installed on the products having this issue, we are not able to easily reproduce the problem so we are not sure if the IDF version is related.

Espressif SoC revision.

ESP32-C3

Operating System used.

macOS

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

Development Kit.

None

Power Supply used.

External 5V

What is the expected behavior?

OTA installs without issues, all the time.

Our goal is to figure out what is going on, make it reproducible, and apply a fix for later updates. It is nice if we could fix the issue in-field, but we expect that is not possible due to the issue being related to the update itself.

What is the actual behavior?

During an OTA, we sometimes get the following failure:

I (5025) esp_https_ota: Starting OTA...
I (5025) esp_https_ota: Writing to partition subtype 17 at offset 0x200000
E (15430) esp-tls-mbedtls: read error :-0x7100:
E (15431) transport_base: esp_tls_conn_read error, errno=Success
E (15431) HTTP_CLIENT: transport_read: error - -1 | ESP_FAIL
E (15439) esp-tls-mbedtls: read error :-0x7100:
E (15443) transport_base: esp_tls_conn_read error, errno=Success
E (15450) HTTP_CLIENT: transport_read: error - -1 | ESP_FAIL
E (15456) esp_https_ota: data read -1, errno 0
E (15464) app_update: OTA update failed (ESP_FAIL), rebooting...

Note the sometimes. About 1% of our products have this issue.

Steps to reproduce.

1. Start OTA over https
2. OTA fails
3. Device reboots

Diagnostic report archive.

No response

More Information.

• About 1% of our products have the issue that they fail to update.
• Products that fail to update are seemingly random and not related to a batch or date. We have not seen relations between the usage of the product (e.g. local HTTP API used or not) and the failures.
• The update sometimes succeeds when we retry it multiple times.
• Products that fail to update can easily be updated in other locations, which may look this a network or environment-related issue.

Our goal is to figure out what is going on, make it reproducible, and apply a fix for later updates. It is nice if we could fix the issue in-field, but we expect that is not possible due to the issue being related to the update itself.

void update_task(void *arg)
{
    // Generate OTA URL
    std::string url = std::string(OTA_URL) + "/" + self->version_to_install.value() + ".bin";

    ESP_LOGW(TAG, "====== OTA UPDATE ======");
    ESP_LOGW(TAG, "Version: %s", self->version_to_install.value().c_str());
    ESP_LOGI(TAG, "Download URL: %s\n", url.c_str());

    // Actually install OTA
    esp_http_client_config_t config = {
        .url        = url.c_str(),
        .cert_pem   = (char *)ca_certificate,
        .timeout_ms = 300000
    };

    esp_https_ota_config_t https_config = {
        .http_config = &config,
    };
    esp_err_t ota_retval = esp_https_ota(&https_config);

    if (ota_retval == ESP_OK)
    {
        ESP_LOGI(TAG, "OTA update successful, rebooting...");
    }
    else
    {
        ESP_LOGE(TAG, "OTA update failed (%s), rebooting...", esp_err_to_name(ota_retval));
    }

    esp_restart();

    // Should not happen, as esp_restart is non-returning, but just in case
    vTaskDelete(NULL);
}

We prefer to no upload the full sdkconfig, here are the most important parts, if you need more please let me know.

sdkconfig

``` # # Build type # CONFIG_APP_BUILD_TYPE_APP_2NDBOOT=y # CONFIG_APP_BUILD_TYPE_RAM is not set CONFIG_APP_BUILD_GENERATE_BINARIES=y CONFIG_APP_BUILD_BOOTLOADER=y CONFIG_APP_BUILD_USE_FLASH_SECTIONS=y # CONFIG_APP_REPRODUCIBLE_BUILD is not set # CONFIG_APP_NO_BLOBS is not set # CONFIG_APP_COMPATIBLE_PRE_V2_1_BOOTLOADERS is not set # CONFIG_APP_COMPATIBLE_PRE_V3_1_BOOTLOADERS is not set # end of Build type

Compiler options

CONFIG_COMPILER_OPTIMIZATION_DEBUG=y

CONFIG_COMPILER_OPTIMIZATION_SIZE is not set

CONFIG_COMPILER_OPTIMIZATION_PERF is not set

CONFIG_COMPILER_OPTIMIZATION_NONE is not set

CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_ENABLE=y

CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_SILENT is not set

CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_DISABLE is not set

CONFIG_COMPILER_FLOAT_LIB_FROM_GCCLIB=y
CONFIG_COMPILER_OPTIMIZATION_ASSERTION_LEVEL=2

CONFIG_COMPILER_OPTIMIZATION_CHECKS_SILENT is not set

CONFIG_COMPILER_HIDE_PATHS_MACROS=y

CONFIG_COMPILER_CXX_EXCEPTIONS is not set

CONFIG_COMPILER_CXX_RTTI is not set

CONFIG_COMPILER_STACK_CHECK_MODE_NONE=y

CONFIG_COMPILER_STACK_CHECK_MODE_NORM is not set

CONFIG_COMPILER_STACK_CHECK_MODE_STRONG is not set

CONFIG_COMPILER_STACK_CHECK_MODE_ALL is not set

CONFIG_COMPILER_WARN_WRITE_STRINGS is not set

CONFIG_COMPILER_DISABLE_GCC12_WARNINGS is not set

CONFIG_COMPILER_DISABLE_GCC13_WARNINGS is not set

CONFIG_COMPILER_DUMP_RTL_FILES is not set

CONFIG_COMPILER_RT_LIB_GCCLIB=y
CONFIG_COMPILER_RT_LIB_NAME="gcc"

end of Compiler options

Partition Table

CONFIG_PARTITION_TABLE_SINGLE_APP=y

CONFIG_PARTITION_TABLE_SINGLE_APP_LARGE is not set

CONFIG_PARTITION_TABLE_TWO_OTA is not set

CONFIG_PARTITION_TABLE_CUSTOM is not set

CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv"
CONFIG_PARTITION_TABLE_FILENAME="partitions_singleapp.csv"
CONFIG_PARTITION_TABLE_OFFSET=0x8000
CONFIG_PARTITION_TABLE_MD5=y

end of Partition Table

CONFIG_OTA_ALLOW_HTTP is not set

ESP HTTP client

CONFIG_ESP_HTTP_CLIENT_ENABLE_HTTPS=y

CONFIG_ESP_HTTP_CLIENT_ENABLE_BASIC_AUTH is not set

CONFIG_ESP_HTTP_CLIENT_ENABLE_DIGEST_AUTH is not set

end of ESP HTTP client

HTTP Server

CONFIG_HTTPD_MAX_REQ_HDR_LEN=512
CONFIG_HTTPD_MAX_URI_LEN=512
CONFIG_HTTPD_ERR_RESP_NO_DELAY=y
CONFIG_HTTPD_PURGE_BUF_LEN=32

CONFIG_HTTPD_LOG_PURGE_DATA is not set

CONFIG_HTTPD_WS_SUPPORT is not set

CONFIG_HTTPD_QUEUE_WORK_BLOCKING is not set

end of HTTP Server

ESP HTTPS OTA

CONFIG_ESP_HTTPS_OTA_DECRYPT_CB is not set

CONFIG_ESP_HTTPS_OTA_ALLOW_HTTP is not set

end of ESP HTTPS OTA

ESP-TLS

CONFIG_ESP_TLS_USING_MBEDTLS=y

CONFIG_ESP_TLS_USE_SECURE_ELEMENT is not set

CONFIG_ESP_TLS_CLIENT_SESSION_TICKETS is not set

CONFIG_ESP_TLS_SERVER is not set

CONFIG_ESP_TLS_PSK_VERIFICATION is not set

CONFIG_ESP_TLS_INSECURE is not set

end of ESP-TLS

mbedTLS

CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y

CONFIG_MBEDTLS_DEFAULT_MEM_ALLOC is not set

CONFIG_MBEDTLS_CUSTOM_MEM_ALLOC is not set

CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y
CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=16384
CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=4096

CONFIG_MBEDTLS_DYNAMIC_BUFFER is not set

CONFIG_MBEDTLS_DEBUG is not set

mbedTLS v3.x related

CONFIG_MBEDTLS_SSL_PROTO_TLS1_3 is not set

CONFIG_MBEDTLS_SSL_VARIABLE_BUFFER_LENGTH is not set

CONFIG_MBEDTLS_X509_TRUSTED_CERT_CALLBACK is not set

CONFIG_MBEDTLS_SSL_CONTEXT_SERIALIZATION is not set

CONFIG_MBEDTLS_SSL_KEEP_PEER_CERTIFICATE=y
CONFIG_MBEDTLS_PKCS7_C=y

end of mbedTLS v3.x related

Certificate Bundle

CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_FULL=y

CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_CMN is not set

CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_NONE is not set

CONFIG_MBEDTLS_CUSTOM_CERTIFICATE_BUNDLE is not set

CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_MAX_CERTS=200

end of Certificate Bundle

CONFIG_MBEDTLS_ECP_RESTARTABLE is not set

CONFIG_MBEDTLS_CMAC_C=y
CONFIG_MBEDTLS_HARDWARE_AES=y
CONFIG_MBEDTLS_HARDWARE_MPI=y
CONFIG_MBEDTLS_HARDWARE_SHA=y
CONFIG_MBEDTLS_ROM_MD5=y

CONFIG_MBEDTLS_ATCA_HW_ECDSA_SIGN is not set

CONFIG_MBEDTLS_ATCA_HW_ECDSA_VERIFY is not set

CONFIG_MBEDTLS_HAVE_TIME=y

CONFIG_MBEDTLS_PLATFORM_TIME_ALT is not set

CONFIG_MBEDTLS_HAVE_TIME_DATE is not set

CONFIG_MBEDTLS_ECDSA_DETERMINISTIC=y
CONFIG_MBEDTLS_SHA512_C=y
CONFIG_MBEDTLS_TLS_SERVER_AND_CLIENT=y

CONFIG_MBEDTLS_TLS_SERVER_ONLY is not set

CONFIG_MBEDTLS_TLS_CLIENT_ONLY is not set

CONFIG_MBEDTLS_TLS_DISABLED is not set

CONFIG_MBEDTLS_TLS_SERVER=y
CONFIG_MBEDTLS_TLS_CLIENT=y
CONFIG_MBEDTLS_TLS_ENABLED=y

TLS Key Exchange Methods

CONFIG_MBEDTLS_PSK_MODES is not set

CONFIG_MBEDTLS_KEY_EXCHANGE_RSA=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ELLIPTIC_CURVE=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_RSA=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_ECDSA=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDH_ECDSA=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDH_RSA=y

end of TLS Key Exchange Methods

CONFIG_MBEDTLS_SSL_RENEGOTIATION=y
CONFIG_MBEDTLS_SSL_PROTO_TLS1_2=y

CONFIG_MBEDTLS_SSL_PROTO_GMTSSL1_1 is not set

CONFIG_MBEDTLS_SSL_PROTO_DTLS is not set

CONFIG_MBEDTLS_SSL_ALPN=y
CONFIG_MBEDTLS_CLIENT_SSL_SESSION_TICKETS=y
CONFIG_MBEDTLS_SERVER_SSL_SESSION_TICKETS=y

Symmetric Ciphers

CONFIG_MBEDTLS_AES_C=y

CONFIG_MBEDTLS_CAMELLIA_C is not set

CONFIG_MBEDTLS_DES_C is not set

CONFIG_MBEDTLS_BLOWFISH_C is not set

CONFIG_MBEDTLS_XTEA_C is not set

CONFIG_MBEDTLS_CCM_C=y
CONFIG_MBEDTLS_GCM_C=y

CONFIG_MBEDTLS_NIST_KW_C is not set

end of Symmetric Ciphers

CONFIG_MBEDTLS_RIPEMD160_C is not set

Certificates

CONFIG_MBEDTLS_PEM_PARSE_C=y
CONFIG_MBEDTLS_PEM_WRITE_C=y
CONFIG_MBEDTLS_X509_CRL_PARSE_C=y
CONFIG_MBEDTLS_X509_CSR_PARSE_C=y

end of Certificates

CONFIG_MBEDTLS_ECP_C=y

CONFIG_MBEDTLS_DHM_C is not set

CONFIG_MBEDTLS_ECDH_C=y
CONFIG_MBEDTLS_ECDSA_C=y

CONFIG_MBEDTLS_ECJPAKE_C is not set

CONFIG_MBEDTLS_ECP_DP_SECP192R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP224R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP256R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP384R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP521R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP192K1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP224K1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP256K1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_BP256R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_BP384R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_BP512R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_CURVE25519_ENABLED=y
CONFIG_MBEDTLS_ECP_NIST_OPTIM=y
CONFIG_MBEDTLS_ECP_FIXED_POINT_OPTIM=y

CONFIG_MBEDTLS_POLY1305_C is not set

CONFIG_MBEDTLS_CHACHA20_C is not set

CONFIG_MBEDTLS_HKDF_C is not set

CONFIG_MBEDTLS_THREADING_C is not set

CONFIG_MBEDTLS_LARGE_KEY_SOFTWARE_MPI is not set

end of mbedTLS

</p>
</details> 

[Ashish285]

Hi @DCSBL , thanks for creating this issue.

The error reported is MBEDTLS_ERR_SSL_BAD_INPUT_DATA which can be from few possible reasons:

1. SSL context corruption mid transfer: Due to some heap overflow from separate task
2. Server side issue where server transfers more data than MBEDTLS_SSL_IN_CONTENT_LEN

I understand that it might be hard to reproduce the issue but will it be possible for you to enabled mbedTLS debug logs and try to capture a failure scenario?

You can enable the debug logs by idf.py menuconfig -> Component config -> mbedTLS -> Enable mbedTLS debugging.

You may also try running only the OTA task and see if that fixes the issue or let us know if you can replicate the same behaviour with some publicly accessible server (such as Github) and we can try the same on our end. Thanks!