Modify to "String HTTPClient::getString(size_t size)"

[Rob58329]

Related area

Significantly reduce delay

Hardware specification

All ESP32 boards

Is your feature request related to a problem?

In cases where Server sends no Content-Length header, the above getString() function gets "_size=-1", and will wait as long as 2 minutes before returning.

Describe the solution you'd like

Further to: #2667

re. https://github.com/espressif/arduino-esp32/blob/master/libraries/HTTPClient/src/HTTPClient.h
and https://github.com/espressif/arduino-esp32/blob/master/libraries/HTTPClient/src/HTTPClient.cpp

In cases where Server sends no Content-Length header, the above getString() function gets "_size=-1", and will wait as long as 2 minutes before returning.

eg. URL=http://api.openweathermap.org/data/2.5/weather?lon=-1.264&lat=52.371&units=metric&lang=en&appid=???

(The http.setConnectTimeout(3000) and http.setTimeout(3000) commands do not change this.)

A fix for this is to set a limit on the minimum data size you need (ie. the maximum to receive) as follow:

Modify "HTTPClient.h" to say: String getString(size_t size=0);
Modify / add 1 line to "HTTPClient.cpp" to:

  String HTTPClient::getString(size_t size) {
    if ((_size==-1) && (size!=0)) _size=size;
    ...

This allows existing "String s=http.getString()" commands to still work fine.
But will also allow the new "String s=http.getString(500)" command which for the above URL example returns the data I need and reduces the wait time to under 1 second.

Although this modification is not as universal as implementing a selectable-timeout on this function, it is far simpler.

I can write a PR for this if someone is able to consider approving/commiting it?

Describe alternatives you've considered

No response

Additional context

No response

I have checked existing list of Feature requests and the Contribution Guide

• I confirm I have checked existing list of Feature requests and Contribution Guide.

[TD-er]

HTTPClient::writeToStream does call functions from _client, which has its own timeout.
So it is a bit strange it will not obey those timeouts.

[Jason2866]

Please test with Arduino Core 3.2.0-rc2 if the behaviour is still there.

[Rob58329]

@Jason2866: Sadly the "http.setTimeout(1000)" still does not work for "http.getString()" (which instead times out after 120 seconds).

{I am using "https://github.com/espressif/arduino-esp32/tree/3.2.0-RC2" as at 16Mar25.)
(I am using an ESP32-Dev-Module.)

The below sketch demonstrates this:

#include <WiFi.h>
#include <HTTPClient.h>

const char *WIFI_AP_SSID="????????"; 
const char *WIFI_AP_PW="????????";

String weatherAPIKey = String("0123456789abcdef0123456789abcdef"); // use your own API key
String weatherURL    = String("http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000"); // open weather api
String weatherUnit   = String("metric");
String weatherLang   = String("en");

void setup() {
  Serial.begin(115200); delay(3000);
  Serial.println("Booting...");
  WiFi.begin(WIFI_AP_SSID,WIFI_AP_PW);
  if (WL_CONNECTED == WiFi.waitForConnectResult()) { // attempt to connect for 10s
    Serial.print("Connected to: "); Serial.print(WiFi.SSID());
    Serial.print(", Local IP=");    Serial.println(WiFi.localIP());
    getWeatherData(weatherUnit, weatherLang, weatherURL, weatherAPIKey);
  } else Serial.println("No Wifi!");
}

void loop() {}

// https://github.com/espressif/arduino-esp32/issues/10375
// Modify "HTTPClient.h" to say: String getString(size_t size=0);
// Modify/add-1-line to "HTTPClient.cpp" to:
//  String HTTPClient::getString(size_t size) {
//    if ((_size==-1) && (size!=0)) _size=size;
//    if...

void getWeatherData(String units, String lang, String url, String apiKey) {
  HTTPClient http; // Use Weather API for live data if WiFi is connected
  http.setConnectTimeout(3000); // 3 second max timeout
  String weatherQueryURL = url + String("&units=") + units + String("&lang=") + lang + String("&appid=") + apiKey;
  Serial.print("weatherQueryURL="); Serial.println(weatherQueryURL);
  http.setTimeout(1000); // doesn't seem to work!
  http.begin(weatherQueryURL.c_str());
  int httpResponseCode = http.GET();
  Serial.printf("httpResponseCode=%i\n",httpResponseCode);
  if (httpResponseCode == 200) {
    Serial.println("Asking for getString() [takes 2mins, but should take ~1ms]...");
    // http.setTimeout(1000); // doesn't seem to work!
    uint32_t mstart=millis();
    String payload=http.getString();      // "Time=121367ms" which is about 2 minutes
    //String payload=http.getString(500); // "Time=1ms" // can use if we modify "HTTPClient.h"/"HTTPClient.cpp" as detailed above
    Serial.printf("Time=%ums\n",millis()-mstart);
    Serial.println("Got\n---"); Serial.println(payload);
    http.end();
  } else Serial.println("Bad Response (we were expecting 200)");
}

[Jason2866]

Closing since issue cant not be reproduced with Arduino Core 3.2.0. Using the provided example sketch with a valid API key results in.

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4660
load:0x40078000,len:15568
ho 0 tail 12 room 4
load:0x40080400,len:4
load:0x40080404,len:3152
entry 0x400805a0
Booting...
Connected to: Jason2866, Local IP=192.168.2.156
weatherQueryURL=http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=removed
httpResponseCode=200
Asking for getString() [takes 2mins, but should take ~1ms]...
Time=0ms
Got
---
{"coord":{"lon":-1,"lat":50},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}],"base":"stations","main":{"temp":8.72,"feels_like":4.42,"temp_min":8.72,"temp_max":8.72,"pressure":1018,"humidity":74,"sea_level":1018,"grnd_level":1018},"visibility":10000,"wind":{"speed":10.44,"deg":316,"gust":11.69},"clouds":{"all":2},"dt":1743195086,"sys":{"country":"FR","sunrise":1743140948,"sunset":1743186523},"timezone":0,"id":3016536,"name":"Gatteville-le-Phare","cod":200}

[Rob58329]

@Jason2866: I still get a 2minute responce time (as at 10Apr25):

I am using "https://github.com/espressif/arduino-esp32/tree/3.2.0-RC2" as at 10Apr25.
(I've also tried with "https://github.com/espressif/arduino-esp32/tree/3.2.0" but get pretty identical results.)

I am using an ESP32-Dev-Module.
I am using: #define OPENWEATHERMAP_APIKEY "f058fe1cad2afe8e2ddc5d063a64cecb" // use your own API key

My SSID (test1) is a mobile-hotspot connected to a 4G-LTE network (O2 in the UK).

-------------------------------------- Simple Output ---------------------------------------

Booting...
Connected to: test1, Local IP=192.168.51.32
weatherQueryURL=http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=f058fe1cad2afe8e2ddc5d063a64cecb
httpResponseCode=200
Asking for getString() [takes 2mins, but should take ~1ms]...
Time=121029ms
Got
---
1f2
{"coord":{"lon":-1,"lat":50},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"base":"stations","main":{"temp":9.24,"feels_like":6.05,"temp_min":9.24,"temp_max":9.24,"pressure":1029,"humidity":79,"sea_level":1029,"grnd_level":1029},"visibility":10000,"wind":{"speed":6.72,"deg":60,"gust":7.38},"clouds":{"all":100},"dt":1744295754,"sys":{"country":"FR","sunrise":1744262467,"sunset":1744310941},"timezone":3600,"id":3016536,"name":"Gatteville-le-Phare","cod":200}
0

--------------------------------------- Verbose Output ---------------------------------------

( 277.6 KB)
Allocated Bytes   :    34508 B (  33.7 KB)
Minimum Free Bytes:   278256 B ( 271.7 KB)
Largest Free Block:   110580 B ( 108.0 KB)
------------------------------------------
Flash Info:
------------------------------------------
Chip Size         :  4194304 B (4 MB)
Block Size        :    65536 B (  64.0 KB)
Sector Size       :     4096 B (   4.0 KB)
Page Size         :      256 B (   0.2 KB)
Bus Speed         : 80 MHz
Bus Mode          : QIO
------------------------------------------
Partitions Info:
------------------------------------------
              nvs : addr: 0x00009000, size:    20.0 KB, type: DATA, subtype: NVS
          otadata : addr: 0x0000E000, size:     8.0 KB, type: DATA, subtype: OTA
             app0 : addr: 0x00010000, size:  1280.0 KB, type:  APP, subtype: OTA_0
             app1 : addr: 0x00150000, size:  1280.0 KB, type:  APP, subtype: OTA_1
           spiffs : addr: 0x00290000, size:  1408.0 KB, type: DATA, subtype: SPIFFS
         coredump : addr: 0x003F0000, size:    64.0 KB, type: DATA, subtype: COREDUMP
------------------------------------------
Software Info:
------------------------------------------
Compile Date/Time : Apr 10 2025 15:51:57
Compile Host OS   : windows
ESP-IDF Version   : v5.4-690-gd4aa25a38e-dirty
Arduino Version   : 3.2.0
------------------------------------------
Board Info:
------------------------------------------
Arduino Board     : ESP32_DEV
Arduino Variant   : esp32
Arduino FQBN      : esp32:ESP32:esp32:JTAGAdapter=default,PSRAM=disabled,PartitionScheme=default,CPUFreq=240,FlashMode=qio,FlashFreq=80,FlashSize=4M,UploadSpeed=921600,LoopCore=1,EventsCore=1,DebugLevel=verbose,EraseFlash=none,ZigbeeMode=default
============ Before Setup End ============
[   734][V][esp32-hal-uart.c:552] uartBegin(): UART0 baud(115200) Mode(800001c) rxPin(3) txPin(1)
[   743][V][esp32-hal-uart.c:656] uartBegin(): UART0 not installed. Starting installation
[   753][V][esp32-hal-uart.c:666] uartBegin(): UART0 RX FIFO full threshold set to 120 (value requested: 120 || FIFO Max = 128)
[   766][V][esp32-hal-uart.c:687] uartBegin(): Setting UART0 to use REF_TICK clock
[   776][V][esp32-hal-uart.c:738] uartBegin(): UART0 initialization done.
[   791][V][esp32-hal-uart.c:785] uartSetRxFIFOFull(): UART0 RX FIFO Full value set to 120 from a requested value of 120
Booting...
[  3817][V][NetworkEvents.cpp:117] _checkForEvent(): Network Event: 101 - WIFI_READY
[  3892][V][STA.cpp:186] _onStaEvent(): STA Started
[  3897][V][NetworkEvents.cpp:117] _checkForEvent(): Network Event: 110 - STA_START
[  3905][V][STA.cpp:110] _onStaArduinoEvent(): Arduino STA Event: 110 - STA_START
[  3955][V][STA.cpp:206] _onStaEvent(): STA Connected: SSID: sg1, BSSID: ca:27:c2:4b:90:70, Channel: 8, Auth: WPA2_PSK
[  3966][V][NetworkEvents.cpp:117] _checkForEvent(): Network Event: 112 - STA_CONNECTED
[  3974][V][STA.cpp:110] _onStaArduinoEvent(): Arduino STA Event: 112 - STA_CONNECTED
[  4978][V][NetworkInterface.cpp:78] _onIpEvent(): sta Got New IP: 192.168.51.32 MASK: 255.255.255.0 GW: 192.168.51.153
[  4989][V][NetworkEvents.cpp:117] _checkForEvent(): Network Event: 115 - STA_GOT_IP
[  4996][V][STA.cpp:110] _onStaArduinoEvent(): Arduino STA Event: 115 - STA_GOT_IP
[  5004][V][STA.cpp:171] _onStaArduinoEvent(): STA IP: 192.168.51.32, MASK: 255.255.255.0, GW: 192.168.51.153
Connected to: test1, Local IP=192.168.51.32
weatherQueryURL=http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=f058fe1cad2afe8e2ddc5d063a64cecb
[  5026][V][HTTPClient.cpp:242] beginInternal(): url: http://api.openweathermap.org/data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=f058fe1cad2afe8e2ddc5d063a64cecb
[  5047][D][HTTPClient.cpp:293] beginInternal(): protocol: http, host: api.openweathermap.org port: 80 url: /data/2.5/weather?lon=-1.000&lat=50.000&units=metric&lang=en&appid=f058fe1cad2afe8e2ddc5d063a64cecb
[  5065][D][HTTPClient.cpp:574] sendRequest(): request type: 'GET' redirCount: 0

[  5073][D][NetworkManager.cpp:83] hostByName(): Clearing DNS cache
[  5359][D][NetworkManager.cpp:127] hostByName(): DNS found IPv4 5.9.68.251
[  5394][D][HTTPClient.cpp:1112] connect():  connected to api.openweathermap.org:80
[  5474][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
[  5481][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Server: openresty'
[  5488][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Date: Thu, 10 Apr 2025 14:53:07 GMT'
[  5497][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Content-Type: application/json; charset=utf-8'
[  5507][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'X-Cache-Key: /data/2.5/weather?lang=en&lat=50&lon=-1&units=metric'
[  5519][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Access-Control-Allow-Origin: *'
[  5528][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Access-Control-Allow-Credentials: true'
[  5537][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Access-Control-Allow-Methods: GET, POST'
[  5546][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: 'Connection: close'
[  5553][V][HTTPClient.cpp:1201] handleHeaderResponse(): RX: ''
[  5559][D][HTTPClient.cpp:1257] handleHeaderResponse(): code: 200
[  5565][D][HTTPClient.cpp:618] sendRequest(): sendRequest code=200

httpResponseCode=200
Asking for getString() [takes 2mins, but should take ~1ms]...
[126603][D][NetworkClient.cpp:576] connected(): Disconnected: RES: 0, ERR: 128
[126611][V][HTTPClient.cpp:1399] writeToStreamDataBlock(): connection closed or file end (written: 510).
[126620][D][HTTPClient.cpp:393] disconnect(): tcp is closed

Time=121054ms
Got
---
1f2
{"coord":{"lon":-1,"lat":50},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"base":"stations","main":{"temp":9.24,"feels_like":6.05,"temp_min":9.24,"temp_max":9.24,"pressure":1029,"humidity":79,"sea_level":1029,"grnd_level":1029},"visibility":10000,"wind":{"speed":6.72,"deg":60,"gust":7.38},"clouds":{"all":100},"dt":1744296787,"sys":{"country":"FR","sunrise":1744262467,"sunset":1744310941},"timezone":3600,"id":3016536,"name":"Gatteville-le-Phare","cod":200}
0


[126666][D][HTTPClient.cpp:393] disconnect(): tcp is closed

=========== After Setup Start ============
INTERNAL Memory Info:
------------------------------------------
Total Size        :   326036 B ( 318.4 KB)
Free Bytes        :   229784 B ( 224.4 KB)
Allocated Bytes   :    86004 B (  84.0 KB)
Minimum Free Bytes:   217112 B ( 212.0 KB)
Largest Free Block:   110580 B ( 108.0 KB)
------------------------------------------
GPIO Info:
------------------------------------------
GPIO : BUS_TYPE[bus/unit][chan]
--------------------------------------  
   1 : UART_TX[0]
   3 : UART_RX[0]
============ After Setup End =============