Bluetooth looses data using library V3.0.4

[EricHarbers]

Board

ESP32

Device Description

The ESP32 chip is connected to a Bluetooth Serial device, eg a PC

Hardware Configuration

The ESP32 chip is connected to a serial device, using a UART

Version

v3.0.4

IDE Name

Arduino IDE

Operating System

Windows 10

Flash frequency

40 MHz

PSRAM enabled

yes

Upload speed

230400

Description

Today I installed the ESP32 library V3.0.4 for Arduino.
After I create a version of my project for the ESP32 project I use, my Bluetooth Serial communication looses data. The data will be transferred from a serial uart at 230400 Baud.
The problem started since the V3.0.4 version of the library. With previous versions I had no problems for Bluetooth.
I have seen that between the old version and the V3.0.4 version, the file BluetoothSerial.cpp has been changed significiantly.
With the V3.0.4 version, my bluetooth looses bytes at the speed of 230400 baud.

Sketch

/*
                              +------------------+
+-----------------------------| CPP SOURCE FILE  |-----------------------------+
|                             +------------------+                             |
|                                                                              |
|       project       : ESP32Control                                           |
|       filename      : Bluetooth.cpp                                          |
|       initiator     : Eric Harbers                                           |
|                                                                              |
+------------------------------------------------------------------------------+
Copyright (C) 2024, Enraf-Nonius, Rotterdam, Netherlands


 ________________________
| REVISION HISTORY       |
|________________________|_____________________________________________
|Version  |    Date      |     Revision by         |     Description
|_________|______________|_________________________|___________________
|  V1.00  |  2024-04-01  |  Eric Harbers           | * Original version
|  V1.01  |  2024-04-25  |  Eric Harbers           | * Check for 6 valid digits in BTConfirmRequestCallback
|         |              |                         | 
|         |              |                         | 
|         |              |                         | 
|_________|______________|_________________________|___________________

*/

#include "Bluetooth.h"
#include "MyomedProtocol.h"
#include "Message.h"

BluetoothSerial SerialBT;
t_Bluetooth bluetooth;
uint8_t ReceiveBuffer[512];

#ifdef MYOMEDPROTOCOL_ACTIVE
uint8_t mp_send_Buffer[1024];
uint8_t mp_receive_Buffer[1024];
MyomedProtocol mp_send(mp_send_Buffer, sizeof(mp_send_Buffer));
MyomedProtocol mp_receive(mp_receive_Buffer, sizeof(mp_receive_Buffer));
#endif

void bluetooth_connect();
void bt_spp_cb(esp_spp_cb_event_t event, esp_spp_cb_param_t *param);
void bluetooth_receiveBT(const uint8_t *buffer, size_t size);
void bluetooth_receiveSerial();
void bluetooth_receiveErrorSerial(hardwareSerial_error_t e);
uint8_t bluetooth_sendcommand(MyomedCommand &c);
uint8_t bluetooth_receivecommand(MyomedCommand &c);
uint8_t bluetooth_receivesample(MyomedSample &s);

void BTConfirmRequestCallback(uint32_t keyVal) {
  SerialBT.confirmReply(true);
  if (keyVal < 1000000) // must be max 6 digits
  {
    Serial1.printf("BTConfirmRequestCallback val %u\r\n", keyVal);
    bluetooth.keyVal = keyVal;
    bluetooth.status |= BLUETOOTH_STATUS_PIN;
  }
}

void BTAuthCompleteCallback(bool success) {
  bluetooth.status &= ~BLUETOOTH_STATUS_PIN;
  bluetooth.keyVal = (uint32_t)-1;
}

void bluetooth_init() {

  memset(&bluetooth, 0, sizeof(t_Bluetooth));
  bluetooth.keyVal = (uint32_t)-1;
  strcpy(bluetooth.localName, "Myomed");
  strcpy(bluetooth.remoteName, "raspberrypi");
  strcpy(bluetooth.pin, "1234");
  memcpy(bluetooth.check, "12345678", BLUETOOTH_CHECK_LEN);
  Serial.setRxBufferSize(1024);
  Serial.onReceive(bluetooth_receiveSerial);
  Serial.onReceiveError(bluetooth_receiveErrorSerial);
  SerialBT.register_callback(bt_spp_cb);
  SerialBT.onConfirmRequest(BTConfirmRequestCallback);
  SerialBT.onAuthComplete(BTAuthCompleteCallback);
  SerialBT.onData(bluetooth_receiveBT);
  SerialBT.enableSSP();
  SerialBT.begin(bluetooth.localName, true);
#ifdef MYOMEDPROTOCOL_ACTIVE
  mp_send.setCommandFunction(bluetooth_sendcommand);
  mp_receive.setCommandFunction(bluetooth_receivecommand);
  //mp_receive.setSampleFunction(bluetooth_receivesample);
#endif

  bluetooth_connect();
}

void bluetooth_connect() {
  if (SerialBT.connect(bluetooth.remoteName))
  {
    bluetooth.status |= BLUETOOTH_STATUS_CONNECTED;
  }
  else
  {
    bluetooth.status &= ~BLUETOOTH_STATUS_CONNECTED;
  }
  memcpy((void *)&bluetooth.macAddress, (void *)esp_bt_dev_get_address(), sizeof(bluetooth.macAddress));
}

void bluetooth_receiveBT(const uint8_t *buffer, size_t size) {
#ifdef MYOMEDPROTOCOL_ACTIVE
  mp_send.parse(buffer, size);
#endif
  Serial.write(buffer, size);
}

void bluetooth_receiveSerial() {
  size_t len;

  while ((len = Serial.read(ReceiveBuffer, sizeof(ReceiveBuffer))) > 0) {
#ifdef MYOMEDPROTOCOL_ACTIVE
    mp_receive.parse(ReceiveBuffer, len);
#endif
    SerialBT.write(ReceiveBuffer, len);
  }
}

void bluetooth_receiveErrorSerial(hardwareSerial_error_t e) {
  static char* errorList[] = {
    "UART_NO_ERROR",
    "UART_BREAK_ERROR",
    "UART_BUFFER_FULL_ERROR",
    "UART_FIFO_OVF_ERROR",
    "UART_FRAME_ERROR",
    "UART_PARITY_ERROR"
  };
  
//  Serial1.printf("error = %s\r\n", errorList[e]);
}

uint8_t bluetooth_sendcommand(MyomedCommand &c) {
  Serial1.printf("Send: ");
  Serial1.printf("Cmd = %X, ", c.getCmd());
  Serial1.printf("Data = %X\r\n", c.getData());
  return 0;
}

uint8_t bluetooth_receivecommand(MyomedCommand &c) {
  uint16_t cmd = c.getCmd();

  switch (cmd) {
    case 0x100:
      break;

    default:
      Serial1.printf("Receive: ");
      Serial1.printf("Cmd = %X, ", cmd);
      Serial1.printf("Data = %X\r\n", c.getData());
      return 1;
  }
  return 0;
}

uint8_t bluetooth_receivesample(MyomedSample &s) {
  static size_t cnt = 0;
  size_t i;
  bool res = true;

  for (i=0; i<2; i++) {
    unsigned int v = s[i];
    char low = v & 0xFF;
    char high = (v >> 8) & 0xFF;

    res &= (low == high);
  }
  if (!res) {
    cnt++;
  }
  else
    cnt = 0;
  return 0;
}

void bt_spp_cb(esp_spp_cb_event_t event, esp_spp_cb_param_t *param) {
  //Serial1.printf("Event = %u\r\n", event);
  switch (event) {
    case ESP_SPP_SRV_OPEN_EVT://Client connection open
      bluetooth.status |= BLUETOOTH_STATUS_CONNECTED;
      break;

    case ESP_SPP_CLOSE_EVT://Client connection closed
      bluetooth.status &= ~BLUETOOTH_STATUS_CONNECTED;
      break;
  }
}

void bluetooth_tick() {
}

Debug Message

I see no specific debug messages, but I loose data in my Bluetooth communication so that there will be a corrupted communication between my ESP32 and the PC. This was not the case with previous lib versions.

Other Steps to Reproduce

I have solved the problem by increasing the TX_QUEUE_SIZE in the V3.0.4 lib file BluetoothSerial.cpp. See line 44:
The original TX_QUEUE_SIZE was: 32
I have increased the TX_QUEUE_SIZE to: 128

This change solved my problem.

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[me-no-dev]

can you please provide a minimal sketch to reproduce the issue. One that we can compile and verify

[EricHarbers]

ESP32Control.zip
Attached the zip file containing my project.
In the zip file there is also the file sections.ld, in which I've added a section:
.version_info : ALIGN(0x10)
{

KEEP(*(.versionSection)) /* keep my variable even if not referenced */

} >default_rodata_seg
This ld file is placed in the arduino library map.

[EricHarbers]

MyoConnect.zip
Also the zip file MyoConnect.zip. Please use that on the pc and start MyoConnect.exe.
The PC bluetooth name can be any name.
In the Debug screen, you will see messages like: a 2-byte hex value and then: the command does not exist. (Maybe in dutch, i am not sure). This is the indication that the protocol has been disturbed.

[EricHarbers]

No, you miss the device that is serial connected to the ESP32.
It will not communicate now.
I'll change the code for you. Be patient.

[EricHarbers]

I don't know how to let you get our problem, because I miss a part of our device that is specific.
Can't you simply send data between the ESP32 and a PC using a bluetooth channel. The data should be read by the ESP32 from a serial input at 230400 bps. That is what my device does.

[EricHarbers]

You have to simply send data between the ESP32 and a PC using a bluetooth channel. The data should be read by the ESP32 from a serial input at 230400 bps. That is what my device does.

[EricHarbers]

The data loose is still coming, despite the bigger buffer I used
the problem started since version V3.0.4.
Can someone take a look for it?

[EricHarbers]

I've investigated some more time in the problem.
I'm back to ESP32 BSP V3.0.3. That solved all my problems with bluetooth and serial UART.

With BSP V3.0.4 I had the following problems:

• The serial UART looses data from time to time. (That was the problem mentioned above, for bluetooth (but it occurred on the UART))
• During Bluetooth communication, the ESP32 disconnected the bluetooth connection.
• When the ESP32 was paired with my PC, it couldn't connect with bluetooth. It gave Error 1 (unknown socket error) or Error 4 and stayed disconnected.
• Only with an unpaired ESP32 it was possible to communicate through bluetooth.

[me-no-dev]

there are no changes on our end between those versions, which suggests that maybe something in IDF changed. If so, maybe it's already fixed or will be soon fixed. Please try the new versions when they come out

[EricHarbers]

On your BluetoothSerial.cpp, I've seen a lot of changed code between V3.0.3 and V3.0.4.
I could see that because I have a copy of bluetoothserial.cpp V3.0.3.

I don't know the changes with serial uart.

[me-no-dev]

are you sure you've seen changes in BluetoothSerial between 3.0.3 and 3.0.4? 3.0.3 was released on July 17th and last change is in June 19th

[EricHarbers]

Dear me-no-dev,

Yes, I am absolutely sure that the code for BluetoothSerial has been changed between V3.0.3 and V3.0.4. That's why I initially came with the bufsize change.
With V3.0.3 it works fine.
Next week, at my work, I'll take an extra check.

[EricHarbers]

No, you have right.
I've checked the code of V3.0.3 and it looks the same like V3.0.4.
Sorry for the mistake.

But I don't know why V3.0.3 BT will work correctly and V3.0.4 BT not.

[me-no-dev]

Probably something in ESP-IDF driver changed. That is why I suggested for you to try new versions when we release them and see if the issue has been resolved

[EricHarbers]

I've installed BSP 3.0.5 and tested that with my ESP32 project.
My data loose problem still exists.
The data protocol I've build up to the BT serial layer will not accept the stream. This means for me, that data will be lost or changed in the BT serial driver, or somewhere below that.
If I go back to BSP 3.0.3 all works fine.

[GitHubLionel]

I have quite the same problem with data receive from UART. With version 3.0.3 all is good, but in version 3.0.5 the data received is sometimes smaller than 0x80. For example, normal data is 0x93 but sometimes I receive 0x13 instead.
I am totally helpless in the face of this problem.
EDIT : after many search, I found the problem.
In file esp32-hal-uart.c, on lines 508+, we have this code :

  if (baudrate <= 250000) {
    uart_config.source_clk = UART_SCLK_REF_TICK;  // valid for ESP32, S2 - MAX supported baud rate is 250 Kbps
  } else {
    uart_config.source_clk = UART_SCLK_APB;  // baudrate may change with the APB Frequency!
  }

And this is the problem because my speed is 256 kbaud. If I take 128 kbaud, no problem.
In old code (v 3.0.3), we just have this line of code :

uart_config.source_clk = UART_SCLK_DEFAULT;

[SuGlider]

Adding this case for testing NimBLE future library. @lucasssvaz PTAL.

[Jason2866]

Is the issue still there with actual Arduino core 3.1.3 and 3.2.0-rc2?

[GitHubLionel]

Hi,

No, problem is not solved and I think to know where is the problem.
When I start my application, I have a low baudrate (600 bauds) to configure my peripheral. Then after, I change to a high baudrate (256000 or 512000 bauds). Code is like:

Serial.begin(600);
... configuration
Serial.begin(256000);

The problem is that uart_config.source_clk is not change and stay to old value.
This code in esp32-hal-uart.c is not correct:

      if (uart->_baudrate != baudrate) {
        if (ESP_OK != uart_set_baudrate(uart_nr, baudrate)) {
          log_e("UART%d changing baudrate failed.", uart_nr);
          retCode = false;
        } else {
          log_i("UART%d changed baudrate to %d", uart_nr, baudrate);
          uart->_baudrate = baudrate;  **======> ERROR, source_clk not updated**
        }
      }

[lucasssvaz]

@SuGlider Could you PTAL in this uart issue ?

[Jason2866]

Can you try with Serial.updateBaudRate(256000) as done in example https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/Serial_All_CPU_Freqs/Serial_All_CPU_Freqs.ino

[GitHubLionel]

I tried. Work fine with Serial.updateBaudRate(256000) in place of Serial.begin(256000) 👍

[GitHubLionel]

So raw correction is to add this lines:

#if !SOC_UART_SUPPORT_XTAL_CLK
  soc_module_clk_t newClkSrc = baudrate <= REF_TICK_BAUDRATE_LIMIT ? SOC_MOD_CLK_REF_TICK : SOC_MOD_CLK_APB;
  uart_ll_set_sclk(UART_LL_GET_HW(uart->num), newClkSrc);
#endif

Before change the baudrate:

      if (uart->_baudrate != baudrate) {
#if !SOC_UART_SUPPORT_XTAL_CLK
  soc_module_clk_t newClkSrc = baudrate <= REF_TICK_BAUDRATE_LIMIT ? SOC_MOD_CLK_REF_TICK : SOC_MOD_CLK_APB;
  uart_ll_set_sclk(UART_LL_GET_HW(uart->num), newClkSrc);
#endif
        if (ESP_OK != uart_set_baudrate(uart_nr, baudrate)) {
          log_e("UART%d changing baudrate failed.", uart_nr);
          retCode = false;
        } else {
          log_i("UART%d changed baudrate to %d", uart_nr, baudrate);
          uart->_baudrate = baudrate;
        }
      }

[Jason2866]

No, when changing the baudrate the command Serial.updateBaudRate(x) needs to be used.

[GitHubLionel]

Ok, in that case, we need a warning for the begin() method to say that if you just want to change the Baudrate you must use updateBaudRate() method even if uartBegin() seems to allow this change.

[Jason2866]

Documentation is Open Source too. A PR is welcome

[SuGlider]

Are we hijacking this issue? @GitHubLionel - does the UART reported problem relate to this BT Serial issue?
Otherwise we need to verify if this BT issue is still valid. IDF has done many changes to Bluedroid Stack, which may affect BluetoothSerial.

[Jason2866]

@EricHarbers Is the orig. issue fixed?

[EricHarbers]

@Jason2866 Due to issue #10420 I can't currently test the problem properly. Maybe the problem is issue #10420 itself.
If you can fix issue #10420 first, then I can test this issue.

[SuGlider]

As said in #10420 (comment), HardwareSerial::onReceive() works like an ISR. Therefore is should just store the UART received data into some buffer and exit. The onReceive function may turn on some flag or a FreeRTOS Semaphore. You can also send the incomming UART data to a FreeRTOS Queue, if necessary.

On a separated FreeRTOS Task, it can check the flag or the Semaphore or block an infinite loop that checks if there is data in the Queue. Then process this data and send it to the SerialBT receiving end.

The current code is not recomented.

void bluetooth_receiveSerial() {
  size_t len;

  while ((len = Serial.read(ReceiveBuffer, sizeof(ReceiveBuffer))) > 0) {
#ifdef MYOMEDPROTOCOL_ACTIVE
    mp_receive.parse(ReceiveBuffer, len);
#endif
    SerialBT.write(ReceiveBuffer, len);
  }
}

[Jason2866]

Closing since fixed.