Intermittent BT Navigator aborts caused by action server acknowledgement delays

[prabhav-a]

Bug report

BT Navigator intermittently aborts due to action server acknowledgement timeouts. This behavior is inconsistent — sometimes navigation works as expected, and other times the timeouts occur frequently.

I’ve been using Nav2 with Fast-RTPS for about four months without major issues, but over the past two weeks, these timeouts have become much more common. I’ve found several related issues reported earlier, but most discussions seem to have ended without a clear resolution.I understand that CycloneDDS is the recommended middleware for Nav2, but due to system constraints, I’m unable to switch from Fast-RTPS. My main questions are:

• Is this indeed a DDS-related issue, and has it been addressed in more recent versions?
• If so, are there any specific configurations or workarounds that could help mitigate it?

Our system runs approximately 70 nodes, and it’s consistently the Nav2 nodes whose actions or services intermittently fail to be discovered or acknowledge goals. While I’m not suggesting that the problem lies within Nav2 itself, I’d like to understand what aspects of Nav2’s communication pattern (e.g., action or service usage, QoS profiles) might make it more sensitive to Fast-RTPS issues. Also during this time the CPU Usage was less than 50%.

[bt_navigator] [WARN] Timed out while waiting for action server to acknowledge goal request for compute_path_to_pose
[bt_navigator] [WARN] Timed out while waiting for action server to acknowledge goal request for follow_path
[bt_navigator] [WARN] Node timed out while executing service call to local_costmap/clear_entirely_local_costmap
[bt_navigator] [WARN] [BehaviorTreeEngine] : Behavior Tree tick rate 100.00 was exceeded!
[bt_navigator] [ERROR] Goal failed

• Operating System: Ubuntu 22.04
• Computer: Nvidia Jetson Orin
• ROS2 Version: Humble
• Version or commit hash: Tried with latest humble binary (1.1.18) and also tried by building it from source of 1.1.19
• DDS implementation: Fast-RTPS

[mini-1235]

but due to system constraints, I’m unable to switch from Fast-RTPS

I don’t quite understand why switching to CycloneDDS isn’t possible. I have personally tried it on Jetson Orin and it worked fine for me.

[bt_navigator] [WARN] Timed out while waiting for action server to acknowledge goal request for compute_path_to_pose
[bt_navigator] [WARN] Timed out while waiting for action server to acknowledge goal request for follow_path
[bt_navigator] [WARN] Node timed out while executing service call to local_costmap/clear_entirely_local_costmap
[bt_navigator] [WARN] [BehaviorTreeEngine] : Behavior Tree tick rate 100.00 was exceeded!
[bt_navigator] [ERROR] Goal failed

I’ve been using Nav2 with Fast-RTPS for about four months without major issues, but over the past two weeks, these timeouts have become much more common.

I don't think we can reproduce this, but since you mentioned it worked well with older versions of Fast-DDS, could you try downgrading it if switching to CycloneDDS is impossible?

[prabhav-a]

Thank you for the quick response!

I don’t quite understand why switching to CycloneDDS isn’t possible. I have personally tried it on Jetson Orin and it worked fine for me.

I’m also using micro-ROS, which currently does not support CycloneDDS, so unfortunately switching middleware isn’t an option for our setup at the moment.

Also, just to clarify — I didn’t mean that it worked better with an older version of FastDDS. However, if there’s a particular FastDDS version known to be more stable with Nav2, I’d be happy to try downgrading or testing with that specific version. Please let me know if you have any recommendations.

Additionally, do you think this behavior could indeed be DDS-related, or might there be another possible.

[mini-1235]

Additionally, do you think this behavior could indeed be DDS-related, or might there be another possible.

Honestly, it’s difficult to say. It could be related to other nodes in your network or the way Nav2 is being used or DDS as you mentioned. You might need to do some debugging to pinpoint the cause. For example, you could try to reduce the number of nodes in the network

[tonynajjar]

[bt_navigator] [WARN] Node timed out while executing service call to local_costmap/clear_entirely_local_costmap

The server callback needs to acquire the costmap lock before clearing it so this could be because it takes more time for your costmap to update than your configured BT server timeout. You can figure out if that's the case with logs detailling how much time it took to acquire the lock.
I had a similar problem with the getCostsCallback

[prabhav-a]

[bt_navigator] [WARN] Node timed out while executing service call to local_costmap/clear_entirely_local_costmap

The server callback needs to acquire the costmap lock before clearing it so this could be because it takes more time for your costmap to update than your configured BT server timeout. You can figure out if that's the case with logs detailling how much time it took to acquire the lock. I had a similar problem with the getCostsCallback

Thanks for the insight! That makes sense — I’ll check if the delay is due to the costmap lock.

However, I’ve also noticed that compute_path_to_pose and follow_path sometimes time out as well, and the behavior seems completely random. Do you think this could indicate a broader timing issue.

I can try enabling detailed logs or timing traces if that helps narrow it down.