TransportGetAllocationStatsAction may cause significant load on elected master

[idegtiarenko]

Elasticsearch Version

8.14

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

TransportGetAllocationStatsAction runs on elected master so it is theoretically possible to overload it by executing node stats requests around various nodes in cluster, especially in a clusters with many shards as complexity is proportional to the shard count.

The aggregated result of the computation is small (5 numbers per node), we should consider caching it for small period of time (1 minute?) and reuse it between different calls during.

elasticsearch/server/src/main/java/org/elasticsearch/action/admin/cluster/node/stats/TransportNodesStatsAction.java

Lines 88 to 96 in 5e52059

	if (NodesStatsRequestParameters.Metric.ALLOCATIONS.containedIn(metrics)) {
	client.execute(
	TransportGetAllocationStatsAction.TYPE,
	new TransportGetAllocationStatsAction.Request(new TaskId(clusterService.localNode().getId(), task.getId())),
	listener.delegateFailure((l, r) -> {
	ActionListener.respondAndRelease(l, newResponse(request, merge(responses, r.getNodeAllocationStats()), failures));
	})
	);
	} else {

Steps to Reproduce

n/a

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[mhl-b]

Do we need client caching too? Server caching should help when there are few nodes with large number of shards. But if there many nodes that frequently poll stats, networking overhead will cripple up. May be distribute load, client picks random node from cluster and if that node does not have it in cache, forward to master and populate cache.

[DaveCTurner]

TransportGetAllocationStatsAction$Response is pretty small, ~100B/node or thereabouts, I think caching it on the master is sufficient.

[shreedaddy]

TransportGetAllocationStatsAction$Response is pretty small, ~100B/node or thereabouts, I think caching it on the master is sufficient.

Please let me know if this issue is open. If it is can you please assign it to me.

[DaveCTurner]

Still available @shreedaddy thanks for the offer. We can't assign issues to folks outside the @elastic org but if you want to contribute a PR then please feel free.

[shreedaddy]

Still available @shreedaddy thanks for the offer. We can't assign issues to folks outside the @elastic org but if you want to contribute a PR then please feel free.

Will get started.