Sporadic "CT: Map insertion failed" errors, mostly on short-lived connections

[lgyurci]

Is there an existing issue for this?

• I have searched the existing issues

Version

equal or higher than v1.16.0 and lower than v1.17.0

What happened?

This bug report is related to #33074 , where in the same context, "No mapping for NAT masquerade" errors occured. After updating from 1.15.x to 1.16.0, that issue got resolved, and instead, flows get dropped at about the same rate with the reason "CT: Map insertion failed". In #33115 , a specific race condition got "fixed", and my guess is, that if the port allocation now fails on rev snat, cilium does not try again, and drops the flow.

How can we reproduce the issue?

Try hosting a highly loaded DNS server on a kubernetes cluster with cilium as the CNI. Or create and close a lot of connections quickly in other ways (while the CT table is highly loaded), since the issue happens "randomly" on new connections. More ideas in #33074 .

Cilium Version

Client: 1.16.0 8299999 2024-07-23T22:22:14-07:00 go version go1.22.5 linux/amd64
Daemon: 1.16.0 8299999 2024-07-23T22:22:14-07:00 go version go1.22.5 linux/amd64

Kernel Version

Linux 5.14.0-427.16.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 8 17:48:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

v1.28.13+rke2r1

Regression

It did work differently in 1.15.x, but to my best knowledge, never worked properly.

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[marseel]

👋 Have you taken a look at https://docs.cilium.io/en/stable/operations/troubleshooting/#handling-drop-ct-map-insertion-failed maybe? I wonder if that potentially helps.

It would be great if you could include sysdump.

[lgyurci]

Hi. There are a lot of sensitive data in the sysdump, which I would rather not publish. Is there any specific file or section that you are interested in?

We have followed through with the common troubleshooting steps, and I'm pretty sure this has nothing to do with the CT completely filling up, like in the issue described in the reference you have sent to me. Please refer to #33074 for more information, this is essentially the same issue, just behaving differently, probably because of the "fix" in #33115 .

[pasteley]

We are currently running 1.16.4 and encountered with the same problem. It's mainly reproduced at agent on node with pods that handles dns queries.

On metric cilium_drop_count_total{reason="CT: Map insertion failed"} we observe such drops ones per 5 sec.

From my point of view it's unlikely a bpf map scaling problem.
We can see that map grows to ~2.37M entries for Non-TCP:

# cilium status --verbose | grep "connection tracking"
  Non-TCP connection tracking   2370229
  TCP connection tracking       4740459

At the same time, ~122k alive non-TCP connections with significant GC activity with ~200k deleted non-TCP entries:

# cilium metrics list | grep conntrack
cilium_datapath_conntrack_dump_resets_total              area="conntrack" family="ipv6" name="dump_interrupts"                                     0.000000
cilium_datapath_conntrack_gc_duration_seconds            family="ipv6" protocol="TCP" status="completed"                                           75.917479
cilium_datapath_conntrack_gc_duration_seconds            family="ipv6" protocol="non-TCP" status="completed"                                       1084.062838
cilium_datapath_conntrack_gc_entries                     family="ipv6" protocol="TCP" status="alive"                                               7299.000000
cilium_datapath_conntrack_gc_entries                     family="ipv6" protocol="TCP" status="deleted"                                             11182.000000
cilium_datapath_conntrack_gc_entries                     family="ipv6" protocol="non-TCP" status="alive"                                           120216.000000
cilium_datapath_conntrack_gc_entries                     family="ipv6" protocol="non-TCP" status="deleted"                                         219242.000000
cilium_datapath_conntrack_gc_key_fallbacks_total         family="ipv6" protocol="TCP"                                                              0.000000
cilium_datapath_conntrack_gc_key_fallbacks_total         family="ipv6" protocol="non-TCP"                                                          0.000000
cilium_datapath_conntrack_gc_runs_total                  family="ipv6" protocol="TCP" status="completed"                                           555.000000
cilium_datapath_conntrack_gc_runs_total                  family="ipv6" protocol="non-TCP" status="completed"                                       555.000000

On atempt to list them using cilium bpf ct list global cilium issues error (we are running ipv6-only):
Error: Unable to open /sys/fs/bpf/tc/globals/cilium_ct4_global: loading pinned map /sys/fs/bpf/tc/globals/cilium_ct4_global: no such file or directory

And another strange fact that cilium_ct6_global appears to be empty:

# ls -l /sys/fs/bpf/tc/globals/cilium_ct6*
-rw------- 1 root root 0 May  9  2024 /sys/fs/bpf/tc/globals/cilium_ct6_global

In cilium monitor drops looks like this (with pod on current node as dst):

xx drop (CT: Map insertion failed, -16) flow 0xb9b89bf3 to endpoint 1134, ifindex 2, file bpf_lxc.c:1705, , identity 16777247->591038: 2a00:<removed>:2df2 -> 2a00:<removed>:e602
xx drop (CT: Map insertion failed, -16) flow 0x5d63f125 to endpoint 0, ifindex 2, file bpf_host.c:468, , identity 16777247->unknown: 2a00:<removed>:9d17 -> 2a00:<removed>:e602

[vonzepp]

We have a similar issue with pods, serving DNS requests (A lot of UDP requests).
cilium_datapath_conntrack_gc_runs_total and cilium_drop_count_total{reason="CT: Map insertion failed"} have a strong correlation.
Not all Garbage collections lead to drops with reason="CT: Map insertion failed". But all drops were exaclty at a time of GC.
cilium version: 1.16.4

[brb]

@pasteley Could you attach a sysdump? If not feasible, then bpftool prog and bpftool map outputs?

[pasteley]

@pasteley Could you attach a sysdump? If not feasible, then bpftool prog and bpftool map outputs?

bpftool_prog
bpftool_map

[brb]

@pasteley Thanks. I wanted to validate that BPF progs are using a single instance of the relevant CT BPF map. All looks good.

[NeonSludge]

Seeing the same issue in our cluster (AlmaLinux 9.4, vanilla kernel, k8s 1.30.4, Cilium 1.16.4) on nodes with high connection churn (CT map pressure is low).
The CT map insertion failures in our case also have a strong correlation to CT GC runs.