Websocket support in layer 7 NetworkPolicy

[cortopy]

Proposal / RFE

Is your feature request related to a problem?

Even though websockets are Layer 7, Cilium doesn't seem to support them in Network Policies.

Furthermore, if http rules are present, they interfere with the evaluation of requests to websockets in a way that all wss traffic gets dropped.

Ideally, I would expect CIlium to support the following:

• ignores websockets in L7 evaluation (the bare minimum if full support is not possible)
• applies http rules to websockets too so that can traffic can be forwarded
• there is some kind of wss ruleset in addition to http and kafka

Describe the solution you'd like

I have a CiliumNetworkPolicy like this

spec:
  endpointSelector:
    matchLabels:
      app: foo
  ingress:
    - fromEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: contour
          k8s:app: envoy
      toPorts:
      - ports:
        - port: "8080"
          protocol: TCP
        rules:
          http:
            - headers:
                - X-Forwarded-For: <my-ip>

for an application which uses websockets. The goal of this policy is to only allow private addresses to access the app.

My problem is that this app has an endpoint which is a websocket. When this policy is deployed, I can see that all http traffic gets through. However for the websocket endpoint I experience the following:

• cilium monitor shows all the http requests, but is totally silent for websocket
• requests to the websocket endpoint get a 403 response (which is not logged anywhere i can see)

[joestringer]

For reference, we use Envoy for the HTTP L7 filtering and it looks like Envoy has supported websockets for a while now: envoyproxy/envoy#1274 . Not sure what exactly is necessary on the Cilium filters side to enable this functionality.

@cortopy are there some standard demo apps available that could be used to easily test this? Step-by-step instructions would help a Cilium contributor to pick this up and work on it.

[cortopy]

There's no issue with Envoy or Cilium provided there are no L7 rules. It looks to me like a case where the evaluation of the rules are creating the problem and not Cilium networking itself.

I'm very confident the issue is with the evaluation of rules because as soon as I remove all the network policies in the namespace, I can see the application fine

This happened to me when trying to deploy theia. This is an excellent example to try because the app has both http and wss endpoints. Here is one of the official containers

[aditighag]

@cortopy Is the "my-ip" in the L7 rule above a cluster IP or an external IP? Have you tried creating an L3+L4 policy [1] to achieve the same?

[1] https://docs.cilium.io/en/v1.8/policy/language/#

[cortopy]

@aditighag sorry the delay. It was a cluster IP.

I've moved on to a different CNI as cilium was giving me other issues and I can't try the L3-L4 rules

[aditighag]

I triaged the issue, and the fix was discussed during the community meeting on Sept 28. I'll send out a PR in the v1.10 time frame.

[rpardini]

I was just hit in a very similar way when applying

io.cilium.proxy-visibility: "<Egress/53/UDP/DNS>,<Egress/80/TCP/HTTP>,<Ingress/5000/TCP/HTTP>"

to a Pod with a websocket app running on port 5000.
HTTP requests flow and visible normally, but the websocket connection simply fails.
App deployed is https://github.com/rpardini/fluidsignalr -- a simple SignalR app, nothing special about the deployment and service.
I've had to configure the Ingress (nginx) for websocket to work either way.

Cilium 1.10.6, mixed arm64/amd64 cluster otherwise working perfectly.