Add liveness and readiness probes to API deployment

[droctothorpe]

Idiomatically, production pods should leverage liveness probes. Liveness probes ensure that pods are restarted if the application is unresponsive but the main process is still running.

For pods running in production, you should always define a liveness probe. Without one, Kubernetes has no way of knowing whether your app is still alive or not. As long as the process is still running, Kubernetes will consider the container to be healthy.
-- From Kubernetes in Action by Marko Luksa

Dask Gateway Server includes a health check endpoint, but kubelet isn't checking it.

This PR updates the Helm deployment for Dask Gateway Server to include liveness and readiness probes.

I tested this on my local machine using Docker Desktop Kubernetes and confirmed that everything works.

Major kudos to @suchitpandya for identifying this discrepancy.

[jcrist]

Thanks @droctothorpe! I added the health check route, but completely forgot to add it to the chart. Good catch.

[droctothorpe]

Happy to contribute in whatever small way I can. Updated per your specifications and validated:

[jcrist]

LGTM. Thanks!

[dhirschfeld]

I'm not an expert (by any means) but I've read that you have to be cautious with using liveness probes:
https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html

Not sure if it's relevant here (prob depends on the implementation of the /api/health endpoint), but perhaps worth keeping in mind...

[jcrist]

Yeah, the important thing is that checks for the liveness probe don't rely on any downstream services, they only check the health of the pod itself. Otherwise you run the risk of cascading failures. We're fine here in this regard.

[consideRatio]

To reduce the issues of working with Helm's templates, I suggest opting for doing something more like this, and moving httpGet to values.yaml, removing enabled: true in favor of letting it being declared or not indicate if it should be included or not.

Hmm... Or maybe not if switching it on and off is a important matter to make it easy. But I wouldn't say it is.

If everything is to be made confiugrable in a Helm chart, it becomes messy otherwise, and if not, it becomes a big driver of new issues from users.

{{- with .Values.gateway.livenessProbe }}
livenessProbe:
  {{- .Values.gateway.livenessProbe | toYaml | nindent 12 }}
{{- end }}