Flink Kubernetes Operator Examples

Overview

This directory contains few examples for the Flink Kubernetes Operator. These examples should only serve as starting points when familiarizing yourself with the Flink Kubernetes Operator and users are expected to extend these based on their production needs.

Usage

Please refer to the Quick Start for setting up your environment to run the examples.

Examples

Basic Application Deployment example

This is a simple deployment defined by a minimal deployment file. The configuration contains the following:

• Defines the job to run
• Assigns the resources available for the job
• Defines the parallelism used

To run the job submit the yaml file using kubectl:

kubectl apply -f basic.yaml

Checkpointing & High Availability

Basic example to configure Flink Deployments in HA mode. The example shows how to set savepoint directory, checkpoint directory and HA. To try out this run the following command:

kubectl apply -f basic-checkpoint-ha.yaml

Basic Session Deployment example

This example shows how to create a basic Session Cluster and then how to submit specific jobs to this cluster if needed.

Without jobs

The Flink Deployment could be created without any jobs. In this case the Flink jobs could be created later by submitting the jobs separately.

The Flink Deployment configuration contains the following:

• The name of the Flink Deployment
• The resources available for the Flink Deployment

The Flink Deployment configuration does NOT contain the following:

• The job to run
• Any job specific configurations

To create a Flink Deployment with the specific resources without any jobs run the following command:

kubectl apply -f basic-session-deployment-only.yaml

Adding jobs

For an existing Flink Deployment another configuration could be used to create new jobs. This configuration should contain the following:

• The Flink Deployment to use
• The job to run
• Any job specific configurations

If the Flink Deployment is created by basic-session-deployment-only.yaml new job could be added by the following command:

kubectl apply -f basic-session-job-only.yaml

Creating Deployment and Jobs together

Alternatively the Flink Deployment and the Flink Session Job configurations can be submitted together.

To try out this run the following command:

kubectl apply -f basic-session-deployment-and-job.yaml

SQL runner

For running Flink SQL scripts check this example.

Python example

For running Flink Python jobs check this example.

Advanced examples

There are typical requirements for production systems and the examples below contain configuration files showing how to archive some of these.

Custom logging

This example adds specific logging configuration for the Flink Deployment using the logConfiguration property. To try out this run the following command:

kubectl apply -f custom-logging.yaml

Pod templates

Flink Kubernetes Operator provides the possibility to simplify the deployment descriptors by using Pod Templates.

This example shows how these templates are created and used. To try out this run the following command:

kubectl apply -f pod-template.yaml

Ingress

Flink's Web UI access can be configured by the Ingress entries.

Basic Ingress example

Simple domain based ingress configuration could be tried out by running this command:

kubectl apply -f basic-ingress.yaml

Advanced Ingress example

It is possible to generate path based routing. To try out this run the following command:

kubectl apply -f advanced-ingress.yaml

Horizontal Pod Autoscaler

It is possible to provide the Horizontal Pod Autoscaler configuration through the yaml files. To run this example Kubernetes 1.23 or newer is needed, so the autoscaling/v2 is available.

The feature is experimental so use it with caution but here is an example configuration:

kubectl apply -f hpa/basic-hpa.yaml

Using Kustomize

This example shows how to override default values using kustomize

For the detailed description of advanced configuration techniques follow this link.