Skip to content

Commit 9bb83c0

Browse files
gyforagyfora
authored
[FLINK-27009] Add example for running SQL scripts using the operator
1 parent f4b196f commit 9bb83c0

File tree

9 files changed

+465
-0
lines changed

9 files changed

+465
-0
lines changed

Diff for: ‎examples/flink-sql-runner-example/Dockerfile

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
################################################################################
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
################################################################################
18+
19+
FROM flink:1.15.0
20+
21+
RUN mkdir /opt/flink/usrlib
22+
ADD target/flink-sql-runner-example-*.jar /opt/flink/usrlib/sql-runner.jar
23+
ADD sql-scripts /opt/flink/usrlib/sql-scripts

Diff for: ‎examples/flink-sql-runner-example/README.md

+80
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Flink Kubernetes Operator SQL Example
21+
22+
## Overview
23+
24+
This is an end-to-end example of running Flink SQL scripts using the Flink Kubernetes Operator.
25+
26+
It is only intended to serve as a showcase of how Flink SQL can be executed on the operator and users are expected to extend the implementation and dependencies based on their production needs.
27+
28+
Currently, it is not planned to add direct API support for SQL submission to the Kubernetes operator due to the complexity of image and dependency management that is specific to each use-case.
29+
At the same time we are confident that using these examples as a starting point the operator would cover all user needs. If Apache Flink itself extends the SQL support for Application mode in the future, the operator will aim to support that.
30+
31+
*What's in this example?*
32+
33+
1. SQL Script runner Flink Java application
34+
2. DockerFile to build custom image with script runner + SQL scripts
35+
3. Example YAML for submitting scripts using the operator
36+
37+
## How does it work?
38+
39+
As Flink doesn't support submitting SQL scripts directly as jobs, we have created a simple Flink Java application that takes the user script and executes it using the `TableEnvironment#executeSql` method.
40+
41+
The SQL Runner will allow us to execute SQL scripts as if they were simple Flink Application jars, something that already works quite well with the operator. We package the included SQL Runner implementation together with the SQL scripts under `sql-scripts` into a docker image and we use it in our `FlinkDeployment` yaml file.
42+
43+
***Note:*** *While the included SqlRunner should work for most simple cases, it is not expected to be very robust or battle tested. If you find any bugs or limitations, feel free to open Jira tickets and bugfix PRs.*
44+
45+
## Usage
46+
47+
The following steps assume that you have the Flink Kubernetes Operator installed and running in your environment.
48+
49+
**Step 1**: Build Sql Runner maven project
50+
```bash
51+
cd examples/flink-sql-runner-example
52+
mvn clean package
53+
```
54+
55+
**Step 2**: Add your SQL script files under the `sql-scripts` directory
56+
57+
**Step 3**: Build docker image
58+
```bash
59+
# Uncomment when building for local minikube env:
60+
# eval $(minikube docker-env)
61+
62+
DOCKER_BUILDKIT=1 docker build . -t flink-sql-runner-example:latest
63+
```
64+
This step will create an image based on an official Flink base image including the SQL runner jar and your user scripts.
65+
66+
**Step 4**: Create FlinkDeployment Yaml and Submit
67+
68+
Edit the included `sql-example.yaml` so that the `job.args` section points to the SQL script that you wish to execute, then submit it.
69+
70+
```bash
71+
kubectl apply -f sql-example.yaml
72+
```
73+
74+
## Connectors and other extensions
75+
76+
This example will only work with the very basic table and connector types out of the box, however enabling new ones is very easy.
77+
78+
Simply find your [required connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) and add it as compile dependency to the `flink-sql-runner-example` project `pom.xml`. This will ensure that is packaged into the sql-runner fatjar and will be available for you on the cluster.
79+
80+
Once you dive deeper you will quickly find that the SqlRunner implementation is very basic and might not cover your more advanced needs. Feel free to simply extend or customise the code as necessary for your requirements.

Diff for: ‎examples/flink-sql-runner-example/pom.xml

+134
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
20+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
21+
<modelVersion>4.0.0</modelVersion>
22+
23+
<parent>
24+
<groupId>org.apache.flink</groupId>
25+
<artifactId>flink-kubernetes-operator-parent</artifactId>
26+
<version>1.1-SNAPSHOT</version>
27+
<relativePath>../..</relativePath>
28+
</parent>
29+
30+
<artifactId>flink-sql-runner-example</artifactId>
31+
<name>Flink SQL Runner Example</name>
32+
33+
<dependencies>
34+
<!-- Apache Flink dependencies -->
35+
<!-- These dependencies are provided, because they should not be packaged into the JAR file. -->
36+
<dependency>
37+
<groupId>org.apache.flink</groupId>
38+
<artifactId>flink-streaming-java</artifactId>
39+
<version>${flink.version}</version>
40+
<scope>provided</scope>
41+
</dependency>
42+
<dependency>
43+
<groupId>org.apache.flink</groupId>
44+
<artifactId>flink-table-api-java</artifactId>
45+
<version>${flink.version}</version>
46+
<scope>provided</scope>
47+
</dependency>
48+
49+
<!-- Add connector dependencies here. They must be in the default scope (compile). -->
50+
51+
<!-- Example:
52+
53+
<dependency>
54+
<groupId>org.apache.flink</groupId>
55+
<artifactId>flink-connector-kafka</artifactId>
56+
<version>${flink.version}</version>
57+
</dependency>
58+
-->
59+
60+
<!-- Add logging framework, to produce console output when running in the IDE. -->
61+
<!-- These dependencies are excluded from the application JAR by default. -->
62+
<dependency>
63+
<groupId>org.slf4j</groupId>
64+
<artifactId>slf4j-api</artifactId>
65+
<version>${slf4j.version}</version>
66+
<scope>provided</scope>
67+
</dependency>
68+
<dependency>
69+
<groupId>org.apache.logging.log4j</groupId>
70+
<artifactId>log4j-slf4j-impl</artifactId>
71+
<version>${log4j.version}</version>
72+
<scope>runtime</scope>
73+
</dependency>
74+
<dependency>
75+
<groupId>org.apache.logging.log4j</groupId>
76+
<artifactId>log4j-api</artifactId>
77+
<version>${log4j.version}</version>
78+
<scope>runtime</scope>
79+
</dependency>
80+
<dependency>
81+
<groupId>org.apache.logging.log4j</groupId>
82+
<artifactId>log4j-core</artifactId>
83+
<version>${log4j.version}</version>
84+
<scope>runtime</scope>
85+
</dependency>
86+
</dependencies>
87+
88+
<build>
89+
<plugins>
90+
<plugin>
91+
<groupId>org.apache.maven.plugins</groupId>
92+
<artifactId>maven-shade-plugin</artifactId>
93+
<version>3.1.1</version>
94+
<executions>
95+
<!-- Run shade goal on package phase -->
96+
<execution>
97+
<phase>package</phase>
98+
<goals>
99+
<goal>shade</goal>
100+
</goals>
101+
<configuration>
102+
<artifactSet>
103+
<excludes>
104+
<exclude>org.apache.flink:flink-shaded-force-shading</exclude>
105+
<exclude>com.google.code.findbugs:jsr305</exclude>
106+
<exclude>org.slf4j:*</exclude>
107+
<exclude>org.apache.logging.log4j:*</exclude>
108+
</excludes>
109+
</artifactSet>
110+
<filters>
111+
<filter>
112+
<!-- Do not copy the signatures in the META-INF folder.
113+
Otherwise, this might cause SecurityExceptions when using the JAR. -->
114+
<artifact>*:*</artifact>
115+
<excludes>
116+
<exclude>META-INF/*.SF</exclude>
117+
<exclude>META-INF/*.DSA</exclude>
118+
<exclude>META-INF/*.RSA</exclude>
119+
</excludes>
120+
</filter>
121+
</filters>
122+
<transformers>
123+
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
124+
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
125+
<mainClass>org.apache.flink.examples.SqlRunner</mainClass>
126+
</transformer>
127+
</transformers>
128+
</configuration>
129+
</execution>
130+
</executions>
131+
</plugin>
132+
</plugins>
133+
</build>
134+
</project>

Diff for: ‎examples/flink-sql-runner-example/sql-example.yaml

+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
################################################################################
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
################################################################################
18+
19+
apiVersion: flink.apache.org/v1beta1
20+
kind: FlinkDeployment
21+
metadata:
22+
name: sql-example
23+
spec:
24+
image: flink-sql-runner-example:latest
25+
flinkVersion: v1_15
26+
flinkConfiguration:
27+
taskmanager.numberOfTaskSlots: "1"
28+
serviceAccount: flink
29+
jobManager:
30+
resource:
31+
memory: "2048m"
32+
cpu: 1
33+
taskManager:
34+
resource:
35+
memory: "2048m"
36+
cpu: 1
37+
job:
38+
jarURI: local:///opt/flink/usrlib/sql-runner.jar
39+
args: ["/opt/flink/usrlib/sql-scripts/simple.sql"]
40+
parallelism: 1
41+
upgradeMode: stateless

Diff for: ‎examples/flink-sql-runner-example/sql-scripts/simple.sql

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an "AS IS" BASIS,
14+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*/
18+
19+
CREATE TABLE blackhole_table (
20+
name STRING,
21+
age INT
22+
) WITH (
23+
'connector' = 'blackhole'
24+
);
25+
26+
INSERT INTO blackhole_table
27+
VALUES ('fred flintstone', 35), ('barney rubble', 32);

Diff for: ‎examples/flink-sql-runner-example/sql-scripts/statement-set.sql

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an "AS IS" BASIS,
14+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*/
18+
19+
CREATE TABLE blackhole_table (
20+
name STRING,
21+
age INT
22+
) WITH (
23+
'connector' = 'blackhole'
24+
);
25+
CREATE TABLE blackhole_table2 (
26+
name STRING,
27+
age INT
28+
) WITH (
29+
'connector' = 'blackhole'
30+
);
31+
32+
EXECUTE STATEMENT SET
33+
BEGIN
34+
INSERT INTO blackhole_table
35+
VALUES ('fred flintstone', 35), ('barney rubble', 32);
36+
INSERT INTO blackhole_table2
37+
VALUES ('fred flintstone', 35), ('barney rubble', 32);
38+
END;

0 commit comments

Comments
 (0)