Use the Confluent Terraform provider to deploy and manage Confluent infrastructure. The Confluent Terraform provider automates the workflow for managing environments, Apache Kafka® clusters, Kafka topics, and other resources in Confluent Cloud.
confluent login --save
Install Terraform version manager tfutils/tfenv
Alternatively, install the Terraform CLI
To ensure you're using the acceptable version of Terraform you may run the following command:
terraform version
Your output should resemble:
Terraform v0.14.0 # any version >= v0.14.0 is OK
- Open the Confluent Cloud Console
- In the top right menu, select "Cloud API Keys"
- Choose "Add Key" and select "Granular Access"
- For Service Account, select "Create a New One" and name is <YOUR_NAME>-terraform-workshop-SA
- Download your key
- In the top right menu, select "Accounts & Access", select "Access" tab
- Click on the organization and select "Add Role Assignment"
- Select the account you created (service account) and select "Organization Admin". Click Save
In the initial setup, you'll be provisioning the following resources:
- A Basic Cluster (region and cloud provider can be altered in configurations/initial-setup/clusters.tf)
And using the following data sources:
- Cloud API Key and Secret (created above)
- Environment (your choice, ID provided in environment variables)
change to the correct directory
cd configurations/initial-setup
initialize the project
terraform init
export TF_VAR_confluent_cloud_api_key="<CONFLUENT_CLOUD_API_KEY>"
export TF_VAR_confluent_cloud_api_secret="<CONFLUENT_CLOUD_API_SECRET>"
export TF_VAR_confluent_cloud_env_id="<CONFLUENT_CLOUD_ENVIRONMENT_ID>"
export TF_VAR_confluent_cloud_cluster_name="<YOUR_NAME>_inventory"
validate the terraform configs
terraform validate
deploy the terraform planned resources
terraform apply
In this step, you'll be provisioning the following resources:
- Topic (called orders)
- API Keys, Service Accounts and Access Control Lists (to grant necessary permissions)
And using the following data sources:
- Basic Cluster (created above)
- Cloud API Key and Secret (created above)
- Environment (your choice, ID provided in environment variables)
change to the correct directory
cd ../topic-management
initialize the project
terraform init
Obtain the cluster id of the cluster you created in the previous section.
We will need to set a specific name for the service account we use, as service accounts are at the environment level, so any clusters within an environment cannot have conflicting service account names.
export TF_VAR_confluent_cloud_cluster_id="<CONFLUENT_CLOUD_CLUSTER_ID>"
export TF_VAR_confluent_cloud_topic_manager_name="<YOUR_NAME>-topic-manager"
export TF_VAR_confluent_cloud_topic_consumer_name="<YOUR_NAME>-topic-consumer"
export TF_VAR_confluent_cloud_topic_producer_name="<YOUR_NAME>-topic-producer"
validate the terraform configs
terraform validate
deploy the terraform planned resources
terraform apply
In this step, you'll be provisioning the following resources:
- Datagen Source Connector
- API Keys, Service Accounts and Access Control Lists (to grant necessary permissions)
And using the following data sources:
- Basic Cluster (created above)
- Cloud API Key and Secret (created above)
- Environment (your choice, ID provided in environment variables)
- Topic (orders, created above)
change to the correct directory
cd ../connect-management
initialize the project
terraform init
export TF_VAR_confluent_cloud_connect_manager_name="<YOUR_NAME>-connect-manager"
export TF_VAR_confluent_cloud_application_connector_name="<YOUR_NAME>-application-connector"
validate the terraform configs
terraform validate
deploy the terraform planned resources
terraform apply
in configurations/connect-management run:
terraform destroy
in configurations/topic-management run:
terraform destroy
in configurations/initial-setup run:
terraform destroy
Confluent Provider on Terraform Registry
Confluent Terraform Provider Docs
Confluent Example Configurations
Confluent Terraform Provider Webinar & Demo