How to Deploy Apache Kafka Cluster on Kubernetes

In this article we are going to cover Deploying a Kafka Cluster on Kubernetes | How to Deploy Apache Kafka Cluster on Kubernetes.

Apache Kafka is a distributed event streaming platform capable of handling large-scale data streams in real-time. Deploying Kafka on Kubernetes provides scalability, fault tolerance, and ease of management. In this guide, we will walk through the step-by-step process of deploying a Kafka cluster on Kubernetes using ZooKeeper for coordination.

Prerequisites

Before starting, ensure you have the following:

• A Kubernetes cluster (Minikube, EKS, GKE, or AKS)
• kubectl installed and configured
• Sufficient resources (at least 3 worker nodes with 4GB RAM each)
• Basic knowledge of Kubernetes concepts like Deployments, Services, and ConfigMaps

Step #1:Set Up the Namespace in Kubernetes

Create a separate namespace for Kafka to keep resources organized.

kubectl create namespace kafka

Step #2: Deploy Zookeeper on Kubernetes

Kafka relies on Zookeeper for cluster coordination. Deploy a single-instance Zookeeper service.

zookeeper.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper-1
  namespace: kafka
  labels:
    app: zookeeper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - name: zookeeper
        image: confluentinc/cp-zookeeper:7.3.2
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_CLIENT_PORT
          value: "2181"
        - name: ZOOKEEPER_SERVER_ID
          value: "1"
        - name: ZOOKEEPER_SERVERS
          value: "zookeeper-1:2888:3888"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper-1
  namespace: kafka
spec:
  ports:
  - port: 2181
    targetPort: 2181
  selector:
    app: zookeeper

Apply:

kubectl apply -f zookeeper.yaml

Description:

Deployment:

• Runs Zookeeper (confluentinc/cp-zookeeper:7.3.2) with:
  • Client port: 2181
  • Server ID: 1
  • Cluster config: zookeeper-1:2888:3888

Service:

• Exposes Zookeeper on port 2181 for client connections.

Step #3:Deploy Kafka Brokers on Kubernetes

Deploy a Kafka broker that connects to Zookeeper.

Kafka Broker 1

kafka-broker-1.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-broker-1
  namespace: kafka
  labels:
    app: kafka-broker-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker-1
  template:
    metadata:
      labels:
        app: kafka-broker-1
    spec:
      hostname: kafka-broker-1
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:7.3.2
        ports:
        - containerPort: 9092
        - containerPort: 29092
        env:
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zookeeper-1:2181"
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
        - name: KAFKA_INTER_BROKER_LISTENER_NAME
          value: "PLAINTEXT"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "PLAINTEXT://kafka-broker-1:29092,PLAINTEXT_HOST://kafka-broker-1:9092"
        - name: KAFKA_BROKER_ID
          value: "1"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-broker-1
  namespace: kafka
spec:
  ports:
  - name: external
    port: 9092
    targetPort: 9092
  - name: internal
    port: 29092
    targetPort: 29092
  selector:
    app: kafka-broker-1
  type: ClusterIP

Apply:

kubectl apply -f kafka-broker-1.yaml

Kafka Broker 2

kafka-broker-2.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-broker-2
  namespace: kafka
  labels:
    app: kafka-broker-2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker-2
  template:
    metadata:
      labels:
        app: kafka-broker-2
    spec:
      hostname: kafka-broker-2
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:7.3.2
        ports:
        - containerPort: 9093
        - containerPort: 29093
        env:
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zookeeper-1:2181"
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
        - name: KAFKA_INTER_BROKER_LISTENER_NAME
          value: "PLAINTEXT"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "PLAINTEXT://kafka-broker-2:29093,PLAINTEXT_HOST://kafka-broker-2:9093"
        - name: KAFKA_BROKER_ID
          value: "2"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-broker-2
  namespace: kafka
spec:
  ports:
  - name: external
    port: 9093
    targetPort: 9093
  - name: internal
    port: 29093
    targetPort: 29093
  selector:
    app: kafka-broker-2
  type: ClusterIP

Apply:

kubectl apply -f kafka-broker-2.yaml

Kafka Broker 3

kafka-broker-3.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-broker-3
  namespace: kafka
  labels:
    app: kafka-broker-3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker-3
  template:
    metadata:
      labels:
        app: kafka-broker-3
    spec:
      hostname: kafka-broker-3
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:7.3.2
        ports:
        - containerPort: 9094
        - containerPort: 29094
        env:
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zookeeper-1:2181"
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
        - name: KAFKA_INTER_BROKER_LISTENER_NAME
          value: "PLAINTEXT"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "PLAINTEXT://kafka-broker-3:29094,PLAINTEXT_HOST://kafka-broker-3:9094"
        - name: KAFKA_BROKER_ID
          value: "3"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-broker-3
  namespace: kafka
spec:
  ports:
  - name: external
    port: 9094
    targetPort: 9094
  - name: internal
    port: 29094
    targetPort: 29094
  selector:
    app: kafka-broker-3
  type: ClusterIP

Apply:

kubectl apply -f kafka-broker-3.yaml

Description:

• Image: confluentinc/cp-kafka:7.3.2
• Namespace: kafka
• Connected to Zookeeper (zookeeper-1:2181) for coordination.
• Each broker has a unique ID (KAFKA_BROKER_ID):
  • Broker 1 → ID: 1 (Ports: 9092, 29092)
  • Broker 2 → ID: 2 (Ports: 9093, 29093)
  • Broker 3 → ID: 3 (Ports: 9094, 29094)
• Exposed Listeners:
  • Internal (2909X) → Used for inter-broker communication.
  • External (909X) → Used for client connections.
• Service Type: ClusterIP (internal access only).

Step #4:Deploy Kafka Monitoring UI on Kubernetes

Kafka UI is a web interface to manage and monitor Kafka clusters.

kafka-ui.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-monitoring-ui
  namespace: kafka
  labels:
    app: kafka-ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-ui
  template:
    metadata:
      labels:
        app: kafka-ui
    spec:
      containers:
      - name: kafka-ui
        image: provectuslabs/kafka-ui:latest
        ports:
        - containerPort: 8080
        env:
        - name: KAFKA_CLUSTERS_0_NAME
          value: "local"
        - name: KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
          value: "kafka-broker-1:29092,kafka-broker-2:29093,kafka-broker-3:29094"
        - name: KAFKA_CLUSTERS_0_ZOOKEEPER
          value: "zookeeper-1:2181"
        - name: AUTH_TYPE
          value: "LOGIN_FORM"
        - name: SPRING_SECURITY_USER_NAME
          value: "<Enter username what you want ?>"
        - name: SPRING_SECURITY_USER_PASSWORD
          value: "<Enter password what you want ?>"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-ui
  namespace: kafka
spec:
  ports:
  - port: 8082
    targetPort: 8080
  selector:
    app: kafka-ui
  type: LoadBalancer

Give the Username and Password in above YAML

Apply:

kubectl apply -f kafka-ui.yaml

Description:

• Image: provectuslabs/kafka-ui:latest
• Namespace: kafka
• Exposed on Port 8080 (mapped to 8082 via Service).
• Connected to Kafka Brokers (kafka-broker-1, kafka-broker-2, kafka-broker-3) via internal ports (2909X).
• Connected to Zookeeper (zookeeper-1:2181).
• Authentication Enabled (LOGIN_FORM) with configurable username/password.
• Service Type: LoadBalancer for external access.

Verify the Deployments:

kubectl get pods -n kafka

Expose Kafka UI using port forwarding:

kubectl port-forward --address 0.0.0.0 svc/kafka-ui 8082:8082 -n kafka

Now you can access Kafka UI at http://public-ip:8082:

Give the Username and Password which you gave in the kafka-ui.yaml

Step #5:Initialize Kafka Topics in Kubernetes

Create an initialization job to preconfigure Kafka topics.

init-kafka.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: init-kafka
  namespace: kafka
spec:
  template:
    spec:
      containers:
      - name: init-kafka
        image: confluentinc/cp-kafka:7.3.2
        command:
        - /bin/sh
        - -c
        - |
          kafka-topics --bootstrap-server kafka-broker-1:29092 --list && \
          kafka-topics --bootstrap-server kafka-broker-1:29092 --create --if-not-exists --topic test-event --replication-factor 3 --partitions 3 && \
          kafka-topics --bootstrap-server kafka-broker-1:29092 --list
      restartPolicy: OnFailure

Apply:

kubectl apply -f init-kafka.yaml

Description:

• Image: confluentinc/cp-kafka:7.3.2
• Namespace: kafka
• One-time execution (Job runs once and exits).
• Bootstrap Server: kafka-broker-1:29092
• Operations Performed:
  1. Lists existing Kafka topics.
  2. Creates test-event topic (if it doesn’t exist) with:
     • Replication Factor: 3
     • Partitions: 3
  3. Lists topics again for verification.
• Restart Policy: OnFailure (Retries on failure).

Check the Topics:

Check the Brokers:

Conclusion:

Deploying a Kafka cluster on Kubernetes enhances its scalability and resilience. By setting up Zookeeper, Kafka brokers, Kafka UI, and initializing topics, you can efficiently manage real-time data streaming applications. Further improvements, such as Helm charts, StatefulSets, and persistent storage, can help ensure a production-ready deployment.

Related Articles:

Monitoring Apache Kafka Using Prometheus and Grafana

Reference:

Why run Apache Kafka on Kubernetes?