In this article, We are going to perform How to Install Apache Kafka on Ubuntu 20.04 LTS or any Cloud Instance like Amazon EC2, Azure VM, Google Compute Engine,etc. with preinstalled Ubuntu.
Table of Contents
Introduction
Kafka is most popular open source software which provides a framework for storing, reading and analyzing steaming data.To know more about Kafka visit official documentation page.
Prerequisites
- Ubuntu 20.04 with Minimal Installation
- SSH Access with Sudo previleges
- Firewall Ports: 2181,9092
- JDK 1.8 or higher version
Install JDK on Ubuntu
Please follow below article to download and install Oracle JAVA 8 on Ubuntu 20.04 LTS Manually.
How to Install Oracle JAVA 8 on Ubuntu 20.04 LTS
OR
you can install OpenJDK 8 or OpenJDK 11
$ sudo apt-get update
$ sudo apt install openjdk-8-jdk
$ sudo apt install openjdk-11-jdk
$ java -version
java version "1.8.0_251" Java(TM) SE Runtime Environment (build 1.8.0_251-b07) Java HotSpot(TM) 64-Bit Server VM (build 25.251-b07, mixed mode)
Step 1: How to Install Apache Kafka on Ubuntu 20.04 LTS
To download the binary from offical website. Please use this link to download and to prompts to download page.
$ sudo wget https://downloads.apache.org/kafka/2.4.1/kafka_2.13-2.4.1.tgz
Now to un-tar the archive file and move to another location:
$ sudo tar xzf kafka_2.13-2.4.1.tgz
$ sudo mv kafka_2.13-2.4.1 /opt/kafka
Step 2: Creating zookeeper and Kafka Systemd Unit Files
Create the systemd unit file for zookeeper service
$ sudo nano /etc/systemd/system/zookeeper.service
paste the below lines
[Unit] Description=Apache Zookeeper service Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Reload the daemon to take effect
$ sudo systemctl daemon-reload
Create the systemd unit file for kafka service
$ sudo nano /etc/systemd/system/kafka.service
paste the below lines
[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/opt/jdk/jdk1.8.0_251"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Note: Modify the value of JAVA_HOME value,If the path of your Java installation is different.
Reload the daemon to take effect
$ sudo systemctl daemon-reload
Step 3: Start ZooKeeper and Kafka Service
Lets start zookeeper service first
$ sudo systemctl start zookeeper
Start the kafka service
$ sudo systemctl start kafka
Check the status of zookeeper service if it started
$ sudo systemctl status zookeeper
Output:
● zookeeper.service - Apache Zookeeper service Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: enabled) Active: active (running) since Sat 2020-06-11 09:52:40 UTC; 22min ago Docs: http://zookeeper.apache.org Main PID: 25638 (java) Tasks: 30 (limit: 9513) CGroup: /system.slice/zookeeper.service └─25638 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true
Check the status of kafka service if it started
$ sudo systemctl status kafka
Output:
● kafka.service - Apache Kafka Service Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled) Active: active (running) since Sat 2020-06-11 09:52:48 UTC; 24min ago Docs: http://kafka.apache.org/documentation.html Main PID: 26027 (java) Tasks: 66 (limit: 9513) CGroup: /system.slice/kafka.service └─26027 /opt/jdk/jdk1.8.0_241/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -D
To Start Kafka And ZooKeeper Server in the background(without creating systemd unit file)
We have shell script to run the kafka and zookeeper server in backend:
Create a file named kafkastart.sh and copy the below script:
#!/bin/bash sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties > /dev/null 2>&1 & sleep 5 sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties > /dev/null 2>&1 &
After give the executable permissions to the file:
sudo chmod +x kafkastart.sh
Successfully We have covered how to install apache kafka on ubuntu 20.04 LTS.
Step 4: Creating Topic in Kafka
Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:
$ cd /opt/kafka
$ sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic DevOps
Output:
Created topic "DevOps".
To check the list of topics created.
$ sudo bin/kafka-topics.sh --list --zookeeper localhost:2181
Output:
DevOps
Step 5: To send some messages
To send some messages for created Topic.
$ sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic DevOps
its prompt for messages to type:
> Hi > How are you?
Step 6: To Start consumer
Using below command we can see the list of messages:
$ sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic DevOps --from-beginning > Hi > How are you?
Step 7: To Connect Kafka from remote machine
To connect, create Topic and send messages from remote server. Please follow below steps.
Go to the below path:
cd /opt/kafka/config
Now look for server.properties and make some configuration changes:
sudo vi server.properties
In this properties file uncomment as mentioned below:
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<HOST IP>:9092
Step 8: To Delete any Topic
If you want to delete any created topic use below command:
$ sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>
Conclusion:
In this article, We have performed ,How to Install Apache Kafka on Ubuntu 20.04 LTS and also covered creating systemd unit file for zookeeper and kafka service, start zookeeper, start kafka, create a topic, delete topic, start kafka and zookeeper service.
