In this article, We are going to perform How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS.
Table of Contents
Introduction
Kafka is most popular open source software which provides a framework for storing, reading and analyzing steaming data.To know more about Kafka visit official documentation page.
Prerequisites
- Ubuntu 18.04/16.04 with Minimal Installation
- SSH Access with Sudo previleges
- Firewall Ports: 2181,9092
- JDK 1.8 or higher version
Install JDK on Ubuntu
Please follow below article to download and install Oracle JAVA 8 on Ubuntu 18.04/16.04 LTS Manually.
Download and Install Oracle Java 8 on Ubuntu 18.04/16.04 LTS
OR
you can install OpenJDK 8
$ sudo apt-get update
$ sudo apt install openjdk-8-jdk
$ java -version
java version "1.8.0_241" Java(TM) SE Runtime Environment (build 1.8.0_241-b07) Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)
Step 1: Download and Install Apache Kafka on Ubuntu
To download the binary from offical website. Please use this link to download and to prompts to download page.
$ sudo wget https://downloads.apache.org/kafka/2.4.1/kafka_2.13-2.4.1.tgz
Now to un-tar the archive file and move to another location:
$ sudo tar xzf kafka_2.13-2.4.1.tgz
$ sudo mv kafka_2.13-2.4.1 /opt/kafka
Step 2: Creating zookeeper and Kafka Systemd Unit Files
Create the systemd unit file for zookeeper service
$ sudo nano /etc/systemd/system/zookeeper.service
paste the below lines
[Unit] Description=Apache Zookeeper service Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Reload the daemon to take effect
$ sudo systemctl daemon-reload
Create the systemd unit file for kafka service
$ sudo nano /etc/systemd/system/kafka.service
paste the below lines
[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/opt/jdk/jdk1.8.0_241"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Note: Modify the value of JAVA_HOME value,If the path of your Java installation is different.
Reload the daemon to take effect
$ sudo systemctl daemon-reload
Step 3: Start ZooKeeper and Kafka Service
Lets start zookeeper service first
$ sudo systemctl start zookeeper
Start the kafka service
$ sudo systemctl start kafka
Check the status of zookeeper service if it started
$ sudo systemctl status zookeeper
Output:
● zookeeper.service - Apache Zookeeper service
Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2020-04-11 09:52:40 UTC; 22min ago
Docs: http://zookeeper.apache.org
Main PID: 25638 (java)
Tasks: 30 (limit: 9513)
CGroup: /system.slice/zookeeper.service
└─25638 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true
Check the status of kafka service if it started
$ sudo systemctl status kafka
Output:
● kafka.service - Apache Kafka Service
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2020-04-11 09:52:48 UTC; 24min ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 26027 (java)
Tasks: 66 (limit: 9513)
CGroup: /system.slice/kafka.service
└─26027 /opt/jdk/jdk1.8.0_241/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -D
To Start Kafka And ZooKeeper Server in the background(without creating systemd unit file)
We have shell script to run the kafka and zookeeper server in backend:
Create a file named kafkastart.sh and copy the below script:
#!/bin/bash sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties > /dev/null 2>&1 & sleep 5 sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties > /dev/null 2>&1 &
After give the executable permissions to the file:
sudo chmod +x kafkastart.sh
Successfully We have covered install apache kafka on ubuntu.
Step 4: Creating Topic in Kafka
Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:
$ cd /opt/kafka
$ sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic DevOps
Output:
Created topic "DevOps".
To check the list of topics created.
$ sudo bin/kafka-topics.sh --list --zookeeper localhost:2181
Output:
DevOps
Step 5: To send some messages
To send some messages for created Topic.
$ sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic DevOps
its prompt for messages to type:
> Hi > How are you?
Step 6: To Start consumer
Using below command we can see the list of messages:
$ sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic DevOps --from-beginning > Hi > How are you?
Step 7: To Connect Kafka from remote machine
To connect, create Topic and send messages from remote server. Please follow below steps.
Go to the below path:
cd /opt/kafka/config
Now look for server.properties and make some configuration changes:
sudo vi server.properties
In this properties file uncomment as mentioned below:
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<HOST IP>:9092
Step 8: To Delete any Topic
If you want to delete any created topic use below command:
$ sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>
Conclusion:
In this article, We have performed ,How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS and also covered creating systemd unit file for zookeeper and kafka service, start zookeeper, start kafka, create a topic, delete topic, start kafka and zookeeper service.
Hi Sagar,
Thanks for the article. The article has no flaw at all. The same is exactly worked with Ubuntu20.04 too…
Thanks for your article..
Thanks for your reply.
Good and easy to follow steps.
kafkastart.sh was handy.
Would be helpful you could also give a batch file to stop the
services something like the “kafkastop.sh” so that the sequence
be followed to stop the Zookeeper and then Kafka.
Thanks for your suggestion, will add it.
Tq, this is really helpful.. –zookeeper is deprecated but still can be used.
thanks