In this article, We are going to perform How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS.

Introduction

Kafka is most popular open source software which provides a framework for storing, reading and analyzing steaming data.To know more about Kafka visit official documentation page.

Prerequisites

• Ubuntu 18.04/16.04 with Minimal Installation
• SSH Access with Sudo previleges
• Firewall Ports: 2181,9092
• JDK 1.8 or higher version

Install JDK on Ubuntu

Please follow below article to download and install Oracle JAVA 8 on Ubuntu 18.04/16.04 LTS Manually.

Download and Install Oracle Java 8 on Ubuntu 18.04/16.04 LTS

OR

you can install OpenJDK 8

$ sudo apt-get update

$ sudo apt install openjdk-8-jdk

$ java -version

java version "1.8.0_241"

Java(TM) SE Runtime Environment (build 1.8.0_241-b07)

Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

Step 1: Download  and Install Apache Kafka on Ubuntu

To download the binary from offical website. Please use this link to download and to prompts to download page.

$ sudo wget https://downloads.apache.org/kafka/2.4.1/kafka_2.13-2.4.1.tgz

Now to un-tar the archive file and move to another location:

$ sudo tar xzf kafka_2.13-2.4.1.tgz

$ sudo mv kafka_2.13-2.4.1 /opt/kafka

Step 2: Creating zookeeper and Kafka Systemd Unit Files

Create the systemd unit file for zookeeper service

$ sudo nano  /etc/systemd/system/zookeeper.service

paste the below lines

[Unit]
Description=Apache Zookeeper service
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Reload the daemon to take effect

$ sudo systemctl daemon-reload

Create the systemd unit file for kafka service

$ sudo nano /etc/systemd/system/kafka.service

paste the below lines

[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/opt/jdk/jdk1.8.0_241"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Note: Modify the value of JAVA_HOME value,If the path of your Java installation is different.

Reload the daemon to take effect

$ sudo systemctl daemon-reload

Step 3: Start ZooKeeper and Kafka Service

Lets start zookeeper service first

$ sudo systemctl start zookeeper

Start the kafka service

$ sudo systemctl start kafka

Check the status of  zookeeper service if it started

$ sudo systemctl status zookeeper

Output:

● zookeeper.service - Apache Zookeeper service
Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2020-04-11 09:52:40 UTC; 22min ago
Docs: http://zookeeper.apache.org
Main PID: 25638 (java)
Tasks: 30 (limit: 9513)
CGroup: /system.slice/zookeeper.service
└─25638 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true

Check the status of  kafka service if it started

$ sudo systemctl status kafka

Output:

● kafka.service - Apache Kafka Service
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2020-04-11 09:52:48 UTC; 24min ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 26027 (java)
Tasks: 66 (limit: 9513)
CGroup: /system.slice/kafka.service
└─26027 /opt/jdk/jdk1.8.0_241/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -D

To Start Kafka And ZooKeeper Server in the background(without creating systemd unit file)

We have shell script to run the kafka and zookeeper server in backend:

Create a file named kafkastart.sh and copy the below script:

#!/bin/bash
sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties > /dev/null 2>&1 &
sleep 5
sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties > /dev/null 2>&1 &

After give the executable permissions to the file:

sudo chmod +x kafkastart.sh

Successfully We have covered install apache kafka on ubuntu.

Step 4: Creating  Topic in Kafka

Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:

$ cd /opt/kafka

$ sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic DevOps

Output:

Created topic "DevOps".

To check the list of topics created.

$ sudo bin/kafka-topics.sh --list --zookeeper localhost:2181

Output:

DevOps

Step 5: To send some messages

To send some messages for created Topic.

$ sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic DevOps

its prompt for messages to type:

> Hi
> How are you?

Step 6: To Start consumer

Using below command we can see the list of messages:

$ sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic DevOps --from-beginning

> Hi
> How are you?

Step 7: To Connect Kafka from remote machine

To connect, create Topic and send messages from remote server. Please follow below steps.

Go to the below path:

cd /opt/kafka/config

Now look for server.properties and make some configuration changes:

sudo vi server.properties

In this properties file uncomment as mentioned  below:

listeners=PLAINTEXT://:9092

advertised.listeners=PLAINTEXT://<HOST IP>:9092

Step 8: To Delete any Topic

If you want to delete any created  topic use below command:

$ sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>

Conclusion:

In this article, We have performed ,How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS and also covered creating systemd unit file for zookeeper and kafka service, start zookeeper, start kafka, create a topic, delete topic, start kafka and zookeeper service.

Related Articles:

How to Install Apache Kafka on Ubuntu 20.04 LTS

How to Install MariaDB on Ubuntu 18.04/16.04 LTS