How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS

In this article, We are going to perform How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS.

Introduction

Kafka is most popular open source software which provides a framework for storing, reading and analyzing steaming data.To know more about Kafka visit official documentation page.

Prerequisites

  • Ubuntu 18.04/16.04 with Minimal Installation
  • SSH Access with Sudo previleges
  • Firewall Ports: 2181,9092
  • JDK 1.8 or higher version

Install JDK on Ubuntu

Please follow below article to download and install Oracle JAVA 8 on Ubuntu 18.04/16.04 LTS Manually.

Download and Install Oracle Java 8 on Ubuntu 18.04/16.04 LTS

OR

you can install OpenJDK 8

$ sudo apt-get update
$ sudo apt install openjdk-8-jdk
To check the java version, Here i have install installed Oracle Java 8 on my system.
$ java -version
Output:
java version "1.8.0_241"

Java(TM) SE Runtime Environment (build 1.8.0_241-b07)

Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

Step 1: Download  and Install Apache Kafka on Ubuntu

To download the binary from offical website. Please use this link to download and to prompts to download page.

$ sudo wget https://downloads.apache.org/kafka/2.4.1/kafka_2.13-2.4.1.tgz

Now to un-tar the archive file and move to another location:

$ sudo tar xzf kafka_2.13-2.4.1.tgz
$ sudo mv kafka_2.13-2.4.1 /opt/kafka

Step 2: Creating zookeeper and Kafka Systemd Unit Files

Create the systemd unit file for zookeeper service

$ sudo nano  /etc/systemd/system/zookeeper.service

paste the below lines

/etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper service
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Reload the daemon to take effect

$ sudo systemctl daemon-reload

Create the systemd unit file for kafka service

$ sudo nano /etc/systemd/system/kafka.service

paste the below lines

/etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/opt/jdk/jdk1.8.0_241"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Note: Modify the value of JAVA_HOME value,If the path of your Java installation is different.

Reload the daemon to take effect

$ sudo systemctl daemon-reload

Step 3: Start ZooKeeper and Kafka Service

Lets start zookeeper service first

$ sudo systemctl start zookeeper

Start the kafka service

$ sudo systemctl start kafka

Check the status of  zookeeper service if it started

$ sudo systemctl status zookeeper

Output:

● zookeeper.service - Apache Zookeeper service
Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2020-04-11 09:52:40 UTC; 22min ago
Docs: http://zookeeper.apache.org
Main PID: 25638 (java)
Tasks: 30 (limit: 9513)
CGroup: /system.slice/zookeeper.service
└─25638 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true

Check the status of  kafka service if it started

$ sudo systemctl status kafka

Output:

● kafka.service - Apache Kafka Service
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2020-04-11 09:52:48 UTC; 24min ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 26027 (java)
Tasks: 66 (limit: 9513)
CGroup: /system.slice/kafka.service
└─26027 /opt/jdk/jdk1.8.0_241/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -D

To Start Kafka And ZooKeeper Server in the background(without creating systemd unit file)

We have shell script to run the kafka and zookeeper server in backend:

Create a file named kafkastart.sh and copy the below script:

#!/bin/bash
sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties > /dev/null 2>&1 &
sleep 5
sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties > /dev/null 2>&1 &

After give the executable permissions to the file:

sudo chmod +x kafkastart.sh

Successfully We have covered install apache kafka on ubuntu.

Step 4: Creating  Topic in Kafka

Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:

$ cd /opt/kafka
$ sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic DevOps
Output:
Created topic "DevOps".

To check the list of topics created.

$ sudo bin/kafka-topics.sh --list --zookeeper localhost:2181
Output:
DevOps

Step 5: To send some messages

To send some messages for created Topic.

$ sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic DevOps

its prompt for messages to type:

> Hi
> How are you?

Step 6: To Start consumer

Using below command we can see the list of messages:

$ sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic DevOps --from-beginning

> Hi
> How are you?

Step 7: To Connect Kafka from remote machine

To connect, create Topic and send messages from remote server. Please follow below steps.

Go to the below path:

cd /opt/kafka/config

Now look for server.properties and make some configuration changes:

sudo vi server.properties

In this properties file uncomment as mentioned  below:

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<HOST IP>:9092

Step 8: To Delete any Topic

If you want to delete any created  topic use below command:

$ sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>

Conclusion:

In this article, We have performed ,How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS and also covered creating systemd unit file for zookeeper and kafka service, start zookeeper, start kafka, create a topic, delete topic, start kafka and zookeeper service.

Related Articles:

How to Install Apache Kafka on Ubuntu 20.04 LTS

How to Install MariaDB on Ubuntu 18.04/16.04 LTS

Sivasai Sagar

I am working as DevOps Engineer and having 5 years of Experience. Likes to share knowledge.

6 thoughts on “How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS”

  1. Hi Sagar,

    Thanks for the article. The article has no flaw at all. The same is exactly worked with Ubuntu20.04 too…

    Thanks for your article..

    Reply
  2. Good and easy to follow steps.

    kafkastart.sh was handy.
    Would be helpful you could also give a batch file to stop the
    services something like the “kafkastop.sh” so that the sequence
    be followed to stop the Zookeeper and then Kafka.

    Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Share via
Copy link
Powered by Social Snap