How to Install Apache Kafka on CentOS 8

In this article, We are going to perform How to Install Apache Kafka on CentOS 8 or any other cloud platform like Amazon EC2, Azure VM, Google Compute Cloud, etc., with preinstalled CentOS 8

Introduction

Kafka is most popular open source software which provides a framework for storing, reading and analyzing steaming data.To know more about Kafka visit official documentation page.

Prerequisites

  • CentOS 8 with Minimal Installation
  • SSH Access with Sudo previleges
  • Firewall Ports: 2181,9092
  • JDK 1.8 or higher version

Install JDK on Ubuntu

Please follow below article to download and install Oracle JAVA 11 on CentOS 8 LTS Manually.

How to Download and Install Oracle Java 11 on CentOS 8

OR

you can install OpenJDK 11

$ sudo yum update
$ sudo dnf install java-11-openjdk wget vim

To check the java version, Here i have install installed Oracle Java 8 on my system.

$ java -version

Output:

java version "11.0.6" 2020-01-14 LTS

Java(TM) SE Runtime Environment 18.9 (build 11.0.6+8-LTS)

Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.6+8-LTS, mixed mode)

Step 1: Download and Install Apache Kafka on CentOS 8

To download the binary from offical website. Please use this link to download and to prompts to download page.

$ sudo wget https://downloads.apache.org/kafka/2.4.1/kafka_2.13-2.4.1.tgz

Now to un-tar the archive file and move to another location:

$ sudo tar xzf kafka_2.13-2.4.1.tgz

Move apache kafka setup to /opt/kafka directory

$ sudo mv kafka_2.13-2.4.1 /opt/kafka

Step 2: Creating zookeeper and Kafka Systemd Unit Files

Create the systemd unit file for zookeeper service

$ sudo nano  /etc/systemd/system/zookeeper.service

paste the below lines

/etc/systemd/system/zookeeper.service
[Unit]

Description=Apache Zookeeper service

Documentation=http://zookeeper.apache.org

Requires=network.target remote-fs.target

After=network.target remote-fs.target

[Service]

Type=simple

ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties

ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh

Restart=on-abnormal

[Install]

WantedBy=multi-user.target

Reload the daemon to take effect

$ sudo systemctl daemon-reload

Create the systemd unit file for kafka service

$ sudo nano /etc/systemd/system/kafka.service

paste the below lines

/etc/systemd/system/kafka.service
[Unit]

Description=Apache Kafka Service

Documentation=http://kafka.apache.org/documentation.html

Requires=zookeeper.service

[Service]

Type=simple

Environment="JAVA_HOME=/opt/jdk/jdk-11.0.6"

ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]

WantedBy=multi-user.target

Note: Modify the value of JAVA_HOME value,If the path of your Java installation is different.

Reload the daemon to take effect

$ sudo systemctl daemon-reload

Step 3: Start ZooKeeper and Kafka Service

Lets start zookeeper service first

$ sudo systemctl start zookeeper

Start the kafka service

$ sudo systemctl start kafka

Check the status of  zookeeper service if it started

$ sudo systemctl status zookeeper

 

Check the status of  kafka service if it started

$ sudo systemctl status kafka

 

To Start Kafka And ZooKeeper Server in the background(without creating systemd unit file)

We have shell script to run the kafka and zookeeper server in backend:

Create a file named kafkastart.sh and copy the below script:

#!/bin/bash

sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties > /dev/null 2>&1 &

sleep 5

sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties > /dev/null 2>&1 &

After give the executable permissions to the file:

$ sudo chmod +x kafkastart.sh

Successfully We have covered install apache kafka on CentOS 8.

Step 4: Creating  Topic in Kafka

Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:

$ cd /opt/kafka
$ sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic DevOps

Output:

Created topic “DevOps”.

To check the list of topics created.

$ sudo bin/kafka-topics.sh --list --zookeeper localhost:2181

Output:

DevOps

Step 5: To send some messages

To send some messages for created Topic.

$ sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic DevOps

its prompt for messages to type:

> Hi

> How are you?

Step 6: To Start consumer

Using below command we can see the list of messages:

$ sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic DevOps --from-beginning

 

> Hi

> How are you?

Step 7: To Connect Kafka from remote machine

To connect, create Topic and send messages from remote server. Please follow below steps.

Go to the below path:

$ cd /opt/kafka/config

Now look for server.properties and make some configuration changes:

$ sudo vi server.properties

In this properties file uncomment as mentioned  below:

listeners=PLAINTEXT://:9092

advertised.listeners=PLAINTEXT://<HOST IP>:9092
Step 8: To Delete any Topic

If you want to delete any created  topic use below command:

$ sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>

Conclusion:

In this article, We have performed ,How to Install Apache Kafka on CentOS 8 and also covered creating systemd unit file for zookeeper and kafka service, start zookeeper, start kafka, create a topic, delete topic, start kafka and zookeeper service.

Related Articles:

How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS

Shivdas Kanade

I am Shivdas Kanade working as DevOps Engineer. Passionate about Linux, Open Source and DevOps Tools. Believe in sharing knowledge.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Shares
Share via
Copy link