In this article, We are going to perform How to Install Apache Kafka on CentOS 8 or any other cloud platform like Amazon EC2, Azure VM, Google Compute Cloud, etc., with preinstalled CentOS 8
Table of Contents
Introduction
Kafka is most popular open source software which provides a framework for storing, reading and analyzing steaming data.To know more about Kafka visit official documentation page.
Prerequisites
- CentOS 8 with Minimal Installation
- SSH Access with Sudo previleges
- Firewall Ports: 2181,9092
- JDK 1.8 or higher version
Install JDK on Ubuntu
Please follow below article to download and install Oracle JAVA 11 on CentOS 8 LTS Manually.
How to Download and Install Oracle Java 11 on CentOS 8
OR
you can install OpenJDK 11
$ sudo yum update
$ sudo dnf install java-11-openjdk wget vim
To check the java version, Here i have install installed Oracle Java 8 on my system.
$ java -version
Output:
java version "11.0.6" 2020-01-14 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.6+8-LTS) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.6+8-LTS, mixed mode)
Step 1: Download and Install Apache Kafka on CentOS 8
To download the binary from offical website. Please use this link to download and to prompts to download page.
$ sudo wget https://downloads.apache.org/kafka/2.4.1/kafka_2.13-2.4.1.tgz
Now to un-tar the archive file and move to another location:
$ sudo tar xzf kafka_2.13-2.4.1.tgz
Move apache kafka setup to /opt/kafka directory
$ sudo mv kafka_2.13-2.4.1 /opt/kafka
Step 2: Creating zookeeper and Kafka Systemd Unit Files
Create the systemd unit file for zookeeper service
$ sudo nano /etc/systemd/system/zookeeper.service
paste the below lines
[Unit] Description=Apache Zookeeper service Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Reload the daemon to take effect
$ sudo systemctl daemon-reload
Create the systemd unit file for kafka service
$ sudo nano /etc/systemd/system/kafka.service
paste the below lines
[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/opt/jdk/jdk-11.0.6"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Note: Modify the value of JAVA_HOME value,If the path of your Java installation is different.
Reload the daemon to take effect
$ sudo systemctl daemon-reload
Step 3: Start ZooKeeper and Kafka Service
Lets start zookeeper service first
$ sudo systemctl start zookeeper
Start the kafka service
$ sudo systemctl start kafka
Check the status of zookeeper service if it started
$ sudo systemctl status zookeeper
Check the status of kafka service if it started
$ sudo systemctl status kafka
To Start Kafka And ZooKeeper Server in the background(without creating systemd unit file)
We have shell script to run the kafka and zookeeper server in backend:
Create a file named kafkastart.sh and copy the below script:
#!/bin/bash sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties > /dev/null 2>&1 & sleep 5 sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties > /dev/null 2>&1 &
After give the executable permissions to the file:
$ sudo chmod +x kafkastart.sh
Successfully We have covered install apache kafka on CentOS 8.
Step 4: Creating Topic in Kafka
Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:
$ cd /opt/kafka
$ sudo bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic DevOps
Output:
Created topic “DevOps”.
To check the list of topics created.
$ sudo bin/kafka-topics.sh --list --zookeeper localhost:2181
Output:
DevOps
Step 5: To send some messages
To send some messages for created Topic.
$ sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic DevOps
its prompt for messages to type:
> Hi > How are you?
Step 6: To Start consumer
Using below command we can see the list of messages:
$ sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic DevOps --from-beginning
> Hi > How are you?
Step 7: To Connect Kafka from remote machine
To connect, create Topic and send messages from remote server. Please follow below steps.
Go to the below path:
$ cd /opt/kafka/config
Now look for server.properties and make some configuration changes:
$ sudo vi server.properties
In this properties file uncomment as mentioned below:
listeners=PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://<HOST IP>:9092
Step 8: To Delete any Topic
If you want to delete any created topic use below command:
$ sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>
Conclusion:
In this article, We have performed ,How to Install Apache Kafka on CentOS 8 and also covered creating systemd unit file for zookeeper and kafka service, start zookeeper, start kafka, create a topic, delete topic, start kafka and zookeeper service.
