In this article We are going to perform How to Install Apache Kafka on Ubuntu 22.04 LTS and also covered creating systemd unit file for zookeeper and Kafka service, create a topic in Kafka,send messages in Kafka, Start consumer in Kafka.
Table of Contents
What is Kafka ?
Kafka is an open source software which provides a framework for storing, reading and analysing streaming data.
Being open source means that it is essentially free to use and has a large network of users and developers who contribute towards updates, new features and offering support for new users.
Kafka is designed to be run in a “distributed” environment, which means that rather than sitting on one user’s computer, it runs across several (or many) servers, leveraging the additional processing power and storage capacity that this brings.
Why do you need Kafka it depends on your use case
Some use cases I can think of :
- you can have a streaming app that captures logs of all the apps in a company and all can be redirected to common logging space.
- Real-time transactions you can leverage kafka.
- You can build set of micro services that perform complex business logic to communicate through Kafka .
You can easily integrate kafka to various big data processing frameworks like apache spark , Apache storm seamlessly.
You can know more about it here : Apache Kafka
Prerequisites
- Create Ubuntu Instance 22.04 LTS
- SSH Access with Sudo privileges
- Firewall Port 9092
- JDK 1.8 or higher version
Install OpenJDK on Ubuntu 22.04 LTS
To update system packages
sudo apt-get update
you can install OpenJDK 8 or OpenJDK 11
sudo apt install openjdk-8-jdk
or
sudo apt install openjdk-11-jdk
To check the java version, Here I have install installed Oracle Java 11 on my system.
java -version
Output:
Step #1:Install Apache Kafka on Ubuntu 22.04 LTS
To download the Kafka binary from offical website. Please use this Kafka official download page and to prompts to download page and you can download Kafka using wget
sudo wget https://downloads.apache.org/kafka/3.5.0/kafka_2.12-3.5.0.tgzNow to un-tar or Unzip the archive file and move to another location:
sudo tar xzf kafka_2.12-3.5.0.tgz
sudo mv kafka_2.12-3.5.0 /opt/kafka
Step #2:Creating Zookeeper and Kafka Systemd Unit Files in Ubuntu 22.04 LTS
Create the systemd unit file for zookeeper service
sudo nano /etc/systemd/system/zookeeper.service
paste the below lines
/etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper service
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Reload the daemon to take effect
sudo systemctl daemon-reload
Create the systemd unit file for kafka service
sudo nano /etc/systemd/system/kafka.service
paste the below lines
[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Note: Modify the value of JAVA_HOME value, If the path of your Java installation is different path like If your java version is 1.8 the use Environment=”JAVA_HOME=/opt/jdk/jdk1.8.0_251″ this path.
Reload the daemon to take effect
sudo systemctl daemon-reload
Step #3:To Start ZooKeeper and Kafka Service and Check its Status
Lets start zookeeper service first
sudo systemctl start zookeeper
Check the status of zookeeper service if it started
sudo systemctl status zookeeper
Output:
Start the kafka service
sudo systemctl start kafka
Check the status of kafka service if it started
sudo systemctl status kafka
Output:
To Start Kafka And ZooKeeper Server in the background(without creating systemd unit file)
We have shell script to run the kafka and zookeeper server in backend:
Create a file named kafkastart.sh and copy the below script:
#!/bin/bash
sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties > /dev/null 2>&1 &
sleep 5
sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties > /dev/null 2>&1 &After give the executable permissions to the file:
sudo chmod +x kafkastart.sh
Successfully We have covered how to install apache kafka on ubuntu 22.04 LTS.
Step #4:Creating Topic in Kafka
Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:
cd /opt/kafka
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic DevopsHint
Output:
Explanation of command
- –create :- It is used for create a new topic
- –replication-factor :- It is used for how many copies of data will be created.
- –partitions :- It is used for set the partitions options as the number of brokers you want your data to be split between.
- –topic :- It is used for name of the topic
To check the list of topics created.
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
Output:
Step #5:To send some messages using Kafka
To send some messages for created Topic.
sudo bin/kafka-console-producer.sh —broker-list localhost:9092 —topic DevopsHint
its prompt for messages to type:
> hello world! > How are you? > This is DevopsHint > Bye…
Step #6:To Start consumer in Kafka
Using below command we can see the list of messages:
sudo bin/kafka-console-consumer.sh ---bootstrap-server localhost:9092 ---topic DevopsHint ---from-beginningOutput
Step #7:To Connect Kafka from remote machine
To connect, create Topic and send messages from remote server. Please follow below steps.
Go to the below path:
cd /opt/kafka/config
Now look for server.properties and make some configuration changes:
sudo vi server.properties
In this properties file uncomment as mentioned below:
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<HOST IP>:9092Step #8:To Delete any Topics in Kafka
If you want to delete any created topic use below command:
sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>
We have covered How to Install Apache Kafka on Ubuntu 22.04 LTS.
Conclusion:
In this article, We have covered ,How to Install Apache Kafka on Ubuntu 22.04 LTS and also covered creating systemd unit file for zookeeper and kafka service, start zookeeper, start kafka, create a topic,To send some messages, To start consumer.
Related Articles:
