Install Apache Kafka on Ubuntu 18.04/16.04 LTS

In this article, We are going to perform How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS.

Introduction

Kafka is most popular open source software which provides a framework for storing, reading and analyzing steaming data.To know more about Kafka visit official documentation page.

Prerequisites

  • Ubuntu 18.04/16.04 with Minimal Installation
  • SSH Access with Sudo previleges
  • Firewall Ports: 2181,9092
  • JDK 1.8 or higher version

Install JDK on Ubuntu

Please follow below article to download and install Oracle JAVA 8 on Ubuntu 18.04/16.04 LTS Manually.

Download and Install Oracle Java 8 on Ubuntu 18.04/16.04 LTS

$ sudo apt-get update

Step 1: Download Apache Kafka

To download the binary from offical website. Please use this link to download and to prompts to download page.

$ sudo wget https://www-us.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz

Now to un-tar the archive file and move to another location:

$ sudo tar -xzf kafka_2.12-2.3.0.tgz
$ sudo mv kafka_2.12-2.3.0 /opt/kafka

Step 2: To Start ZooKeeper

To start kafka we need to run zookeeper first. Apache Kafka has provided a package within to run zookeeper server.

Note : The whole article default folder to execute command’s will be “/opt/kafka/”

$ cd /opt/kafka/
$ sudo bin/zookeeper-server-start.sh config/zookeeper.properties

Output:

[2019-12-04 11:25:38,202] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-12-04 11:25:38,214] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-04 11:25:38,214] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-04 11:25:38,215] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-04 11:25:38,215] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2019-12-04 11:25:38,271] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
.
.
.
.
.
[2019-12-04 11:27:59,934] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-04 11:27:59,958] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2019-12-04 11:27:59,963] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

Now we have to start kafka server.

Step 3: To Start Kafka

$ sudo bin/kafka-server-start.sh config/server.properties

Output:

[2019-12-04 11:31:55,277] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-12-04 11:31:56,543] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2019-12-04 11:31:56,544] INFO starting (kafka.server.KafkaServer)
[2019-12-04 11:31:56,545] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2019-12-04 11:31:56,616] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
.
.
.
.
.
[2019-12-04 11:31:56,771] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-12-04 11:31:56,803] INFO [SocketServer brokerId=0] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
[2019-12-04 11:31:56,829] INFO Kafka version: 2.2.1 (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-04 11:31:56,829] INFO Kafka commitId: 55783d3133a5a49a (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-04 11:31:56,831] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Step 4: To Create a Topic

Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:

$ cd /opt/kafka
$ sudo bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic DevOps
Output:
Created topic "DevOps".

To check the list of topics created.

$ sudo bin/kafka-topics.sh --list --bootstrap-server localhost:9092
Output:
DevOps

Step 5: To send some messages

To send some messages for created Topic.

$ sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic DevOps

its prompt for messages to type:

> Hi
> How are you?

Step 6: To Start consumer

Using below command we can see the list of messages:

$ sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic DevOps --from-beginning

> Hi
> How are you?

Step 7: To Start Kafka And ZooKeeper Server

We have shell script to run the kafka and zookeeper server in backend:

Create a file named kafkastart.sh and copy the below script:

#!/bin/bash
sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/kafka/config/zookeeper.properties > /dev/null 2>&1 &
sleep 5
sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties > /dev/null 2>&1 &

After give the executable permissions to the file:

sudo chmod +x kafkastart.sh

Step 8: To Connect Kafka from remote machine

To connect, create Topic and send messages from remote server. Please follow below steps.

Go to the below path:

cd /opt/kafka/config

Now look for server.properties and make some configuration changes:

sudo vi server.properties

In this properties file uncomment as mentioned  below:

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<HOST IP>:9092

Step 9: To Delete any Topic

If you want to delete any created  topic use below command:

$ sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>

Conclusion:

In this article, We have performed ,How to Install Apache Kafka on Ubuntu 18.04/16.04 LTS and also covered to start zookeeper, start kafka, create a topic, delete topic, start kafka and zookeeper service.

Related Articles:

How to Install MariaDB on Ubuntu 18.04/16.04 LTS

Sivasai Sagar

I am working as DevOps Engineer and having 5 years of Experience. Likes to share knowledge.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Shares
Share via
Copy link