In this article we will learn How to Monitor Kafka logs using Elastic Stack. Kafka and the Elastic Stack (Elasticsearch, Logstash, Kibana, and Filebeat) form a powerful combination for real-time data processing and log analysis. Kafka acts as a message broker, enabling seamless data flow between different components, while the Elastic Stack collects, processes, stores, and visualizes logs. This guide walks you through installing and configuring these technologies on an Ubuntu system, ensuring smooth data ingestion and visualization.
Table of Contents
Prerequisites
- AWS Account with Ubuntu 24.04 LTS EC2 Instance.
- At least 2 CPU cores and 4 GB of RAM for smooth performance.
- Java and Apache installed.
Step #1:Setting Up Ubuntu EC2 Instance
Update the Package List to ensure you have the latest versions.
sudo apt update
Elasticsearch requires Java, so we need to install OpenJDK 17.
sudo apt install -y openjdk-17-jdk
Install the Apache web server.
sudo apt install -y apache2
Step #2:Install and Configure Elasticsearch
Import the Elasticsearch GPG key.
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
Add the Elasticsearch repository.
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
Now lets update the package list again. The repository is added to the system’s package sources.
sudo apt update
Install Elasticsearch.
sudo apt install -y elasticsearch
Modify Elasticsearch configuration for remote access.
sudo nano /etc/elasticsearch/elasticsearch.yml
Find the network.host setting, uncomment it, and set it to 0.0.0.0 to bind to all available IP addresses and uncomment the discovery section to specify the initial nodes for cluster formation discovery.seed_hosts: []
For a basic setup (not recommended for production), disable security features.
xpack.security.enabled: false
Save and exit the editor.
Enable and start Elasticsearch.
sudo systemctl enable elasticsearch sudo systemctl start elasticsearch
Check the status of the elasticsearch to ensure it is running.
sudo systemctl status elasticsearch
Send a GET request to check if Elasticsearch is running and responding. If successful, you should see a JSON response with cluster information.
curl -X GET "localhost:9200"
You can access it using browser with your Public IP address:9200 port which is a default port for Elasticsearch.
Step #3:Install and Configure Logstash
Logstash processes logs before sending them to Elasticsearch. Install it using following command.
sudo apt install -y logstash
Create a configuration file.
sudo nano /etc/logstash/conf.d/apache.conf
Add the following configuration to collect logs from Kafka, parse them, and send them to Elasticsearch.
input {
kafka {
bootstrap_servers => "localhost:9092"
topics => "apache"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
target => "geoip"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-apache-%{+YYYY.MM.dd}"
}
}
Enable and start Logstash
sudo systemctl enable logstash
sudo systemctl start logstash
Checks the status of Logstash.
sudo systemctl status logstash
Step #4:Install and Configure Kibana
Kibana provides visualization for Elasticsearch data. Install Kibana on the system.
sudo apt install -y kibana
Open the Kibana configuration file for editing.
sudo nano /etc/kibana/kibana.yml
Uncomment and adjust the following lines to bind Kibana to all IP addresses and connect it to Elasticsearch.
server.port: 5601 server.host: "0.0.0.0" elasticsearch.hosts: ["http://localhost:9200"]
Enable and start Kibana.
sudo systemctl enable kibana
sudo systemctl start kibana
Checks the status of Kibana.
sudo systemctl status kibana
Access the Kibana interface by navigating to http://<your-server-ip>:5601 in your web browser. Click on Explore on my own.
This will open the Kibana dashboard where you can start exploring your data.
Step #5:Install and Configure Filebeat
Filebeat collects and forwards log data to Elasticsearch or Logstash. Install Filebeat on the system.
sudo apt install -y filebeat
Enable the Apache module in Filebeat.
sudo filebeat modules enable apache
Configure the Apache module.
sudo nano /etc/filebeat/modules.d/apache.yml
Ensure the following configuration is enabled to send Apache logs.
- module: apache
access:
enabled: true
var.paths: ["/var/log/apache2/access.log*"]
error:
enabled: true
var.paths: ["/var/log/apache2/error.log*"]
Save and exit the file.
Edit the Filebeat configuration file to ship logs to Kafka.
sudo nano /etc/filebeat/filebeat.yml
Change the type to log and change enable to true from false also comment out the id: my-filestream-id and give the path to our custome log file.
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/apache2/access.log
Comment out the Elasticsearch output section. and write the kafka output section as shown below.
output.kafka:
codec.format:
string: '%{[@timestamp]} %{[message]}'
hosts: ["localhost:9092"]
topic: apache
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
Test the configuration.
sudo filebeat test config
Start and enable the Filebeat service.
sudo systemctl enable filebeat
sudo systemctl start filebeat
Checks the status of filebeat.
sudo systemctl status filebeat
Step #6:Install and Configure Kafka
Kafka is responsible for streaming log data from Filebeat to Logstash. Install Zookeeper first.
sudo apt install -y zookeeperd
Download Kafka from Apache’s official servers.
wget https://downloads.apache.org/kafka/3.7.2/kafka_2.13-3.7.2.tgz
Extract the downloaded Kafka archive.
tar -xvzf kafka_2.13-3.7.2.tgz
Copy the extracted Kafka files to a standard installation location.
sudo cp -r kafka_2.13-3.7.2 /opt/kafka
Start the Kafka server.
sudo /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Verify the topic.
sudo /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
First open your browser and navigate to http://<your-server-ip>. You should see the default Apache welcome page.
Lets consume messages from Kafka.
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic apache --from-beginning
Ensure Elasticsearch is receiving data from Filebeat by checking the indices.
curl -XGET "localhost:9200/_cat/indices?v"You should see output indicating the presence of indices created by logstash.
Step #7:Visualizing Data in Kibana
Go to Menu bar from top-left corner and select Stack Management under the management section.
Under “Kibana” section, click on “Data views”.
Click on “Create data view”.
Enter logstash-* (the index name used in Logstash output) in the Index pattern name field and click on Save data view to Kibana.
Scroll down and click on the Logs option in Obeservability in the left-hand navigation menu. If the menu is collapsed, click the Expand icon to reveal the options.
Go to All logs as shown below.
Next go to the Data Views.
Select logstash-* as a data view.
Kibana displays logs data from the last 15 minutes, visualized as a histogram along with individual log messages below. (You may need to adjust the time range.)
To produce a custom test messages run the following command.
/opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic apacheType a message like shown below.
Welcome to devopshint
We are generating a Test message to Kafka
Now go back to kibana and refresh the logs, you will see the test message logs their.
Conclusion:
By following these steps, you can successfully Monitor Kafka logs with the Elastic Stack on an Ubuntu system. This setup enables real-time log processing and visualization using Elasticsearch, Logstash, Kibana, and Filebeat. Kafka acts as a central message broker, ensuring efficient log streaming. With this infrastructure in place, you can monitor and analyze logs in real time, making it easier to detect and resolve issues.
Related Articles:
How to Install Elastic Stack on Ubuntu 24.04 LTS
Send Java Gradle App Logs to Elastic Stack
Send Java Maven App Logs to Elastic Stack
Reference:
