In this article we will learn how to Monitor Nginx Logs with Elastic Stack and Filebeat on Ubuntu 24.04 | Send Nginx Logs to Elastic Stack and Filebeat. Monitoring Nginx logs is crucial for tracking errors, analyzing traffic patterns, and identifying security threats. In this guide, we’ll set up the Elastic Stack (Elasticsearch, Kibana, and Filebeat) on Ubuntu 24.04 and configure Filebeat to collect Nginx logs. By the end, you’ll have a Kibana dashboard visualizing your Nginx logs.
Table of Contents
Prerequisites
- AWS account with an Ubuntu 24.04 EC2 instance.
- Instance with at least 2 CPU cores and 4 GB of RAM for optimal performance.
- Java and Nginx installed.
Step #1:Set Up Ubuntu EC2 Instance
Update the Package List to ensure you have the latest versions.
sudo apt update
Elasticsearch requires Java, so we need to install OpenJDK 11.
sudo apt install -y openjdk-11-jdk
Install the Nginx web server.
sudo apt install nginx -y
Check the status of the Nginx to ensure it is running.
sudo systemctl status nginx
Open your browser and navigate to http://<your-server-ip>. The default Nginx welcome page should appear.
Step #2:Install Elasticsearch on Ubuntu 24.04 LTS
Import the Elasticsearch GPG key.
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
Add the Elasticsearch repository.
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
Now lets update the package list again. The repository is added to the system’s package sources.
sudo apt update
Install Elasticsearch.
sudo apt install -y elasticsearch
Enable and start Elasticsearch.
sudo systemctl enable elasticsearch sudo systemctl start elasticsearch
Check the status of the elasticsearch to ensure it is running.
sudo systemctl status elasticsearch
Modify Elasticsearch configuration for remote access.
sudo nano /etc/elasticsearch/elasticsearch.yml
Find the network.host setting, uncomment it, and set it to 0.0.0.0 to bind to all available IP addresses and uncomment the discovery section to specify the initial nodes for cluster formation discovery.seed_hosts: []
For a basic setup (not recommended for production), disable security features.
xpack.security.enabled: false
Save and exit the editor.
Restart Elasticsearch to apply the changes.
sudo systemctl restart elasticsearch
Send a GET request to check if Elasticsearch is running and responding. If successful, you should see a JSON response with cluster information.
curl -X GET "localhost:9200"
You can access it using browser with your Public IP address:9200 port which is a default port for Elasticksearch.
Step #3:Install Kibana on Ubuntu 24.04 LTS
Kibana provides visualization for Elasticsearch data. Install Kibana on the system.
sudo apt install -y kibana
Enable and start Kibana.
sudo systemctl enable kibana
sudo systemctl start kibana
Checks the status of Kibana.
sudo systemctl status kibana
Open the Kibana configuration file for editing.
sudo nano /etc/kibana/kibana.yml
Uncomment and adjust the following lines to bind Kibana to all IP addresses and connect it to Elasticsearch.
server.port: 5601 server.host: "0.0.0.0" elasticsearch.hosts: ["http://localhost:9200"]
Restart Kibana to apply the changes.
sudo systemctl restart kibana
Access the Kibana interface by navigating to http://<your-server-ip>:5601 in your web browser. This will open the Kibana dashboard where you can start exploring your data.
You can start by adding integrations or Explore on my own.
Step #4:Install Filebeat on Ubuntu 24.04 LTS
Filebeat collects and forwards log data to Elasticsearch or Logstash. Install Filebeat on the system.
sudo apt install -y filebeat
No need to edit the filebeat configuration as by default it is configured to send logs to Elasticsearch.
Enable the Nginx module in Filebeat.
sudo filebeat modules enable nginx
Configure the Nginx module for log collection.
sudo nano /etc/filebeat/modules.d/nginx.yml
Ensure the following configuration is enabled to send Nginx logs.
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/nginx/access.log*"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/nginx/error.log*"]
# Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
ingress_controller:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
Save and exit the file.
Test the configuration.
sudo filebeat test config
Apply Filebeat setup changes.
sudo filebeat setup
Start and enable the Filebeat service.
sudo systemctl enable filebeat
sudo systemctl start filebeat
Checks the status of filebeat.
sudo systemctl status filebeat
Ensure Elasticsearch is receiving data from Filebeat by checking the indices.
curl -XGET "localhost:9200/_cat/indices?v"You should see output indicating the presence of indices created by Filebeat.
Step #5:Verify Nginx Logs in Kibana
Now go back to Kibana. Scroll down and click on the Logs option in Obeservability in the left-hand navigation menu. If the menu is collapsed, click the Expand icon at the bottom left to reveal the options.
Kibana displays Nginx logs data from the last 15 minutes, visualized as a histogram along with individual log messages below. (You may need to adjust the time range.)
Step #6:Generating 404 Error in Nginx for Testing in Elastic Stack
To generate a 404 Not Found error and see it in Kibana, access the following page on browser.
http://<public-ip-address>/this-page-does-not-existThis request will be logged in Nginx access log and should be visible in Kibana.
Now refresh the kibana logs page.
You can even see the details of Nginx logs. You can see the details of our Cloud provider also some other details.
Conclusion:
In this guide, we successfully installed the Elastic Stack (Elasticsearch, Kibana, and Filebeat) to monitor Nginx logs on Ubuntu 24.04. We configured Filebeat to collect Nginx access and error logs, ensuring seamless data ingestion and visualization. With this setup, you can efficiently track web traffic, detect errors, and improve server performance.
Related Articles:
How to Install Elastic Stack on Ubuntu 24.04 LTS
Install Elastic Stack on Amazon Linux 2
Set Up ELK Stack (Elasticsearch, Logstash and Kibana) On Windows
Reference:
