In this article, we will learn how to automate log rotation using a shell script, focusing on MongoDB log files as an example. Log rotation is essential for maintaining system performance and ensuring that log files do not consume excessive disk space. By automating this process, you can manage log files more efficiently without manual intervention.
Table of Contents
Prerequisites
- AWS Account with Ubuntu 24.04 LTS EC2 Instance.
- Basic knowledge of MongoDB and Shell scripting.
Step #1:Install MongoDB on Ubuntu
Update the system before starting installation process.
sudo apt update 
To import the MongoDB public GPG key, run the following command.
curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg \
--dearmor
Create the /etc/apt/sources.list.d/mongodb-org-7.0.list file for Ubuntu 24.04.
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
Reload the local package database using following command.
sudo apt-get update
Install the latest stable version of MongoDB using following command.
sudo apt-get install -y mongodb-org
After this reload the daemon service and also enable the mongod and after that start the mongod.
sudo systemctl daemon-reload
sudo systemctl enable mongod
sudo systemctl start mongod
Check the status to if MongoDB is running successfully.
sudo systemctl status mongod
Step #2:Set Up MongoDB Authentication
Next set up MongoDB authentication and create a user. Begin by connecting with mongosh.
mongoshAfter connecting to your MongoDB instance with mongosh, switch to the admin database using the command.
use adminCreate a new user with a username, password, and appropriate roles.
db.createUser({
user: "fosstechnix", pwd: "devops",
roles: [ { role: "userAdminAnyDatabase", db: "admin" }, { role: "dbAdminAnyDatabase", db: "admin" }, { role: "readWriteAnyDatabase", db: "admin" } ]
})
Exit the shell after creating the user.
exit
Step #3:Create the Script File
First, we need to create a new shell script file. Open your terminal and run the following command to create a file named log_rotate.sh
nano log_rotate.sh
Step #4:Write the Script for Log Rotation
Add the following code to the file.
#!/bin/bash
# Exit immediately if a command exits with a non-zero status.
set -e
# Check if mongosh is installed
if ! command -v mongosh &> /dev/null; then
echo "mongosh could not be found. Please install it before running this script."
exit 1
fi
# Define MongoDB connection details (improve security with environment variables)
MONGODB_USERNAME="${MONGODB_USERNAME:-}"
MONGODB_PASSWORD="${MONGODB_PASSWORD:-}"
if [[ -z "$MONGODB_USERNAME" || -z "$MONGODB_PASSWORD" ]]; then
echo "MongoDB username or password is not set. Please set the MONGODB_USERNAME and MONGODB_PASSWORD environment variables."
exit 1
fi
# Trigger MongoDB log rotation
mongo_command="db.adminCommand({ logRotate: 1 })"
if ! mongosh_output=$(mongosh --eval "$mongo_command" -u "$MONGODB_USERNAME" -p "$MONGODB_PASSWORD" 2>&1); then
echo "Failed to rotate MongoDB logs. Error: $mongosh_output"
exit 1
fi
echo "Log rotation command executed. Waiting for the rotation to complete..."
# Wait for log rotation to complete (adjust based on expected rotation time)
sleep 120
# Define search criteria for old log files
log_dir="/var/log/mongodb/"
log_prefix="mongod.log.2024"
max_file_age=5 # Files older than 5 days will be removed
# Find and delete old log files
if find "$log_dir" -name "${log_prefix}*" -mtime +"$max_file_age" -exec rm -rf {} \;; then
echo "MongoDB log rotation and cleanup completed successfully."
else
echo "Error occurred during the cleanup of old log files."
exit 1
fi
save the file and exit.
Explanation of the script:
- Exit on Error:
set -eEnsures that the script will exit immediately if any command returns a non-zero exit status, which helps to avoid running subsequent commands if an error occurs. - Check for mongosh Installation:
command -v mongosh &> /dev/nullChecks ifmongoshis installed. If not, the script outputs an error message and exits. - Define MongoDB Connection Details:
MONGODB_USERNAME="${MONGODB_USERNAME:-}"andMONGODB_PASSWORD="${MONGODB_PASSWORD:-}"Retrieves the values of the environment variablesMONGODB_USERNAMEandMONGODB_PASSWORD. If these variables are not set, they default to empty strings. - The
ifcondition checks if either variable is unset or empty. If so, the script outputs an error message and exits. - Trigger MongoDB Log Rotation:
mongo_command="db.adminCommand({ logRotate: 1 })"Defines the MongoDB command to rotate logs. mongosh --eval "$mongo_command" -u "$MONGODB_USERNAME" -p "$MONGODB_PASSWORD"Executes the log rotation command usingmongoshwith the provided username and password. If the command fails, the error message is captured, and the script exits with an error.- Wait for Log Rotation:
sleep 120Pauses the script for 120 seconds to allow the log rotation to complete. This duration can be adjusted based on the expected time for the rotation to finish. - Define and Delete Old Log Files:
log_dir="/var/log/mongodb/"Specifies the directory where MongoDB log files are stored. log_prefix="mongod.log.2024"Specifies the prefix of the log files to be targeted. This pattern will match log files generated in 2024.max_file_age=5: Specifies that files older than 5 days should be removed.find "$log_dir" -name "${log_prefix}*" -mtime +"$max_file_age" -exec rm -rf {} \;: Finds and deletes log files in the specified directory that match the prefix and are older than the specified number of days.- The
ifcondition checks the success of thefindcommand. If it completes successfully, a success message is printed. If it fails, an error message is printed and the script exits.
Step #5:Set Up Environment Variables
To enhance security, set the MongoDB username and password as environment variables.
export MONGODB_USERNAME="fosstechnix"
export MONGODB_PASSWORD="devops"
Step #6:Make the Script Executable
Make the script executable by changing its permissions.
chmod +x log_rotate.sh
Step #7:Run the Script
To run the script, use the following command in the terminal, We need to provide a directory as an argument when running the script.
./log_rotate.sh
As you can see our log rotation and cleanup is completed.
Step #8:Verify Log Rotation
List the files in the MongoDB log directory to see if new log files have been created.
ls -lh /var/log/mongodb/
Look for files with the pattern mongod.log.2024*. You should see the latest log file and any rotated log files. As shown above mongod.log is latest log file and mongod.log.2024-06-05T07-32-34 is rotated log file.
Conclusion:
In conclusion, automating log rotation using a shell script ensures efficient log management and prevents log files from consuming excessive disk space. By following the steps outlined in this article, you can set up a reliable system to rotate and clean up MongoDB logs automatically. This practice not only enhances system performance but also simplifies log maintenance, allowing you to focus on more critical tasks.
Related Articles:
Shell Script to renew SSL Certificate
Reference:
