Parse and Delete Linux Logs using Shell Script

In this article we will learn How to Parse and Delete Logs using Shell Script. In Linux systems, log files play a crucial role in monitoring system activity, diagnosing issues, and maintaining security. However, manually searching through log files for specific information can be time-consuming and tedious. We’ll explore how to streamline log searching using Bash scripting, making the process more efficient and user-friendly.

Prerequisites

• AWS Account with Ubuntu 24.04 LTS EC2 Instance.
• Basic knowledge of Shell scripting.

Step #1:Create the script for parsing logs

Open the terminal and use the nano command to create a new file.

nano logfile.sh

write a script to parse log files.

#!/bin/bash

logfile1=/var/log/syslog
logfile2=/var/log/cloud-init.log

mydatexpr=`date +%b\ %d`

for log in $logfile{1,2}
do
echo $log BEGIN
egrep "$mydatexpr" $log
echo $log END
done

Save the file and exit the editor.

Explanation of the script:

1. Shebang Line: This line specifies the shell interpreter to be used, which is bash.
2. Variables Initialization: These lines initialize variables logfile1 and logfile2 with the paths to the syslog and cloud-init log files, respectively.
3. Date Expression: This line uses the date command to get the current month and day (+%b %d) and assigns it to the variable mydatexpr. This will be used to search for logs with the same date format.
4. For Loop: This loop iterates over each log file specified in $logfile1 and $logfile2
5. Log Search: This line uses egrep to search for log entries containing the specified date expression ($mydatexpr) in the current log file ($log).
6. Echo Commands: This line prints the log file being processed.

Make the file executable by making it executable using the chmod command.

chmod +x logfile.sh

Run the script by executing the following command.

./logfile.sh

As you can see it successfully parse the logs. The script output shows the search results and confirms it found relevant entries. The script successfully searched /var/log/syslog and /var/log/cloud-init.log for lines containing today’s date.

Step #2:Create the script to delete log files

Open the terminal and use the nano command to create a new file.

nano logdelete.sh

write a script to delete log file.

#!/bin/bash

# Get the filename as an argument
logfile="/var/log/syslog"

# Check if a filename is provided
if [ -z "$logfile" ]; then
  echo "Error: Please specify a filename as an argument."
  exit 1
fi

# Check if the file exists
if [ ! -f "$logfile" ]; then
  echo "Error: File '$logfile' does not exist."
  exit 1
fi

# Create a backup with timestamp in the same directory
backup_file="$logfile.bak-$(date +%Y-%m-%d)"
cp "$logfile" "$backup_file"

if [ $? -eq 0 ]; then
  echo "Created backup: $backup_file"
  # Delete the original file (confirm before deletion)
  read -p "Are you sure you want to delete '$logfile' (y/N)? " answer
  if [ "$answer" == "y" ] || [ "$answer" == "Y" ]; then
    rm "$logfile"
    echo "Deleted: $logfile"
  else
    echo "Deletion cancelled."
  fi
else
  echo "Error: Failed to create backup."
fi

Save the file and exit the editor.

Explanation of the script:

1. Shebang: Specifies the shell interpreter to be used for executing the script.
2. Variable Initialization: Assigns the path of the log file to the variable logfile which will be deleted.
3. Check for Provided Filename: Checks if a filename is provided as an argument. If not, it displays an error message and exits the script with a non-zero status.
4. Check for File Existence: Checks if the specified file exists. If not, it displays an error message and exits the script.
5. Backup Creation: Creates a backup of the original file with a timestamp in the same directory.
6. Backup Success Check: Checks if the backup creation was successful. If successful, it displays a message confirming the creation of the backup.
7. Confirmation for Deletion: Prompts the user for confirmation before deleting the original file. If confirmed, it deletes the file and displays a message. If not confirmed, it cancels the deletion and displays a message.
8. Error Handling: Displays an error message if the backup creation fails.

Change the file permissions to make it executable using the chmod command.

chmod +x logdelete.sh

Run the script by executing the following command.

sudo ./logdelete.sh

Output:

Now lets see what will our logfile will return after running it again since syslog is deleted.

Run the following command.

./logfile.sh

As you can see it says file doesn’t exit and return the logs of cloud-init.log only.

Conclusion:

In conclusion, shell scripting provides a powerful tool for automating logs management tasks on Linux servers. we,ve explored parsing logs with shell scripts and managing them safely. We learned techniques for searching and extracting information from logs as well as best practices for deletion. The provided script demonstrates a cautious approach to deleting logs with confirmation and backup creation. This automation not only saves time but also ensures efficient logs management practices, contributing to the overall health and security of your system.

Related Articles:

MySQL Database Backup using Restore Shell Script

Reference:

Bash reference manual