Apache Log File Rotation Perl Script

Written by Tony Bhimani
June 27, 2006

Requirements
*nix based Operating System
Perl and Cwd

Download the source code: xlogrotate.tar.gz

The Apache log file rotation script is something I wrote in my early Linux years. Being unaware of any included log rotation scripts or programs for Linux that could rotate, compress, and restart Apache so it could regain the file handle to the logs, I wrote this script in Perl to manage all the log files across all the domains I host on this server. This script, which I dubbed xlogrotate, uses a flat file to store a list all the log files it needs to rotate and compress. xlogrotate is scheduled via cron to run on the first day of each month at midnight. It'll iterate through each log file entry by tarring it as a temporary file, compressing the temporary file with gzip, and then renaming it to its destination file name which is created using its original file name and last modification date. In the event that the destination file name already exists it'll append a sequential integer to keep the file names distinct so they don't overwrite each other. This is necessary if log files need to be rotated on an hourly basis or at the least more than once a day. The xlogrotate script features the use of regular expressions to extract file names and paths from a log file string and parsing the log file list looking for comments and blank lines, file I/O operations for reading a flat file line by line, reading file attributes such as date and time using stat, and issuing (shelling) system commands from within the Perl script. It's a nifty little log rotation script that can be customized to handle more than just Apache web server logs.

Installation

Extract the contents of the gzipped tar archive using tar zxvf xlogrotate.tar.gz to /root/scripts or another directory of your choice. You'll need to edit the xlogrotate.lst file and add the full path of each Apache log file you want to rotate and compress. Put each log file on a separate line and don't include any spaces in front or behind the entry. I didn't include any code for whitespace trimming but you can add that on your own if you wish. Within the flat file list you can use comments by adding a line with a # in front and blank lines are also allowed so everything isn't all squished together. xlogrotate requires Cwd to get the current working directory so it can find the flat file list. Cwd can be installed from cpan. If you need to, be sure to update the shebang in xlogrotate with the path of your Perl interpreter if it isn't /usr/bin/perl. Finally, set xlogrotate as executable with chmod 755 xlogrotate.

Setting up the Cron job

Assuming you installed the script and log flat file to /root/scripts/logrotate then you can add a cron job for xlogrotate to /etc/crontab like so.

00 0 1 * * root /root/scripts/logrotate/xlogrotate > /dev/null 2>&1

The script will run on the first day of each month at 12:00am and won't send any status output as emails to root. I suggest you receive the emails so that you know the cron job ran as scheduled. If for some reason it didn't run then you can start it manually from command-line. To receive the emails remove the > /dev/null 2>&1 from the crontab entry.

Summary

Log files should be rotated on a regular basis for several reasons. The primary reason would be to manage the size of the log files. By rotating logs you won't let the size of them grow out of control. This can be especially important when your web site gets a massive amount of hits. Another reason would be that archiving becomes a lot easier. Instead of browsing through a giant log file looking for a specific date you can easily look in your archives broken down by month, year, or even day depending on the frequency you set for rotation. Apache and system performance may also become an issue. If you never rotate your logs then reading from or writing to an extremely large file can impact system performance. Parsing smaller files is always quicker and less taxing on system resources. Whatever the reason, xlogrotate will create backups of your Apache logs and archive them thus making them easier to manage and saving disk space by compressing them. The script can be customized by also handling rotation of system logs, mail server logs, ftp logs, or any other kind of log file out there.

Source Code:

#!/usr/bin/perl

use strict;
use Cwd;

my ($logfile,$path,$filename);

# open the log file list
open LIST, "< " . getcwd() . "/xlogrotate.lst" or die "Cannot open list file! - $!\n";

# keep reading the list till we're done
while (<LIST>) {
  $logfile = $_;        # read a line
  chomp $logfile;       # cut off the trailing newline character

  # if the line starts with a pound (#) then skip it, it's a comment
  # also skip all blank lines
  if (($logfile !~ /^\s*#.+/) && ($logfile !~ /^\s*$/)) {
    # make sure the log file exists
    if (-e $logfile) {
      # split the logfile string into its path and file name
      ($path,$filename) = $logfile =~ m|^(.*[/\\])([^/\\]+­?)$|;

      # get array of file attributes
      my @stats = stat($logfile);
      # index 9 of array is file last modify time - use localtime to get each piece
      my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($stats[9]);
      # create file date string for output naming
      my $filedate = sprintf("%d%02d%02d",$year+1900,$mon,$mday);

      # temporary tar file name
      my $tmpname = $path . "tmp.tar";
      # destination file name
      my $newname = $filename . "." . $filedate . ".tar.gz";
      # full path of destination file
      my $dstfile = $path . $newname;

      # if the destination file exists we don't want to overwrite it
      # so a number is appended to check for distinct file names
      my $filenum = 1;
      if (-e $dstfile) {
        while (true) {
          # make a new destination file name
          $newname = $filename . "." . $filedate . "-" . $filenum . ".tar.gz";
          $dstfile = $path . $newname;
          if (-e $dstfile) {
            $filenum++;
          } else {
            last;
          }
        }
      }

      # create the tar file, compress it, and rename the temp name to the final name
      system("tar -P -cf $tmpname $logfile");
      system("gzip $tmpname");
      system("mv -f $tmpname.gz $dstfile");

      # print the working message, remove the old log file, and create a new empty one
      print "creating backup: $dstfile\n";
      system("rm $logfile");
      system("touch $logfile");
    }
  }
}

# close the log file list
close LIST;

# restart the apache service using graceful (existing connections aren't terminated)
print "restarting apache...\n";
system("/etc/init.d/httpd graceful");

How would you rate the usefulness of this content? Poor 1 2 3 4 5 6 7 8 9 Outstanding Optional: Tell us why you rated the content this way. Characters remaining: 1024

Average rating: 4.38 out of 9. 1 2 3 4 5 6 7 8 9 13 people have rated this content.