Monitoring Logrotate with Outlyer

Logs are a set of records generated by operating systems, services, and applications that provide a valuable timeline of events for troubleshooting when an issue is discovered.

Logs are typically stored in files (log files) in the filesystem and they can grow in size very quickly. If log files grow indefinitely and a machine runs out of disk space, chances are that the machine will crash, causing major issues to the services and applications running on it. Sometimes, due to lack of disk space, a machine can crash so bad that it’s not even possible to log into it via ssh anymore. In order to avoid that, a process called log rotation can be applied.

Log rotation is an automated process used in system administration in which dated log files are archived. For example, after a log rotation, a log file named access.log becomes access.log.1 (or access.log.1.gz, if compression is enabled) and a new log file access.log is set up for new log entries. In Linux, log rotation is typically performed by the logrotate tool, which allows automatic rotation, compression, removal, and mailing of log files.

Logrotate Configuration

On Ubuntu and others Linux distributions, logrotate is installed by default and used by many system tools (like apt, dpkg, and others) for log rotation. The logrotate global configuration file is generally located at /etc/logrotate.conf and specific logrotate configurations for each service are stored in the /etc/logrotate.d/ directory. By default, the logrotate binary is daily invoked by the /etc/cron.daily/logrotate cron job.

Logrotate provides many rotation settings and they are all documented in the logrotate man page. Let’s take a look at a sample logrotate configuration file for Nginx:

/var/log/nginx/*.log {
        daily
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`
        endscript
}

The first line specifies log file names (or patterns) for which the configuration will apply to. In this case, this same logrotate configuration applies for both Nginx access.log and error.log, as both are located at /var/log/nginx/ directory by default.

Basically, this configuration means:

  1. daily: rotate both Nginx log files daily;
  2. missingok: if a log file does not exist, does not issue an error;
  3. rotate 52: the log files will go through 52 daily rotations before start being removed from the filesystem;
  4. compress: archived log files are compressed with gzip;
  5. delaycompress: postpone compression of the previous log file to the next rotation cycle (this only has effect when used in combination with compress). For example, by using this option, the final log files for Nginx access log will be access.log, access.log.1 (note there is no compression here, the compression was delayed), access.log.2.gz, access.log.3.gz, …, access.log.52.gz;
  6. notifempty: do not rotate the log file if it is empty;
  7. create 640 root adm: specifies the mode, owner, and group in which the new log file will be created;
  8. sharedscripts: the postrotate script will only be run once (after the old logs have been compressed), not once for each log which is rotated;
  9. postrotate/endscript: after the log file has been rotated (but before the old version of the log has been compressed), the command block will be executed, which will re-open Nginx logs in response to the USR1 signal.

It’s very important to note that logrotate is not a daemon that is permanently running and comparing the current date and time with logrotate configuration files (hourly, daily, weekly, monthly) to perform rotations. In other words, logrotate requires someone external (like cron) invoking the logrotate binary passing a logrotate configuration file as an argument so that rotations actually take place. This means that if you remove the /etc/cron.daily/logrotate default cron job that daily executes the logrotate binary, log rotations will never be performed, even though the configuration files state hourly, daily, weekly, or monthly.

The question now is, how does logrotate know when the last rotation was performed for each log file? Basically, logrotate keeps a file at /var/lib/logrotate/status with a list of each log file name and the respective date and time in which the last rotation was performed. This file is known as logrotate state file (in some Linux distributions, it may be located at /var/lib/logrotate.status). Below is how logrotate state file looks like:

logrotate state -- version 2
"/var/log/syslog" 2018-9-13-2:21:54
"/var/log/nginx/access.log" 2018-9-13-2:0:0
"/var/log/nginx/error.log" 2018-9-13-2:0:0
"/var/log/dpkg.log" 2018-9-13-2:21:54
...

So, if the default logrotate cron job runs daily and a logrotate configuration file uses the hourly directive, would the log file be hourly rotated? The answer is no. In order for that to work, you would have to set up a cron job that hourly invoked the logrotate binary.

What if you want to rotate a log file every 30 minutes? Well, as there is no “30 minutes” configuration in logrotate, you would need to set up a cron job to run every 30 minutes invoking the logrotate binary and forcing a rotation by using the logrotate -f option.

So, why do you need monitoring for logrotate and the log files it manages? Essentially, because if rotations silently fail (and they do, believe me!) for whatever reason (like a misleading configuration), your log files will end up consuming all available disk space and your machines will crash. When that happens, in order to figure out what caused the rotation to fail, you have to manually run logrotate in debug mode so it provides verbose output. In a real-world scenario, a system administrator needs to get notified as fast as possible when something goes wrong in the infrastructure.

How to Monitor Logrotate with Outlyer?

As soon as you install the logrotate integration for Outlyer, you will get an out-of-the-box logrotate dashboard and alert template.

The logrotate dashboard allows to easily track when the last rotation was performed as well as the size of each log file managed by logrotate (the ones listed in the logrotate state file):

Outlyer Logrotate Dashboard Outlyer Logrotate Dashboard

Besides, Outlyer can send alert notifications when a rotation fails within a period of time (in seconds) or a log file size becomes greater than a specific limit (in bytes):

Logrotate Alerts Example Logrotate Alerts Example