Log rotation: the two ways it fails silently

2026-03-31

Rotation usually fails in one of two directions, and both are invisible until you need the logs: either the file is never rotated and the disk fills, or it is rotated and the application keeps writing to the old inode, so the "current" log stays empty forever.

Reopen, do not copy-truncate

The correct sequence is: rename the file, create a new one, tell the process to reopen. The second option, copytruncate, copies the file and then truncates it in place — which means anything written between the copy and the truncate is lost, and on a busy service that is a real window, occurring every single rotation.

/var/log/app/*.log {
    weekly
    rotate 8
    compress
    delaycompress
    missingok
    notifempty
    create 0640 app app
    postrotate
        systemctl reload app >/dev/null 2>&1 || true
    endscript
}

• delaycompress — do not compress the most recent rotated file. Some daemons hold the descriptor a while longer than you think, and compressing under them produces a truncated archive.
• create 0640 app app — without the mode and owner, the new file is created with the default and the service, running as a non-root user, silently cannot write to it.
• missingok — otherwise a service that has never logged yet makes the whole nightly run fail, and the mail nobody reads says "error: stat of /var/log/x failed".

How to signal each daemon is a per-application detail worth looking up rather than guessing: nginx wants USR1 (reopen logs, no restart), rsyslog wants a restart of its own unit, and anything under systemd should not be rotated at all.

Do not rotate what journald owns

If a service logs to the journal, journald already does retention by size and age. Putting logrotate on /var/log/journal or forcing an app from journal to a file just so it can be rotated buys disk churn and a second retention policy that disagrees with the first:

# /etc/systemd/journald.conf.d/size.conf
[Journal]
SystemMaxUse=500M
MaxRetentionSec=3month

The failure mode worth knowing: Storage=auto keeps the journal volatile until /var/log/journal exists, so the journal is empty after a reboot — which is precisely when it is wanted.

Testing without waiting a week

logrotate -d /etc/logrotate.d/app        # dry run, prints decisions
logrotate -f -v /etc/logrotate.d/app     # force one rotation now
ls -i /var/log/app/access.log            # inode before and after
ls -l /proc/$(pgrep -f 'app .*worker' | head -1)/fd | grep log

That last line is the actual test: if the process still has the rotated file open by descriptor, your postrotate signal is not working, and the new file will stay empty while the deleted one grows. This is the failure that hides for months, and it hides from every monitoring system that only watches disk usage — because the growing file is unlinked and invisible to du while still eating space.

← manuals