Notes

systemd timers that quietly stopped firing

2026-10-28

A timer can be enabled, active and still never run again after the box resumes from suspend. systemctl list-timers will show a next elapse time in the past if the unit missed its window and Persistent= is not set.

[Timer]
OnCalendar=Mon *-*-* 03:15:00
Persistent=true      # catch up after downtime
RandomizedDelaySec=12m

Then check what the service actually did, not what the timer intended:

systemctl status backup.timer
journalctl -u backup.service --since "7 days ago" --no-pager | tail -20

The other common cause is a service that exits 0 while doing nothing useful, because the script returns early on a missing mount. Put the mount under RequiresMountsFor= in the unit instead of testing for the directory.

PostgreSQL backup checklist

2026-10-11

A backup that has never been restored is a rumour. Restore to a scratch instance on a schedule, not once at setup.
Keep pg_wal archiving separate from the base backup; a base backup without the WAL range covering it cannot be replayed to a consistent point.
Record the version of pg_dump used. Restoring a dump taken by a newer server with an older client is a fun afternoon.
Verify the checksum after transfer, not before: corruption usually happens on the destination media.
Test that the backup user cannot read the data directory directly. If it can, the backup is one compromised account away from being public.

Triage for a noisy access log

2026-09-30

Before reaching for fail2ban, work out what the traffic actually is. Most log noise is scanners looking for well-known paths, and it costs nothing to serve.

awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -20
awk '$9 == 404 {print $1}' access.log | sort | uniq -c | sort -rn | head

If the same source asks for hundreds of unrelated paths in one connection, it is a scanner and rate limiting is enough. If a single path is requested at high rate from many addresses, something is being looked for specifically, and that is worth reading the referer field for.

Only after that: is the traffic reaching the application at all, or is it dying on a static location? Rate limiting at the wrong layer just moves the cost.

Longer procedures live under /docs/.

← back