restic backups on a timer, and the restore drill

2026-10-19

Two repositories, not one

One repository on a second machine is a backup with a shared failure domain: the same outage, the same account, the same provider dispute takes both. Use two destinations with different failure domains and let restic copy do the second leg, so the backup run itself only talks to one endpoint.

RESTIC_REPOSITORY=sftp:backup@host-a:/srv/restic/%h
RESTIC_REPOSITORY_B=s3:s3.example.org/bucket-name
# both repos must be initialised with the SAME password, otherwise copy fails

Keep the password in a root-only file, not in the unit file, and keep a copy of it somewhere that is not covered by the backup — a backup you cannot decrypt is a well-preserved pile of entropy.

The run

#!/bin/sh
set -eu
export RESTIC_REPOSITORY RESTIC_PASSWORD_FILE=/root/.config/restic/pass

restic backup --tag daily \
  --one-file-system \
  --exclude-caches \
  --exclude-file /etc/restic/excludes \
  /etc /var /home

restic forget --prune \
  --keep-within 14d --keep-weekly 6 --keep-monthly 12 --keep-yearly 3

--one-file-system stops the backup from quietly climbing into a mounted remote share and backing it up twice.
Prune policy is a statement about how far back you want to answer "what did this look like in March". Write it down as an intention, not as whatever the defaults were.
Prune rewrites pack files. On a slow remote repository that is the most expensive step of the whole run — run it weekly, not after every backup.

Integrity checks that fit in the runtime

A full restic check reads the entire repository, which on a large repo is not something you will schedule honestly. Two cheaper things that catch most real problems:

# structural check plus a sample of data, weekly
restic check --read-data-subset=2%

# the cheap one, after every run, catches a truncated upload
restic snapshots --latest 1

The failure mode that --read-data-subset misses is silent corruption of the part not sampled. That is what the second repository is for: two independent copies make the "both corrupted in the same way" case very small.

The drill

Scheduled, in the calendar, not "when I remember":

Monthly. Restore one file you did not choose, by opening the repository and searching for it. This tests that you can find things, which is the part that fails under pressure.
Quarterly. Full restore of one host onto a scratch VM, then start the services. This is the only test that catches "the backup works but the application needs three steps that were never written down".
Once a year. Restore using only the second repository, having deleted the first from your local cache config. Repositories that have only ever been read from one endpoint develop invisible dependencies on that endpoint.

The unit

[Unit]
Description=restic backup
RequiresMountsFor=/var/backups

[Service]
Type=oneshot
Nice=19
IOSchedulingClass=idle
ExecStart=/usr/local/sbin/restic-run
OnFailure=notify@%n.service

Then the timer with Persistent=true, so a machine that was off overnight catches up instead of silently skipping. And wire OnFailure= to something that reaches you: a backup that fails quietly for eleven weeks is discovered exactly once, and always at the worst moment.

← manuals