A small backup setup that actually gets tested
Untested backups are a rumour. This is the setup I use on fleets of one to thirty machines: restic to cheap object storage, a systemd timer, and a restore check that fails loudly when it should.
Repository layout
One repository per host, not per service. Hosts are the unit that gets rebuilt, and a single repository keeps restore simple. Keys are per host too, so a compromised box cannot delete another box's history if the storage policy forbids it.
export RESTIC_REPOSITORY=s3:https://s3.example.net/backups/host01
export RESTIC_PASSWORD_FILE=/etc/restic/pass
restic initThe timer
Twice daily, with a randomised delay so thirty machines do not stampede the storage at 03:00 sharp:
[Timer]
OnCalendar=*-*-* 03,15:17
RandomizedDelaySec=37min
Persistent=trueThe part everyone skips
Every month, restore the newest snapshot of /etc and /var/lib/postgresql
into a temporary directory and compare a manifest. If the restore fails, the unit fails, and the
alerting path is the same one that pages for disk space.
restic restore latest --target /var/tmp/restore-check \
--include /etc --include /var/lib/postgresql
test -f /var/tmp/restore-check/etc/hostname
pg_verifybackup /var/tmp/restore-check/var/lib/postgresql/latestTwo details make this worth the effort. First, pruning and checking are separate jobs: running
restic check --read-data weekly is expensive, and doing it after every backup is worse
than useless. Second, retention must be explicit — --keep-daily 7 --keep-weekly 4
--keep-monthly 6 is a reasonable default for machines under a terabyte.
What I learned the hard way
A backup that has never been restored will contain exactly the files you did not need, and will be
missing the database WAL because the dump ran while the service was writing. Stop the service, or use
a proper snapshot, but do not trust a tar of a live data directory.
Storage is cheap; the drill is not. Automate the drill anyway.