What a power cut actually does to a server

2026-01-17

Write this down while it is fresh, because the useful part is not that the power went out but which assumptions turned out to be wrong.

What happened

Supply dropped at 03:40. The UPS took the load, the monitor daemon noticed ONBATT, and nothing else happened until the batteries were empty at 04:12. The machine lost power mid-write. There was no shutdown sequence at all — the daemon was running and configured, and it was configured as a slave with no master and nothing to tell it what to do.

What survived, and why

• ext4 with a journal came back clean. The journal replayed and the filesystem was consistent — not lucky, just doing the thing it does.
• SQLite databases did not. A file that is written with several separate write()s and no fsync can land in a state where one part of a record is on disk and the neighbour is not. Integrity check reported corruption in one table.
• Databases with a write-ahead log survived, which is exactly what they are for. Recovery took seconds and needed nothing from me.
• The backup that was due at 03:00 had not run, because the machine had been rebooted for a kernel update the night before and the timer had no Persistent=true. So the corrupted SQLite file had nothing current to restore from.

Three independent small things — no master, no fsync, no catch-up — and only the combination made it a problem. That is the whole lesson.

Shutdown ordering is the actual product

A UPS buys minutes, and minutes are only useful if something spends them well. The sequence has to stop things in dependency order, and the two mistakes people make are the same two: databases stopped after the filesystem they live on, and network services stopped before the clients that need to flush to them.

# /etc/nut/nut.conf
MODE=standalone            # or netmaster on the box the UPS is plugged into

# /etc/nut/upsmon.conf
MONITOR ups@localhost 1 master secretpw slave
SHUTDOWNCMD "/sbin/shutdown -h +0"
POWEROFFWAIT 120
OFFDELAY 30
DNSENABLED no

The interesting values: OFFDELAY is how long to wait after ordering the shutdown before cutting power to the outlets, and POWEROFFWAIT is how long to wait for the machine to actually go down. Defaults assume a fast machine with nothing running. A database that needs ninety seconds to shut down cleanly, followed by an outlet that cuts power at sixty, turns your graceful shutdown into the same power cut you were trying to avoid.

What the UPS is plugged into

The box should be on battery. So should whatever it depends on to shut down sanely — and this is where it goes wrong in practice: the switch or access point on mains power means the machine is alive, the network is dead, and every timeout fires at once while the clock is ticking down. Conversely, a disk enclosure on mains power gets pulled out from under a machine that is still writing.

The only test that counts

Unplug it, deliberately, during the day, with a stopwatch and a second machine watching:

1. Does the monitor daemon notice, and how fast?
2. Does the shutdown sequence start, and does each service stop in the order you think it does? journalctl -b -1 afterwards answers this precisely.
3. Does the outlet actually cut power, and does the machine come back on its own when supply returns, with the right boot settings?
4. How long was left on the battery, really? Batteries lie: a three-year-old battery is a fraction of its label, and runtime falls off a cliff under load rather than linearly.

Run it once a year. The drift is silent: batteries age, load grows, and a service added last month may be the one that now takes four minutes to stop.

← manuals