On 22 October I opened my computer to a notification that Omarchy, the Arch-based Linux distro I run, had an update waiting. My plan for the morning was dull and specific. screenrecord.in was about to start charging for its paid tier, I had changes to make in the recorder before that went live, and the sensible order was to push the system update first and spend the rest of the day on the app. The update ran, and somewhere inside it something happened to the encrypted SSD that holds my OS and most of my Docker containers. The volume wouldn’t come back afterwards. I don’t know what the update did to it, and I’ve stopped guessing. I spent about an hour on the usual rescue attempts and then gave up. An hour is about all the patience I have for a disk that has stopped cooperating. I wasn’t scared at that point, because I had backups. That confidence lasted until the reinstall.

First I flashed the SSD and put Omarchy back on it, and the install that came up was unusable. It was slow in a way I couldn’t pin on the disk or the network or my own impatience, and I had no appetite for debugging an installation that hadn’t given me anything yet. So I formatted the SSD again, set up encryption from scratch and installed a second time. That one was fine, which is the most annoying kind of bug, the kind that fixes itself and teaches you nothing. Then came the boring part. Docker images, one at a time, and some of them I had built myself and no longer remembered how. A forgotten base image, a flag I had to work out from an error message, a volume that had to exist before the container would start. It took a couple of days just to gather everything in one place and another couple to get it all running, and around the fifth day I had my apps back.

What didn’t come back was some of the analytics data and a set of customer chats, and that’s the part that still sits badly with me. The backup system I had was real and it worked, but it covered the things that were easy to list and missed the ones that lived inside somebody else’s schema. Nobody had ever asked me to restore a chat log before, so nothing had ever failed loudly enough to tell me. Then the funny and depressing part. The mirror SSD had been sitting in the same machine since day one, and I had never got it working because its SATA cable was dead. That’s a part that costs about a hundred and fifty rupees. I never spent twenty minutes finding out why a disk I had already paid for was doing nothing. When everything else fell over, I finally opened the case and replaced the cable, and now the mirror holds a real-time rsync of the OS drive.

Where things stand now

  1. Scripts dump Postgres, ClickHouse and MySQL every fifteen minutes into one directory. That’s fifteen minutes of possible loss instead of the couple of days it used to be.
  2. Another script walks the parent directory of every app looking for SQLite files and copies them into a parallel directory, so apps that keep their state in a file stop being invisible. It also copies them to Cloudflare R2.
  3. The OS SSD mirrors to the second SSD in the same case with rsync, which is the reason I finally fixed a cable.
  4. I’ve actually restored from these backups to check the speed, and a restore now takes minutes.
Monitoring
  1. Exception tracking moved in-house to GlitchTip, because I’d rather own the noise than rent it.
  2. Kuma watches the services. That’s a better arrangement than btop, which only ever told me what I already suspected while I was staring at it.

I know what the mirror doesn’t do. Two disks in the same case protect me from a dead cable and a bad update. They do nothing about the machine catching fire or being stolen, so I’ve moved that problem rather than deleted it. The backup I was relaxed about had never been asked to prove itself. The first time it was, it turned out to cover only the easy half of my data.

If your own restore ever went sideways in a way you did not see coming, my handle is @troysk704 and I would like to hear it.