Notes from running production.

I'm Ivo, a platform engineer and SRE. I write down what broke, what fixed it, and the numbers.

storage-node, from ZFS + NFS as Kubernetes Storage: Point-in-Time Recovery Without the Cost
# app bug corrupted the database files, noticed 30 minutes later
$ zfs rollback zroot/k8s-data@2026-04-15-09-00
# cluster hardware gone: clone this morning's offsite snapshot
$ zfs clone remote/k8s-data@2026-04-15 zroot/recovery

Writing