Backups

OverviewCopy Anchor Link

Two systemd timers on the node, installed by Ansible (ansible/roles/backup), back up the cluster:

Each run:

1. takes consistent online copies (no app is stopped);
2. packs them in a .tar.gz with a SHA256SUMS file and encrypts it with age;
3. uploads it to the private backups container of the polinetworkbackups storage account, using the id-k3s-backup managed identity;
4. publishes its success time to Prometheus.

RetentionCopy Anchor Link

The storage account only accepts traffic from the K3s subnet, so download backups from the node.

MonitoringCopy Anchor Link

The BackupStale alert fires on Telegram when there has been no successful application backup for 7 hours or no control-plane backup for 26 hours. BackupMetricsMissing fires if the metrics disappear altogether.

From the node:

# Next and last runs
systemctl list-timers 'k3s-*-backup.timer'

# Logs of the last run
sudo journalctl -u k3s-application-backup.service -n 50

# Run a backup now
sudo systemctl start k3s-application-backup.service

Adding a new appCopy Anchor Link

Volumes are not backed up automatically. If you add an app with data worth keeping (see Add Storage), add it to ansible/roles/backup/templates/k3s-application-backup.sh.j2 in polinetwork-cd, using the right tool for the data:

• a database: its own dump/backup command, through kubectl exec (like PostgreSQL and InfluxDB);
• an SQLite file: the sqlite_backup helper, which takes a consistent copy and archives the other files in the volume;
• Redis: the redis_snapshot helper.

Then run the playbook (see K3s Node): verify.yml takes a fresh backup and checks it.

RestoringCopy Anchor Link

1. Get the archiveCopy Anchor Link

The last 7 days are on the node, in /srv/standard/backups/k3s-applications/, readable only by root. Copy the one you need to your home:

sudo cp /srv/standard/backups/k3s-applications/<archive>.tar.gz.age ~
sudo chown pnadmin ~/<archive>.tar.gz.age

For older backups, list and download them from Blob Storage on the node, with the backup identity (its client ID is backup_identity_client_id in ansible/vars/main.yml):

TOKEN=$(curl -s -H Metadata:true \
  "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fstorage.azure.com%2F&client_id=<backup-identity-client-id>" \
  | jq -r .access_token)

# List
curl -s -H "Authorization: Bearer $TOKEN" -H "x-ms-version: 2023-11-03" \
  "https://polinetworkbackups.blob.core.windows.net/backups?restype=container&comp=list&prefix=k3s-applications/k3s01/" \
  | grep -o '<Name>[^<]*</Name>'

# Download
curl -s -H "Authorization: Bearer $TOKEN" -H "x-ms-version: 2023-11-03" \
  -o ~/<archive>.tar.gz.age \
  "https://polinetworkbackups.blob.core.windows.net/backups/k3s-applications/k3s01/<archive>.tar.gz.age"

2. Decrypt it on your machineCopy Anchor Link

Copy the encrypted archive to your machine and decrypt it there with age and the identity from Key Vault:

scp pnadmin@10.43.1.4:<archive>.tar.gz.age .

# umask 077: the identity file is readable only by you
(umask 077; az keyvault secret show --vault-name kv-pn-infra --name backup-age-identity \
  --query value -o tsv > age-identity.txt)

mkdir restore
age --decrypt --identity age-identity.txt <archive>.tar.gz.age | tar -xz -C restore
(cd restore && sha256sum -c SHA256SUMS)

rm age-identity.txt

restore/ now contains postgres-dumpall.sql, redis.rdb, bot-ts-redis.rdb, influxdb-backup.tar, uptime-kuma.db, grafana.db and the *-files.tar archives.

3. Restore the dataCopy Anchor Link

For PostgreSQL, the dump recreates every database (--clean --if-exists), so you can stream it into the running server:

ssh pnadmin@10.43.1.4 \
  "k3s kubectl exec -i -n postgres deploy/postgres -- sh -c 'psql --username=\"\$POSTGRES_USER\" --dbname=postgres'" \
  < restore/postgres-dumpall.sql

For the other services, stop the app (kubectl scale --replicas=0), put the files back in its volume under /srv/<fast|standard>/volumes/<namespace>/<pvc>/, and scale it up again. Follow the upstream docs for Redis and InfluxDB.

The control-plane backup is only needed to rebuild the node from scratch: it restores the K3s datastore, including the service-account signing key that Azure workload identity trusts.