panel/docs/BACKUP.md

Backups and disaster recovery

Two different things need backing up:

1. Instance backups — game saves/worlds, made by the panel itself.
2. Panel state — the Postgres database + the controller's data/ directory. The panel does NOT back these up for you.

Instance backups (panel-side)

POST /api/instances/{id}/backups (dashboard Backups tab, or panelctl backup create <agent-id> <instance-id> [description]) spawns a tar-sidecar container that mounts the instance's volumes and writes a timestamped tarball to the agent's --backup-dir (default ./data/backups/<instance-id>/ on the agent host — not the controller, unless they're the same box).

Restore stops the game container first, then reverses the flow: dashboard, or panelctl backup restore <instance-id> <backup-id>. panelctl backup list <instance-id> lists.

Scheduling: the scheduler has a backup action type, so a nightly auto-backup is one schedule row:

{
  "instance_id": "mc-1",
  "trigger_kind": "cron",
  "cron_spec": "0 0 4 * * *",
  "action": {"type": "backup", "description": "Nightly auto-backup"}
}

Retention is manual. Nothing prunes old tarballs — list, pick, delete (UI or DELETE /api/instances/{id}/backups/{bkpId}, which removes the DB row). Watch your disk if you schedule backups.

Instance backup tarballs live outside Postgres; copy data/backups/ off-host if you want them to survive the machine.

Panel state: what to save

What	Where	Why
Postgres dump	pg_dump (below)	users, agents, instances, assigned ports, schedules, backup index, Steam credential ciphertext
data/ca/	controller host	internal CA — all agent certs chain to it, and the Steam-credential encryption key is derived from it. Losing it = re-pairing every agent + re-entering Steam creds.
data/admin-token	controller host	out-of-band admin auth (regenerable, but scripts may pin it)
data/certs/	each agent host	that agent's cert (regenerable by re-pairing)
data/instance-meta/	each agent host	rehydrated from the controller on connect — no need to back up
data/instances/ + named volumes	each agent host	the actual game data — covered by instance backups above, or copy raw

pg_dump procedure

Managed panel-postgres container (from install.sh / compose):

docker exec panel-postgres pg_dump -U panel -d panel -Fc \
  > panel-$(date +%F).dump

External Postgres: pg_dump -h dbhost -U panel -d panel -Fc > panel.dump.

Cron it; the dump is small (panel metadata, not game data).

DR runbook — restore to a new host

Assumes you have: a Postgres dump, a copy of the controller's data/ directory (at minimum data/ca/), and (optionally) instance backup tarballs / raw instance data from the agent hosts.

1. Fresh install, controller only:

   PANEL_SKIP_AGENT=1 bash install.sh
   sudo systemctl stop panel-controller
   

2. Restore the database (drop the schema the fresh boot created):

   docker exec -i panel-postgres psql -U panel -d postgres \
     -c 'DROP DATABASE panel;' -c 'CREATE DATABASE panel OWNER panel;'
   docker exec -i panel-postgres pg_restore -U panel -d panel < panel-YYYY-MM-DD.dump
   

3. Restore data/ over the fresh one — data/ca/ is the critical piece; keep ownership (chown -R panel:panel /opt/panel/data).
4. Start the controller: sudo systemctl start panel-controller. Because the users table is non-empty, no new admin is bootstrapped — your old logins work.
5. Agents:
   • Same agent hosts, new controller IP/name → update --controller in their units; their existing certs still chain to the restored CA. If the controller's hostname changed, mint the server cert with the new name (-tls-hostnames) — the CA is what matters.
   • New agent hosts → pair them fresh (PAIRING.md) using the same agent-id as before so instance rows re-attach, then restore game data into --data-root / named volumes (or use the panel's Restore on each instance from copied-in tarballs under --backup-dir/<instance-id>/).
6. Verify: dashboard lists agents as connected, instances present; start one instance and check its world loaded.

What is NOT covered

• No automatic offsite copies — everything above is manual/cron.
• Instance backup tarballs sit on the same disk as the instances by default; point --backup-dir at other storage if that worries you.
• Steam Guard sentry blobs restore with the DB + data/ca; if either is lost you'll redo the Steam login ceremony (ADMIN.md).