panel v0.9.2 — open-source game server manager

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 01:06:16 -07:00
commit 0f6aea796c
2164 changed files with 301480 additions and 0 deletions
@@ -0,0 +1,106 @@
+# Backups and disaster recovery
+
+Two different things need backing up:
+
+1. **Instance backups** — game saves/worlds, made by the panel itself.
+2. **Panel state** — the Postgres database + the controller's `data/`
+   directory. The panel does NOT back these up for you.
+
+## Instance backups (panel-side)
+
+`POST /api/instances/{id}/backups` (dashboard Backups tab, or
+`panelctl backup create <agent-id> <instance-id> [description]`) spawns
+a tar-sidecar container that mounts the instance's volumes and writes a
+timestamped tarball to the **agent's** `--backup-dir` (default
+`./data/backups/<instance-id>/` on the agent host — not the controller,
+unless they're the same box).
+
+Restore stops the game container first, then reverses the flow:
+dashboard, or `panelctl backup restore <instance-id> <backup-id>`.
+`panelctl backup list <instance-id>` lists.
+
+**Scheduling:** the scheduler has a `backup` action type, so a nightly
+auto-backup is one schedule row:
+
+```json
+{
+  "instance_id": "mc-1",
+  "trigger_kind": "cron",
+  "cron_spec": "0 0 4 * * *",
+  "action": {"type": "backup", "description": "Nightly auto-backup"}
+}
+```
+
+**Retention is manual.** Nothing prunes old tarballs — list, pick,
+delete (UI or `DELETE /api/instances/{id}/backups/{bkpId}`, which
+removes the DB row). Watch your disk if you schedule backups.
+
+Instance backup tarballs live outside Postgres; copy
+`data/backups/` off-host if you want them to survive the machine.
+
+## Panel state: what to save
+
+| What | Where | Why |
+|---|---|---|
+| Postgres dump | `pg_dump` (below) | users, agents, instances, assigned ports, schedules, backup index, Steam credential ciphertext |
+| `data/ca/` | controller host | internal CA — all agent certs chain to it, and the Steam-credential encryption key is derived from it. **Losing it = re-pairing every agent + re-entering Steam creds.** |
+| `data/admin-token` | controller host | out-of-band admin auth (regenerable, but scripts may pin it) |
+| `data/certs/` | each agent host | that agent's cert (regenerable by re-pairing) |
+| `data/instance-meta/` | each agent host | rehydrated from the controller on connect — no need to back up |
+| `data/instances/` + named volumes | each agent host | the actual game data — covered by instance backups above, or copy raw |
+
+## pg_dump procedure
+
+Managed `panel-postgres` container (from `install.sh` / compose):
+
+```bash
+docker exec panel-postgres pg_dump -U panel -d panel -Fc \
+  > panel-$(date +%F).dump
+```
+
+External Postgres: `pg_dump -h dbhost -U panel -d panel -Fc > panel.dump`.
+
+Cron it; the dump is small (panel metadata, not game data).
+
+## DR runbook — restore to a new host
+
+Assumes you have: a Postgres dump, a copy of the controller's `data/`
+directory (at minimum `data/ca/`), and (optionally) instance backup
+tarballs / raw instance data from the agent hosts.
+
+1. **Fresh install, controller only:**
+   ```bash
+   PANEL_SKIP_AGENT=1 bash install.sh
+   sudo systemctl stop panel-controller
+   ```
+2. **Restore the database** (drop the schema the fresh boot created):
+   ```bash
+   docker exec -i panel-postgres psql -U panel -d postgres \
+     -c 'DROP DATABASE panel;' -c 'CREATE DATABASE panel OWNER panel;'
+   docker exec -i panel-postgres pg_restore -U panel -d panel < panel-YYYY-MM-DD.dump
+   ```
+3. **Restore `data/`** over the fresh one — `data/ca/` is the critical
+   piece; keep ownership (`chown -R panel:panel /opt/panel/data`).
+4. **Start the controller:** `sudo systemctl start panel-controller`.
+   Because the users table is non-empty, no new admin is bootstrapped —
+   your old logins work.
+5. **Agents:**
+   - Same agent hosts, new controller IP/name → update `--controller`
+     in their units; their existing certs still chain to the restored CA.
+     If the controller's *hostname* changed, mint the server cert with
+     the new name (`-tls-hostnames`) — the CA is what matters.
+   - New agent hosts → pair them fresh ([PAIRING.md](PAIRING.md)) using
+     the **same agent-id** as before so instance rows re-attach, then
+     restore game data into `--data-root` / named volumes (or use the
+     panel's Restore on each instance from copied-in tarballs under
+     `--backup-dir/<instance-id>/`).
+6. **Verify:** dashboard lists agents as connected, instances present;
+   start one instance and check its world loaded.
+
+## What is NOT covered
+
+- No automatic offsite copies — everything above is manual/cron.
+- Instance backup tarballs sit on the same disk as the instances by
+  default; point `--backup-dir` at other storage if that worries you.
+- Steam Guard sentry blobs restore with the DB + `data/ca`; if either is
+  lost you'll redo the Steam login ceremony ([ADMIN.md](ADMIN.md)).