# Backups and disaster recovery

Two different things need backing up:

1. **Instance backups** — game saves/worlds, made by the panel itself.
2. **Panel state** — the Postgres database + the controller's `data/`
   directory. The panel does NOT back these up for you.

## Instance backups (panel-side)

`POST /api/instances/{id}/backups` (dashboard Backups tab, or
`panelctl backup create <agent-id> <instance-id> [description]`) spawns
a tar-sidecar container that mounts the instance's volumes and writes a
timestamped tarball to the **agent's** `--backup-dir` (default
`./data/backups/<instance-id>/` on the agent host — not the controller,
unless they're the same box).

Restore stops the game container first, then reverses the flow:
dashboard, or `panelctl backup restore <instance-id> <backup-id>`.
`panelctl backup list <instance-id>` lists.

**Scheduling:** the scheduler has a `backup` action type, so a nightly
auto-backup is one schedule row:

```json
{
  "instance_id": "mc-1",
  "trigger_kind": "cron",
  "cron_spec": "0 0 4 * * *",
  "action": {"type": "backup", "description": "Nightly auto-backup"}
}
```

**Retention is manual.** Nothing prunes old tarballs — list, pick,
delete (UI or `DELETE /api/instances/{id}/backups/{bkpId}`, which
removes the DB row). Watch your disk if you schedule backups.

Instance backup tarballs live outside Postgres; copy
`data/backups/` off-host if you want them to survive the machine.

## Panel state: what to save

| What | Where | Why |
|---|---|---|
| Postgres dump | `pg_dump` (below) | users, agents, instances, assigned ports, schedules, backup index, Steam credential ciphertext |
| `data/ca/` | controller host | internal CA — all agent certs chain to it, and the Steam-credential encryption key is derived from it. **Losing it = re-pairing every agent + re-entering Steam creds.** |
| `data/admin-token` | controller host | out-of-band admin auth (regenerable, but scripts may pin it) |
| `data/certs/` | each agent host | that agent's cert (regenerable by re-pairing) |
| `data/instance-meta/` | each agent host | rehydrated from the controller on connect — no need to back up |
| `data/instances/` + named volumes | each agent host | the actual game data — covered by instance backups above, or copy raw |

## pg_dump procedure

Managed `panel-postgres` container (from `install.sh` / compose):

```bash
docker exec panel-postgres pg_dump -U panel -d panel -Fc \
  > panel-$(date +%F).dump
```

External Postgres: `pg_dump -h dbhost -U panel -d panel -Fc > panel.dump`.

Cron it; the dump is small (panel metadata, not game data).

## DR runbook — restore to a new host

Assumes you have: a Postgres dump, a copy of the controller's `data/`
directory (at minimum `data/ca/`), and (optionally) instance backup
tarballs / raw instance data from the agent hosts.

1. **Fresh install, controller only:**
   ```bash
   PANEL_SKIP_AGENT=1 bash install.sh
   sudo systemctl stop panel-controller
   ```
2. **Restore the database** (drop the schema the fresh boot created):
   ```bash
   docker exec -i panel-postgres psql -U panel -d postgres \
     -c 'DROP DATABASE panel;' -c 'CREATE DATABASE panel OWNER panel;'
   docker exec -i panel-postgres pg_restore -U panel -d panel < panel-YYYY-MM-DD.dump
   ```
3. **Restore `data/`** over the fresh one — `data/ca/` is the critical
   piece; keep ownership (`chown -R panel:panel /opt/panel/data`).
4. **Start the controller:** `sudo systemctl start panel-controller`.
   Because the users table is non-empty, no new admin is bootstrapped —
   your old logins work.
5. **Agents:**
   - Same agent hosts, new controller IP/name → update `--controller`
     in their units; their existing certs still chain to the restored CA.
     If the controller's *hostname* changed, mint the server cert with
     the new name (`-tls-hostnames`) — the CA is what matters.
   - New agent hosts → pair them fresh ([PAIRING.md](PAIRING.md)) using
     the **same agent-id** as before so instance rows re-attach, then
     restore game data into `--data-root` / named volumes (or use the
     panel's Restore on each instance from copied-in tarballs under
     `--backup-dir/<instance-id>/`).
6. **Verify:** dashboard lists agents as connected, instances present;
   start one instance and check its world loaded.

## What is NOT covered

- No automatic offsite copies — everything above is manual/cron.
- Instance backup tarballs sit on the same disk as the instances by
  default; point `--backup-dir` at other storage if that worries you.
- Steam Guard sentry blobs restore with the DB + `data/ca`; if either is
  lost you'll redo the Steam login ceremony ([ADMIN.md](ADMIN.md)).