107 lines
4.6 KiB
Markdown
107 lines
4.6 KiB
Markdown
# Backups and disaster recovery
|
|
|
|
Two different things need backing up:
|
|
|
|
1. **Instance backups** — game saves/worlds, made by the panel itself.
|
|
2. **Panel state** — the Postgres database + the controller's `data/`
|
|
directory. The panel does NOT back these up for you.
|
|
|
|
## Instance backups (panel-side)
|
|
|
|
`POST /api/instances/{id}/backups` (dashboard Backups tab, or
|
|
`panelctl backup create <agent-id> <instance-id> [description]`) spawns
|
|
a tar-sidecar container that mounts the instance's volumes and writes a
|
|
timestamped tarball to the **agent's** `--backup-dir` (default
|
|
`./data/backups/<instance-id>/` on the agent host — not the controller,
|
|
unless they're the same box).
|
|
|
|
Restore stops the game container first, then reverses the flow:
|
|
dashboard, or `panelctl backup restore <instance-id> <backup-id>`.
|
|
`panelctl backup list <instance-id>` lists.
|
|
|
|
**Scheduling:** the scheduler has a `backup` action type, so a nightly
|
|
auto-backup is one schedule row:
|
|
|
|
```json
|
|
{
|
|
"instance_id": "mc-1",
|
|
"trigger_kind": "cron",
|
|
"cron_spec": "0 0 4 * * *",
|
|
"action": {"type": "backup", "description": "Nightly auto-backup"}
|
|
}
|
|
```
|
|
|
|
**Retention is manual.** Nothing prunes old tarballs — list, pick,
|
|
delete (UI or `DELETE /api/instances/{id}/backups/{bkpId}`, which
|
|
removes the DB row). Watch your disk if you schedule backups.
|
|
|
|
Instance backup tarballs live outside Postgres; copy
|
|
`data/backups/` off-host if you want them to survive the machine.
|
|
|
|
## Panel state: what to save
|
|
|
|
| What | Where | Why |
|
|
|---|---|---|
|
|
| Postgres dump | `pg_dump` (below) | users, agents, instances, assigned ports, schedules, backup index, Steam credential ciphertext |
|
|
| `data/ca/` | controller host | internal CA — all agent certs chain to it, and the Steam-credential encryption key is derived from it. **Losing it = re-pairing every agent + re-entering Steam creds.** |
|
|
| `data/admin-token` | controller host | out-of-band admin auth (regenerable, but scripts may pin it) |
|
|
| `data/certs/` | each agent host | that agent's cert (regenerable by re-pairing) |
|
|
| `data/instance-meta/` | each agent host | rehydrated from the controller on connect — no need to back up |
|
|
| `data/instances/` + named volumes | each agent host | the actual game data — covered by instance backups above, or copy raw |
|
|
|
|
## pg_dump procedure
|
|
|
|
Managed `panel-postgres` container (from `install.sh` / compose):
|
|
|
|
```bash
|
|
docker exec panel-postgres pg_dump -U panel -d panel -Fc \
|
|
> panel-$(date +%F).dump
|
|
```
|
|
|
|
External Postgres: `pg_dump -h dbhost -U panel -d panel -Fc > panel.dump`.
|
|
|
|
Cron it; the dump is small (panel metadata, not game data).
|
|
|
|
## DR runbook — restore to a new host
|
|
|
|
Assumes you have: a Postgres dump, a copy of the controller's `data/`
|
|
directory (at minimum `data/ca/`), and (optionally) instance backup
|
|
tarballs / raw instance data from the agent hosts.
|
|
|
|
1. **Fresh install, controller only:**
|
|
```bash
|
|
PANEL_SKIP_AGENT=1 bash install.sh
|
|
sudo systemctl stop panel-controller
|
|
```
|
|
2. **Restore the database** (drop the schema the fresh boot created):
|
|
```bash
|
|
docker exec -i panel-postgres psql -U panel -d postgres \
|
|
-c 'DROP DATABASE panel;' -c 'CREATE DATABASE panel OWNER panel;'
|
|
docker exec -i panel-postgres pg_restore -U panel -d panel < panel-YYYY-MM-DD.dump
|
|
```
|
|
3. **Restore `data/`** over the fresh one — `data/ca/` is the critical
|
|
piece; keep ownership (`chown -R panel:panel /opt/panel/data`).
|
|
4. **Start the controller:** `sudo systemctl start panel-controller`.
|
|
Because the users table is non-empty, no new admin is bootstrapped —
|
|
your old logins work.
|
|
5. **Agents:**
|
|
- Same agent hosts, new controller IP/name → update `--controller`
|
|
in their units; their existing certs still chain to the restored CA.
|
|
If the controller's *hostname* changed, mint the server cert with
|
|
the new name (`-tls-hostnames`) — the CA is what matters.
|
|
- New agent hosts → pair them fresh ([PAIRING.md](PAIRING.md)) using
|
|
the **same agent-id** as before so instance rows re-attach, then
|
|
restore game data into `--data-root` / named volumes (or use the
|
|
panel's Restore on each instance from copied-in tarballs under
|
|
`--backup-dir/<instance-id>/`).
|
|
6. **Verify:** dashboard lists agents as connected, instances present;
|
|
start one instance and check its world loaded.
|
|
|
|
## What is NOT covered
|
|
|
|
- No automatic offsite copies — everything above is manual/cron.
|
|
- Instance backup tarballs sit on the same disk as the instances by
|
|
default; point `--backup-dir` at other storage if that worries you.
|
|
- Steam Guard sentry blobs restore with the DB + `data/ca`; if either is
|
|
lost you'll redo the Steam login ceremony ([ADMIN.md](ADMIN.md)).
|