panel v0.9.1 — open-source game server manager

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 23:25:11 -07:00
commit 03a281d009
2161 changed files with 300880 additions and 0 deletions
@@ -0,0 +1,823 @@
+# panel
+
+An open-source, modular game-server management panel — AMP-class features,
+distributed Controller/Agent architecture, CEL-driven automation, full
+instance lifecycle (install → run → update → backup → restore), live
+dashboard, and CLI.
+
+Written in Go. Single static binary per component. Docker-backed runtime.
+26 modules ship in-tree, including the awkward Windows-only games
+(Conan Exiles, Empyrion, V Rising, Sons of the Forest, DayZ, Soulmask,
+Project Zomboid, Windrose) under Wine + Xvfb. The agent **auto-builds**
+every module's Docker image from its in-tree `Dockerfile` the first time
+an instance needs it — there is no manual per-module `docker build` step.
+
+## 60-second quickstart
+
+One box, one line. Docker is installed for you if missing; Postgres runs
+as a managed container; controller + agent become systemd units (Linux)
+or Scheduled Tasks (Windows), and the local agent is auto-paired over mTLS.
+
+```bash
+# Linux (Debian/Ubuntu/Fedora/RHEL/Raspbian) — run as root:
+curl -fsSL https://raw.githubusercontent.com/dbledeez/panel/main/install.sh | sudo bash
+```
+
+```powershell
+# Windows (Docker Desktop must already be running):
+powershell -ExecutionPolicy Bypass -File install.ps1
+```
+
+Then open `http://<host>:8080/` and log in with the `ADMIN BOOTSTRAP`
+credentials printed in the controller log (or pre-seed via
+`PANEL_ADMIN_PASSWORD`). Full options, docker-compose, and from-source
+paths: [docs/INSTALL.md](docs/INSTALL.md).
+
+```
+                    ┌─────────────────────────┐
+                    │       Operator          │
+                    │  (browser / panelctl)   │
+                    └────────────┬────────────┘
+                                 │ HTTPS / gRPC
+                    ┌────────────▼────────────┐
+                    │       Controller        │
+                    │  • Web dashboard :8080  │
+                    │  • Agent gRPC    :8443  │
+                    │  • Postgres             │
+                    │  • Scheduler (cron+CEL) │
+                    │  • Event bus + SSE      │
+                    │  • Session auth         │
+                    └────────────┬────────────┘
+                                 │ gRPC bidi (mTLS)
+            ┌────────────────────┼────────────────────┐
+            │                    │                    │
+   ┌────────▼────────┐  ┌────────▼────────┐  ┌────────▼────────┐
+   │    Agent A      │  │    Agent B      │  │    Agent C      │
+   │  • Docker       │  │  • Docker       │  │  • Docker       │
+   │  • Modules      │  │  • Modules      │  │  • Modules      │
+   │  • RCON clients │  │  • RCON clients │  │  • RCON clients │
+   │  • State tracker│  │  • State tracker│  │  • State tracker│
+   └─────────────────┘  └─────────────────┘  └─────────────────┘
+```
+
+---
+
+## Documentation
+
+| Doc | Covers |
+|---|---|
+| [docs/INSTALL.md](docs/INSTALL.md) | Installers, docker-compose, from-source, flags reference, systemd, reverse proxy + TLS, firewall |
+| [docs/NETWORKING.md](docs/NETWORKING.md) | Per-game port table, router port-forwarding, what the panel auto-allocates |
+| [docs/PAIRING.md](docs/PAIRING.md) | mTLS agent pairing, multi-host setup |
+| [docs/ADMIN.md](docs/ADMIN.md) | First-admin bootstrap, password reset, Steam sign-in, Steam credentials for paid games |
+| [docs/BACKUP.md](docs/BACKUP.md) | Backups, pg_dump, disaster-recovery runbook |
+| [docs/architecture.md](docs/architecture.md) | The real stack, component by component |
+
+## Table of contents
+
+- [Status](#status)
+- [Why panel](#why-panel)
+- [Architecture](#architecture)
+- [Quick start (dev)](#quick-start-dev)
+- [Production-ish layout](#production-ish-layout)
+- [Modules](#modules)
+- [Module manifest](#module-manifest)
+- [Wine/Xvfb modules](#winexvfb-modules)
+- [`panelctl` CLI](#panelctl-cli)
+- [HTTP API](#http-api)
+- [Admin HTTP (out-of-band repair)](#admin-http-out-of-band-repair)
+- [Scheduler](#scheduler)
+- [Mods & Workshop](#mods--workshop)
+- [Backups](#backups)
+- [File manager](#file-manager)
+- [Repository layout](#repository-layout)
+- [Data directory layout](#data-directory-layout)
+- [Build](#build)
+- [Working with the embedded UI](#working-with-the-embedded-ui)
+- [Troubleshooting](#troubleshooting)
+- [Contributing / writing a new module](#contributing--writing-a-new-module)
+- [License](#license)
+
+---
+
+## Status
+
+panel is a working private deployment driving real game servers — not a
+toy. Feature completeness varies by game; the headline state is:
+
+- ✅ Manifest-driven module plugins (26 shipped — see [Modules](#modules))
+- ✅ mTLS agent pairing: one-time token minted in the UI → `agent
+  --pair-token` fetches the CA + a signed cert and exits; the agent then
+  connects with mTLS and rogue agents are rejected at the TLS handshake
+  (see [docs/PAIRING.md](docs/PAIRING.md))
+- ✅ Agent auto-builds `panel-*` module images from the in-tree Dockerfiles
+  on first use — pull-through for third-party images, no manual builds
+- ✅ Distributed Controller + Agent over gRPC (bidi streams, reconnect +
+  rehydrate, sidecar metadata survives restarts)
+- ✅ Docker runtime with bind-mount and named-volume support, host
+  networking, per-instance port allocation
+- ✅ Wine + Xvfb pattern for Windows-only dedicated servers (Empyrion,
+  Conan Exiles, V Rising, Sons of the Forest, DayZ, Soulmask, Project
+  Zomboid, Windrose, Dragonwilds)
+- ✅ Full state tracking: RCON polling + log-line event matching → typed
+  PlayerEvent + AppState bus
+- ✅ Telnet **and** Source RCON adapters (Minecraft, CS-derived games,
+  Rust, Valheim w/ plugins, 7DTD, Conan, etc.) with redial-on-EOF
+- ✅ Scheduler: 6-field cron triggers **and** CEL event triggers with
+  sustained-for windows
+- ✅ Config-file templating with auto-generated secrets; per-game custom
+  Config tabs over `ServerSettings.ini` / `GameUserSettings.ini` /
+  `serverconfig.xml` / properties / JSON
+- ✅ File manager: web UI browses container volumes via `docker exec` +
+  `docker cp`, bind-mount fallback, path-traversal defended, multi-root
+  per module
+- ✅ Update providers: **Direct HTTP**, **GitHub releases**, **SteamCMD
+  sidecar** (with platform overrides for Windows-on-Linux), **Steam beta
+  branches** (e.g. Conan Legacy UE4, Conan Enhanced UE5)
+- ✅ Backup / restore via tar-sidecar containers
+- ✅ Postgres persistence (agents, instances, schedules, backups, users)
+- ✅ Local auth: argon2id + session cookies, admin bootstrap, change
+  password, optional Sign-in-with-Steam linking
+- ✅ Live event bus with SSE stream (`Last-Event-ID` resume + snapshot) +
+  `panelctl watch`
+- ✅ Template Library: 236-entry game catalog (`static/catalog.json`,
+  distilled from CubeCoders' MIT templates by `tools/amp-distill`) with an
+  admin one-click module scaffolder (`POST /api/scaffold-module`)
+- ✅ Update-check (check-only): agent compares installed Steam build id vs
+  `app_info` (15-min cache) → Updates-tab panel + card badge
+- ✅ Prometheus `/metrics` (admin-token / loopback gated)
+- ✅ Mods tab: ARK CurseForge / DayZ Workshop / Conan Workshop / V Rising
+  PlasmaCore / Empyrion scenarios — search, install, version-pin
+- ✅ ARK SA cluster registry with cross-server transfer + admin rebind
+- 🟡 Cross-platform: Linux works for everything except modules that
+  require nested KVM. Primary dev host is Windows + Docker Desktop.
+- ❌ OIDC / WebAuthn / Casbin ACL (local auth only; role = admin / user)
+- ❌ SFTP (web FM covers the common case)
+- ❌ Host-mode runtime (cgroups v2 / Windows Job Objects)
+
+## Why panel
+
+Pterodactyl/Pelican is good but Docker-only and PHP, AMP is closed-source
+and per-machine, GameServerApp and friends are SaaS. panel is what falls
+out if you want:
+
+- One static Go binary per component, no Composer / npm / Node runtime
+  on the host.
+- Modules as plain YAML — drop a `module.yaml` in `modules/<id>/`,
+  restart the agent, the game appears in the dashboard.
+- Real distributed multi-host: Controller is one box, Agents run
+  wherever Docker runs.
+- A scheduler that can react to *the world* — "stop the server if
+  it's been empty for 30 minutes", "restart at 3am only on Tuesdays" —
+  rather than just cron.
+- First-class Wine/Xvfb support: Windows-only dedicated servers run on
+  Linux Docker hosts without per-game artisanal scripting.
+- Out-of-band repair via a token-authed `/admin/v1/*` HTTP surface —
+  the dashboard breaks, you SSH in, and `panelctl admin recreate` /
+  `set-config` / `ark-rebind` still work.
+
+Designed for someone running ~10 game servers across 2-4 boxes for
+themselves and a couple of communities, not for a hosting company. No
+multi-tenancy, no billing.
+
+## Architecture
+
+Two long-running components plus optional sidecars.
+
+**Controller** — `./controller/cmd/controller`
+- HTTP :8080 (configurable) — dashboard, REST API, SSE event stream
+- gRPC :8443 — bidi stream from each Agent
+- Postgres for durable state (no SQLite default — explicitly not sharded)
+- Embedded migrations, embedded static dashboard (default UI =
+  `static/new.html` shell + `static/rcc.js` + `static/rcc.css`, served
+  hash-stamped with immutable caching via `//go:embed`; the legacy
+  single-file `static/index.html` is the frozen "classic" UI, opt-in via
+  the `panel_ui=classic` cookie)
+- Scheduler in-process (cron + CEL)
+- Prometheus `GET /metrics` (admin-token or loopback gated)
+
+**Agent** — `./agent/cmd/agent`
+- Dials Controller, stays connected with reconnect + heartbeat
+- Loads modules from `--modules-dir` at boot (default `./modules`)
+- Drives the local Docker daemon (`/var/run/docker.sock` or
+  `npipe:////./pipe/docker_engine` on Windows)
+- Spawns short-lived sidecar containers for SteamCMD updates and tar
+  backup/restore
+- Per-instance state tracker: RCON polling, log-line tail, event
+  pattern matching → typed events streamed back to Controller
+
+**Sidecars (per-game)**
+- `steamcmd-sidecar` — SteamCMD with `+force_install_dir` mounted at
+  the instance's game-content volume; supports `+@sSteamCmdForcePlatformType
+  windows` for cross-platform Windows depots, beta branch pinning, and
+  Workshop downloads.
+- `tar-sidecar` — exec'd against the volumes for backup/restore so the
+  game container doesn't have to expose its filesystem.
+
+**RCON adapters** (`agent/internal/rcon/`)
+- `source_rcon` — Source-RCON wire protocol (Minecraft Java, Conan,
+  Factorio, Palworld, Project Zomboid)
+- `telnet` — Telnet-style password gate then line protocol (7DTD, Empyrion)
+- `websocket_rcon` — Rust's WebRCON
+- `stdio` — attach to the container's stdin (Terraria, Valheim, Vein,
+  Enshrouded)
+- `be_rcon` — BattlEye RCON over UDP (DayZ)
+- `docker_exec_rcon` — exec a CLI inside the container (ARK SA)
+
+All adapters auto-redial on EOF (Conan and Palworld lessons baked in).
+
+## Quick start (dev)
+
+Prereqs: Go 1.26+, Docker Desktop (Windows) or Docker Engine (Linux),
+[`buf`](https://buf.build/docs/installation/) on PATH for proto stubs.
+
+```bash
+# 1) Postgres (single dev container)
+docker compose -f docker-compose.dev.yml up -d
+
+# 2) Generate protobuf stubs (uses remote plugins — no local protoc needed)
+buf generate
+
+# 3) Start the controller. First run prints an ADMIN BOOTSTRAP banner
+#    with the generated admin password — copy it out of stderr.
+go run ./controller/cmd/controller
+
+# 4) Start an agent (any host with Docker + reachable to controller)
+go run ./agent/cmd/agent --insecure \
+  --modules-dir=./modules \
+  --data-root=./data/instances \
+  --meta-dir=./data/instance-meta \
+  --backup-dir=./data/backups
+
+# 5) Open http://localhost:8080/, log in as admin@panel.local with
+#    the bootstrap password.
+```
+
+If you'd rather pre-seed the password instead of reading it from logs:
+
+```bash
+PANEL_ADMIN_PASSWORD=changeme go run ./controller/cmd/controller
+```
+
+## Production-ish layout
+
+This is the layout the maintainer runs. Not prescriptive — adapt freely.
+
+- One **Controller** box with Postgres co-located. Reverse-proxied
+  behind Nginx Proxy Manager / Caddy / Cloudflare. Public dashboard
+  URL has a real cert; gRPC :8443 is on the LAN only.
+- N **Agent** boxes, each with Docker. Agents don't need to be reachable
+  from the internet — they dial out to the Controller's gRPC port.
+- A `data/` directory per machine (Controller + each Agent) under
+  whatever path you want. **Don't put it in the repo working tree** —
+  `data/` is gitignored for a reason: it holds per-instance state,
+  RCON passwords, mTLS material, and game saves.
+- One systemd unit per binary on Linux, one Service on Windows
+  (NSSM works fine).
+
+The agent is intentionally stateless beyond `data/instance-meta/*.json`
+(rehydrated from the Controller on connect). Wipe the meta directory
+and the agent rebuilds it from the Controller's authoritative DB.
+
+## Modules
+
+Game modules live under `modules/<id>/` and are picked up by every
+agent at boot. `panel-*` images are **auto-built by the agent** from the
+module's in-tree `Dockerfile` the first time an instance needs them
+(they are never published to a registry); third-party images are pulled. Each ships with a `module.yaml` and (where needed) an
+`entrypoint.sh` and `Dockerfile`.
+
+| Module | Image / build | RCON | Notes |
+|---|---|---|---|
+| `7dtd` | `panel-7dtd` (auto-built) | telnet | Canonical-nim freeze, cluster playbook |
+| `ark-sa` | `acekorneya/asa_server:latest` | `docker_exec_rcon` | Cluster registry, CurseForge mods, transfer flow |
+| `barotrauma` | `panel-barotrauma` (auto-built) | log only | Workshop mods |
+| `conan-exiles` | `panel-conan-exiles` (auto-built, Wine) | `source_rcon` | UE4 Legacy / UE5 Enhanced edition picker, Workshop |
+| `core-keeper` | `panel-core-keeper` (auto-built) | log only | |
+| `dayz` | `panel-dayz` (auto-built, Wine) | `be_rcon` | Steam Workshop manager; `requires_steam_login` |
+| `demo` | `alpine:3.21` | none | Lifecycle / event sandbox |
+| `dragonwilds` | `panel-dragonwilds` (auto-built, Wine) | log only | |
+| `empyrion` | `panel-empyrion` (auto-built, Wine) | telnet | Custom scenario manager, EAH bridge, discovery streamer |
+| `empyrion-bridge` | multi-stage dotnet (auto-built) | bridge | EWA gateway sidecar |
+| `enshrouded` | `panel-enshrouded` (auto-built) | stdio | |
+| `factorio` | `panel-factorio` (auto-built) | `source_rcon` | |
+| `minecraft-bedrock` | `itzg/minecraft-bedrock-server` | log only | |
+| `minecraft-java` | `itzg/minecraft-server:latest` | `source_rcon` | |
+| `palworld` | `panel-palworld` (auto-built) | `source_rcon` | RCON redial-on-EOF |
+| `project-zomboid` | `panel-project-zomboid` (auto-built, Wine) | `source_rcon` | |
+| `rust` | `panel-rust` (auto-built) | `websocket_rcon` | Rust+ port |
+| `satisfactory` | `panel-satisfactory` (auto-built) | none | |
+| `sons-of-the-forest` | `panel-sons-of-the-forest` (auto-built, Wine) | log only | |
+| `soulmask` | `panel-soulmask` (auto-built, Wine) | log only | |
+| `steamcmd-test` | `alpine:3.21` + SteamCMD | none | HLDS smoke test (~930 MB) |
+| `terraria` | `panel-terraria` (auto-built) | stdio | |
+| `v-rising` | `panel-v-rising` (auto-built, Wine) | log only | PlasmaCore mods |
+| `valheim` | `panel-valheim` (auto-built) | stdio | |
+| `vein` | `panel-vein` (auto-built) | stdio | |
+| `windrose` | `panel-windrose` (auto-built, Wine) | log only | |
+
+Per-game gotchas live in [`memory/gotchas.md`](memory/gotchas.md) — read
+that before debugging any specific game.
+
+## Module manifest
+
+Every module is one declarative YAML file. The schema is in
+[`pkg/module/manifest.go`](pkg/module/manifest.go). A condensed example:
+
+```yaml
+id: my-game
+name: "My Game"
+version: 0.1.0
+supported_modes: [docker]
+
+runtime:
+  docker:
+    image: itzg/my-game-server:latest        # or local-built image tag
+    network_mode: host                       # optional
+    browseable_root: /data                    # File-manager root
+    browseable_roots:                         # Optional multi-root
+      - { name: "Saves", path: /data, hint: "world data" }
+      - { name: "Game",  path: /game, hint: "engine binaries" }
+    env:                                      # Pre-declare every env key
+      MAX_PLAYERS: "20"                       #   you want forwarded;
+      ADMIN_PASSWORD: ""                      #   resolve.go filters by
+    volumes:                                  #   this map.
+      - { type: volume, name: "panel-$INSTANCE_ID-data", container: /data }
+      - { target: "$DATA_PATH/logs", container: /log }   # bind mount
+
+ports:
+  - { name: game,  proto: tcp, default: 25565, required: true,  env: GAME_PORT }
+  - { name: query, proto: udp, default: 25566, required: false, env: QUERY_PORT }
+  - { name: rcon,  proto: tcp, default: 25575, internal: true,  env: RCON_PORT }
+
+resources:
+  min_ram_mb: 2048
+  recommended_ram_mb: 4096
+
+rcon:
+  adapter: source_rcon         # or telnet, stdio, websocket_rcon, be_rcon, docker_exec_rcon
+  host_port: rcon
+  password_secret: RCON_PASSWORD
+  password_literal: "fallback"
+  commands:
+    list:      "list"
+    broadcast: "say {msg}"
+    kick:      "kick {player}"
+    save:      "save-all"
+    shutdown:  "stop"
+
+state_sources:
+  - type: rcon
+    command: list
+    every: 60s
+    parse:
+      kind: regex
+      pattern: '(?P<n>\d+)\s+of\s+(?P<m>\d+) players'
+      fields: { players_online: 1, players_max: 2 }
+
+events:
+  join:  { pattern: "(?P<name>\\S+) joined", kind: join }
+  leave: { pattern: "(?P<name>\\S+) left",   kind: leave }
+  chat:  { pattern: "<(?P<name>[^>]+)> (?P<msg>.+)", kind: chat }
+
+config_files:
+  - path: server.properties
+    format: properties
+    template: templates/server.properties.tmpl
+
+config_values:                  # Drive the Config tab in the dashboard
+  - key: MAX_PLAYERS
+    label: "Max players"
+    description: "Player slot limit."
+    default: "20"
+  - key: ADMIN_PASSWORD
+    label: "Admin password"
+    description: "In-game admin auth."
+    default: ""
+
+secrets:
+  - { name: admin_token, generated: true }
+
+update_providers:
+  - id: stable
+    kind: steamcmd
+    app_id: "294420"
+    install_path: /data
+    skip_validate: false        # Set true for app_ids whose validate hangs
+```
+
+Key rules the loader enforces:
+
+- Only env keys pre-declared under `runtime.docker.env` get forwarded
+  to the container. Adding a `config_values` entry without the matching
+  env declaration silently drops the value — this trips people up.
+- `volumes:` entries use `$INSTANCE_ID` and `$DATA_PATH` substitutions
+  resolved by the agent at create time.
+- `internal: true` ports are not exposed by the allocator but still get
+  their `env:` substituted.
+- `update_providers` are picked from a dropdown in the UI; multiple
+  providers per module is fine (Conan ships `enhanced` and `legacy`).
+
+## Wine/Xvfb modules
+
+The Windows-only dedicated servers (Empyrion, Conan, V Rising, Sons of
+the Forest, DayZ, Soulmask, Project Zomboid, Windrose, Dragonwilds) run
+inside a Linux container under `xvfb-run` + `wine`. The pattern:
+
+```
+panel-<game>:latest    # custom image: debian + wineHQ + xvfb + setpriv
+   ↓ docker run
+container starts as root
+   ↓ stage 1: chown volumes, symlink /Logs → save volume, drop caps
+exec setpriv → user `panel`
+   ↓ stage 2 (panel user): stamp env into game INI, then
+xvfb-run -a wine /game/.../Win64-Shipping.exe ARGS
+```
+
+Per-game entrypoints live under `modules/<id>/entrypoint.sh`. They all:
+- Refuse to run wine as root (wine policy).
+- Allocate a fresh X display via `xvfb-run -a` to avoid colliding with
+  host Xvfb (a bug we hit when host networking is on — see
+  `memory/gotchas.md` "network_mode: host Xvfb collision").
+- Stamp non-empty env values into the game's INI/JSON config files
+  *idempotently* — empty env means "don't touch this key", so operators
+  who edit the INI directly via the Files tab won't get clobbered.
+- Trap SIGTERM and call `wineserver -k` for clean shutdown.
+
+The shared SteamCMD sidecar runs `+@sSteamCmdForcePlatformType windows`
+to fetch the Windows depot onto a Linux volume.
+
+## `panelctl` CLI
+
+Two surfaces in one binary at `./controller/cmd/panelctl`.
+
+**Operator gRPC** — talks to the Controller's gRPC Panel service. Day-to-day
+server lifecycle.
+
+```
+panelctl agents
+panelctl instances [agent-id]
+panelctl create   <agent-id> <instance-id> <module-id>
+panelctl start    <agent-id> <instance-id>
+panelctl stop     <agent-id> <instance-id> [grace-seconds]
+panelctl delete   <agent-id> <instance-id> [--purge]
+panelctl update   <agent-id> <instance-id> [provider-id]
+panelctl rcon     <agent-id> <instance-id> <command...>
+panelctl watch    [instance-id]
+panelctl backup   create  <agent-id> <instance-id> [description]
+panelctl backup   restore <instance-id> <backup-id>
+panelctl modules
+```
+
+**Admin HTTP** — out-of-band repair. Drives the recreate dance
+(env-config edits, ARK cluster rebinds, generic recreate with mount
+overrides) without needing a dashboard session. Auth is loopback-bypass
+when run on the Controller host, OR a long-lived token at
+`<data-dir>/admin-token` (auto-generated on first start, mode 0600).
+
+```
+panelctl admin token-show
+panelctl admin status
+panelctl admin instances
+panelctl admin ark-rebind <instance-id>
+panelctl admin set-config <instance-id> KEY=VALUE [KEY=VALUE ...]
+panelctl admin del-config <instance-id> KEY [KEY ...]
+panelctl admin recreate   <instance-id> [--mount CONTAINER_PATH=HOST_PATH ...]
+```
+
+Common flags:
+
+```
+--http URL              Default http://localhost:8080
+--admin-token TOK       Or $PANEL_ADMIN_TOKEN, or read from <data-dir>/admin-token
+--data-dir DIR          Default ./data
+```
+
+## HTTP API
+
+All `/api/*` require an authenticated session cookie. `/api/login` is
+the only unauthenticated endpoint.
+
+| Method | Path | What it does |
+|---|---|---|
+| `POST` | `/api/login` | Issues session cookie |
+| `POST` | `/api/logout` | Revokes session |
+| `GET`  | `/api/me` | Current user + role |
+| `POST` | `/api/change-password` | Rotate own password |
+| `GET`  | `/api/agents` | Live-connected agents |
+| `GET`  | `/api/modules` | Union of modules across agents |
+| `GET`  | `/api/instances[?agent_id=]` | Persisted instance rows |
+| `POST` | `/api/instances` | Create |
+| `POST` | `/api/instances/{id}/start` | Start |
+| `POST` | `/api/instances/{id}/stop[?grace=N]` | Stop (container retained) |
+| `DELETE` | `/api/instances/{id}[?purge=true]` | Delete instance + optionally volumes |
+| `POST` | `/api/instances/{id}/rcon` | Ad-hoc RCON exec |
+| `POST` | `/api/instances/{id}/update` | Trigger update provider |
+| `GET`  | `/api/instances/{id}/files[?path=]` | List |
+| `GET`  | `/api/instances/{id}/files/read?path=` | Read file |
+| `POST` | `/api/instances/{id}/files` | Write file |
+| `DELETE` | `/api/instances/{id}/files?path=` | Delete |
+| `GET`  | `/api/instances/{id}/backups` | List backups |
+| `POST` | `/api/instances/{id}/backups` | Create backup |
+| `POST` | `/api/instances/{id}/backups/{bkpId}/restore` | Restore |
+| `DELETE` | `/api/instances/{id}/backups/{bkpId}` | Remove DB row |
+| `GET`  | `/api/schedules[?instance_id=]` | List schedules |
+| `POST` | `/api/schedules` | Create (cron **or** event trigger) |
+| `DELETE` | `/api/schedules/{id}` | Remove |
+| `POST` | `/api/schedules/{id}/{enable\|disable}` | Toggle |
+| `GET`  | `/api/events[?instance=&agent=]` | SSE stream (supports `Last-Event-ID` / `?last_event_id=` resume) |
+| `GET`  | `/api/template-catalog` | 236-entry game template catalog |
+| `POST` | `/api/scaffold-module` | Admin: scaffold a new `modules/<id>/` from a catalog entry |
+| `GET`  | `/api/instances/{id}/update-check` | Last cached update-check result |
+| `POST` | `/api/instances/{id}/update-check` | Run a check (installed build vs Steam `app_info`) |
+| `GET`  | `/api/update-checks` | All instances' update-check state |
+| `GET`  | `/metrics` | Prometheus metrics (outside `/api`; admin token or loopback only) |
+
+## Admin HTTP (out-of-band repair)
+
+Sits **outside** the session middleware. Auth: loopback-bypass OR
+`X-Panel-Admin-Token: <hex>` header (constant-time compared against
+`<data-dir>/admin-token`, auto-generated mode 0600).
+
+| Method | Path | What it does |
+|---|---|---|
+| `GET`  | `/admin/v1/health` | `{ok, version, hostname, loopback, time}` |
+| `GET`  | `/admin/v1/instances` | All instances + `config_values` + `assigned_ports` + `ark_cluster_id` |
+| `POST` | `/admin/v1/instances/{id}/ark-cluster-rebind` | Re-runs the cluster mount recreate against the registry's current cluster id |
+| `POST` | `/admin/v1/instances/{id}/config` | Body `{"set":{...},"delete":[...]}` — merges config_values, runs the env-config recreate dance |
+| `POST` | `/admin/v1/instances/{id}/recreate` | Body `{"mount_overrides":{...}}` — generic stop → delete-preserve → create → start |
+
+The point of this surface: when the dashboard is broken, you can SSH
+in, hit `/admin/v1/*` with curl, and rescue an instance.
+
+## Scheduler
+
+Two trigger kinds.
+
+**cron** — robfig/cron v3, 6-field (seconds resolution):
+
+```json
+{
+  "instance_id": "mc-1",
+  "trigger_kind": "cron",
+  "cron_spec": "0 0 3 * * *",
+  "action": {"type": "instance", "op": "restart", "grace_seconds": 30}
+}
+```
+
+**event** — CEL expression over the live event bus, with optional
+sustain-for window. The classic example: stop the server if it's been
+empty for 30 minutes.
+
+```json
+{
+  "instance_id": "mc-1",
+  "trigger_kind": "event",
+  "event_spec": "event_type == \"app_state\" && players_online == 0",
+  "event_sustained_seconds": 1800,
+  "action": {"type": "instance", "op": "stop", "grace_seconds": 60}
+}
+```
+
+CEL input fields available at evaluation time:
+
+```
+event_type      "join" | "leave" | "chat" | "app_state" | "log_match" | …
+instance_id     string
+agent_id        string
+players_online  int
+players_max     int
+uptime_seconds  int
+instance_status "running" | "stopped" | "starting" | "exited"
+exit_code       int     (only on exit events)
+player_kind     "join" | "leave" | "chat"
+player_name     string
+player_id       string
+player_detail   string
+log_line        string  (raw matched line)
+log_stream      "stdout" | "stderr"
+```
+
+Action types:
+
+```json
+{"type": "rcon",     "command": "saveworld"}
+{"type": "instance", "op": "start" | "stop" | "restart", "grace_seconds": 30}
+```
+
+## Mods & Workshop
+
+Per-game module integrations live in
+`controller/cmd/controller/<game>mods.go`. The current set:
+
+- **ARK SA** — CurseForge API, search + install + version pin. Cluster
+  registry tracks shared `ClusterDirOverride` mounts; rebind via Admin
+  HTTP rewrites all member instances atomically.
+- **DayZ** — Steam Workshop item search, mod-priority load order, deploy
+  via SteamCMD sidecar.
+- **Conan Exiles** — Steam Workshop with `requiredtags[]=Enhanced|Legacy`
+  filtering driven by the instance's `EDITION` config_value, so UE5
+  servers don't see UE4-only mods. `ServerModList=` stamped into
+  ServerSettings.ini and `ConanSandbox/Mods/modlist.txt`.
+- **V Rising** — PlasmaCore + community .pak drops, install path
+  pinned per-instance.
+- **Empyrion** — custom scenario manager (zip uploads + scenario
+  selector), discovery streamer for in-game POI markers.
+- **Barotrauma** — Steam Workshop subscriptions written into the
+  client/server `WorkshopMods/` tree.
+
+Volumes are reused across instances where it's safe: e.g. all Conan
+instances on a given agent share `panel-conan-workshop` for downloaded
+Workshop content, mounted at the game's expected `/game/steamapps/workshop`
+path.
+
+## Backups
+
+`POST /api/instances/{id}/backups` spawns a `tar-sidecar` container
+that mounts the same volumes as the game container (read-only on the
+data side) and writes a tarball to the **agent's** `--backup-dir`
+(default `./data/backups/<instance-id>/`). Restore reverses the flow,
+with the game container stopped first.
+
+Backups are timestamped and labeled with an optional description, and
+can be scheduled (scheduler action `{"type":"backup"}`). Retention is
+manual right now — list, pick, delete. See [docs/BACKUP.md](docs/BACKUP.md).
+
+## File manager
+
+Three operations against an instance's `browseable_root` (or any
+`browseable_roots[i].path`):
+
+- **List** (`GET /files?path=`) — `docker exec ls -la --full-time`
+  parsed into typed entries.
+- **Read** (`GET /files/read?path=`) — `docker cp` to a temp file,
+  stream back. Caps at a sane size; bigger files prompt for download.
+- **Write** (`POST /files`) — temp file, `docker cp`, `chown`.
+- **Delete** (`DELETE /files`) — `docker exec rm`. Symlinks are
+  resolved with `realpath` first; path traversal blocked at the
+  manifest's declared roots.
+
+Bind-mounted volumes get a fast-path that skips `docker cp` entirely.
+
+## Repository layout
+
+```
+panel/
+├── agent/                       # Per-host agent
+│   ├── cmd/agent/               # Binary entrypoint
+│   └── internal/
+│       ├── dispatch/            # gRPC reply handlers — one file per concern
+│       ├── module/              # Module loader + resolver
+│       ├── rcon/                # Source/Telnet/MC RCON adapters
+│       ├── runtime/             # Docker driver
+│       ├── state/               # Per-instance state tracker
+│       └── updater/             # SteamCMD/HTTP/GitHub sidecars
+├── controller/
+│   ├── cmd/
+│   │   ├── controller/          # Web + gRPC server
+│   │   │   └── static/          # Embedded dashboard (new.html + rcc.js/rcc.css;
+│   │   │                        #   index.html = frozen classic; catalog.json)
+│   │   └── panelctl/            # CLI
+│   └── internal/
+│       ├── db/                  # SQL migrations + queries
+│       ├── scheduler/           # cron + CEL
+│       └── eventbus/            # SSE + agent fanout
+├── pkg/module/                  # Shared module-manifest parsing
+├── proto/panel/v1/              # gRPC contract (buf-managed)
+├── modules/                     # 26 modules — drop-in YAML + scripts
+│   └── <game>/
+│       ├── module.yaml
+│       ├── Dockerfile           # if a custom image is needed
+│       └── entrypoint.sh
+├── docs/                        # INSTALL, NETWORKING, PAIRING, ADMIN, BACKUP, architecture
+├── docker-compose.dev.yml       # Postgres only
+├── go.work                      # Multi-module workspace
+└── memory/                      # Maintainer's working notes (not user docs)
+```
+
+## Data directory layout
+
+`data/` is gitignored. On a fresh install you'll grow:
+
+```
+data/
+├── ark-clusters.json            # Operator-curated cluster config (TRACKED)
+├── arkcluster/                  # Per-cluster shared state
+├── backups/                     # Tarballs from tar-sidecar
+├── ca/                          # Internal CA private key + server cert (mTLS)
+├── certs/                       # Issued agent certs
+├── instance-meta/<id>.json      # Per-instance metadata (rcon password, paths)
+└── instances/<id>/              # Bind-mount volumes per instance
+    ├── logs/
+    └── ...
+```
+
+`admin-token` lives at `<data-dir>/admin-token` (mode 0600,
+auto-generated on first controller start).
+
+## Build
+
+Multi-module Go workspace. The root `go.work` ties together
+`agent/`, `controller/`, `pkg/`, `proto/`.
+
+```bash
+# Generate proto stubs (once after a clone, and after any .proto change)
+buf generate
+
+# Linux/Mac:
+go build -o bin/controller ./controller/cmd/controller
+go build -o bin/agent      ./agent/cmd/agent
+go build -o bin/panelctl   ./controller/cmd/panelctl
+
+# Windows (PowerShell):
+go build -o controller.exe .\controller\cmd\controller
+go build -o agent.exe      .\agent\cmd\agent
+go build -o panelctl.exe   .\controller\cmd\panelctl
+```
+
+Cross-compile a Linux agent from a Windows host:
+
+```powershell
+$env:GOOS="linux"; $env:GOARCH="amd64"
+go build -o bin/agent-linux ./agent/cmd/agent
+Remove-Item Env:GOOS, Env:GOARCH
+```
+
+## Working with the embedded UI
+
+The default dashboard is split across three embedded files in
+`controller/cmd/controller/static/`: `new.html` (shell), `rcc.js`
+(all feature JS), `rcc.css` (design system) — plain JS + CSS, no build
+step. `/rcc.js` and `/rcc.css` are served hash-stamped with immutable
+cache headers, so browsers pick up new builds without cache pain. The
+legacy single-file `index.html` (~24k lines) is the **frozen classic UI**
+— it carries a freeze notice, is only served on `panel_ui=classic`, and
+no new features land there.
+
+- Everything is embedded via `//go:embed`, so editing any static file
+  requires a controller rebuild + restart.
+- **Before any controller build that touched the UI**, run the parse gate:
+  `node controller/.test/gate-parse.mjs` — a single JS syntax error blanks
+  the whole dashboard and `go build` cannot catch it.
+- Per-game Config tabs are keyed by module id with a small schema
+  (`groups[].fields[]` with `{prop, label, type, options, hint}`) that the
+  form renderer turns into INI/JSON read+write against the game's config
+  file via the file manager API.
+- Module icon/art/accent + readiness regex are manifest-driven:
+  `appearance:` and `ready_pattern:` in `module.yaml`, served via
+  `GET /api/modules` (inline maps remain as fallback).
+- The shared form for `config_values` (env-driven settings) is rendered
+  on the dashboard's *Config Values* sub-tab; this is separate from the
+  game-specific config-file editors above.
+
+## Troubleshooting
+
+- **"agent connected but no modules"** — agent's `--modules-dir` is
+  wrong, or the manifest failed to parse. Check the agent's stderr.
+- **"create succeeded but instance never starts"** — for SteamCMD-driven
+  modules, the install volume is empty. Click *Update* once first; the
+  entrypoints exit `EX_CONFIG (78)` with a hint when they can't find
+  the dedicated server binary.
+- **"environment variable I added to module.yaml never reaches the
+  container"** — you forgot to pre-declare the key under
+  `runtime.docker.env`. resolve.go silently filters unknown keys.
+- **"dashboard broken, can't recover"** — `panelctl admin recreate
+  <instance-id>`. If the controller is also down, the data directory
+  + Postgres dump is enough to spin up a replacement.
+- **Wine modules: container stays "running" but the game never binds
+  ports** — usually a top-level launcher stub problem. Invoke the deep
+  `Binaries/Win64/<Game>Server-Win64-Shipping.exe` directly (see Conan's
+  entrypoint for the canonical pattern).
+- **`network_mode: host` + Xvfb collision** — when two host-net
+  containers both run xvfb on the same display number you'll see "Server
+  is already active for display" in stderr. Always use `xvfb-run -a` to
+  auto-allocate.
+
+The maintainer's running list of caught-on-fire-once notes lives in
+[`memory/gotchas.md`](memory/gotchas.md).
+
+## Contributing / writing a new module
+
+1. Create `modules/<id>/module.yaml`. Easiest is to copy the closest
+   existing module — Source-RCON game? Start from Conan's. Wine? V
+   Rising's. Plain Linux dedicated server? Factorio's.
+2. If you need a custom image, add a `Dockerfile` next to it. Include
+   `setpriv`, `xvfb-run`, and `wine` for Windows-on-Linux modules.
+3. Pre-declare every env key that any `config_values` entry maps to
+   inside `runtime.docker.env`. Empty defaults are fine.
+4. Restart the agent — modules load on boot. The new game appears in
+   the dashboard's create-instance dropdown.
+5. Test the lifecycle end-to-end: create → update → start → connect a
+   client → stop → delete (no purge) → recreate → verify world saves
+   survived.
+6. If the game has a Workshop / mods scene, add a `<game>mods.go` to
+   `controller/cmd/controller/` and surface it in the dashboard's per-game
+   Mods tab.
+
+PRs welcome. The maintainer's editorial preferences are recorded in
+[`memory/ux_rules.md`](memory/ux_rules.md) — read it before redesigning
+any tab.
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE).