panel/README.md

# panel

An open-source, modular game-server management panel — AMP-class features,
distributed Controller/Agent architecture, CEL-driven automation, full
instance lifecycle (install → run → update → backup → restore), live
dashboard, and CLI.

Written in Go. Single static binary per component. Docker-backed runtime.
26 modules ship in-tree, including the awkward Windows-only games
(Conan Exiles, Empyrion, V Rising, Sons of the Forest, DayZ, Soulmask,
Project Zomboid, Windrose) under Wine + Xvfb. The agent **auto-builds**
every module's Docker image from its in-tree `Dockerfile` the first time
an instance needs it — there is no manual per-module `docker build` step.

## 60-second quickstart

One box, one line. Docker is installed for you if missing; Postgres runs
as a managed container; controller + agent become systemd units (Linux)
or Scheduled Tasks (Windows), and the local agent is auto-paired over mTLS.

```bash
# Linux (Debian/Ubuntu/Fedora/RHEL/Raspbian) — run as root:
curl -fsSL https://raw.githubusercontent.com/dbledeez/panel/main/install.sh | sudo bash
```

```powershell
# Windows (Docker Desktop must already be running):
powershell -ExecutionPolicy Bypass -File install.ps1
```

Then open `http://<host>:8080/` and log in with the `ADMIN BOOTSTRAP`
credentials printed in the controller log (or pre-seed via
`PANEL_ADMIN_PASSWORD`). Full options, docker-compose, and from-source
paths: [docs/INSTALL.md](docs/INSTALL.md).

```
                    ┌─────────────────────────┐
                    │       Operator          │
                    │  (browser / panelctl)   │
                    └────────────┬────────────┘
                                 │ HTTPS / gRPC
                    ┌────────────▼────────────┐
                    │       Controller        │
                    │  • Web dashboard :8080  │
                    │  • Agent gRPC    :8443  │
                    │  • Postgres             │
                    │  • Scheduler (cron+CEL) │
                    │  • Event bus + SSE      │
                    │  • Session auth         │
                    └────────────┬────────────┘
                                 │ gRPC bidi (mTLS)
            ┌────────────────────┼────────────────────┐
            │                    │                    │
   ┌────────▼────────┐  ┌────────▼────────┐  ┌────────▼────────┐
   │    Agent A      │  │    Agent B      │  │    Agent C      │
   │  • Docker       │  │  • Docker       │  │  • Docker       │
   │  • Modules      │  │  • Modules      │  │  • Modules      │
   │  • RCON clients │  │  • RCON clients │  │  • RCON clients │
   │  • State tracker│  │  • State tracker│  │  • State tracker│
   └─────────────────┘  └─────────────────┘  └─────────────────┘
```

---

## Documentation

| Doc | Covers |
|---|---|
| [docs/INSTALL.md](docs/INSTALL.md) | Installers, docker-compose, from-source, flags reference, systemd, reverse proxy + TLS, firewall |
| [docs/NETWORKING.md](docs/NETWORKING.md) | Per-game port table, router port-forwarding, what the panel auto-allocates |
| [docs/PAIRING.md](docs/PAIRING.md) | mTLS agent pairing, multi-host setup |
| [docs/ADMIN.md](docs/ADMIN.md) | First-admin bootstrap, password reset, Steam sign-in, Steam credentials for paid games |
| [docs/BACKUP.md](docs/BACKUP.md) | Backups, pg_dump, disaster-recovery runbook |
| [docs/architecture.md](docs/architecture.md) | The real stack, component by component |

## Table of contents

- [Status](#status)
- [Why panel](#why-panel)
- [Architecture](#architecture)
- [Quick start (dev)](#quick-start-dev)
- [Production-ish layout](#production-ish-layout)
- [Modules](#modules)
- [Module manifest](#module-manifest)
- [Wine/Xvfb modules](#winexvfb-modules)
- [`panelctl` CLI](#panelctl-cli)
- [HTTP API](#http-api)
- [Admin HTTP (out-of-band repair)](#admin-http-out-of-band-repair)
- [Scheduler](#scheduler)
- [Mods & Workshop](#mods--workshop)
- [Backups](#backups)
- [File manager](#file-manager)
- [Repository layout](#repository-layout)
- [Data directory layout](#data-directory-layout)
- [Build](#build)
- [Working with the embedded UI](#working-with-the-embedded-ui)
- [Troubleshooting](#troubleshooting)
- [Contributing / writing a new module](#contributing--writing-a-new-module)
- [License](#license)

---

## Status

panel is a working private deployment driving real game servers — not a
toy. Feature completeness varies by game; the headline state is:

- ✅ Manifest-driven module plugins (26 shipped — see [Modules](#modules))
- ✅ mTLS agent pairing: one-time token minted in the UI → `agent
  --pair-token` fetches the CA + a signed cert and exits; the agent then
  connects with mTLS and rogue agents are rejected at the TLS handshake
  (see [docs/PAIRING.md](docs/PAIRING.md))
- ✅ Agent auto-builds `panel-*` module images from the in-tree Dockerfiles
  on first use — pull-through for third-party images, no manual builds
- ✅ Distributed Controller + Agent over gRPC (bidi streams, reconnect +
  rehydrate, sidecar metadata survives restarts)
- ✅ Docker runtime with bind-mount and named-volume support, host
  networking, per-instance port allocation
- ✅ Wine + Xvfb pattern for Windows-only dedicated servers (Empyrion,
  Conan Exiles, V Rising, Sons of the Forest, DayZ, Soulmask, Project
  Zomboid, Windrose, Dragonwilds)
- ✅ Full state tracking: RCON polling + log-line event matching → typed
  PlayerEvent + AppState bus
- ✅ Telnet **and** Source RCON adapters (Minecraft, CS-derived games,
  Rust, Valheim w/ plugins, 7DTD, Conan, etc.) with redial-on-EOF
- ✅ Scheduler: 6-field cron triggers **and** CEL event triggers with
  sustained-for windows
- ✅ Config-file templating with auto-generated secrets; per-game custom
  Config tabs over `ServerSettings.ini` / `GameUserSettings.ini` /
  `serverconfig.xml` / properties / JSON
- ✅ File manager: web UI browses container volumes via `docker exec` +
  `docker cp`, bind-mount fallback, path-traversal defended, multi-root
  per module
- ✅ Update providers: **Direct HTTP**, **GitHub releases**, **SteamCMD
  sidecar** (with platform overrides for Windows-on-Linux), **Steam beta
  branches** (e.g. Conan Legacy UE4, Conan Enhanced UE5)
- ✅ Backup / restore via tar-sidecar containers
- ✅ Postgres persistence (agents, instances, schedules, backups, users)
- ✅ Local auth: argon2id + session cookies, admin bootstrap, change
  password, optional Sign-in-with-Steam linking
- ✅ Live event bus with SSE stream (`Last-Event-ID` resume + snapshot) +
  `panelctl watch`
- ✅ Template Library: 236-entry game catalog (`static/catalog.json`,
  distilled from CubeCoders' MIT templates by `tools/amp-distill`) with an
  admin one-click module scaffolder (`POST /api/scaffold-module`)
- ✅ Update-check (check-only): agent compares installed Steam build id vs
  `app_info` (15-min cache) → Updates-tab panel + card badge
- ✅ Prometheus `/metrics` (admin-token / loopback gated)
- ✅ Mods tab: ARK CurseForge / DayZ Workshop / Conan Workshop / V Rising
  PlasmaCore / Empyrion scenarios — search, install, version-pin
- ✅ ARK SA cluster registry with cross-server transfer + admin rebind
- 🟡 Cross-platform: Linux works for everything except modules that
  require nested KVM. Primary dev host is Windows + Docker Desktop.
- ❌ OIDC / WebAuthn / Casbin ACL (local auth only; role = admin / user)
- ❌ SFTP (web FM covers the common case)
- ❌ Host-mode runtime (cgroups v2 / Windows Job Objects)

## Why panel

Pterodactyl/Pelican is good but Docker-only and PHP, AMP is closed-source
and per-machine, GameServerApp and friends are SaaS. panel is what falls
out if you want:

- One static Go binary per component, no Composer / npm / Node runtime
  on the host.
- Modules as plain YAML — drop a `module.yaml` in `modules/<id>/`,
  restart the agent, the game appears in the dashboard.
- Real distributed multi-host: Controller is one box, Agents run
  wherever Docker runs.
- A scheduler that can react to *the world* — "stop the server if
  it's been empty for 30 minutes", "restart at 3am only on Tuesdays" —
  rather than just cron.
- First-class Wine/Xvfb support: Windows-only dedicated servers run on
  Linux Docker hosts without per-game artisanal scripting.
- Out-of-band repair via a token-authed `/admin/v1/*` HTTP surface —
  the dashboard breaks, you SSH in, and `panelctl admin recreate` /
  `set-config` / `ark-rebind` still work.

Designed for someone running ~10 game servers across 2-4 boxes for
themselves and a couple of communities, not for a hosting company. No
multi-tenancy, no billing.

## Architecture

Two long-running components plus optional sidecars.

**Controller** — `./controller/cmd/controller`
- HTTP :8080 (configurable) — dashboard, REST API, SSE event stream
- gRPC :8443 — bidi stream from each Agent
- Postgres for durable state (no SQLite default — explicitly not sharded)
- Embedded migrations, embedded static dashboard (default UI =
  `static/new.html` shell + `static/rcc.js` + `static/rcc.css`, served
  hash-stamped with immutable caching via `//go:embed`; the legacy
  single-file `static/index.html` is the frozen "classic" UI, opt-in via
  the `panel_ui=classic` cookie)
- Scheduler in-process (cron + CEL)
- Prometheus `GET /metrics` (admin-token or loopback gated)

**Agent** — `./agent/cmd/agent`
- Dials Controller, stays connected with reconnect + heartbeat
- Loads modules from `--modules-dir` at boot (default `./modules`)
- Drives the local Docker daemon (`/var/run/docker.sock` or
  `npipe:////./pipe/docker_engine` on Windows)
- Spawns short-lived sidecar containers for SteamCMD updates and tar
  backup/restore
- Per-instance state tracker: RCON polling, log-line tail, event
  pattern matching → typed events streamed back to Controller

**Sidecars (per-game)**
- `steamcmd-sidecar` — SteamCMD with `+force_install_dir` mounted at
  the instance's game-content volume; supports `+@sSteamCmdForcePlatformType
  windows` for cross-platform Windows depots, beta branch pinning, and
  Workshop downloads.
- `tar-sidecar` — exec'd against the volumes for backup/restore so the
  game container doesn't have to expose its filesystem.

**RCON adapters** (`agent/internal/rcon/`)
- `source_rcon` — Source-RCON wire protocol (Minecraft Java, Conan,
  Factorio, Palworld, Project Zomboid)
- `telnet` — Telnet-style password gate then line protocol (7DTD, Empyrion)
- `websocket_rcon` — Rust's WebRCON
- `stdio` — attach to the container's stdin (Terraria, Valheim, Vein,
  Enshrouded)
- `be_rcon` — BattlEye RCON over UDP (DayZ)
- `docker_exec_rcon` — exec a CLI inside the container (ARK SA)

All adapters auto-redial on EOF (Conan and Palworld lessons baked in).

## Quick start (dev)

Prereqs: Go 1.26+, Docker Desktop (Windows) or Docker Engine (Linux),
[`buf`](https://buf.build/docs/installation/) on PATH for proto stubs.

```bash
# 1) Postgres (single dev container)
docker compose -f docker-compose.dev.yml up -d

# 2) Generate protobuf stubs (uses remote plugins — no local protoc needed)
buf generate

# 3) Start the controller. First run prints an ADMIN BOOTSTRAP banner
#    with the generated admin password — copy it out of stderr.
go run ./controller/cmd/controller

# 4) Start an agent (any host with Docker + reachable to controller)
go run ./agent/cmd/agent --insecure \
  --modules-dir=./modules \
  --data-root=./data/instances \
  --meta-dir=./data/instance-meta \
  --backup-dir=./data/backups

# 5) Open http://localhost:8080/, log in as admin@panel.local with
#    the bootstrap password.
```

If you'd rather pre-seed the password instead of reading it from logs:

```bash
PANEL_ADMIN_PASSWORD=changeme go run ./controller/cmd/controller
```

## Production-ish layout

This is the layout the maintainer runs. Not prescriptive — adapt freely.

- One **Controller** box with Postgres co-located. Reverse-proxied
  behind Nginx Proxy Manager / Caddy / Cloudflare. Public dashboard
  URL has a real cert; gRPC :8443 is on the LAN only.
- N **Agent** boxes, each with Docker. Agents don't need to be reachable
  from the internet — they dial out to the Controller's gRPC port.
- A `data/` directory per machine (Controller + each Agent) under
  whatever path you want. **Don't put it in the repo working tree** —
  `data/` is gitignored for a reason: it holds per-instance state,
  RCON passwords, mTLS material, and game saves.
- One systemd unit per binary on Linux, one Service on Windows
  (NSSM works fine).

The agent is intentionally stateless beyond `data/instance-meta/*.json`
(rehydrated from the Controller on connect). Wipe the meta directory
and the agent rebuilds it from the Controller's authoritative DB.

## Modules

Game modules live under `modules/<id>/` and are picked up by every
agent at boot. `panel-*` images are **auto-built by the agent** from the
module's in-tree `Dockerfile` the first time an instance needs them
(they are never published to a registry); third-party images are pulled. Each ships with a `module.yaml` and (where needed) an
`entrypoint.sh` and `Dockerfile`.

| Module | Image / build | RCON | Notes |
|---|---|---|---|
| `7dtd` | `panel-7dtd` (auto-built) | telnet | Canonical-nim freeze, cluster playbook |
| `ark-sa` | `acekorneya/asa_server:latest` | `docker_exec_rcon` | Cluster registry, CurseForge mods, transfer flow |
| `barotrauma` | `panel-barotrauma` (auto-built) | log only | Workshop mods |
| `conan-exiles` | `panel-conan-exiles` (auto-built, Wine) | `source_rcon` | UE4 Legacy / UE5 Enhanced edition picker, Workshop |
| `core-keeper` | `panel-core-keeper` (auto-built) | log only | |
| `dayz` | `panel-dayz` (auto-built, Wine) | `be_rcon` | Steam Workshop manager; `requires_steam_login` |
| `demo` | `alpine:3.21` | none | Lifecycle / event sandbox |
| `dragonwilds` | `panel-dragonwilds` (auto-built, Wine) | log only | |
| `empyrion` | `panel-empyrion` (auto-built, Wine) | telnet | Custom scenario manager, EAH bridge, discovery streamer |
| `empyrion-bridge` | multi-stage dotnet (auto-built) | bridge | EWA gateway sidecar |
| `enshrouded` | `panel-enshrouded` (auto-built) | stdio | |
| `factorio` | `panel-factorio` (auto-built) | `source_rcon` | |
| `minecraft-bedrock` | `itzg/minecraft-bedrock-server` | log only | |
| `minecraft-java` | `itzg/minecraft-server:latest` | `source_rcon` | |
| `palworld` | `panel-palworld` (auto-built) | `source_rcon` | RCON redial-on-EOF |
| `project-zomboid` | `panel-project-zomboid` (auto-built, Wine) | `source_rcon` | |
| `rust` | `panel-rust` (auto-built) | `websocket_rcon` | Rust+ port |
| `satisfactory` | `panel-satisfactory` (auto-built) | none | |
| `sons-of-the-forest` | `panel-sons-of-the-forest` (auto-built, Wine) | log only | |
| `soulmask` | `panel-soulmask` (auto-built, Wine) | log only | |
| `steamcmd-test` | `alpine:3.21` + SteamCMD | none | HLDS smoke test (~930 MB) |
| `terraria` | `panel-terraria` (auto-built) | stdio | |
| `v-rising` | `panel-v-rising` (auto-built, Wine) | log only | PlasmaCore mods |
| `valheim` | `panel-valheim` (auto-built) | stdio | |
| `vein` | `panel-vein` (auto-built) | stdio | |
| `windrose` | `panel-windrose` (auto-built, Wine) | log only | |

Per-game gotchas live in [`memory/gotchas.md`](memory/gotchas.md) — read
that before debugging any specific game.

## Module manifest

Every module is one declarative YAML file. The schema is in
[`pkg/module/manifest.go`](pkg/module/manifest.go). A condensed example:

```yaml
id: my-game
name: "My Game"
version: 0.1.0
supported_modes: [docker]

runtime:
  docker:
    image: itzg/my-game-server:latest        # or local-built image tag
    network_mode: host                       # optional
    browseable_root: /data                    # File-manager root
    browseable_roots:                         # Optional multi-root
      - { name: "Saves", path: /data, hint: "world data" }
      - { name: "Game",  path: /game, hint: "engine binaries" }
    env:                                      # Pre-declare every env key
      MAX_PLAYERS: "20"                       #   you want forwarded;
      ADMIN_PASSWORD: ""                      #   resolve.go filters by
    volumes:                                  #   this map.
      - { type: volume, name: "panel-$INSTANCE_ID-data", container: /data }
      - { target: "$DATA_PATH/logs", container: /log }   # bind mount

ports:
  - { name: game,  proto: tcp, default: 25565, required: true,  env: GAME_PORT }
  - { name: query, proto: udp, default: 25566, required: false, env: QUERY_PORT }
  - { name: rcon,  proto: tcp, default: 25575, internal: true,  env: RCON_PORT }

resources:
  min_ram_mb: 2048
  recommended_ram_mb: 4096

rcon:
  adapter: source_rcon         # or telnet, stdio, websocket_rcon, be_rcon, docker_exec_rcon
  host_port: rcon
  password_secret: RCON_PASSWORD
  password_literal: "fallback"
  commands:
    list:      "list"
    broadcast: "say {msg}"
    kick:      "kick {player}"
    save:      "save-all"
    shutdown:  "stop"

state_sources:
  - type: rcon
    command: list
    every: 60s
    parse:
      kind: regex
      pattern: '(?P<n>\d+)\s+of\s+(?P<m>\d+) players'
      fields: { players_online: 1, players_max: 2 }

events:
  join:  { pattern: "(?P<name>\\S+) joined", kind: join }
  leave: { pattern: "(?P<name>\\S+) left",   kind: leave }
  chat:  { pattern: "<(?P<name>[^>]+)> (?P<msg>.+)", kind: chat }

config_files:
  - path: server.properties
    format: properties
    template: templates/server.properties.tmpl

config_values:                  # Drive the Config tab in the dashboard
  - key: MAX_PLAYERS
    label: "Max players"
    description: "Player slot limit."
    default: "20"
  - key: ADMIN_PASSWORD
    label: "Admin password"
    description: "In-game admin auth."
    default: ""

secrets:
  - { name: admin_token, generated: true }

update_providers:
  - id: stable
    kind: steamcmd
    app_id: "294420"
    install_path: /data
    skip_validate: false        # Set true for app_ids whose validate hangs
```

Key rules the loader enforces:

- Only env keys pre-declared under `runtime.docker.env` get forwarded
  to the container. Adding a `config_values` entry without the matching
  env declaration silently drops the value — this trips people up.
- `volumes:` entries use `$INSTANCE_ID` and `$DATA_PATH` substitutions
  resolved by the agent at create time.
- `internal: true` ports are not exposed by the allocator but still get
  their `env:` substituted.
- `update_providers` are picked from a dropdown in the UI; multiple
  providers per module is fine (Conan ships `enhanced` and `legacy`).

## Wine/Xvfb modules

The Windows-only dedicated servers (Empyrion, Conan, V Rising, Sons of
the Forest, DayZ, Soulmask, Project Zomboid, Windrose, Dragonwilds) run
inside a Linux container under `xvfb-run` + `wine`. The pattern:

```
panel-<game>:latest    # custom image: debian + wineHQ + xvfb + setpriv
   ↓ docker run
container starts as root
   ↓ stage 1: chown volumes, symlink /Logs → save volume, drop caps
exec setpriv → user `panel`
   ↓ stage 2 (panel user): stamp env into game INI, then
xvfb-run -a wine /game/.../Win64-Shipping.exe ARGS
```

Per-game entrypoints live under `modules/<id>/entrypoint.sh`. They all:
- Refuse to run wine as root (wine policy).
- Allocate a fresh X display via `xvfb-run -a` to avoid colliding with
  host Xvfb (a bug we hit when host networking is on — see
  `memory/gotchas.md` "network_mode: host Xvfb collision").
- Stamp non-empty env values into the game's INI/JSON config files
  *idempotently* — empty env means "don't touch this key", so operators
  who edit the INI directly via the Files tab won't get clobbered.
- Trap SIGTERM and call `wineserver -k` for clean shutdown.

The shared SteamCMD sidecar runs `+@sSteamCmdForcePlatformType windows`
to fetch the Windows depot onto a Linux volume.

## `panelctl` CLI

Two surfaces in one binary at `./controller/cmd/panelctl`.

**Operator gRPC** — talks to the Controller's gRPC Panel service. Day-to-day
server lifecycle.

```
panelctl agents
panelctl instances [agent-id]
panelctl create   <agent-id> <instance-id> <module-id>
panelctl start    <agent-id> <instance-id>
panelctl stop     <agent-id> <instance-id> [grace-seconds]
panelctl delete   <agent-id> <instance-id> [--purge]
panelctl update   <agent-id> <instance-id> [provider-id]
panelctl rcon     <agent-id> <instance-id> <command...>
panelctl watch    [instance-id]
panelctl backup   create  <agent-id> <instance-id> [description]
panelctl backup   restore <instance-id> <backup-id>
panelctl modules
```

**Admin HTTP** — out-of-band repair. Drives the recreate dance
(env-config edits, ARK cluster rebinds, generic recreate with mount
overrides) without needing a dashboard session. Auth is loopback-bypass
when run on the Controller host, OR a long-lived token at
`<data-dir>/admin-token` (auto-generated on first start, mode 0600).

```
panelctl admin token-show
panelctl admin status
panelctl admin instances
panelctl admin ark-rebind <instance-id>
panelctl admin set-config <instance-id> KEY=VALUE [KEY=VALUE ...]
panelctl admin del-config <instance-id> KEY [KEY ...]
panelctl admin recreate   <instance-id> [--mount CONTAINER_PATH=HOST_PATH ...]
```

Common flags:

```
--http URL              Default http://localhost:8080
--admin-token TOK       Or $PANEL_ADMIN_TOKEN, or read from <data-dir>/admin-token
--data-dir DIR          Default ./data
```

## HTTP API

All `/api/*` require an authenticated session cookie. `/api/login` is
the only unauthenticated endpoint.

| Method | Path | What it does |
|---|---|---|
| `POST` | `/api/login` | Issues session cookie |
| `POST` | `/api/logout` | Revokes session |
| `GET`  | `/api/me` | Current user + role |
| `POST` | `/api/change-password` | Rotate own password |
| `GET`  | `/api/agents` | Live-connected agents |
| `GET`  | `/api/modules` | Union of modules across agents |
| `GET`  | `/api/instances[?agent_id=]` | Persisted instance rows |
| `POST` | `/api/instances` | Create |
| `POST` | `/api/instances/{id}/start` | Start |
| `POST` | `/api/instances/{id}/stop[?grace=N]` | Stop (container retained) |
| `DELETE` | `/api/instances/{id}[?purge=true]` | Delete instance + optionally volumes |
| `POST` | `/api/instances/{id}/rcon` | Ad-hoc RCON exec |
| `POST` | `/api/instances/{id}/update` | Trigger update provider |
| `GET`  | `/api/instances/{id}/files[?path=]` | List |
| `GET`  | `/api/instances/{id}/files/read?path=` | Read file |
| `POST` | `/api/instances/{id}/files` | Write file |
| `DELETE` | `/api/instances/{id}/files?path=` | Delete |
| `GET`  | `/api/instances/{id}/backups` | List backups |
| `POST` | `/api/instances/{id}/backups` | Create backup |
| `POST` | `/api/instances/{id}/backups/{bkpId}/restore` | Restore |
| `DELETE` | `/api/instances/{id}/backups/{bkpId}` | Remove DB row |
| `GET`  | `/api/schedules[?instance_id=]` | List schedules |
| `POST` | `/api/schedules` | Create (cron **or** event trigger) |
| `DELETE` | `/api/schedules/{id}` | Remove |
| `POST` | `/api/schedules/{id}/{enable\|disable}` | Toggle |
| `GET`  | `/api/events[?instance=&agent=]` | SSE stream (supports `Last-Event-ID` / `?last_event_id=` resume) |
| `GET`  | `/api/template-catalog` | 236-entry game template catalog |
| `POST` | `/api/scaffold-module` | Admin: scaffold a new `modules/<id>/` from a catalog entry |
| `GET`  | `/api/instances/{id}/update-check` | Last cached update-check result |
| `POST` | `/api/instances/{id}/update-check` | Run a check (installed build vs Steam `app_info`) |
| `GET`  | `/api/update-checks` | All instances' update-check state |
| `GET`  | `/metrics` | Prometheus metrics (outside `/api`; admin token or loopback only) |

## Admin HTTP (out-of-band repair)

Sits **outside** the session middleware. Auth: loopback-bypass OR
`X-Panel-Admin-Token: <hex>` header (constant-time compared against
`<data-dir>/admin-token`, auto-generated mode 0600).

| Method | Path | What it does |
|---|---|---|
| `GET`  | `/admin/v1/health` | `{ok, version, hostname, loopback, time}` |
| `GET`  | `/admin/v1/instances` | All instances + `config_values` + `assigned_ports` + `ark_cluster_id` |
| `POST` | `/admin/v1/instances/{id}/ark-cluster-rebind` | Re-runs the cluster mount recreate against the registry's current cluster id |
| `POST` | `/admin/v1/instances/{id}/config` | Body `{"set":{...},"delete":[...]}` — merges config_values, runs the env-config recreate dance |
| `POST` | `/admin/v1/instances/{id}/recreate` | Body `{"mount_overrides":{...}}` — generic stop → delete-preserve → create → start |

The point of this surface: when the dashboard is broken, you can SSH
in, hit `/admin/v1/*` with curl, and rescue an instance.

## Scheduler

Two trigger kinds.

**cron** — robfig/cron v3, 6-field (seconds resolution):

```json
{
  "instance_id": "mc-1",
  "trigger_kind": "cron",
  "cron_spec": "0 0 3 * * *",
  "action": {"type": "instance", "op": "restart", "grace_seconds": 30}
}
```

**event** — CEL expression over the live event bus, with optional
sustain-for window. The classic example: stop the server if it's been
empty for 30 minutes.

```json
{
  "instance_id": "mc-1",
  "trigger_kind": "event",
  "event_spec": "event_type == \"app_state\" && players_online == 0",
  "event_sustained_seconds": 1800,
  "action": {"type": "instance", "op": "stop", "grace_seconds": 60}
}
```

CEL input fields available at evaluation time:

```
event_type      "join" | "leave" | "chat" | "app_state" | "log_match" | …
instance_id     string
agent_id        string
players_online  int
players_max     int
uptime_seconds  int
instance_status "running" | "stopped" | "starting" | "exited"
exit_code       int     (only on exit events)
player_kind     "join" | "leave" | "chat"
player_name     string
player_id       string
player_detail   string
log_line        string  (raw matched line)
log_stream      "stdout" | "stderr"
```

Action types:

```json
{"type": "rcon",     "command": "saveworld"}
{"type": "instance", "op": "start" | "stop" | "restart", "grace_seconds": 30}
```

## Mods & Workshop

Per-game module integrations live in
`controller/cmd/controller/<game>mods.go`. The current set:

- **ARK SA** — CurseForge API, search + install + version pin. Cluster
  registry tracks shared `ClusterDirOverride` mounts; rebind via Admin
  HTTP rewrites all member instances atomically.
- **DayZ** — Steam Workshop item search, mod-priority load order, deploy
  via SteamCMD sidecar.
- **Conan Exiles** — Steam Workshop with `requiredtags[]=Enhanced|Legacy`
  filtering driven by the instance's `EDITION` config_value, so UE5
  servers don't see UE4-only mods. `ServerModList=` stamped into
  ServerSettings.ini and `ConanSandbox/Mods/modlist.txt`.
- **V Rising** — PlasmaCore + community .pak drops, install path
  pinned per-instance.
- **Empyrion** — custom scenario manager (zip uploads + scenario
  selector), discovery streamer for in-game POI markers.
- **Barotrauma** — Steam Workshop subscriptions written into the
  client/server `WorkshopMods/` tree.

Volumes are reused across instances where it's safe: e.g. all Conan
instances on a given agent share `panel-conan-workshop` for downloaded
Workshop content, mounted at the game's expected `/game/steamapps/workshop`
path.

## Backups

`POST /api/instances/{id}/backups` spawns a `tar-sidecar` container
that mounts the same volumes as the game container (read-only on the
data side) and writes a tarball to the **agent's** `--backup-dir`
(default `./data/backups/<instance-id>/`). Restore reverses the flow,
with the game container stopped first.

Backups are timestamped and labeled with an optional description, and
can be scheduled (scheduler action `{"type":"backup"}`). Retention is
manual right now — list, pick, delete. See [docs/BACKUP.md](docs/BACKUP.md).

## File manager

Three operations against an instance's `browseable_root` (or any
`browseable_roots[i].path`):

- **List** (`GET /files?path=`) — `docker exec ls -la --full-time`
  parsed into typed entries.
- **Read** (`GET /files/read?path=`) — `docker cp` to a temp file,
  stream back. Caps at a sane size; bigger files prompt for download.
- **Write** (`POST /files`) — temp file, `docker cp`, `chown`.
- **Delete** (`DELETE /files`) — `docker exec rm`. Symlinks are
  resolved with `realpath` first; path traversal blocked at the
  manifest's declared roots.

Bind-mounted volumes get a fast-path that skips `docker cp` entirely.

## Repository layout

```
panel/
├── agent/                       # Per-host agent
│   ├── cmd/agent/               # Binary entrypoint
│   └── internal/
│       ├── dispatch/            # gRPC reply handlers — one file per concern
│       ├── module/              # Module loader + resolver
│       ├── rcon/                # Source/Telnet/MC RCON adapters
│       ├── runtime/             # Docker driver
│       ├── state/               # Per-instance state tracker
│       └── updater/             # SteamCMD/HTTP/GitHub sidecars
├── controller/
│   ├── cmd/
│   │   ├── controller/          # Web + gRPC server
│   │   │   └── static/          # Embedded dashboard (new.html + rcc.js/rcc.css;
│   │   │                        #   index.html = frozen classic; catalog.json)
│   │   └── panelctl/            # CLI
│   └── internal/
│       ├── db/                  # SQL migrations + queries
│       ├── scheduler/           # cron + CEL
│       └── eventbus/            # SSE + agent fanout
├── pkg/module/                  # Shared module-manifest parsing
├── proto/panel/v1/              # gRPC contract (buf-managed)
├── modules/                     # 26 modules — drop-in YAML + scripts
│   └── <game>/
│       ├── module.yaml
│       ├── Dockerfile           # if a custom image is needed
│       └── entrypoint.sh
├── docs/                        # INSTALL, NETWORKING, PAIRING, ADMIN, BACKUP, architecture
├── docker-compose.dev.yml       # Postgres only
├── go.work                      # Multi-module workspace
└── memory/                      # Maintainer's working notes (not user docs)
```

## Data directory layout

`data/` is gitignored. On a fresh install you'll grow:

```
data/
├── ark-clusters.json            # Operator-curated cluster config (TRACKED)
├── arkcluster/                  # Per-cluster shared state
├── backups/                     # Tarballs from tar-sidecar
├── ca/                          # Internal CA private key + server cert (mTLS)
├── certs/                       # Issued agent certs
├── instance-meta/<id>.json      # Per-instance metadata (rcon password, paths)
└── instances/<id>/              # Bind-mount volumes per instance
    ├── logs/
    └── ...
```

`admin-token` lives at `<data-dir>/admin-token` (mode 0600,
auto-generated on first controller start).

## Build

Multi-module Go workspace. The root `go.work` ties together
`agent/`, `controller/`, `pkg/`, `proto/`.

```bash
# Generate proto stubs (once after a clone, and after any .proto change)
buf generate

# Linux/Mac:
go build -o bin/controller ./controller/cmd/controller
go build -o bin/agent      ./agent/cmd/agent
go build -o bin/panelctl   ./controller/cmd/panelctl

# Windows (PowerShell):
go build -o controller.exe .\controller\cmd\controller
go build -o agent.exe      .\agent\cmd\agent
go build -o panelctl.exe   .\controller\cmd\panelctl
```

Cross-compile a Linux agent from a Windows host:

```powershell
$env:GOOS="linux"; $env:GOARCH="amd64"
go build -o bin/agent-linux ./agent/cmd/agent
Remove-Item Env:GOOS, Env:GOARCH
```

## Working with the embedded UI

The default dashboard is split across three embedded files in
`controller/cmd/controller/static/`: `new.html` (shell), `rcc.js`
(all feature JS), `rcc.css` (design system) — plain JS + CSS, no build
step. `/rcc.js` and `/rcc.css` are served hash-stamped with immutable
cache headers, so browsers pick up new builds without cache pain. The
legacy single-file `index.html` (~24k lines) is the **frozen classic UI**
— it carries a freeze notice, is only served on `panel_ui=classic`, and
no new features land there.

- Everything is embedded via `//go:embed`, so editing any static file
  requires a controller rebuild + restart.
- **Before any controller build that touched the UI**, run the parse gate:
  `node controller/.test/gate-parse.mjs` — a single JS syntax error blanks
  the whole dashboard and `go build` cannot catch it.
- Per-game Config tabs are keyed by module id with a small schema
  (`groups[].fields[]` with `{prop, label, type, options, hint}`) that the
  form renderer turns into INI/JSON read+write against the game's config
  file via the file manager API.
- Module icon/art/accent + readiness regex are manifest-driven:
  `appearance:` and `ready_pattern:` in `module.yaml`, served via
  `GET /api/modules` (inline maps remain as fallback).
- The shared form for `config_values` (env-driven settings) is rendered
  on the dashboard's *Config Values* sub-tab; this is separate from the
  game-specific config-file editors above.

## Troubleshooting

- **"agent connected but no modules"** — agent's `--modules-dir` is
  wrong, or the manifest failed to parse. Check the agent's stderr.
- **"create succeeded but instance never starts"** — for SteamCMD-driven
  modules, the install volume is empty. Click *Update* once first; the
  entrypoints exit `EX_CONFIG (78)` with a hint when they can't find
  the dedicated server binary.
- **"environment variable I added to module.yaml never reaches the
  container"** — you forgot to pre-declare the key under
  `runtime.docker.env`. resolve.go silently filters unknown keys.
- **"dashboard broken, can't recover"** — `panelctl admin recreate
  <instance-id>`. If the controller is also down, the data directory
  + Postgres dump is enough to spin up a replacement.
- **Wine modules: container stays "running" but the game never binds
  ports** — usually a top-level launcher stub problem. Invoke the deep
  `Binaries/Win64/<Game>Server-Win64-Shipping.exe` directly (see Conan's
  entrypoint for the canonical pattern).
- **`network_mode: host` + Xvfb collision** — when two host-net
  containers both run xvfb on the same display number you'll see "Server
  is already active for display" in stderr. Always use `xvfb-run -a` to
  auto-allocate.

The maintainer's running list of caught-on-fire-once notes lives in
[`memory/gotchas.md`](memory/gotchas.md).

## Contributing / writing a new module

1. Create `modules/<id>/module.yaml`. Easiest is to copy the closest
   existing module — Source-RCON game? Start from Conan's. Wine? V
   Rising's. Plain Linux dedicated server? Factorio's.
2. If you need a custom image, add a `Dockerfile` next to it. Include
   `setpriv`, `xvfb-run`, and `wine` for Windows-on-Linux modules.
3. Pre-declare every env key that any `config_values` entry maps to
   inside `runtime.docker.env`. Empty defaults are fine.
4. Restart the agent — modules load on boot. The new game appears in
   the dashboard's create-instance dropdown.
5. Test the lifecycle end-to-end: create → update → start → connect a
   client → stop → delete (no purge) → recreate → verify world saves
   survived.
6. If the game has a Workshop / mods scene, add a `<game>mods.go` to
   `controller/cmd/controller/` and surface it in the dashboard's per-game
   Mods tab.

PRs welcome. The maintainer's editorial preferences are recorded in
[`memory/ux_rules.md`](memory/ux_rules.md) — read it before redesigning
any tab.

## License

Apache-2.0 — see [LICENSE](LICENSE).