342 lines
28 KiB
Markdown
342 lines
28 KiB
Markdown
# 7 Days to Die — Cluster Playbook
|
||
|
||
> The single operator + architecture doc for the Refuge 7DTD cluster: how shared
|
||
> characters/inventory work, how new servers auto-fix themselves on create, the
|
||
> one rule you must never break, and exactly what we do at the 3.0 wipe.
|
||
>
|
||
> Deep "why" lives in `CANONICAL_NIM_FREEZE.md` (root cause) and the manual
|
||
> fallback in `nim-freeze/REMAP_RUNBOOK.md`. **This doc is the one to read first.**
|
||
|
||
---
|
||
|
||
## 1. The one thing to understand
|
||
|
||
7DTD stores everything — your inventory, the blocks in the world, the trees —
|
||
by a **numeric id**, not by name. The engine hands out those numbers automatically
|
||
when it boots, and the assignment **drifts** whenever the *build* changes. "Build"
|
||
= the exact set of mods + their load order + the RefugeBot DLL version + the engine
|
||
version. Nothing else matters.
|
||
|
||
* **Same build everywhere → same numbers everywhere.** Inventory transfers cleanly,
|
||
worlds load. (Proven: two cluster servers on the same build produce byte-identical
|
||
id tables.)
|
||
* **Different builds → different numbers → "morph".** A glass block saved as id 21895
|
||
on one server reads as a car hood on another that put a different block at 21895.
|
||
Characters can even reset to level 1.
|
||
|
||
Everything below — the canonical table, the stamp, the decoration remap, the
|
||
auto-provision — exists for **one** reason: to bridge servers/worlds that were
|
||
baked under *different* builds so they all agree on the numbers.
|
||
|
||
---
|
||
|
||
## 2. The golden rule (non-negotiable)
|
||
|
||
> **Every server in the cluster must run the byte-identical build at all times.**
|
||
|
||
Same mods, same load order, same RefugeBot build, same game version — on **all**
|
||
members, always. Break this and inventory morphs. The build *can* evolve over time,
|
||
but only as a **coordinated, all-servers-at-once maintenance event** (see §6), never
|
||
one server at a time.
|
||
|
||
---
|
||
|
||
## 3. The two moving parts
|
||
|
||
| Part | What it is | Who maintains it |
|
||
|---|---|---|
|
||
| **The canonical** | One frozen `name ↔ id` table (`blockmappings.nim` + `itemmappings.nim`) every server stamps on, so the **shared player inventory** (`.ttp`) decodes identically everywhere. Lives at `/cluster/canonical/` (host: `…/7dtdcluster/<id>/canonical/`). | `nim-freeze/rebuild.py` (re-anchor / extend) |
|
||
| **The decoration remap** | A world bakes its **decorations** (`decoration.7dt` / `multiblocks.7dt`) with the engine's *native* ids at generation. Those must be rewritten to the canonical's ids or the world NREs on load. | `nim-freeze/remap_7dt.py`, now **automatic** (see §4) |
|
||
|
||
Why both: the canonical fixes **inventory**, the remap fixes **the world the
|
||
decorations live in**. The deep mechanics are in `CANONICAL_NIM_FREEZE.md`.
|
||
|
||
---
|
||
|
||
## 4. How it works now — auto-provision on create (the steady state)
|
||
|
||
You don't run any of the remap tooling by hand anymore. The game container's
|
||
entrypoint + the agent do it for you. A brand-new clustered world becomes
|
||
canon-compatible **on its own**, in two boots:
|
||
|
||
1. **Gen boot.** You create the server (cluster picked), Update (downloads the
|
||
game), and Start. The engine generates the world and bakes its decorations
|
||
with native ids. The world is up but *not yet aligned* to the cluster.
|
||
2. **Align boot (automatic).** The agent's auto-provision watcher notices the
|
||
fresh world's decorations are in the native band and **bounces the container
|
||
once**. On that boot, the entrypoint remaps `decoration.7dt` + `multiblocks.7dt`
|
||
native → canonical (using the world's own native table), drops a
|
||
`.canon-provisioned` marker, and stamps the canonical. The world comes back
|
||
**clean, on canon, joinable, transfer-safe.**
|
||
|
||
**What you see (visibility):** in the server's **Console** —
|
||
`[panel] auto-provision: aligning the new world's decorations to the cluster…`
|
||
then `remapped decoration.7dt native->canonical` then
|
||
`decorations aligned to cluster canonical — world is now canon-compatible` then
|
||
`StartGame done`. The card briefly shows **"aligning decorations to cluster…"**.
|
||
|
||
**It is a strict no-op for everything else:** existing canon worlds
|
||
(season10/creative/insane/pvp) and non-cluster servers never enter the remap
|
||
branch (their decorations are already in the canon band), so recreating/restarting
|
||
them does nothing. The remap is **stable** — once a world is canon, it stays canon
|
||
across every save and reboot (the engine preserves loaded decoration ids; it only
|
||
writes native ids at *generation*).
|
||
|
||
**The backstop:** even if the auto-bounce is ever missed, the entrypoint aligns the
|
||
world on its *next* ordinary restart. The auto-bounce just makes it zero-touch.
|
||
|
||
### Files that implement it (for the next engineer)
|
||
* `modules/7dtd/entrypoint.sh` — the remap block (right before the canonical-stamp
|
||
guard) + the `.panel-save-base` writer. Gated on the **decoration band** only.
|
||
* `modules/7dtd/Dockerfile` — adds `python3-minimal` + the vendored remapper.
|
||
* `modules/7dtd/nim-freeze/remap_7dt.vendored.py` — self-contained in-container remapper.
|
||
* `agent/internal/dispatch/decoremap_provision.go` — the watcher (single-flight,
|
||
probes the running container, bounces once on a native-band world).
|
||
|
||
---
|
||
|
||
## 5. Adding a new server (operator steps)
|
||
|
||
1. **Add server** in the panel, module **7 Days to Die**, and **pick the cluster**
|
||
in the create form. (This wires the shared Player folder + the canonical mount.)
|
||
2. **Update** (installs the game) → **Start**. Let it generate the world.
|
||
3. **RWG worlds only:** once the world has generated, set the **"Shared player world"**
|
||
in the **Cluster tab** to the new (seed-named) world. The picker writes
|
||
`CLUSTER_PLAYER_SAVE` and triggers a recreate, so the entrypoint stamps + aligns the
|
||
right dir. **Also point Region Medic at the same `<World>/<GameName>`** (Cluster tab →
|
||
Region Medic card), or Medic keeps healing the old, now-deleted world. Until both are
|
||
set, alignment can't target the world.
|
||
4. That's it. The world auto-aligns to the cluster (watch the Console). When you see
|
||
`StartGame done` after the alignment bounce, it's joinable and transfer-safe.
|
||
|
||
> **Is the new world done? Don't let players join until all three pass:**
|
||
> 1. Console shows `StartGame done` *after* the `auto-provision: aligning…` line.
|
||
> 2. Save dir has a `.canon-provisioned` file.
|
||
> 3. Decorations read in the **24xxx** band — verify it yourself (read-only):
|
||
> ```sh
|
||
> docker run --rm -v panel-<inst>-saves:/sv:ro debian:12-slim sh -c \
|
||
> 'f="/sv/.local/share/7DaysToDie/Saves/<World>/<GameName>/decoration.7dt"; \
|
||
> echo "first deco id = $(( $(od -An -tu2 -j17 -N2 "$f") & 0x7FFF ))"'
|
||
> ```
|
||
> `>= 24000` = canon (good). `< 24000` = still native (NOT aligned — do not let players in).
|
||
>
|
||
> **If after 2 boots there's still no `.canon-provisioned` / the band is native** (the
|
||
> §12 #3 case): do **one** more panel **Restart** — the entrypoint backstop aligns it on
|
||
> that boot — and re-check. Still native after that → manual repair via
|
||
> `nim-freeze/REMAP_RUNBOOK.md`.
|
||
|
||
---
|
||
|
||
## 6. Changing mods (a maintenance event, not a casual edit)
|
||
|
||
Because of the golden rule, a mod change is a **whole-cluster** operation:
|
||
|
||
1. Update **every** server's mods together (same set, same order, same RefugeBot
|
||
build). Never one server at a time.
|
||
2. **Rebuild the canonical** so it contains the new names (else they append per-world
|
||
again → morph). This is **two** steps — ⚠️ **`rebuild.py` does NOT discover new mod
|
||
content; it only re-anchors an existing name universe.** Doing 2b without 2a silently
|
||
produces an incomplete canonical and re-introduces the exact wipe bug this system
|
||
prevents.
|
||
* **2a — Re-harvest the complete name universe** (the hard part). Blocks = the
|
||
`<base>:<shape>` cross-product (`shapes.xml` × the 10 shape-helper bases) ∪ the
|
||
engine's authoritative block dump (telnet `exportcurrentconfigs`, or the runtime
|
||
`Block` list) ∪ every named block; items = every `<item name=>` ∪ every
|
||
`<item_modifier name=>`. The original harvest ran as two ultracode workflows; its
|
||
inputs/outputs live on figaro at `~/nimcanon/registry/` but **the generator is not
|
||
committed** — exact greps/commands are in `CANONICAL_NIM_FREEZE.md` → "After a mod
|
||
change". **Do not attempt a mod change without re-running this harvest.**
|
||
* **2b — Re-anchor** against season10's CURRENT, STOPPED `.nim` (the
|
||
`West Apeeni Mountains/MyGame` anchor — verify 38,626/2,719+ first, §13) with the new
|
||
universe: `nim-freeze/rebuild.py` preserves every existing id and appends new names
|
||
at fresh ids (so old worlds + the ~88 shared characters still decode). Verify
|
||
`GATE PASS: True`, then copy into `/cluster/canonical/` (chmod 644).
|
||
3. **Re-stamp every member** so they pick up the new canonical — a panel **Restart**
|
||
is enough (the entrypoint re-stamps from `/cluster/canonical` on every boot; season10
|
||
too, by Restart only — never Rebuild). New worlds genned after the change auto-align
|
||
(§4); existing worlds keep working (their content + the canonical's existing ids are
|
||
unchanged).
|
||
|
||
Adding content is safe (append). Removing/reordering content is where ids shift —
|
||
that's exactly what a **wipe** (§7) resets cleanly.
|
||
|
||
---
|
||
|
||
## 7. The 3.0 wipe roadmap (the easy button)
|
||
|
||
A version wipe is the moment to make the whole thing **bulletproof and remap-free**,
|
||
because you're discarding all the old worlds anyway. Plan:
|
||
|
||
1. **Freeze + hash the 3.0 build.** Lock the exact mod set + load order + RefugeBot DLL +
|
||
engine version, and record a hash of the artifact set (e.g. `tar c Mods/ | sha256sum`)
|
||
into a committed `3.0-build.lock`. That snapshot *is* the contract going forward.
|
||
2. **Wipe all worlds — through the panel, never docker CLI.** Per server: delete the old
|
||
`Saves/<World>/<GameName>` dirs (mount the saves volume, per the runbook's volume-access
|
||
pattern) and set a fresh `world_seed` via `env-config`, or delete-preserve-volumes +
|
||
recreate. Characters too — `.ttp` is build-dependent; a wipe is a fresh start for
|
||
inventories (carrying them across a build change needs the `.7rg`/`.ttp` codec work in §8).
|
||
3. **Re-anchor the canonical to the LIVE 3.0 runtime.** "Capture the current runtime's
|
||
table" = the `REMAP_RUNBOOK.md` **"CAPTURE the world's native table"** sub-procedure: on
|
||
a fresh 3.0 world, Stop → move the canonical aside (back it up first!) → write the 8-byte
|
||
empty `.nim` → Start (the engine writes the world's authoritative native table) → copy it
|
||
out → restore the canonical. Feed that as the new anchor to `rebuild.py` together with a
|
||
freshly-harvested 3.0 universe (§6 step 2a). Now **`canonical == runtime`** and the gap
|
||
that forces per-world remapping is gone.
|
||
4. **From then on, fresh maps just work with zero remap** — every world is born from the
|
||
same starting numbers. The auto-provision/stamp machinery stays in place as a harmless
|
||
safety net; it simply has nothing to fix.
|
||
5. Only a *future* mod change re-introduces the need to rebuild the canonical (§6).
|
||
|
||
> ⚠️ **The 3.0 wipe is irreversible and depends on the (currently uncommitted) §6 harvest
|
||
> workflow.** Do NOT execute it from this strategy alone — when 3.0 lands, write a dedicated,
|
||
> tested 3.0 runbook with the exact harvest + capture + wipe commands first. The plan above
|
||
> is sound; the runnable commands are the gap to close before you pull the trigger.
|
||
|
||
Net: **today** = same-mods rule + auto-provision handles new servers; **at 3.0** =
|
||
clean re-anchor and new maps need no special handling at all. It's a ~one-evening job
|
||
when 3.0 lands; nothing to prep now.
|
||
|
||
The truly permanent fix that removes even the maintenance events: pin every block/item
|
||
to an explicit `id=` in a top-priority modlet so the runtime can never drift from the
|
||
canonical (untested on this engine version — a worthwhile experiment), or freeze the
|
||
RefugeBot build the canonical was anchored under.
|
||
|
||
---
|
||
|
||
## 8. Known limits / frontier
|
||
|
||
* **RWG needs one pick.** A random-gen world's save folder name is seed-derived and
|
||
unknown until after gen, so the operator sets "Shared player world" once (§5 step 3).
|
||
Auto-align kicks in after that.
|
||
* **Regions + player `.ttp` are not remapped.** The codecs for `Region/*.7rg`
|
||
(explored chunk blocks) and `Player/*.ttp` (inventory) weren't cracked, so alignment
|
||
is done on a **fresh, 0-region world before exploration** (which is exactly when a
|
||
new server provisions). The 87 shared characters are fine (already canon-written).
|
||
Solving these codecs (decompile `Assembly-CSharp.dll`) is what would let you migrate
|
||
characters across a build change instead of wiping them.
|
||
|
||
---
|
||
|
||
## 9. Quick reference
|
||
|
||
| Want to… | Do this |
|
||
|---|---|
|
||
| Add a server | §5 (auto-aligns; RWG needs the one Cluster-tab pick) |
|
||
| Confirm a world is aligned | save dir has `.canon-provisioned`; decorations in 24xxx band |
|
||
| Change mods | §6 (all servers together + rebuild canonical) |
|
||
| Wipe for 3.0 | §7 (freeze build → wipe → re-anchor canonical to live runtime) |
|
||
| Manually fix a stuck world | `nim-freeze/REMAP_RUNBOOK.md` |
|
||
| Understand the root cause | `CANONICAL_NIM_FREEZE.md` |
|
||
| Re-anchor / extend the canonical | `nim-freeze/rebuild.py` |
|
||
| Deploy controller / agent / module | §10 (+ the `panel-deploy` skill) |
|
||
| Recover a wedged agent (config fields won't load / 504s) | restart `panel-agent` on figaro — §10 / §11 |
|
||
| Known issues & open work | §12 (season10 console, the "setting up" indicator, probe-path) |
|
||
| What's deployed vs. uncommitted | §13 |
|
||
|
||
---
|
||
|
||
## 10. Operational architecture & deploy (for the next engineer)
|
||
|
||
**Where it runs**
|
||
|
||
| Piece | Host | Notes |
|
||
|---|---|---|
|
||
| **Controller** (dashboard + gRPC) | kaiten `<user>@<controller-host>` | binary `/home/refuge/panel/bin/controller`, unit `panel-controller.service`, HTTP `:8180`, gRPC `:8443` |
|
||
| **Agent** (owns the docker game containers) | figaro `<user>@<agent-host>` | binary `/home/refuge/panel/bin/agent`, unit `panel-agent.service`, dials the controller |
|
||
| **Cluster shared dir** (the `/cluster` bind) | figaro `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/` | holds `canonical/` + `Player/`; bind-mounted into every member as `/cluster` |
|
||
| **Game containers** | figaro, image `panel-7dtd:latest` | named `panel-<instance>` (e.g. `panel-rg-season-10`) |
|
||
|
||
SSH everywhere with `~/.ssh/bubbly_ed25519`, user `refuge`. **The full deploy recipe is the `panel-deploy` skill — follow it.** The NOPASSWD sudoers is exact-match (single-service `systemctl` forms only).
|
||
|
||
**Panel access & conventions (read before driving anything)**
|
||
* **Dashboard:** `http://<controller-host>:8180` on the LAN (controller HTTP). Log in with your panel account. On kaiten itself the `http://127.0.0.1:8180/api/...` routes are unauthenticated — that's what the read-only diagnostic `curl`s in this doc use.
|
||
* **Server name → ids:** an instance's id is its slug. "Season 10" ⇒ instance id `rg-season-10` ⇒ container `panel-rg-season-10` ⇒ saves volume `panel-rg-season-10-saves`. List them all (with status) via `GET /api/instances`. Full live table in §13.
|
||
* **UI = API:** the `POST /api/instances/<id>/{env-config,start,stop,rebuild}` calls in these docs are exactly what the dashboard's Cluster-tab / Maintenance buttons fire. Use either — just target the right `<id>`.
|
||
* **"Stop / Start / Restart" always means the PANEL action** (agent-owned, clean shutdown). **NEVER `docker stop/start` a `panel-*` container** — the agent owns lifecycle + state reconciliation. Confirm a container is stopped via the grey card, or read-only `docker inspect -f '{{.State.Running}}' panel-<id>` → `false`. (Read-only `docker exec/logs/inspect` and `docker run` sidecars are fine.)
|
||
* **"Restart" vs "Rebuild" (load-bearing):** Restart = panel Stop then Start, **same image** — safe for any server *including season10*. **Rebuild/Recreate = adopts `panel-7dtd:latest`** — needed to pick up a module/image change, but **forbidden for `rg-season-10`** (it must stay on its old image; §13).
|
||
* **New entrypoint env vars MUST be declared in `module.yaml` `env:`** or the agent silently drops them (symptom: the var is empty at runtime even though you set it). `CLUSTER_ID` + `CLUSTER_PLAYER_SAVE` are already declared — add any new `CLUSTER_*` / provisioning var there too.
|
||
|
||
**Three deployable pieces**
|
||
|
||
1. **Controller** (Go + the dashboard HTML in `controller/cmd/controller/static/{new,index}.html`) — `GOOS=linux GOARCH=amd64 go build -o bin/controller-linux ./controller/cmd/controller`, scp to kaiten `bin/controller.new`, bak + swap + `systemctl restart panel-controller.service`.
|
||
* ⚠️ **After every controller deploy, HARD-refresh the browser (Ctrl-Shift-R).** A normal refresh serves the *old* cached JS against the new backend and the page half-breaks (fields won't load, console blank). It looks exactly like a real outage but is just stale cache — rule this out first.
|
||
2. **Agent** (Go) — `… go build -o bin/agent-linux ./agent/cmd/agent`, scp to figaro, bak + swap + `systemctl restart panel-agent.service`.
|
||
* Restarting the agent does **NOT** stop the game containers — it re-attaches/rehydrates them cleanly (stats + rcon + log streams re-open). Safe anytime; it is also the **recovery** for a wedged agent (§11).
|
||
3. **Module** (`modules/7dtd/{entrypoint.sh,Dockerfile,nim-freeze/remap_7dt.vendored.py}`) — sync the module dir Windows → kaiten → figaro, then build the image **on figaro** (where the containers run; kaiten holds the source-of-truth copy but builds nothing):
|
||
```sh
|
||
# from the Windows repo C:\Users\dbled\sources\panel :
|
||
scp -r modules/7dtd/ <user>@<controller-host>:/home/refuge/panel/modules/
|
||
ssh <user>@<controller-host> 'rsync -av /home/refuge/panel/modules/7dtd/ <user>@<agent-host>:/home/refuge/panel/modules/7dtd/'
|
||
ssh <user>@<agent-host> 'cd /home/refuge/panel/modules/7dtd && docker build -t panel-7dtd:latest .'
|
||
```
|
||
Existing containers **pin to the old image ID** — a container only adopts the new image via **Settings → Maintenance → Rebuild** (or delete-preserve-volumes + recreate).
|
||
* ⚠️ **Never Rebuild/recreate `rg-season-10`.** It is the live released server *and* the canonical anchor; it must stay on its current (old) image. `.Config.Image` shows the name `:latest` for every container — verify by image **ID**, not name.
|
||
|
||
**Do NOT `docker stop/start` a `panel-*` game container from the CLI** — let the panel do it (the agent owns lifecycle + state reconciliation). Read-only `docker exec`, `docker logs`, `docker inspect`, and `docker run` sidecars are fine.
|
||
|
||
---
|
||
|
||
## 11. Incident log & landmines
|
||
|
||
### 2026-06-13 — agent-wide hang (ExecCapture connection leak) ✅ FIXED + HARDENED
|
||
|
||
* **Symptom:** config fields wouldn't load on **any** server (the durable `serverconfig.xml` editor → `/api/instances/{id}/files/read` returned **HTTP 504 "timeout waiting for agent response"**); one server's console went blank; the agent was **idle (0% CPU) but unresponsive to file ops**; no panic.
|
||
* **Root cause:** the auto-provision watcher (`agent/internal/dispatch/decoremap_provision.go`) polls a running container with a docker exec (`DockerRuntime.ExecCapture`) every 20s. `ExecCapture` ran `stdcopy.StdCopy(…att.Reader)` on the hijacked exec stream **with no `ctx` enforcement on the Read**. When it caught the just-bounced `cluster-seed-test` mid-restart, the read blocked **forever** and leaked a docker SDK connection each cycle → the SDK client's connection pool exhausted → **every** later docker op (`CopyFileFromContainer` for config reads, a log-stream re-attach) hung. It is **not** a Go-mutex deadlock (the only mutex in `docker.go` — `hijackedStream.mu` — just guards the stdio-attach `Close()` and is unrelated); it's connection-pool starvation.
|
||
* **Recover:** `ssh <user>@<agent-host> 'sudo -n /usr/bin/systemctl restart panel-agent.service'` — clears the leaked connections; game containers keep running. **Confirm recovery:** re-open any server's Config tab (fields load within a few seconds), or on kaiten `curl -s -o /dev/null -w '%{http_code}\n' 'http://127.0.0.1:8180/api/instances/rg-season-10/files/read?path=/game-saves/serverconfig.xml'` → expect **200**, not 504.
|
||
* **Permanent fix (shipped):** `ExecCapture` now runs `StdCopy` on a goroutine and `select`s on `ctx.Done()`; on timeout it `att.Close()`s to unblock the read, then drains the goroutine before reading the buffers (race-free). On the timeout path it returns the partial stdout/stderr with exit code `-1` and `exec copy: <ctx err>`, so callers can tell a wedged exec from a clean one. An exec can no longer hang the whole agent.
|
||
* **Landmine for the future:** *any* agent docker exec/stream read that ignores `ctx` can leak connections and wedge the entire agent the same way. If you add exec-based features, bound the read by `ctx`. (The log-stream demux at `docker.go:271` is a long-lived `Follow` stream — a different model, closes on the docker `rc`'s EOF/ctx — and is fine.)
|
||
* **Recognize it next time:** `files/read` 504s for *every* server + agent at 0% CPU + no crash ⇒ restart the agent first, then hunt for a new unbounded docker read.
|
||
|
||
---
|
||
|
||
## 12. Known issues & open work
|
||
|
||
1. **season10's panel console is blank** (separate from §11; pre-existing). season10's docker json-log has grown to **13 MB** and it is actively spamming exceptions: `No ItemClass entry for type 678xx` (×611) + `Log:Exception` / `ThreadManager:UpdateMainThreadTasks`. Docker's own `--follow --tail 400` serves season10 instantly and the other four servers stream fine, but the agent's SDK log-follow **stalls on season10's stream before forwarding a single line** (blocked-inside, not thrashing — no "log stream ended" retries). Re-attaching (two agent restarts) did **not** fix it. It will likely clear on a **season10 container restart** — a plain panel **Stop → Start**, which keeps season10 on its current image and is safe; this is **not** a Rebuild/recreate (that would adopt `panel-7dtd:latest` and break the canonical anchor — forbidden, see §10/§13). It may re-blank once the exception spam refills the log. **Durable fix:** resolve the `No ItemClass entry` item-config gap (kills the console stall, the log bloat, *and* the exceptions at once), **or** harden the agent log-follow — mechanism not fully pinned (the agent has no pprof endpoint, so no clean goroutine dump was possible). **Do NOT recreate season10 to fix this** (old-image constraint, §10).
|
||
|
||
2. **The "setting up" provisioning indicator doesn't surface.** The agent watcher emits `detail="Finishing cluster setup…"` and the dashboard renders a matching amber "setting up" pill — but the agent's own **state-reconciliation loop emits `detail="announced:running"` a beat later and overwrites it**, so the card flips green prematurely. The **Console line** (`auto-provision: aligning the new world's decorations to the cluster…`) *does* show and survives a refresh — that part works. Proper fix: a **dedicated provisioning state field** the reconciler respects, not the shared `detail` string. (The card-pill matcher is live but dormant in `new.html`/`index.html`; harmless.)
|
||
|
||
3. **Auto-provision watcher probe-path caveat (zero-touch isn't universal yet).** The watcher gates "gen finished" on `main.ttw` **only**; it reads `decoration.7dt` solely to detect the already-canon (≥24000) case. A *missing* `decoration.7dt` still yields `NEEDS_ALIGN` and a bounce — the graceful-stop save writes the native deco, which the entrypoint then remaps on the align boot. For some RWG worlds (e.g. Tuxeno on `cluster-seed-test`/pvp) that two-step doesn't always *complete* zero-touch, but the **entrypoint backstop still aligns the world on its next ordinary restart**, so it ends up correct. Worlds that bake deco at the save path at gen (e.g. refuge-mall) provision fully zero-touch. **Treat the entrypoint remap as the reliable path; the watcher bounce is the optimization.**
|
||
|
||
4. **`No ItemClass entry for type 678xx` on season10** (root of #1). A set of high-band item ids (mod / RefugeBot runtime items) the merged config references but doesn't define. Tracing the missing items cleans up the console stall, the 13 MB log bloat, and the on-server exceptions in one shot. Not urgent — season10 runs fine (20 FPS, players on).
|
||
|
||
---
|
||
|
||
## 13. Handoff status — what's deployed, what's uncommitted
|
||
|
||
**Cluster:** `cl_37e3a7f72cdc`. Shared `Player/` ≈ 88 `.ttp`. Live members:
|
||
|
||
| Server | Instance id | Container / saves volume | Active world (`CLUSTER_PLAYER_SAVE`) | Role |
|
||
|---|---|---|---|---|
|
||
| Season 10 | `rg-season-10` | `panel-rg-season-10` / `panel-rg-season-10-saves` | `West Apeeni Mountains/MyGame` *(canonical anchor)* | **master / anchor — DO NOT Rebuild** |
|
||
| Creative | `creative` | `panel-creative` / `panel-creative-saves` | `Tefasizi County/Refuge` | creative |
|
||
| Insane Survival | `insane-survival` | `panel-insane-survival` / `panel-insane-survival-saves` | `Yixacove Valley/Refuge` | survival |
|
||
| PvP | `cluster-seed-test` | `panel-cluster-seed-test` / `panel-cluster-seed-test-saves` | `Tuxeno Territory/RefugeTHREEPVP` | pvp |
|
||
| Mall | `refuge-mall` | `panel-refuge-mall` / `panel-refuge-mall-saves` | `New Vebape Territory/RefugeMall` | mall |
|
||
|
||
> **Canonical anchor — the single highest-blast-radius fact.** The complete frozen table (**38,626 blocks / 2,719 items**) lives ONLY at season10's `…/Saves/West Apeeni Mountains/MyGame/{block,item}mappings.nim`. season10's *other* save dirs are stale partials (e.g. `Navezgane/MyGame` = 3,356 blocks, `Iupu Territory/Troutdale` = 5,028) and its `CLUSTER_PLAYER_SAVE` env is **empty** (old image, predates the picker — so don't trust the env to find the anchor). Any re-anchor (§6 / §7) MUST copy from exactly `West Apeeni Mountains/MyGame` and **verify the counts read 38,626 / 2,719 (or higher — never lower)** before running `rebuild.py`. Anchoring against a partial silently wipes the ~88 shared characters.
|
||
|
||
**Deployed (live):**
|
||
* **Controller** on kaiten — has the create-form cluster note + the auto-provision "starting" chip + the (dormant) "setting up" card-pill code.
|
||
* **Agent** on figaro — has the auto-provision watcher **and** the hardened `ExecCapture` (§11).
|
||
* **Image `panel-7dtd:latest`** on figaro — has the entrypoint remap block + `python3-minimal` + the vendored `remap_7dt.py`. Running on creative / insane-survival / cluster-seed-test / refuge-mall.
|
||
* **`rg-season-10` deliberately on the OLD image** (no `.panel-save-base`, no remap block) — the live anchor; do not move it.
|
||
* **refuge-mall** — created via the panel this session and registered as a RefugeBot tenant (prod DB row Id=31 + `refugebot.json` in its saves volume), connected.
|
||
|
||
**⚠️ UNCOMMITTED — nothing from this work is in git.** `C:\Users\dbled\sources\panel` is a git working tree but all of the below is uncommitted: the controller dashboard edits, the agent Go (`decoremap_provision.go`, the `dispatch.go` hook + `provisionGuard`, the `ExecCapture` hardening), the module (`entrypoint.sh` remap block, `Dockerfile`, `nim-freeze/remap_7dt.vendored.py`), and these docs. **Rollback points** if a deploy goes bad: `controller.bak-<ts>` on kaiten, `agent.bak-<ts>` on figaro (newest = the binary just before the current one).
|
||
|
||
**If/when you commit,** suggested grouping: (1) canonical + deco freeze/remap, (2) auto-provision (entrypoint + watcher), (3) the `ExecCapture` hardening, (4) docs. Note in the message that the cluster is **live** so a reviewer treats it as load-bearing.
|
||
|
||
---
|
||
|
||
## 14. Disaster recovery — the three irreplaceable trees
|
||
|
||
These three live on figaro and **cannot be regenerated from the repo** — they *are* the source of truth. If figaro's disk dies, this is what hurts:
|
||
|
||
| Tree | Path on figaro | What it is | If lost |
|
||
|---|---|---|---|
|
||
| **The canonical** | `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/canonical/{block,item}mappings.nim` | the frozen stamp source every member uses | rebuildable from season10's `West Apeeni Mountains/MyGame` anchor via `rebuild.py` (needs `~/nimcanon`) |
|
||
| **Shared characters** | `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/Player/` (= `/cluster/Player`) | the ~88 player `.ttp` (every character + inventory) | **unrecoverable** — everyone resets to level 1 |
|
||
| **The rebuild toolchain** | `~/nimcanon/` (`tool/`, `registry/`, `canonical_final/`, `remap/`) | the only parser + name registry + capture tooling to rebuild any of the above | must re-harvest from scratch (the §6 harvest — currently uncommitted) |
|
||
|
||
> ⚠️ **Open risk — confirm these are actually backed up off-figaro.** `~/nimcanon` is persisted out of `/tmp` (survives a reboot) but a disk loss is unaddressed in these docs. **Recommendation:** snapshot all three trees to another host on a schedule (reuse the panel's backup system or a periodic `tar`). Restore order: canonical → Player → validate with an in-game transfer (CANONICAL_NIM_FREEZE "Verify a server is frozen"). Treat this as the **top open risk** until a backup is confirmed.
|