# 7 Days to Die — Cluster Playbook > The single operator + architecture doc for the Refuge 7DTD cluster: how shared > characters/inventory work, how new servers auto-fix themselves on create, the > one rule you must never break, and exactly what we do at the 3.0 wipe. > > Deep "why" lives in `CANONICAL_NIM_FREEZE.md` (root cause) and the manual > fallback in `nim-freeze/REMAP_RUNBOOK.md`. **This doc is the one to read first.** --- ## 1. The one thing to understand 7DTD stores everything — your inventory, the blocks in the world, the trees — by a **numeric id**, not by name. The engine hands out those numbers automatically when it boots, and the assignment **drifts** whenever the *build* changes. "Build" = the exact set of mods + their load order + the RefugeBot DLL version + the engine version. Nothing else matters. * **Same build everywhere → same numbers everywhere.** Inventory transfers cleanly, worlds load. (Proven: two cluster servers on the same build produce byte-identical id tables.) * **Different builds → different numbers → "morph".** A glass block saved as id 21895 on one server reads as a car hood on another that put a different block at 21895. Characters can even reset to level 1. Everything below — the canonical table, the stamp, the decoration remap, the auto-provision — exists for **one** reason: to bridge servers/worlds that were baked under *different* builds so they all agree on the numbers. --- ## 2. The golden rule (non-negotiable) > **Every server in the cluster must run the byte-identical build at all times.** Same mods, same load order, same RefugeBot build, same game version — on **all** members, always. Break this and inventory morphs. The build *can* evolve over time, but only as a **coordinated, all-servers-at-once maintenance event** (see §6), never one server at a time. --- ## 3. The two moving parts | Part | What it is | Who maintains it | |---|---|---| | **The canonical** | One frozen `name ↔ id` table (`blockmappings.nim` + `itemmappings.nim`) every server stamps on, so the **shared player inventory** (`.ttp`) decodes identically everywhere. Lives at `/cluster/canonical/` (host: `…/7dtdcluster//canonical/`). | `nim-freeze/rebuild.py` (re-anchor / extend) | | **The decoration remap** | A world bakes its **decorations** (`decoration.7dt` / `multiblocks.7dt`) with the engine's *native* ids at generation. Those must be rewritten to the canonical's ids or the world NREs on load. | `nim-freeze/remap_7dt.py`, now **automatic** (see §4) | Why both: the canonical fixes **inventory**, the remap fixes **the world the decorations live in**. The deep mechanics are in `CANONICAL_NIM_FREEZE.md`. --- ## 4. How it works now — auto-provision on create (the steady state) You don't run any of the remap tooling by hand anymore. The game container's entrypoint + the agent do it for you. A brand-new clustered world becomes canon-compatible **on its own**, in two boots: 1. **Gen boot.** You create the server (cluster picked), Update (downloads the game), and Start. The engine generates the world and bakes its decorations with native ids. The world is up but *not yet aligned* to the cluster. 2. **Align boot (automatic).** The agent's auto-provision watcher notices the fresh world's decorations are in the native band and **bounces the container once**. On that boot, the entrypoint remaps `decoration.7dt` + `multiblocks.7dt` native → canonical (using the world's own native table), drops a `.canon-provisioned` marker, and stamps the canonical. The world comes back **clean, on canon, joinable, transfer-safe.** **What you see (visibility):** in the server's **Console** — `[panel] auto-provision: aligning the new world's decorations to the cluster…` then `remapped decoration.7dt native->canonical` then `decorations aligned to cluster canonical — world is now canon-compatible` then `StartGame done`. The card briefly shows **"aligning decorations to cluster…"**. **It is a strict no-op for everything else:** existing canon worlds (season10/creative/insane/pvp) and non-cluster servers never enter the remap branch (their decorations are already in the canon band), so recreating/restarting them does nothing. The remap is **stable** — once a world is canon, it stays canon across every save and reboot (the engine preserves loaded decoration ids; it only writes native ids at *generation*). **The backstop:** even if the auto-bounce is ever missed, the entrypoint aligns the world on its *next* ordinary restart. The auto-bounce just makes it zero-touch. ### Files that implement it (for the next engineer) * `modules/7dtd/entrypoint.sh` — the remap block (right before the canonical-stamp guard) + the `.panel-save-base` writer. Gated on the **decoration band** only. * `modules/7dtd/Dockerfile` — adds `python3-minimal` + the vendored remapper. * `modules/7dtd/nim-freeze/remap_7dt.vendored.py` — self-contained in-container remapper. * `agent/internal/dispatch/decoremap_provision.go` — the watcher (single-flight, probes the running container, bounces once on a native-band world). --- ## 5. Adding a new server (operator steps) 1. **Add server** in the panel, module **7 Days to Die**, and **pick the cluster** in the create form. (This wires the shared Player folder + the canonical mount.) 2. **Update** (installs the game) → **Start**. Let it generate the world. 3. **RWG worlds only:** once the world has generated, set the **"Shared player world"** in the **Cluster tab** to the new (seed-named) world. The picker writes `CLUSTER_PLAYER_SAVE` and triggers a recreate, so the entrypoint stamps + aligns the right dir. **Also point Region Medic at the same `/`** (Cluster tab → Region Medic card), or Medic keeps healing the old, now-deleted world. Until both are set, alignment can't target the world. 4. That's it. The world auto-aligns to the cluster (watch the Console). When you see `StartGame done` after the alignment bounce, it's joinable and transfer-safe. > **Is the new world done? Don't let players join until all three pass:** > 1. Console shows `StartGame done` *after* the `auto-provision: aligning…` line. > 2. Save dir has a `.canon-provisioned` file. > 3. Decorations read in the **24xxx** band — verify it yourself (read-only): > ```sh > docker run --rm -v panel--saves:/sv:ro debian:12-slim sh -c \ > 'f="/sv/.local/share/7DaysToDie/Saves///decoration.7dt"; \ > echo "first deco id = $(( $(od -An -tu2 -j17 -N2 "$f") & 0x7FFF ))"' > ``` > `>= 24000` = canon (good). `< 24000` = still native (NOT aligned — do not let players in). > > **If after 2 boots there's still no `.canon-provisioned` / the band is native** (the > §12 #3 case): do **one** more panel **Restart** — the entrypoint backstop aligns it on > that boot — and re-check. Still native after that → manual repair via > `nim-freeze/REMAP_RUNBOOK.md`. --- ## 6. Changing mods (a maintenance event, not a casual edit) Because of the golden rule, a mod change is a **whole-cluster** operation: 1. Update **every** server's mods together (same set, same order, same RefugeBot build). Never one server at a time. 2. **Rebuild the canonical** so it contains the new names (else they append per-world again → morph). This is **two** steps — ⚠️ **`rebuild.py` does NOT discover new mod content; it only re-anchors an existing name universe.** Doing 2b without 2a silently produces an incomplete canonical and re-introduces the exact wipe bug this system prevents. * **2a — Re-harvest the complete name universe** (the hard part). Blocks = the `:` cross-product (`shapes.xml` × the 10 shape-helper bases) ∪ the engine's authoritative block dump (telnet `exportcurrentconfigs`, or the runtime `Block` list) ∪ every named block; items = every `` ∪ every ``. The original harvest ran as two ultracode workflows; its inputs/outputs live on figaro at `~/nimcanon/registry/` but **the generator is not committed** — exact greps/commands are in `CANONICAL_NIM_FREEZE.md` → "After a mod change". **Do not attempt a mod change without re-running this harvest.** * **2b — Re-anchor** against season10's CURRENT, STOPPED `.nim` (the `West Apeeni Mountains/MyGame` anchor — verify 38,626/2,719+ first, §13) with the new universe: `nim-freeze/rebuild.py` preserves every existing id and appends new names at fresh ids (so old worlds + the ~88 shared characters still decode). Verify `GATE PASS: True`, then copy into `/cluster/canonical/` (chmod 644). 3. **Re-stamp every member** so they pick up the new canonical — a panel **Restart** is enough (the entrypoint re-stamps from `/cluster/canonical` on every boot; season10 too, by Restart only — never Rebuild). New worlds genned after the change auto-align (§4); existing worlds keep working (their content + the canonical's existing ids are unchanged). Adding content is safe (append). Removing/reordering content is where ids shift — that's exactly what a **wipe** (§7) resets cleanly. --- ## 7. The 3.0 wipe roadmap (the easy button) A version wipe is the moment to make the whole thing **bulletproof and remap-free**, because you're discarding all the old worlds anyway. Plan: 1. **Freeze + hash the 3.0 build.** Lock the exact mod set + load order + RefugeBot DLL + engine version, and record a hash of the artifact set (e.g. `tar c Mods/ | sha256sum`) into a committed `3.0-build.lock`. That snapshot *is* the contract going forward. 2. **Wipe all worlds — through the panel, never docker CLI.** Per server: delete the old `Saves//` dirs (mount the saves volume, per the runbook's volume-access pattern) and set a fresh `world_seed` via `env-config`, or delete-preserve-volumes + recreate. Characters too — `.ttp` is build-dependent; a wipe is a fresh start for inventories (carrying them across a build change needs the `.7rg`/`.ttp` codec work in §8). 3. **Re-anchor the canonical to the LIVE 3.0 runtime.** "Capture the current runtime's table" = the `REMAP_RUNBOOK.md` **"CAPTURE the world's native table"** sub-procedure: on a fresh 3.0 world, Stop → move the canonical aside (back it up first!) → write the 8-byte empty `.nim` → Start (the engine writes the world's authoritative native table) → copy it out → restore the canonical. Feed that as the new anchor to `rebuild.py` together with a freshly-harvested 3.0 universe (§6 step 2a). Now **`canonical == runtime`** and the gap that forces per-world remapping is gone. 4. **From then on, fresh maps just work with zero remap** — every world is born from the same starting numbers. The auto-provision/stamp machinery stays in place as a harmless safety net; it simply has nothing to fix. 5. Only a *future* mod change re-introduces the need to rebuild the canonical (§6). > ⚠️ **The 3.0 wipe is irreversible and depends on the (currently uncommitted) §6 harvest > workflow.** Do NOT execute it from this strategy alone — when 3.0 lands, write a dedicated, > tested 3.0 runbook with the exact harvest + capture + wipe commands first. The plan above > is sound; the runnable commands are the gap to close before you pull the trigger. Net: **today** = same-mods rule + auto-provision handles new servers; **at 3.0** = clean re-anchor and new maps need no special handling at all. It's a ~one-evening job when 3.0 lands; nothing to prep now. The truly permanent fix that removes even the maintenance events: pin every block/item to an explicit `id=` in a top-priority modlet so the runtime can never drift from the canonical (untested on this engine version — a worthwhile experiment), or freeze the RefugeBot build the canonical was anchored under. --- ## 8. Known limits / frontier * **RWG needs one pick.** A random-gen world's save folder name is seed-derived and unknown until after gen, so the operator sets "Shared player world" once (§5 step 3). Auto-align kicks in after that. * **Regions + player `.ttp` are not remapped.** The codecs for `Region/*.7rg` (explored chunk blocks) and `Player/*.ttp` (inventory) weren't cracked, so alignment is done on a **fresh, 0-region world before exploration** (which is exactly when a new server provisions). The 87 shared characters are fine (already canon-written). Solving these codecs (decompile `Assembly-CSharp.dll`) is what would let you migrate characters across a build change instead of wiping them. --- ## 9. Quick reference | Want to… | Do this | |---|---| | Add a server | §5 (auto-aligns; RWG needs the one Cluster-tab pick) | | Confirm a world is aligned | save dir has `.canon-provisioned`; decorations in 24xxx band | | Change mods | §6 (all servers together + rebuild canonical) | | Wipe for 3.0 | §7 (freeze build → wipe → re-anchor canonical to live runtime) | | Manually fix a stuck world | `nim-freeze/REMAP_RUNBOOK.md` | | Understand the root cause | `CANONICAL_NIM_FREEZE.md` | | Re-anchor / extend the canonical | `nim-freeze/rebuild.py` | | Deploy controller / agent / module | §10 (+ the `panel-deploy` skill) | | Recover a wedged agent (config fields won't load / 504s) | restart `panel-agent` on figaro — §10 / §11 | | Known issues & open work | §12 (season10 console, the "setting up" indicator, probe-path) | | What's deployed vs. uncommitted | §13 | --- ## 10. Operational architecture & deploy (for the next engineer) **Where it runs** | Piece | Host | Notes | |---|---|---| | **Controller** (dashboard + gRPC) | kaiten `@` | binary `/home/refuge/panel/bin/controller`, unit `panel-controller.service`, HTTP `:8180`, gRPC `:8443` | | **Agent** (owns the docker game containers) | figaro `@` | binary `/home/refuge/panel/bin/agent`, unit `panel-agent.service`, dials the controller | | **Cluster shared dir** (the `/cluster` bind) | figaro `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/` | holds `canonical/` + `Player/`; bind-mounted into every member as `/cluster` | | **Game containers** | figaro, image `panel-7dtd:latest` | named `panel-` (e.g. `panel-rg-season-10`) | SSH everywhere with `~/.ssh/bubbly_ed25519`, user `refuge`. **The full deploy recipe is the `panel-deploy` skill — follow it.** The NOPASSWD sudoers is exact-match (single-service `systemctl` forms only). **Panel access & conventions (read before driving anything)** * **Dashboard:** `http://:8180` on the LAN (controller HTTP). Log in with your panel account. On kaiten itself the `http://127.0.0.1:8180/api/...` routes are unauthenticated — that's what the read-only diagnostic `curl`s in this doc use. * **Server name → ids:** an instance's id is its slug. "Season 10" ⇒ instance id `rg-season-10` ⇒ container `panel-rg-season-10` ⇒ saves volume `panel-rg-season-10-saves`. List them all (with status) via `GET /api/instances`. Full live table in §13. * **UI = API:** the `POST /api/instances//{env-config,start,stop,rebuild}` calls in these docs are exactly what the dashboard's Cluster-tab / Maintenance buttons fire. Use either — just target the right ``. * **"Stop / Start / Restart" always means the PANEL action** (agent-owned, clean shutdown). **NEVER `docker stop/start` a `panel-*` container** — the agent owns lifecycle + state reconciliation. Confirm a container is stopped via the grey card, or read-only `docker inspect -f '{{.State.Running}}' panel-` → `false`. (Read-only `docker exec/logs/inspect` and `docker run` sidecars are fine.) * **"Restart" vs "Rebuild" (load-bearing):** Restart = panel Stop then Start, **same image** — safe for any server *including season10*. **Rebuild/Recreate = adopts `panel-7dtd:latest`** — needed to pick up a module/image change, but **forbidden for `rg-season-10`** (it must stay on its old image; §13). * **New entrypoint env vars MUST be declared in `module.yaml` `env:`** or the agent silently drops them (symptom: the var is empty at runtime even though you set it). `CLUSTER_ID` + `CLUSTER_PLAYER_SAVE` are already declared — add any new `CLUSTER_*` / provisioning var there too. **Three deployable pieces** 1. **Controller** (Go + the dashboard HTML in `controller/cmd/controller/static/{new,index}.html`) — `GOOS=linux GOARCH=amd64 go build -o bin/controller-linux ./controller/cmd/controller`, scp to kaiten `bin/controller.new`, bak + swap + `systemctl restart panel-controller.service`. * ⚠️ **After every controller deploy, HARD-refresh the browser (Ctrl-Shift-R).** A normal refresh serves the *old* cached JS against the new backend and the page half-breaks (fields won't load, console blank). It looks exactly like a real outage but is just stale cache — rule this out first. 2. **Agent** (Go) — `… go build -o bin/agent-linux ./agent/cmd/agent`, scp to figaro, bak + swap + `systemctl restart panel-agent.service`. * Restarting the agent does **NOT** stop the game containers — it re-attaches/rehydrates them cleanly (stats + rcon + log streams re-open). Safe anytime; it is also the **recovery** for a wedged agent (§11). 3. **Module** (`modules/7dtd/{entrypoint.sh,Dockerfile,nim-freeze/remap_7dt.vendored.py}`) — sync the module dir Windows → kaiten → figaro, then build the image **on figaro** (where the containers run; kaiten holds the source-of-truth copy but builds nothing): ```sh # from the Windows repo C:\Users\dbled\sources\panel : scp -r modules/7dtd/ @:/home/refuge/panel/modules/ ssh @ 'rsync -av /home/refuge/panel/modules/7dtd/ @:/home/refuge/panel/modules/7dtd/' ssh @ 'cd /home/refuge/panel/modules/7dtd && docker build -t panel-7dtd:latest .' ``` Existing containers **pin to the old image ID** — a container only adopts the new image via **Settings → Maintenance → Rebuild** (or delete-preserve-volumes + recreate). * ⚠️ **Never Rebuild/recreate `rg-season-10`.** It is the live released server *and* the canonical anchor; it must stay on its current (old) image. `.Config.Image` shows the name `:latest` for every container — verify by image **ID**, not name. **Do NOT `docker stop/start` a `panel-*` game container from the CLI** — let the panel do it (the agent owns lifecycle + state reconciliation). Read-only `docker exec`, `docker logs`, `docker inspect`, and `docker run` sidecars are fine. --- ## 11. Incident log & landmines ### 2026-06-13 — agent-wide hang (ExecCapture connection leak) ✅ FIXED + HARDENED * **Symptom:** config fields wouldn't load on **any** server (the durable `serverconfig.xml` editor → `/api/instances/{id}/files/read` returned **HTTP 504 "timeout waiting for agent response"**); one server's console went blank; the agent was **idle (0% CPU) but unresponsive to file ops**; no panic. * **Root cause:** the auto-provision watcher (`agent/internal/dispatch/decoremap_provision.go`) polls a running container with a docker exec (`DockerRuntime.ExecCapture`) every 20s. `ExecCapture` ran `stdcopy.StdCopy(…att.Reader)` on the hijacked exec stream **with no `ctx` enforcement on the Read**. When it caught the just-bounced `cluster-seed-test` mid-restart, the read blocked **forever** and leaked a docker SDK connection each cycle → the SDK client's connection pool exhausted → **every** later docker op (`CopyFileFromContainer` for config reads, a log-stream re-attach) hung. It is **not** a Go-mutex deadlock (the only mutex in `docker.go` — `hijackedStream.mu` — just guards the stdio-attach `Close()` and is unrelated); it's connection-pool starvation. * **Recover:** `ssh @ 'sudo -n /usr/bin/systemctl restart panel-agent.service'` — clears the leaked connections; game containers keep running. **Confirm recovery:** re-open any server's Config tab (fields load within a few seconds), or on kaiten `curl -s -o /dev/null -w '%{http_code}\n' 'http://127.0.0.1:8180/api/instances/rg-season-10/files/read?path=/game-saves/serverconfig.xml'` → expect **200**, not 504. * **Permanent fix (shipped):** `ExecCapture` now runs `StdCopy` on a goroutine and `select`s on `ctx.Done()`; on timeout it `att.Close()`s to unblock the read, then drains the goroutine before reading the buffers (race-free). On the timeout path it returns the partial stdout/stderr with exit code `-1` and `exec copy: `, so callers can tell a wedged exec from a clean one. An exec can no longer hang the whole agent. * **Landmine for the future:** *any* agent docker exec/stream read that ignores `ctx` can leak connections and wedge the entire agent the same way. If you add exec-based features, bound the read by `ctx`. (The log-stream demux at `docker.go:271` is a long-lived `Follow` stream — a different model, closes on the docker `rc`'s EOF/ctx — and is fine.) * **Recognize it next time:** `files/read` 504s for *every* server + agent at 0% CPU + no crash ⇒ restart the agent first, then hunt for a new unbounded docker read. --- ## 12. Known issues & open work 1. **season10's panel console is blank** (separate from §11; pre-existing). season10's docker json-log has grown to **13 MB** and it is actively spamming exceptions: `No ItemClass entry for type 678xx` (×611) + `Log:Exception` / `ThreadManager:UpdateMainThreadTasks`. Docker's own `--follow --tail 400` serves season10 instantly and the other four servers stream fine, but the agent's SDK log-follow **stalls on season10's stream before forwarding a single line** (blocked-inside, not thrashing — no "log stream ended" retries). Re-attaching (two agent restarts) did **not** fix it. It will likely clear on a **season10 container restart** — a plain panel **Stop → Start**, which keeps season10 on its current image and is safe; this is **not** a Rebuild/recreate (that would adopt `panel-7dtd:latest` and break the canonical anchor — forbidden, see §10/§13). It may re-blank once the exception spam refills the log. **Durable fix:** resolve the `No ItemClass entry` item-config gap (kills the console stall, the log bloat, *and* the exceptions at once), **or** harden the agent log-follow — mechanism not fully pinned (the agent has no pprof endpoint, so no clean goroutine dump was possible). **Do NOT recreate season10 to fix this** (old-image constraint, §10). 2. **The "setting up" provisioning indicator doesn't surface.** The agent watcher emits `detail="Finishing cluster setup…"` and the dashboard renders a matching amber "setting up" pill — but the agent's own **state-reconciliation loop emits `detail="announced:running"` a beat later and overwrites it**, so the card flips green prematurely. The **Console line** (`auto-provision: aligning the new world's decorations to the cluster…`) *does* show and survives a refresh — that part works. Proper fix: a **dedicated provisioning state field** the reconciler respects, not the shared `detail` string. (The card-pill matcher is live but dormant in `new.html`/`index.html`; harmless.) 3. **Auto-provision watcher probe-path caveat (zero-touch isn't universal yet).** The watcher gates "gen finished" on `main.ttw` **only**; it reads `decoration.7dt` solely to detect the already-canon (≥24000) case. A *missing* `decoration.7dt` still yields `NEEDS_ALIGN` and a bounce — the graceful-stop save writes the native deco, which the entrypoint then remaps on the align boot. For some RWG worlds (e.g. Tuxeno on `cluster-seed-test`/pvp) that two-step doesn't always *complete* zero-touch, but the **entrypoint backstop still aligns the world on its next ordinary restart**, so it ends up correct. Worlds that bake deco at the save path at gen (e.g. refuge-mall) provision fully zero-touch. **Treat the entrypoint remap as the reliable path; the watcher bounce is the optimization.** 4. **`No ItemClass entry for type 678xx` on season10** (root of #1). A set of high-band item ids (mod / RefugeBot runtime items) the merged config references but doesn't define. Tracing the missing items cleans up the console stall, the 13 MB log bloat, and the on-server exceptions in one shot. Not urgent — season10 runs fine (20 FPS, players on). --- ## 13. Handoff status — what's deployed, what's uncommitted **Cluster:** `cl_37e3a7f72cdc`. Shared `Player/` ≈ 88 `.ttp`. Live members: | Server | Instance id | Container / saves volume | Active world (`CLUSTER_PLAYER_SAVE`) | Role | |---|---|---|---|---| | Season 10 | `rg-season-10` | `panel-rg-season-10` / `panel-rg-season-10-saves` | `West Apeeni Mountains/MyGame` *(canonical anchor)* | **master / anchor — DO NOT Rebuild** | | Creative | `creative` | `panel-creative` / `panel-creative-saves` | `Tefasizi County/Refuge` | creative | | Insane Survival | `insane-survival` | `panel-insane-survival` / `panel-insane-survival-saves` | `Yixacove Valley/Refuge` | survival | | PvP | `cluster-seed-test` | `panel-cluster-seed-test` / `panel-cluster-seed-test-saves` | `Tuxeno Territory/RefugeTHREEPVP` | pvp | | Mall | `refuge-mall` | `panel-refuge-mall` / `panel-refuge-mall-saves` | `New Vebape Territory/RefugeMall` | mall | > **Canonical anchor — the single highest-blast-radius fact.** The complete frozen table (**38,626 blocks / 2,719 items**) lives ONLY at season10's `…/Saves/West Apeeni Mountains/MyGame/{block,item}mappings.nim`. season10's *other* save dirs are stale partials (e.g. `Navezgane/MyGame` = 3,356 blocks, `Iupu Territory/Troutdale` = 5,028) and its `CLUSTER_PLAYER_SAVE` env is **empty** (old image, predates the picker — so don't trust the env to find the anchor). Any re-anchor (§6 / §7) MUST copy from exactly `West Apeeni Mountains/MyGame` and **verify the counts read 38,626 / 2,719 (or higher — never lower)** before running `rebuild.py`. Anchoring against a partial silently wipes the ~88 shared characters. **Deployed (live):** * **Controller** on kaiten — has the create-form cluster note + the auto-provision "starting" chip + the (dormant) "setting up" card-pill code. * **Agent** on figaro — has the auto-provision watcher **and** the hardened `ExecCapture` (§11). * **Image `panel-7dtd:latest`** on figaro — has the entrypoint remap block + `python3-minimal` + the vendored `remap_7dt.py`. Running on creative / insane-survival / cluster-seed-test / refuge-mall. * **`rg-season-10` deliberately on the OLD image** (no `.panel-save-base`, no remap block) — the live anchor; do not move it. * **refuge-mall** — created via the panel this session and registered as a RefugeBot tenant (prod DB row Id=31 + `refugebot.json` in its saves volume), connected. **⚠️ UNCOMMITTED — nothing from this work is in git.** `C:\Users\dbled\sources\panel` is a git working tree but all of the below is uncommitted: the controller dashboard edits, the agent Go (`decoremap_provision.go`, the `dispatch.go` hook + `provisionGuard`, the `ExecCapture` hardening), the module (`entrypoint.sh` remap block, `Dockerfile`, `nim-freeze/remap_7dt.vendored.py`), and these docs. **Rollback points** if a deploy goes bad: `controller.bak-` on kaiten, `agent.bak-` on figaro (newest = the binary just before the current one). **If/when you commit,** suggested grouping: (1) canonical + deco freeze/remap, (2) auto-provision (entrypoint + watcher), (3) the `ExecCapture` hardening, (4) docs. Note in the message that the cluster is **live** so a reviewer treats it as load-bearing. --- ## 14. Disaster recovery — the three irreplaceable trees These three live on figaro and **cannot be regenerated from the repo** — they *are* the source of truth. If figaro's disk dies, this is what hurts: | Tree | Path on figaro | What it is | If lost | |---|---|---|---| | **The canonical** | `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/canonical/{block,item}mappings.nim` | the frozen stamp source every member uses | rebuildable from season10's `West Apeeni Mountains/MyGame` anchor via `rebuild.py` (needs `~/nimcanon`) | | **Shared characters** | `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/Player/` (= `/cluster/Player`) | the ~88 player `.ttp` (every character + inventory) | **unrecoverable** — everyone resets to level 1 | | **The rebuild toolchain** | `~/nimcanon/` (`tool/`, `registry/`, `canonical_final/`, `remap/`) | the only parser + name registry + capture tooling to rebuild any of the above | must re-harvest from scratch (the §6 harvest — currently uncommitted) | > ⚠️ **Open risk — confirm these are actually backed up off-figaro.** `~/nimcanon` is persisted out of `/tmp` (survives a reboot) but a disk loss is unaddressed in these docs. **Recommendation:** snapshot all three trees to another host on a schedule (reuse the panel's backup system or a periodic `tar`). Restore order: canonical → Player → validate with an in-game transfer (CANONICAL_NIM_FREEZE "Verify a server is frozen"). Treat this as the **top open risk** until a backup is confirmed.