# 7 Days to Die — Cluster Playbook

> The single operator + architecture doc for the Refuge 7DTD cluster: how shared
> characters/inventory work, how new servers auto-fix themselves on create, the
> one rule you must never break, and exactly what we do at the 3.0 wipe.
>
> Deep "why" lives in `CANONICAL_NIM_FREEZE.md` (root cause) and the manual
> fallback in `nim-freeze/REMAP_RUNBOOK.md`. **This doc is the one to read first.**

---

## 1. The one thing to understand

7DTD stores everything — your inventory, the blocks in the world, the trees —
by a **numeric id**, not by name. The engine hands out those numbers automatically
when it boots, and the assignment **drifts** whenever the *build* changes. "Build"
= the exact set of mods + their load order + the RefugeBot DLL version + the engine
version. Nothing else matters.

* **Same build everywhere → same numbers everywhere.** Inventory transfers cleanly,
  worlds load. (Proven: two cluster servers on the same build produce byte-identical
  id tables.)
* **Different builds → different numbers → "morph".** A glass block saved as id 21895
  on one server reads as a car hood on another that put a different block at 21895.
  Characters can even reset to level 1.

Everything below — the canonical table, the stamp, the decoration remap, the
auto-provision — exists for **one** reason: to bridge servers/worlds that were
baked under *different* builds so they all agree on the numbers.

---

## 2. The golden rule (non-negotiable)

> **Every server in the cluster must run the byte-identical build at all times.**

Same mods, same load order, same RefugeBot build, same game version — on **all**
members, always. Break this and inventory morphs. The build *can* evolve over time,
but only as a **coordinated, all-servers-at-once maintenance event** (see §6), never
one server at a time.

---

## 3. The two moving parts

| Part | What it is | Who maintains it |
|---|---|---|
| **The canonical** | One frozen `name ↔ id` table (`blockmappings.nim` + `itemmappings.nim`) every server stamps on, so the **shared player inventory** (`.ttp`) decodes identically everywhere. Lives at `/cluster/canonical/` (host: `…/7dtdcluster/<id>/canonical/`). | `nim-freeze/rebuild.py` (re-anchor / extend) |
| **The decoration remap** | A world bakes its **decorations** (`decoration.7dt` / `multiblocks.7dt`) with the engine's *native* ids at generation. Those must be rewritten to the canonical's ids or the world NREs on load. | `nim-freeze/remap_7dt.py`, now **automatic** (see §4) |

Why both: the canonical fixes **inventory**, the remap fixes **the world the
decorations live in**. The deep mechanics are in `CANONICAL_NIM_FREEZE.md`.

---

## 4. How it works now — auto-provision on create (the steady state)

You don't run any of the remap tooling by hand anymore. The game container's
entrypoint + the agent do it for you. A brand-new clustered world becomes
canon-compatible **on its own**, in two boots:

1. **Gen boot.** You create the server (cluster picked), Update (downloads the
   game), and Start. The engine generates the world and bakes its decorations
   with native ids. The world is up but *not yet aligned* to the cluster.
2. **Align boot (automatic).** The agent's auto-provision watcher notices the
   fresh world's decorations are in the native band and **bounces the container
   once**. On that boot, the entrypoint remaps `decoration.7dt` + `multiblocks.7dt`
   native → canonical (using the world's own native table), drops a
   `.canon-provisioned` marker, and stamps the canonical. The world comes back
   **clean, on canon, joinable, transfer-safe.**

**What you see (visibility):** in the server's **Console** —
`[panel] auto-provision: aligning the new world's decorations to the cluster…`
then `remapped decoration.7dt native->canonical` then
`decorations aligned to cluster canonical — world is now canon-compatible` then
`StartGame done`. The card briefly shows **"aligning decorations to cluster…"**.

**It is a strict no-op for everything else:** existing canon worlds
(season10/creative/insane/pvp) and non-cluster servers never enter the remap
branch (their decorations are already in the canon band), so recreating/restarting
them does nothing. The remap is **stable** — once a world is canon, it stays canon
across every save and reboot (the engine preserves loaded decoration ids; it only
writes native ids at *generation*).

**The backstop:** even if the auto-bounce is ever missed, the entrypoint aligns the
world on its *next* ordinary restart. The auto-bounce just makes it zero-touch.

### Files that implement it (for the next engineer)
* `modules/7dtd/entrypoint.sh` — the remap block (right before the canonical-stamp
  guard) + the `.panel-save-base` writer. Gated on the **decoration band** only.
* `modules/7dtd/Dockerfile` — adds `python3-minimal` + the vendored remapper.
* `modules/7dtd/nim-freeze/remap_7dt.vendored.py` — self-contained in-container remapper.
* `agent/internal/dispatch/decoremap_provision.go` — the watcher (single-flight,
  probes the running container, bounces once on a native-band world).

---

## 5. Adding a new server (operator steps)

1. **Add server** in the panel, module **7 Days to Die**, and **pick the cluster**
   in the create form. (This wires the shared Player folder + the canonical mount.)
2. **Update** (installs the game) → **Start**. Let it generate the world.
3. **RWG worlds only:** once the world has generated, set the **"Shared player world"**
   in the **Cluster tab** to the new (seed-named) world. The picker writes
   `CLUSTER_PLAYER_SAVE` and triggers a recreate, so the entrypoint stamps + aligns the
   right dir. **Also point Region Medic at the same `<World>/<GameName>`** (Cluster tab →
   Region Medic card), or Medic keeps healing the old, now-deleted world. Until both are
   set, alignment can't target the world.
4. That's it. The world auto-aligns to the cluster (watch the Console). When you see
   `StartGame done` after the alignment bounce, it's joinable and transfer-safe.

> **Is the new world done? Don't let players join until all three pass:**
> 1. Console shows `StartGame done` *after* the `auto-provision: aligning…` line.
> 2. Save dir has a `.canon-provisioned` file.
> 3. Decorations read in the **24xxx** band — verify it yourself (read-only):
> ```sh
> docker run --rm -v panel-<inst>-saves:/sv:ro debian:12-slim sh -c \
>  'f="/sv/.local/share/7DaysToDie/Saves/<World>/<GameName>/decoration.7dt"; \
>   echo "first deco id = $(( $(od -An -tu2 -j17 -N2 "$f") & 0x7FFF ))"'
> ```
> `>= 24000` = canon (good). `< 24000` = still native (NOT aligned — do not let players in).
>
> **If after 2 boots there's still no `.canon-provisioned` / the band is native** (the
> §12 #3 case): do **one** more panel **Restart** — the entrypoint backstop aligns it on
> that boot — and re-check. Still native after that → manual repair via
> `nim-freeze/REMAP_RUNBOOK.md`.

---

## 6. Changing mods (a maintenance event, not a casual edit)

Because of the golden rule, a mod change is a **whole-cluster** operation:

1. Update **every** server's mods together (same set, same order, same RefugeBot
   build). Never one server at a time.
2. **Rebuild the canonical** so it contains the new names (else they append per-world
   again → morph). This is **two** steps — ⚠️ **`rebuild.py` does NOT discover new mod
   content; it only re-anchors an existing name universe.** Doing 2b without 2a silently
   produces an incomplete canonical and re-introduces the exact wipe bug this system
   prevents.
   * **2a — Re-harvest the complete name universe** (the hard part). Blocks = the
     `<base>:<shape>` cross-product (`shapes.xml` × the 10 shape-helper bases) ∪ the
     engine's authoritative block dump (telnet `exportcurrentconfigs`, or the runtime
     `Block` list) ∪ every named block; items = every `<item name=>` ∪ every
     `<item_modifier name=>`. The original harvest ran as two ultracode workflows; its
     inputs/outputs live on figaro at `~/nimcanon/registry/` but **the generator is not
     committed** — exact greps/commands are in `CANONICAL_NIM_FREEZE.md` → "After a mod
     change". **Do not attempt a mod change without re-running this harvest.**
   * **2b — Re-anchor** against season10's CURRENT, STOPPED `.nim` (the
     `West Apeeni Mountains/MyGame` anchor — verify 38,626/2,719+ first, §13) with the new
     universe: `nim-freeze/rebuild.py` preserves every existing id and appends new names
     at fresh ids (so old worlds + the ~88 shared characters still decode). Verify
     `GATE PASS: True`, then copy into `/cluster/canonical/` (chmod 644).
3. **Re-stamp every member** so they pick up the new canonical — a panel **Restart**
   is enough (the entrypoint re-stamps from `/cluster/canonical` on every boot; season10
   too, by Restart only — never Rebuild). New worlds genned after the change auto-align
   (§4); existing worlds keep working (their content + the canonical's existing ids are
   unchanged).

Adding content is safe (append). Removing/reordering content is where ids shift —
that's exactly what a **wipe** (§7) resets cleanly.

---

## 7. The 3.0 wipe roadmap (the easy button)

A version wipe is the moment to make the whole thing **bulletproof and remap-free**,
because you're discarding all the old worlds anyway. Plan:

1. **Freeze + hash the 3.0 build.** Lock the exact mod set + load order + RefugeBot DLL +
   engine version, and record a hash of the artifact set (e.g. `tar c Mods/ | sha256sum`)
   into a committed `3.0-build.lock`. That snapshot *is* the contract going forward.
2. **Wipe all worlds — through the panel, never docker CLI.** Per server: delete the old
   `Saves/<World>/<GameName>` dirs (mount the saves volume, per the runbook's volume-access
   pattern) and set a fresh `world_seed` via `env-config`, or delete-preserve-volumes +
   recreate. Characters too — `.ttp` is build-dependent; a wipe is a fresh start for
   inventories (carrying them across a build change needs the `.7rg`/`.ttp` codec work in §8).
3. **Re-anchor the canonical to the LIVE 3.0 runtime.** "Capture the current runtime's
   table" = the `REMAP_RUNBOOK.md` **"CAPTURE the world's native table"** sub-procedure: on
   a fresh 3.0 world, Stop → move the canonical aside (back it up first!) → write the 8-byte
   empty `.nim` → Start (the engine writes the world's authoritative native table) → copy it
   out → restore the canonical. Feed that as the new anchor to `rebuild.py` together with a
   freshly-harvested 3.0 universe (§6 step 2a). Now **`canonical == runtime`** and the gap
   that forces per-world remapping is gone.
4. **From then on, fresh maps just work with zero remap** — every world is born from the
   same starting numbers. The auto-provision/stamp machinery stays in place as a harmless
   safety net; it simply has nothing to fix.
5. Only a *future* mod change re-introduces the need to rebuild the canonical (§6).

> ⚠️ **The 3.0 wipe is irreversible and depends on the (currently uncommitted) §6 harvest
> workflow.** Do NOT execute it from this strategy alone — when 3.0 lands, write a dedicated,
> tested 3.0 runbook with the exact harvest + capture + wipe commands first. The plan above
> is sound; the runnable commands are the gap to close before you pull the trigger.

Net: **today** = same-mods rule + auto-provision handles new servers; **at 3.0** =
clean re-anchor and new maps need no special handling at all. It's a ~one-evening job
when 3.0 lands; nothing to prep now.

The truly permanent fix that removes even the maintenance events: pin every block/item
to an explicit `id=` in a top-priority modlet so the runtime can never drift from the
canonical (untested on this engine version — a worthwhile experiment), or freeze the
RefugeBot build the canonical was anchored under.

---

## 8. Known limits / frontier

* **RWG needs one pick.** A random-gen world's save folder name is seed-derived and
  unknown until after gen, so the operator sets "Shared player world" once (§5 step 3).
  Auto-align kicks in after that.
* **Regions + player `.ttp` are not remapped.** The codecs for `Region/*.7rg`
  (explored chunk blocks) and `Player/*.ttp` (inventory) weren't cracked, so alignment
  is done on a **fresh, 0-region world before exploration** (which is exactly when a
  new server provisions). The 87 shared characters are fine (already canon-written).
  Solving these codecs (decompile `Assembly-CSharp.dll`) is what would let you migrate
  characters across a build change instead of wiping them.

---

## 9. Quick reference

| Want to… | Do this |
|---|---|
| Add a server | §5 (auto-aligns; RWG needs the one Cluster-tab pick) |
| Confirm a world is aligned | save dir has `.canon-provisioned`; decorations in 24xxx band |
| Change mods | §6 (all servers together + rebuild canonical) |
| Wipe for 3.0 | §7 (freeze build → wipe → re-anchor canonical to live runtime) |
| Manually fix a stuck world | `nim-freeze/REMAP_RUNBOOK.md` |
| Understand the root cause | `CANONICAL_NIM_FREEZE.md` |
| Re-anchor / extend the canonical | `nim-freeze/rebuild.py` |
| Deploy controller / agent / module | §10 (+ the `panel-deploy` skill) |
| Recover a wedged agent (config fields won't load / 504s) | restart `panel-agent` on figaro — §10 / §11 |
| Known issues & open work | §12 (season10 console, the "setting up" indicator, probe-path) |
| What's deployed vs. uncommitted | §13 |

---

## 10. Operational architecture & deploy (for the next engineer)

**Where it runs**

| Piece | Host | Notes |
|---|---|---|
| **Controller** (dashboard + gRPC) | kaiten `<user>@<controller-host>` | binary `/home/refuge/panel/bin/controller`, unit `panel-controller.service`, HTTP `:8180`, gRPC `:8443` |
| **Agent** (owns the docker game containers) | figaro `<user>@<agent-host>` | binary `/home/refuge/panel/bin/agent`, unit `panel-agent.service`, dials the controller |
| **Cluster shared dir** (the `/cluster` bind) | figaro `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/` | holds `canonical/` + `Player/`; bind-mounted into every member as `/cluster` |
| **Game containers** | figaro, image `panel-7dtd:latest` | named `panel-<instance>` (e.g. `panel-rg-season-10`) |

SSH everywhere with `~/.ssh/bubbly_ed25519`, user `refuge`. **The full deploy recipe is the `panel-deploy` skill — follow it.** The NOPASSWD sudoers is exact-match (single-service `systemctl` forms only).

**Panel access & conventions (read before driving anything)**
* **Dashboard:** `http://<controller-host>:8180` on the LAN (controller HTTP). Log in with your panel account. On kaiten itself the `http://127.0.0.1:8180/api/...` routes are unauthenticated — that's what the read-only diagnostic `curl`s in this doc use.
* **Server name → ids:** an instance's id is its slug. "Season 10" ⇒ instance id `rg-season-10` ⇒ container `panel-rg-season-10` ⇒ saves volume `panel-rg-season-10-saves`. List them all (with status) via `GET /api/instances`. Full live table in §13.
* **UI = API:** the `POST /api/instances/<id>/{env-config,start,stop,rebuild}` calls in these docs are exactly what the dashboard's Cluster-tab / Maintenance buttons fire. Use either — just target the right `<id>`.
* **"Stop / Start / Restart" always means the PANEL action** (agent-owned, clean shutdown). **NEVER `docker stop/start` a `panel-*` container** — the agent owns lifecycle + state reconciliation. Confirm a container is stopped via the grey card, or read-only `docker inspect -f '{{.State.Running}}' panel-<id>` → `false`. (Read-only `docker exec/logs/inspect` and `docker run` sidecars are fine.)
* **"Restart" vs "Rebuild" (load-bearing):** Restart = panel Stop then Start, **same image** — safe for any server *including season10*. **Rebuild/Recreate = adopts `panel-7dtd:latest`** — needed to pick up a module/image change, but **forbidden for `rg-season-10`** (it must stay on its old image; §13).
* **New entrypoint env vars MUST be declared in `module.yaml` `env:`** or the agent silently drops them (symptom: the var is empty at runtime even though you set it). `CLUSTER_ID` + `CLUSTER_PLAYER_SAVE` are already declared — add any new `CLUSTER_*` / provisioning var there too.

**Three deployable pieces**

1. **Controller** (Go + the dashboard HTML in `controller/cmd/controller/static/{new,index}.html`) — `GOOS=linux GOARCH=amd64 go build -o bin/controller-linux ./controller/cmd/controller`, scp to kaiten `bin/controller.new`, bak + swap + `systemctl restart panel-controller.service`.
   * ⚠️ **After every controller deploy, HARD-refresh the browser (Ctrl-Shift-R).** A normal refresh serves the *old* cached JS against the new backend and the page half-breaks (fields won't load, console blank). It looks exactly like a real outage but is just stale cache — rule this out first.
2. **Agent** (Go) — `… go build -o bin/agent-linux ./agent/cmd/agent`, scp to figaro, bak + swap + `systemctl restart panel-agent.service`.
   * Restarting the agent does **NOT** stop the game containers — it re-attaches/rehydrates them cleanly (stats + rcon + log streams re-open). Safe anytime; it is also the **recovery** for a wedged agent (§11).
3. **Module** (`modules/7dtd/{entrypoint.sh,Dockerfile,nim-freeze/remap_7dt.vendored.py}`) — sync the module dir Windows → kaiten → figaro, then build the image **on figaro** (where the containers run; kaiten holds the source-of-truth copy but builds nothing):
   ```sh
   # from the Windows repo C:\Users\dbled\sources\panel :
   scp -r modules/7dtd/ <user>@<controller-host>:/home/refuge/panel/modules/
   ssh <user>@<controller-host> 'rsync -av /home/refuge/panel/modules/7dtd/ <user>@<agent-host>:/home/refuge/panel/modules/7dtd/'
   ssh <user>@<agent-host> 'cd /home/refuge/panel/modules/7dtd && docker build -t panel-7dtd:latest .'
   ```
   Existing containers **pin to the old image ID** — a container only adopts the new image via **Settings → Maintenance → Rebuild** (or delete-preserve-volumes + recreate).
   * ⚠️ **Never Rebuild/recreate `rg-season-10`.** It is the live released server *and* the canonical anchor; it must stay on its current (old) image. `.Config.Image` shows the name `:latest` for every container — verify by image **ID**, not name.

**Do NOT `docker stop/start` a `panel-*` game container from the CLI** — let the panel do it (the agent owns lifecycle + state reconciliation). Read-only `docker exec`, `docker logs`, `docker inspect`, and `docker run` sidecars are fine.

---

## 11. Incident log & landmines

### 2026-06-13 — agent-wide hang (ExecCapture connection leak) ✅ FIXED + HARDENED

* **Symptom:** config fields wouldn't load on **any** server (the durable `serverconfig.xml` editor → `/api/instances/{id}/files/read` returned **HTTP 504 "timeout waiting for agent response"**); one server's console went blank; the agent was **idle (0% CPU) but unresponsive to file ops**; no panic.
* **Root cause:** the auto-provision watcher (`agent/internal/dispatch/decoremap_provision.go`) polls a running container with a docker exec (`DockerRuntime.ExecCapture`) every 20s. `ExecCapture` ran `stdcopy.StdCopy(…att.Reader)` on the hijacked exec stream **with no `ctx` enforcement on the Read**. When it caught the just-bounced `cluster-seed-test` mid-restart, the read blocked **forever** and leaked a docker SDK connection each cycle → the SDK client's connection pool exhausted → **every** later docker op (`CopyFileFromContainer` for config reads, a log-stream re-attach) hung. It is **not** a Go-mutex deadlock (the only mutex in `docker.go` — `hijackedStream.mu` — just guards the stdio-attach `Close()` and is unrelated); it's connection-pool starvation.
* **Recover:** `ssh <user>@<agent-host> 'sudo -n /usr/bin/systemctl restart panel-agent.service'` — clears the leaked connections; game containers keep running. **Confirm recovery:** re-open any server's Config tab (fields load within a few seconds), or on kaiten `curl -s -o /dev/null -w '%{http_code}\n' 'http://127.0.0.1:8180/api/instances/rg-season-10/files/read?path=/game-saves/serverconfig.xml'` → expect **200**, not 504.
* **Permanent fix (shipped):** `ExecCapture` now runs `StdCopy` on a goroutine and `select`s on `ctx.Done()`; on timeout it `att.Close()`s to unblock the read, then drains the goroutine before reading the buffers (race-free). On the timeout path it returns the partial stdout/stderr with exit code `-1` and `exec copy: <ctx err>`, so callers can tell a wedged exec from a clean one. An exec can no longer hang the whole agent.
* **Landmine for the future:** *any* agent docker exec/stream read that ignores `ctx` can leak connections and wedge the entire agent the same way. If you add exec-based features, bound the read by `ctx`. (The log-stream demux at `docker.go:271` is a long-lived `Follow` stream — a different model, closes on the docker `rc`'s EOF/ctx — and is fine.)
* **Recognize it next time:** `files/read` 504s for *every* server + agent at 0% CPU + no crash ⇒ restart the agent first, then hunt for a new unbounded docker read.

---

## 12. Known issues & open work

1. **season10's panel console is blank** (separate from §11; pre-existing). season10's docker json-log has grown to **13 MB** and it is actively spamming exceptions: `No ItemClass entry for type 678xx` (×611) + `Log:Exception` / `ThreadManager:UpdateMainThreadTasks`. Docker's own `--follow --tail 400` serves season10 instantly and the other four servers stream fine, but the agent's SDK log-follow **stalls on season10's stream before forwarding a single line** (blocked-inside, not thrashing — no "log stream ended" retries). Re-attaching (two agent restarts) did **not** fix it. It will likely clear on a **season10 container restart** — a plain panel **Stop → Start**, which keeps season10 on its current image and is safe; this is **not** a Rebuild/recreate (that would adopt `panel-7dtd:latest` and break the canonical anchor — forbidden, see §10/§13). It may re-blank once the exception spam refills the log. **Durable fix:** resolve the `No ItemClass entry` item-config gap (kills the console stall, the log bloat, *and* the exceptions at once), **or** harden the agent log-follow — mechanism not fully pinned (the agent has no pprof endpoint, so no clean goroutine dump was possible). **Do NOT recreate season10 to fix this** (old-image constraint, §10).

2. **The "setting up" provisioning indicator doesn't surface.** The agent watcher emits `detail="Finishing cluster setup…"` and the dashboard renders a matching amber "setting up" pill — but the agent's own **state-reconciliation loop emits `detail="announced:running"` a beat later and overwrites it**, so the card flips green prematurely. The **Console line** (`auto-provision: aligning the new world's decorations to the cluster…`) *does* show and survives a refresh — that part works. Proper fix: a **dedicated provisioning state field** the reconciler respects, not the shared `detail` string. (The card-pill matcher is live but dormant in `new.html`/`index.html`; harmless.)

3. **Auto-provision watcher probe-path caveat (zero-touch isn't universal yet).** The watcher gates "gen finished" on `main.ttw` **only**; it reads `decoration.7dt` solely to detect the already-canon (≥24000) case. A *missing* `decoration.7dt` still yields `NEEDS_ALIGN` and a bounce — the graceful-stop save writes the native deco, which the entrypoint then remaps on the align boot. For some RWG worlds (e.g. Tuxeno on `cluster-seed-test`/pvp) that two-step doesn't always *complete* zero-touch, but the **entrypoint backstop still aligns the world on its next ordinary restart**, so it ends up correct. Worlds that bake deco at the save path at gen (e.g. refuge-mall) provision fully zero-touch. **Treat the entrypoint remap as the reliable path; the watcher bounce is the optimization.**

4. **`No ItemClass entry for type 678xx` on season10** (root of #1). A set of high-band item ids (mod / RefugeBot runtime items) the merged config references but doesn't define. Tracing the missing items cleans up the console stall, the 13 MB log bloat, and the on-server exceptions in one shot. Not urgent — season10 runs fine (20 FPS, players on).

---

## 13. Handoff status — what's deployed, what's uncommitted

**Cluster:** `cl_37e3a7f72cdc`. Shared `Player/` ≈ 88 `.ttp`. Live members:

| Server | Instance id | Container / saves volume | Active world (`CLUSTER_PLAYER_SAVE`) | Role |
|---|---|---|---|---|
| Season 10 | `rg-season-10` | `panel-rg-season-10` / `panel-rg-season-10-saves` | `West Apeeni Mountains/MyGame` *(canonical anchor)* | **master / anchor — DO NOT Rebuild** |
| Creative | `creative` | `panel-creative` / `panel-creative-saves` | `Tefasizi County/Refuge` | creative |
| Insane Survival | `insane-survival` | `panel-insane-survival` / `panel-insane-survival-saves` | `Yixacove Valley/Refuge` | survival |
| PvP | `cluster-seed-test` | `panel-cluster-seed-test` / `panel-cluster-seed-test-saves` | `Tuxeno Territory/RefugeTHREEPVP` | pvp |
| Mall | `refuge-mall` | `panel-refuge-mall` / `panel-refuge-mall-saves` | `New Vebape Territory/RefugeMall` | mall |

> **Canonical anchor — the single highest-blast-radius fact.** The complete frozen table (**38,626 blocks / 2,719 items**) lives ONLY at season10's `…/Saves/West Apeeni Mountains/MyGame/{block,item}mappings.nim`. season10's *other* save dirs are stale partials (e.g. `Navezgane/MyGame` = 3,356 blocks, `Iupu Territory/Troutdale` = 5,028) and its `CLUSTER_PLAYER_SAVE` env is **empty** (old image, predates the picker — so don't trust the env to find the anchor). Any re-anchor (§6 / §7) MUST copy from exactly `West Apeeni Mountains/MyGame` and **verify the counts read 38,626 / 2,719 (or higher — never lower)** before running `rebuild.py`. Anchoring against a partial silently wipes the ~88 shared characters.

**Deployed (live):**
* **Controller** on kaiten — has the create-form cluster note + the auto-provision "starting" chip + the (dormant) "setting up" card-pill code.
* **Agent** on figaro — has the auto-provision watcher **and** the hardened `ExecCapture` (§11).
* **Image `panel-7dtd:latest`** on figaro — has the entrypoint remap block + `python3-minimal` + the vendored `remap_7dt.py`. Running on creative / insane-survival / cluster-seed-test / refuge-mall.
* **`rg-season-10` deliberately on the OLD image** (no `.panel-save-base`, no remap block) — the live anchor; do not move it.
* **refuge-mall** — created via the panel this session and registered as a RefugeBot tenant (prod DB row Id=31 + `refugebot.json` in its saves volume), connected.

**⚠️ UNCOMMITTED — nothing from this work is in git.** `C:\Users\dbled\sources\panel` is a git working tree but all of the below is uncommitted: the controller dashboard edits, the agent Go (`decoremap_provision.go`, the `dispatch.go` hook + `provisionGuard`, the `ExecCapture` hardening), the module (`entrypoint.sh` remap block, `Dockerfile`, `nim-freeze/remap_7dt.vendored.py`), and these docs. **Rollback points** if a deploy goes bad: `controller.bak-<ts>` on kaiten, `agent.bak-<ts>` on figaro (newest = the binary just before the current one).

**If/when you commit,** suggested grouping: (1) canonical + deco freeze/remap, (2) auto-provision (entrypoint + watcher), (3) the `ExecCapture` hardening, (4) docs. Note in the message that the cluster is **live** so a reviewer treats it as load-bearing.

---

## 14. Disaster recovery — the three irreplaceable trees

These three live on figaro and **cannot be regenerated from the repo** — they *are* the source of truth. If figaro's disk dies, this is what hurts:

| Tree | Path on figaro | What it is | If lost |
|---|---|---|---|
| **The canonical** | `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/canonical/{block,item}mappings.nim` | the frozen stamp source every member uses | rebuildable from season10's `West Apeeni Mountains/MyGame` anchor via `rebuild.py` (needs `~/nimcanon`) |
| **Shared characters** | `…/panel/data/7dtdcluster/cl_37e3a7f72cdc/Player/` (= `/cluster/Player`) | the ~88 player `.ttp` (every character + inventory) | **unrecoverable** — everyone resets to level 1 |
| **The rebuild toolchain** | `~/nimcanon/` (`tool/`, `registry/`, `canonical_final/`, `remap/`) | the only parser + name registry + capture tooling to rebuild any of the above | must re-harvest from scratch (the §6 harvest — currently uncommitted) |

> ⚠️ **Open risk — confirm these are actually backed up off-figaro.** `~/nimcanon` is persisted out of `/tmp` (survives a reboot) but a disk loss is unaddressed in these docs. **Recommendation:** snapshot all three trees to another host on a schedule (reuse the panel's backup system or a periodic `tar`). Restore order: canonical → Player → validate with an in-game transfer (CANONICAL_NIM_FREEZE "Verify a server is frozen"). Treat this as the **top open risk** until a backup is confirmed.
