panel/modules/7dtd/CLUSTER_PLAYBOOK.md

7 Days to Die — Cluster Playbook

The single operator + architecture doc for the Refuge 7DTD cluster: how shared characters/inventory work, how new servers auto-fix themselves on create, the one rule you must never break, and exactly what we do at the 3.0 wipe.

Deep "why" lives in CANONICAL_NIM_FREEZE.md (root cause) and the manual fallback in nim-freeze/REMAP_RUNBOOK.md. This doc is the one to read first.

---

1. The one thing to understand

7DTD stores everything — your inventory, the blocks in the world, the trees — by a numeric id, not by name. The engine hands out those numbers automatically when it boots, and the assignment drifts whenever the build changes. "Build" = the exact set of mods + their load order + the RefugeBot DLL version + the engine version. Nothing else matters.

• Same build everywhere → same numbers everywhere. Inventory transfers cleanly, worlds load. (Proven: two cluster servers on the same build produce byte-identical id tables.)
• Different builds → different numbers → "morph". A glass block saved as id 21895 on one server reads as a car hood on another that put a different block at 21895. Characters can even reset to level 1.

Everything below — the canonical table, the stamp, the decoration remap, the auto-provision — exists for one reason: to bridge servers/worlds that were baked under different builds so they all agree on the numbers.

---

2. The golden rule (non-negotiable)

Every server in the cluster must run the byte-identical build at all times.

Same mods, same load order, same RefugeBot build, same game version — on all members, always. Break this and inventory morphs. The build can evolve over time, but only as a coordinated, all-servers-at-once maintenance event (see §6), never one server at a time.

---

3. The two moving parts

Part	What it is	Who maintains it
The canonical	One frozen name ↔ id table (blockmappings.nim + itemmappings.nim) every server stamps on, so the shared player inventory (.ttp) decodes identically everywhere. Lives at /cluster/canonical/ (host: …/7dtdcluster/<id>/canonical/).	nim-freeze/rebuild.py (re-anchor / extend)
The decoration remap	A world bakes its decorations (decoration.7dt / multiblocks.7dt) with the engine's native ids at generation. Those must be rewritten to the canonical's ids or the world NREs on load.	nim-freeze/remap_7dt.py, now automatic (see §4)

Why both: the canonical fixes inventory, the remap fixes the world the decorations live in. The deep mechanics are in CANONICAL_NIM_FREEZE.md.

---

4. How it works now — auto-provision on create (the steady state)

You don't run any of the remap tooling by hand anymore. The game container's entrypoint + the agent do it for you. A brand-new clustered world becomes canon-compatible on its own, in two boots:

1. Gen boot. You create the server (cluster picked), Update (downloads the game), and Start. The engine generates the world and bakes its decorations with native ids. The world is up but not yet aligned to the cluster.
2. Align boot (automatic). The agent's auto-provision watcher notices the fresh world's decorations are in the native band and bounces the container once. On that boot, the entrypoint remaps decoration.7dt + multiblocks.7dt native → canonical (using the world's own native table), drops a .canon-provisioned marker, and stamps the canonical. The world comes back clean, on canon, joinable, transfer-safe.

What you see (visibility): in the server's Console — [panel] auto-provision: aligning the new world's decorations to the cluster… then remapped decoration.7dt native->canonical then decorations aligned to cluster canonical — world is now canon-compatible then StartGame done. The card briefly shows "aligning decorations to cluster…".

It is a strict no-op for everything else: existing canon worlds (season10/creative/insane/pvp) and non-cluster servers never enter the remap branch (their decorations are already in the canon band), so recreating/restarting them does nothing. The remap is stable — once a world is canon, it stays canon across every save and reboot (the engine preserves loaded decoration ids; it only writes native ids at generation).

The backstop: even if the auto-bounce is ever missed, the entrypoint aligns the world on its next ordinary restart. The auto-bounce just makes it zero-touch.

Files that implement it (for the next engineer)

• modules/7dtd/entrypoint.sh — the remap block (right before the canonical-stamp guard) + the .panel-save-base writer. Gated on the decoration band only.
• modules/7dtd/Dockerfile — adds python3-minimal + the vendored remapper.
• modules/7dtd/nim-freeze/remap_7dt.vendored.py — self-contained in-container remapper.
• agent/internal/dispatch/decoremap_provision.go — the watcher (single-flight, probes the running container, bounces once on a native-band world).

---

5. Adding a new server (operator steps)

1. Add server in the panel, module 7 Days to Die, and pick the cluster in the create form. (This wires the shared Player folder + the canonical mount.)
2. Update (installs the game) → Start. Let it generate the world.
3. RWG worlds only: once the world has generated, set the "Shared player world" in the Cluster tab to the new (seed-named) world. The picker writes CLUSTER_PLAYER_SAVE and triggers a recreate, so the entrypoint stamps + aligns the right dir. Also point Region Medic at the same <World>/<GameName> (Cluster tab → Region Medic card), or Medic keeps healing the old, now-deleted world. Until both are set, alignment can't target the world.
4. That's it. The world auto-aligns to the cluster (watch the Console). When you see StartGame done after the alignment bounce, it's joinable and transfer-safe.

Is the new world done? Don't let players join until all three pass:

1. Console shows StartGame done after the auto-provision: aligning… line.
2. Save dir has a .canon-provisioned file.
3. Decorations read in the 24xxx band — verify it yourself (read-only):

docker run --rm -v panel-<inst>-saves:/sv:ro debian:12-slim sh -c \
 'f="/sv/.local/share/7DaysToDie/Saves/<World>/<GameName>/decoration.7dt"; \
  echo "first deco id = $(( $(od -An -tu2 -j17 -N2 "$f") & 0x7FFF ))"'

>= 24000 = canon (good). < 24000 = still native (NOT aligned — do not let players in).

If after 2 boots there's still no .canon-provisioned / the band is native (the §12 #3 case): do one more panel Restart — the entrypoint backstop aligns it on that boot — and re-check. Still native after that → manual repair via nim-freeze/REMAP_RUNBOOK.md.

---

6. Changing mods (a maintenance event, not a casual edit)

Because of the golden rule, a mod change is a whole-cluster operation:

1. Update every server's mods together (same set, same order, same RefugeBot build). Never one server at a time.
2. Rebuild the canonical so it contains the new names (else they append per-world again → morph). This is two steps — ⚠️ rebuild.py does NOT discover new mod content; it only re-anchors an existing name universe. Doing 2b without 2a silently produces an incomplete canonical and re-introduces the exact wipe bug this system prevents.
   • 2a — Re-harvest the complete name universe (the hard part). Blocks = the <base>:<shape> cross-product (shapes.xml × the 10 shape-helper bases) ∪ the engine's authoritative block dump (telnet exportcurrentconfigs, or the runtime Block list) ∪ every named block; items = every <item name=> ∪ every <item_modifier name=>. The original harvest ran as two ultracode workflows; its inputs/outputs live on figaro at ~/nimcanon/registry/ but the generator is not committed — exact greps/commands are in CANONICAL_NIM_FREEZE.md → "After a mod change". Do not attempt a mod change without re-running this harvest.
   • 2b — Re-anchor against season10's CURRENT, STOPPED .nim (the West Apeeni Mountains/MyGame anchor — verify 38,626/2,719+ first, §13) with the new universe: nim-freeze/rebuild.py preserves every existing id and appends new names at fresh ids (so old worlds + the ~88 shared characters still decode). Verify GATE PASS: True, then copy into /cluster/canonical/ (chmod 644).
3. Re-stamp every member so they pick up the new canonical — a panel Restart is enough (the entrypoint re-stamps from /cluster/canonical on every boot; season10 too, by Restart only — never Rebuild). New worlds genned after the change auto-align (§4); existing worlds keep working (their content + the canonical's existing ids are unchanged).

Adding content is safe (append). Removing/reordering content is where ids shift — that's exactly what a wipe (§7) resets cleanly.

---

7. The 3.0 wipe roadmap (the easy button)

A version wipe is the moment to make the whole thing bulletproof and remap-free, because you're discarding all the old worlds anyway. Plan:

1. Freeze + hash the 3.0 build. Lock the exact mod set + load order + RefugeBot DLL + engine version, and record a hash of the artifact set (e.g. tar c Mods/ | sha256sum) into a committed 3.0-build.lock. That snapshot is the contract going forward.
2. Wipe all worlds — through the panel, never docker CLI. Per server: delete the old Saves/<World>/<GameName> dirs (mount the saves volume, per the runbook's volume-access pattern) and set a fresh world_seed via env-config, or delete-preserve-volumes + recreate. Characters too — .ttp is build-dependent; a wipe is a fresh start for inventories (carrying them across a build change needs the .7rg/.ttp codec work in §8).
3. Re-anchor the canonical to the LIVE 3.0 runtime. "Capture the current runtime's table" = the REMAP_RUNBOOK.md "CAPTURE the world's native table" sub-procedure: on a fresh 3.0 world, Stop → move the canonical aside (back it up first!) → write the 8-byte empty .nim → Start (the engine writes the world's authoritative native table) → copy it out → restore the canonical. Feed that as the new anchor to rebuild.py together with a freshly-harvested 3.0 universe (§6 step 2a). Now canonical == runtime and the gap that forces per-world remapping is gone.
4. From then on, fresh maps just work with zero remap — every world is born from the same starting numbers. The auto-provision/stamp machinery stays in place as a harmless safety net; it simply has nothing to fix.
5. Only a future mod change re-introduces the need to rebuild the canonical (§6).

⚠️ The 3.0 wipe is irreversible and depends on the (currently uncommitted) §6 harvest workflow. Do NOT execute it from this strategy alone — when 3.0 lands, write a dedicated, tested 3.0 runbook with the exact harvest + capture + wipe commands first. The plan above is sound; the runnable commands are the gap to close before you pull the trigger.

Net: today = same-mods rule + auto-provision handles new servers; at 3.0 = clean re-anchor and new maps need no special handling at all. It's a ~one-evening job when 3.0 lands; nothing to prep now.

The truly permanent fix that removes even the maintenance events: pin every block/item to an explicit id= in a top-priority modlet so the runtime can never drift from the canonical (untested on this engine version — a worthwhile experiment), or freeze the RefugeBot build the canonical was anchored under.

---

8. Known limits / frontier

• RWG needs one pick. A random-gen world's save folder name is seed-derived and unknown until after gen, so the operator sets "Shared player world" once (§5 step 3). Auto-align kicks in after that.
• Regions + player .ttp are not remapped. The codecs for Region/*.7rg (explored chunk blocks) and Player/*.ttp (inventory) weren't cracked, so alignment is done on a fresh, 0-region world before exploration (which is exactly when a new server provisions). The 87 shared characters are fine (already canon-written). Solving these codecs (decompile Assembly-CSharp.dll) is what would let you migrate characters across a build change instead of wiping them.

---

9. Quick reference

Want to…	Do this
Add a server	§5 (auto-aligns; RWG needs the one Cluster-tab pick)
Confirm a world is aligned	save dir has .canon-provisioned; decorations in 24xxx band
Change mods	§6 (all servers together + rebuild canonical)
Wipe for 3.0	§7 (freeze build → wipe → re-anchor canonical to live runtime)
Manually fix a stuck world	nim-freeze/REMAP_RUNBOOK.md
Understand the root cause	CANONICAL_NIM_FREEZE.md
Re-anchor / extend the canonical	nim-freeze/rebuild.py
Deploy controller / agent / module	§10 (+ the panel-deploy skill)
Recover a wedged agent (config fields won't load / 504s)	restart panel-agent on figaro — §10 / §11
Known issues & open work	§12 (season10 console, the "setting up" indicator, probe-path)
What's deployed vs. uncommitted	§13

---

10. Operational architecture & deploy (for the next engineer)

Where it runs

Piece	Host	Notes
Controller (dashboard + gRPC)	kaiten <user>@<controller-host>	binary /home/refuge/panel/bin/controller, unit panel-controller.service, HTTP :8180, gRPC :8443
Agent (owns the docker game containers)	figaro <user>@<agent-host>	binary /home/refuge/panel/bin/agent, unit panel-agent.service, dials the controller
Cluster shared dir (the /cluster bind)	figaro …/panel/data/7dtdcluster/cl_37e3a7f72cdc/	holds canonical/ + Player/; bind-mounted into every member as /cluster
Game containers	figaro, image panel-7dtd:latest	named panel-<instance> (e.g. panel-rg-season-10)

SSH everywhere with ~/.ssh/bubbly_ed25519, user refuge. The full deploy recipe is the panel-deploy skill — follow it. The NOPASSWD sudoers is exact-match (single-service systemctl forms only).

Panel access & conventions (read before driving anything)

• Dashboard: http://<controller-host>:8180 on the LAN (controller HTTP). Log in with your panel account. On kaiten itself the http://127.0.0.1:8180/api/... routes are unauthenticated — that's what the read-only diagnostic curls in this doc use.
• Server name → ids: an instance's id is its slug. "Season 10" ⇒ instance id rg-season-10 ⇒ container panel-rg-season-10 ⇒ saves volume panel-rg-season-10-saves. List them all (with status) via GET /api/instances. Full live table in §13.
• UI = API: the POST /api/instances/<id>/{env-config,start,stop,rebuild} calls in these docs are exactly what the dashboard's Cluster-tab / Maintenance buttons fire. Use either — just target the right <id>.
• "Stop / Start / Restart" always means the PANEL action (agent-owned, clean shutdown). NEVER docker stop/start a panel-* container — the agent owns lifecycle + state reconciliation. Confirm a container is stopped via the grey card, or read-only docker inspect -f '{{.State.Running}}' panel-<id> → false. (Read-only docker exec/logs/inspect and docker run sidecars are fine.)
• "Restart" vs "Rebuild" (load-bearing): Restart = panel Stop then Start, same image — safe for any server including season10. Rebuild/Recreate = adopts panel-7dtd:latest — needed to pick up a module/image change, but forbidden for rg-season-10 (it must stay on its old image; §13).
• New entrypoint env vars MUST be declared in module.yaml env: or the agent silently drops them (symptom: the var is empty at runtime even though you set it). CLUSTER_ID + CLUSTER_PLAYER_SAVE are already declared — add any new CLUSTER_* / provisioning var there too.

Three deployable pieces

1. Controller (Go + the dashboard HTML in controller/cmd/controller/static/{new,index}.html) — GOOS=linux GOARCH=amd64 go build -o bin/controller-linux ./controller/cmd/controller, scp to kaiten bin/controller.new, bak + swap + systemctl restart panel-controller.service.
   • ⚠️ After every controller deploy, HARD-refresh the browser (Ctrl-Shift-R). A normal refresh serves the old cached JS against the new backend and the page half-breaks (fields won't load, console blank). It looks exactly like a real outage but is just stale cache — rule this out first.
2. Agent (Go) — … go build -o bin/agent-linux ./agent/cmd/agent, scp to figaro, bak + swap + systemctl restart panel-agent.service.
   • Restarting the agent does NOT stop the game containers — it re-attaches/rehydrates them cleanly (stats + rcon + log streams re-open). Safe anytime; it is also the recovery for a wedged agent (§11).
3. Module (modules/7dtd/{entrypoint.sh,Dockerfile,nim-freeze/remap_7dt.vendored.py}) — sync the module dir Windows → kaiten → figaro, then build the image on figaro (where the containers run; kaiten holds the source-of-truth copy but builds nothing):

   # from the Windows repo C:\Users\dbled\sources\panel :
   scp -r modules/7dtd/ <user>@<controller-host>:/home/refuge/panel/modules/
   ssh <user>@<controller-host> 'rsync -av /home/refuge/panel/modules/7dtd/ <user>@<agent-host>:/home/refuge/panel/modules/7dtd/'
   ssh <user>@<agent-host> 'cd /home/refuge/panel/modules/7dtd && docker build -t panel-7dtd:latest .'
   

   Existing containers pin to the old image ID — a container only adopts the new image via Settings → Maintenance → Rebuild (or delete-preserve-volumes + recreate).
   • ⚠️ Never Rebuild/recreate rg-season-10. It is the live released server and the canonical anchor; it must stay on its current (old) image. .Config.Image shows the name :latest for every container — verify by image ID, not name.

Do NOT docker stop/start a panel-* game container from the CLI — let the panel do it (the agent owns lifecycle + state reconciliation). Read-only docker exec, docker logs, docker inspect, and docker run sidecars are fine.

---

11. Incident log & landmines

2026-06-13 — agent-wide hang (ExecCapture connection leak) ✅ FIXED + HARDENED

• Symptom: config fields wouldn't load on any server (the durable serverconfig.xml editor → /api/instances/{id}/files/read returned HTTP 504 "timeout waiting for agent response"); one server's console went blank; the agent was idle (0% CPU) but unresponsive to file ops; no panic.
• Root cause: the auto-provision watcher (agent/internal/dispatch/decoremap_provision.go) polls a running container with a docker exec (DockerRuntime.ExecCapture) every 20s. ExecCapture ran stdcopy.StdCopy(…att.Reader) on the hijacked exec stream with no ctx enforcement on the Read. When it caught the just-bounced cluster-seed-test mid-restart, the read blocked forever and leaked a docker SDK connection each cycle → the SDK client's connection pool exhausted → every later docker op (CopyFileFromContainer for config reads, a log-stream re-attach) hung. It is not a Go-mutex deadlock (the only mutex in docker.go — hijackedStream.mu — just guards the stdio-attach Close() and is unrelated); it's connection-pool starvation.
• Recover: ssh <user>@<agent-host> 'sudo -n /usr/bin/systemctl restart panel-agent.service' — clears the leaked connections; game containers keep running. Confirm recovery: re-open any server's Config tab (fields load within a few seconds), or on kaiten curl -s -o /dev/null -w '%{http_code}\n' 'http://127.0.0.1:8180/api/instances/rg-season-10/files/read?path=/game-saves/serverconfig.xml' → expect 200, not 504.
• Permanent fix (shipped): ExecCapture now runs StdCopy on a goroutine and selects on ctx.Done(); on timeout it att.Close()s to unblock the read, then drains the goroutine before reading the buffers (race-free). On the timeout path it returns the partial stdout/stderr with exit code -1 and exec copy: <ctx err>, so callers can tell a wedged exec from a clean one. An exec can no longer hang the whole agent.
• Landmine for the future: any agent docker exec/stream read that ignores ctx can leak connections and wedge the entire agent the same way. If you add exec-based features, bound the read by ctx. (The log-stream demux at docker.go:271 is a long-lived Follow stream — a different model, closes on the docker rc's EOF/ctx — and is fine.)
• Recognize it next time: files/read 504s for every server + agent at 0% CPU + no crash ⇒ restart the agent first, then hunt for a new unbounded docker read.

---

12. Known issues & open work

1. season10's panel console is blank (separate from §11; pre-existing). season10's docker json-log has grown to 13 MB and it is actively spamming exceptions: No ItemClass entry for type 678xx (×611) + Log:Exception / ThreadManager:UpdateMainThreadTasks. Docker's own --follow --tail 400 serves season10 instantly and the other four servers stream fine, but the agent's SDK log-follow stalls on season10's stream before forwarding a single line (blocked-inside, not thrashing — no "log stream ended" retries). Re-attaching (two agent restarts) did not fix it. It will likely clear on a season10 container restart — a plain panel Stop → Start, which keeps season10 on its current image and is safe; this is not a Rebuild/recreate (that would adopt panel-7dtd:latest and break the canonical anchor — forbidden, see §10/§13). It may re-blank once the exception spam refills the log. Durable fix: resolve the No ItemClass entry item-config gap (kills the console stall, the log bloat, and the exceptions at once), or harden the agent log-follow — mechanism not fully pinned (the agent has no pprof endpoint, so no clean goroutine dump was possible). Do NOT recreate season10 to fix this (old-image constraint, §10).

2. The "setting up" provisioning indicator doesn't surface. The agent watcher emits detail="Finishing cluster setup…" and the dashboard renders a matching amber "setting up" pill — but the agent's own state-reconciliation loop emits detail="announced:running" a beat later and overwrites it, so the card flips green prematurely. The Console line (auto-provision: aligning the new world's decorations to the cluster…) does show and survives a refresh — that part works. Proper fix: a dedicated provisioning state field the reconciler respects, not the shared detail string. (The card-pill matcher is live but dormant in new.html/index.html; harmless.)

3. Auto-provision watcher probe-path caveat (zero-touch isn't universal yet). The watcher gates "gen finished" on main.ttw only; it reads decoration.7dt solely to detect the already-canon (≥24000) case. A missing decoration.7dt still yields NEEDS_ALIGN and a bounce — the graceful-stop save writes the native deco, which the entrypoint then remaps on the align boot. For some RWG worlds (e.g. Tuxeno on cluster-seed-test/pvp) that two-step doesn't always complete zero-touch, but the entrypoint backstop still aligns the world on its next ordinary restart, so it ends up correct. Worlds that bake deco at the save path at gen (e.g. refuge-mall) provision fully zero-touch. Treat the entrypoint remap as the reliable path; the watcher bounce is the optimization.

4. No ItemClass entry for type 678xx on season10 (root of #1). A set of high-band item ids (mod / RefugeBot runtime items) the merged config references but doesn't define. Tracing the missing items cleans up the console stall, the 13 MB log bloat, and the on-server exceptions in one shot. Not urgent — season10 runs fine (20 FPS, players on).

---

13. Handoff status — what's deployed, what's uncommitted

Cluster: cl_37e3a7f72cdc. Shared Player/ ≈ 88 .ttp. Live members:

Server	Instance id	Container / saves volume	Active world (CLUSTER_PLAYER_SAVE)	Role
Season 10	rg-season-10	panel-rg-season-10 / panel-rg-season-10-saves	West Apeeni Mountains/MyGame (canonical anchor)	master / anchor — DO NOT Rebuild
Creative	creative	panel-creative / panel-creative-saves	Tefasizi County/Refuge	creative
Insane Survival	insane-survival	panel-insane-survival / panel-insane-survival-saves	Yixacove Valley/Refuge	survival
PvP	cluster-seed-test	panel-cluster-seed-test / panel-cluster-seed-test-saves	Tuxeno Territory/RefugeTHREEPVP	pvp
Mall	refuge-mall	panel-refuge-mall / panel-refuge-mall-saves	New Vebape Territory/RefugeMall	mall

Canonical anchor — the single highest-blast-radius fact. The complete frozen table (38,626 blocks / 2,719 items) lives ONLY at season10's …/Saves/West Apeeni Mountains/MyGame/{block,item}mappings.nim. season10's other save dirs are stale partials (e.g. Navezgane/MyGame = 3,356 blocks, Iupu Territory/Troutdale = 5,028) and its CLUSTER_PLAYER_SAVE env is empty (old image, predates the picker — so don't trust the env to find the anchor). Any re-anchor (§6 / §7) MUST copy from exactly West Apeeni Mountains/MyGame and verify the counts read 38,626 / 2,719 (or higher — never lower) before running rebuild.py. Anchoring against a partial silently wipes the ~88 shared characters.

Deployed (live):

• Controller on kaiten — has the create-form cluster note + the auto-provision "starting" chip + the (dormant) "setting up" card-pill code.
• Agent on figaro — has the auto-provision watcher and the hardened ExecCapture (§11).
• Image panel-7dtd:latest on figaro — has the entrypoint remap block + python3-minimal + the vendored remap_7dt.py. Running on creative / insane-survival / cluster-seed-test / refuge-mall.
• rg-season-10 deliberately on the OLD image (no .panel-save-base, no remap block) — the live anchor; do not move it.
• refuge-mall — created via the panel this session and registered as a RefugeBot tenant (prod DB row Id=31 + refugebot.json in its saves volume), connected.

⚠️ UNCOMMITTED — nothing from this work is in git. C:\Users\dbled\sources\panel is a git working tree but all of the below is uncommitted: the controller dashboard edits, the agent Go (decoremap_provision.go, the dispatch.go hook + provisionGuard, the ExecCapture hardening), the module (entrypoint.sh remap block, Dockerfile, nim-freeze/remap_7dt.vendored.py), and these docs. Rollback points if a deploy goes bad: controller.bak-<ts> on kaiten, agent.bak-<ts> on figaro (newest = the binary just before the current one).

If/when you commit, suggested grouping: (1) canonical + deco freeze/remap, (2) auto-provision (entrypoint + watcher), (3) the ExecCapture hardening, (4) docs. Note in the message that the cluster is live so a reviewer treats it as load-bearing.

---

14. Disaster recovery — the three irreplaceable trees

These three live on figaro and cannot be regenerated from the repo — they are the source of truth. If figaro's disk dies, this is what hurts:

Tree	Path on figaro	What it is	If lost
The canonical	…/panel/data/7dtdcluster/cl_37e3a7f72cdc/canonical/{block,item}mappings.nim	the frozen stamp source every member uses	rebuildable from season10's West Apeeni Mountains/MyGame anchor via rebuild.py (needs ~/nimcanon)
Shared characters	…/panel/data/7dtdcluster/cl_37e3a7f72cdc/Player/ (= /cluster/Player)	the ~88 player .ttp (every character + inventory)	unrecoverable — everyone resets to level 1
The rebuild toolchain	~/nimcanon/ (tool/, registry/, canonical_final/, remap/)	the only parser + name registry + capture tooling to rebuild any of the above	must re-harvest from scratch (the §6 harvest — currently uncommitted)

⚠️ Open risk — confirm these are actually backed up off-figaro. ~/nimcanon is persisted out of /tmp (survives a reboot) but a disk loss is unaddressed in these docs. Recommendation: snapshot all three trees to another host on a schedule (reuse the panel's backup system or a periodic tar). Restore order: canonical → Player → validate with an in-game transfer (CANONICAL_NIM_FREEZE "Verify a server is frozen"). Treat this as the top open risk until a backup is confirmed.