panel public release
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
# 7DTD Cluster — Canonical-Compatible New World Provisioning (RUNBOOK)
|
||||
# Validated 2026-06-13. Tooling: this dir. Canonical: ~/nimcanon/canonical_final/{block,item}mappings.nim
|
||||
|
||||
## WHERE TO RUN (prerequisites — read first)
|
||||
Run on **figaro** (`<user>@<agent-host>`) as user **refuge**. Requires `~/nimcanon/{tool,canonical_final,remap}` present (the `.nim` parser, the canonical, the cached native table). The repo `remap_7dt.py` hardcodes `sys.path.insert(0,'/home/refuge/nimcanon/tool')` and imports `nimtool` — it will **NOT** run on the Windows dev box or kaiten. (The in-container copy invoked automatically by the entrypoint is the self-contained `remap_7dt.vendored.py` — a different file.)
|
||||
|
||||
Saves live in a **docker volume**, not a host path — `$W` below is only reachable by mounting the volume. Run file ops inside a throwaway container, e.g.:
|
||||
```sh
|
||||
CL=/home/refuge/panel/data/7dtdcluster/cl_37e3a7f72cdc
|
||||
VOL=panel-<inst>-saves # e.g. panel-cluster-seed-test-saves
|
||||
docker run --rm -v $VOL:/sv -v $CL:/cl debian:12-slim sh -c '
|
||||
W="/sv/.local/share/7DaysToDie/Saves/<World>/<GameName>"; <ops on "$W/decoration.7dt" etc.>'
|
||||
```
|
||||
STOP/START **only via the panel** (agent-owned; sends SIGTERM→tini→7DTD clean shutdown, which flushes the native `.nim`). A `docker kill`/`docker stop` would NOT flush and can corrupt the capture.
|
||||
|
||||
## CORRECTED ROOT CAUSE
|
||||
7DTD V2.6(b14) auto-assigns block ids at config-load. The cluster's CURRENT runtime
|
||||
(RefugeBot v1.2.39) puts the tree-decoration band at native ids 21895+, while the
|
||||
season10-anchored CANONICAL puts the same trees at 24141+ (season10's world was baked
|
||||
PRE-RefugeBot-rebuild). At world GENESIS the engine bakes decoration.7dt / multiblocks.7dt
|
||||
BY NUMERIC BLOCK ID using native runtime ids. The entrypoint guard then stamps the
|
||||
canonical .nim over the world's .nim every boot (so inventory/.ttp decode uniformly →
|
||||
clean transfers). MISMATCH: native-baked deco (21895) read against stamped canonical
|
||||
(24141) → null Block → DecoManager.TryAddToOccupiedMap NRE → world load hangs → never
|
||||
GameStartDone. season10/creative load clean only because their deco was baked when the
|
||||
runtime == canonical (pre-rebuild).
|
||||
|
||||
## KEY FACTS
|
||||
- .nim: u32 ver=1, u32 count, recs[u16 id, u16 flag(block=0/item=1), u8 namelen, name]. Parser: ~/nimcanon/tool/nimtool.py
|
||||
- decoration.7dt / multiblocks.7dt: u8 ver=6, u32 count, count*17B recs; block id = u16@+12 & 0x7FFF; high bit + bytes +14/+16 = rotation/placement (PRESERVE).
|
||||
- Engine PRESERVES loaded deco ids on save; only writes native ids at GENERATION. So a canon-remapped decoration.7dt stays canon across reboots → STABLE.
|
||||
- The cluster runtime native table is IDENTICAL across all instances (insane==pvp deco band 33/33). One captured native table is authoritative cluster-wide: ~/nimcanon/remap/cluster_native_deco.nim
|
||||
- Remapper: remap_7dt.py (apply|analyze|decodecheck). Self-verifies byte-safety + 100% canon resolution.
|
||||
|
||||
## PROVISION A NEW CANONICAL-COMPATIBLE WORLD (do this on a FRESH/0-region world)
|
||||
CL=/home/refuge/panel/data/7dtdcluster/cl_37e3a7f72cdc ; VOL=panel-<inst>-saves ; W="<World>/<GameName>"
|
||||
1. Gen the world (set seed, recreate/start). It bakes a NATIVE decoration.7dt.
|
||||
2. CAPTURE the world's native table:
|
||||
a. Stop the instance (panel Stop — NOT `docker stop`).
|
||||
b. **First BACK UP the canonical** (`cp $CL/canonical/*.nim ~/canon-backup/`), THEN move the originals aside so the guard skips the stamp:
|
||||
`mv $CL/canonical/*.nim $CL/canon_test_aside/`
|
||||
⚠️ **The canonical is the SHARED stamp source for EVERY cluster member.** While it is moved aside, do **NOT** boot any other member — it would stamp nothing and could append/diverge (cluster-wide inventory risk). Keep this window short; restore in step 5 and confirm by `sha256sum` before starting anything else.
|
||||
c. printf '\x01\x00\x00\x00\x00\x00\x00\x00' > "$W/blockmappings.nim" ; same for itemmappings.nim
|
||||
d. Start instance → loads clean on native → engine writes the world's native .nim (covers its deco blocks). Graceful-stop to flush.
|
||||
e. Copy out "$W/blockmappings.nim" as <native.nim>.
|
||||
3. REMAP: python3 remap_7dt.py apply <native.nim> ~/nimcanon/canonical_final/blockmappings.nim "$W/decoration.7dt" /tmp/deco.canon.7dt
|
||||
(repeat for multiblocks.7dt if present). Require: "byte-diff outside id field: 0" and "ids unresolved in canon: 0".
|
||||
4. Backup native (cp decoration.7dt decoration.7dt.native-bak) then apply the remapped file; chown 1000:1000.
|
||||
5. mv $CL/canon_test_aside/*.nim $CL/canonical/ # restore the stamp source
|
||||
6. Start instance → guard stamps canon → loads canon-remapped deco → GameStartDone, NRE=0, on canon.
|
||||
7. VERIFY: decoration.7dt id band = 24xxx ; GameStartDone ; NRE=0 ; .nim stamped 38626/2719 at boot.
|
||||
|
||||
## REPAIR AN ALREADY-STUCK WORLD (canon stamped, native deco, NRE) — same as above from step 2.
|
||||
|
||||
## CAVEATS
|
||||
- Region/*.7rg voxel ids + Player/*.ttp item ids are NOT remapped (codecs unsolved). Do the remap on a FRESH 0-region world BEFORE exploration. Existing 87 shared .ttp are fine (written in canon by season10/creative).
|
||||
- Deco block ids must be < 32768 (15-bit field); the tree-deco band is far under.
|
||||
- PERMANENT fix (removes the per-world remap step): pin block ids explicitly in a high-priority modlet so the runtime native table == canonical, OR freeze the RefugeBot build that season10 was baked under. Until then each new world needs this one-time remap.
|
||||
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Rebuild the 7DTD cluster canonical .nim (re-anchor to season10's CURRENT state).
|
||||
|
||||
WHAT THIS DOES
|
||||
Produces a complete, season10-anchored canonical block/item id map. Every id
|
||||
the anchor (season10) currently uses is preserved BYTE-FOR-BYTE; every other
|
||||
name from the "universe" (a prior complete canonical) is appended at a fresh,
|
||||
non-colliding id above the anchor's max. The result is a superset that decodes
|
||||
season10's world + all shared inventories identically, and is COMPLETE so the
|
||||
engine never appends => the table stays frozen forever.
|
||||
|
||||
WHEN TO RUN
|
||||
* season10's live .nim grew (players placed new shapes) and you need to
|
||||
re-stamp it — re-anchor so the canonical preserves the new ids.
|
||||
* You just want to regenerate the deployed canonical from the anchor.
|
||||
NOTE: if MODS CHANGED (new blocks/items added), the universe won't contain the
|
||||
new names. Re-harvest first — see CANONICAL_NIM_FREEZE.md "Rebuilding after a
|
||||
mod change". This script only RE-ANCHORS an existing complete name set.
|
||||
|
||||
USAGE
|
||||
1. STOP season10 (panel Stop) so its .nim is final.
|
||||
2. Copy season10's CURRENT blockmappings.nim + itemmappings.nim out (the ANCHOR):
|
||||
docker run --rm -v panel-rg-season-10-saves:/sv:ro -v /tmp/x:/o debian:12-slim \
|
||||
sh -c 'cp "/sv/.local/share/7DaysToDie/Saves/West Apeeni Mountains/MyGame/blockmappings.nim" /o/anchor_block.nim;
|
||||
cp "/sv/.local/share/7DaysToDie/Saves/West Apeeni Mountains/MyGame/itemmappings.nim" /o/anchor_item.nim'
|
||||
3. python3 rebuild.py anchor_block.nim anchor_item.nim \
|
||||
UNIVERSE_block.nim UNIVERSE_item.nim OUT_DIR
|
||||
(UNIVERSE = a prior complete canonical, e.g. ~/nimcanon/canonical_final/*.nim)
|
||||
4. Confirm "GATE PASS: True". Then copy OUT_DIR/*.nim into the cluster canonical
|
||||
dir and chmod 644:
|
||||
cp OUT_DIR/*.nim /home/refuge/panel/data/7dtdcluster/<cluster_id>/canonical/
|
||||
chmod 644 /home/refuge/panel/data/7dtdcluster/<cluster_id>/canonical/*.nim
|
||||
5. Rebuild each cluster member (POST /api/instances/<id>/rebuild) — the entrypoint
|
||||
guard re-stamps the new canonical on boot. (If a member was STOPPED, /rebuild
|
||||
leaves it stopped — Start it after.)
|
||||
|
||||
.nim BINARY FORMAT (little-endian, no trailing bytes):
|
||||
u32 version(=1), u32 count, then count records of:
|
||||
u16 id, u16 flag (items=1, blocks=0), u8 namelen, <namelen> UTF-8 name bytes
|
||||
Records are stored sorted by ascending id. Ids are SPARSE.
|
||||
"""
|
||||
import struct, sys, hashlib, os
|
||||
|
||||
|
||||
def read_nim(p):
|
||||
b = open(p, "rb").read()
|
||||
ver, cnt = struct.unpack_from("<II", b, 0)
|
||||
off = 8
|
||||
recs = []
|
||||
for _ in range(cnt):
|
||||
i, fl, nl = struct.unpack_from("<HHB", b, off)
|
||||
off += 5
|
||||
nm = b[off:off + nl].decode("utf-8")
|
||||
off += nl
|
||||
recs.append((i, fl, nm))
|
||||
assert off == len(b), f"trailing bytes parsing {p} ({off} != {len(b)})"
|
||||
return recs
|
||||
|
||||
|
||||
def write_nim(recs, p):
|
||||
out = struct.pack("<II", 1, len(recs))
|
||||
for i, fl, nm in recs:
|
||||
nb = nm.encode("utf-8")
|
||||
out += struct.pack("<HHB", i, fl, len(nb)) + nb
|
||||
open(p, "wb").write(out)
|
||||
return out
|
||||
|
||||
|
||||
def build(anchor, universe, flag):
|
||||
"""anchor ids are SACRED (verbatim). append every universe name not present."""
|
||||
recs = list(anchor)
|
||||
present = {nm for _, _, nm in anchor}
|
||||
used = {i for i, _, _ in anchor}
|
||||
nid = max(used) + 1
|
||||
for nm in sorted({nm for _, _, nm in universe} - present):
|
||||
while nid in used:
|
||||
nid += 1
|
||||
if nid >= 65536:
|
||||
raise SystemExit("FATAL: ran out of u16 id space (>65535)")
|
||||
recs.append((nid, flag, nm))
|
||||
used.add(nid)
|
||||
nid += 1
|
||||
recs.sort(key=lambda r: r[0])
|
||||
return recs
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 6:
|
||||
print(__doc__)
|
||||
sys.exit(2)
|
||||
ab, ai, ub, ui, outdir = sys.argv[1:6]
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
anchor_b, anchor_i = read_nim(ab), read_nim(ai)
|
||||
univ_b, univ_i = read_nim(ub), read_nim(ui)
|
||||
fb, fi = build(anchor_b, univ_b, 0), build(anchor_i, univ_i, 1)
|
||||
db = write_nim(fb, os.path.join(outdir, "blockmappings.nim"))
|
||||
di = write_nim(fi, os.path.join(outdir, "itemmappings.nim"))
|
||||
|
||||
fbm = {i: nm for i, _, nm in fb}
|
||||
fim = {i: nm for i, _, nm in fi}
|
||||
alt_b = sum(1 for i, _, nm in anchor_b if fbm.get(i) != nm)
|
||||
alt_i = sum(1 for i, _, nm in anchor_i if fim.get(i) != nm)
|
||||
bids = [i for i, _, _ in fb]
|
||||
bnm = [nm for _, _, nm in fb]
|
||||
iids = [i for i, _, _ in fi]
|
||||
inm = [nm for _, _, nm in fi]
|
||||
compl_b = {nm for _, _, nm in univ_b} <= set(bnm)
|
||||
compl_i = {nm for _, _, nm in univ_i} <= set(inm)
|
||||
|
||||
print(f"blocks={len(fb)} maxid={max(bids)} | items={len(fi)} maxid={max(iids)}")
|
||||
print(f"block sha256 = {hashlib.sha256(db).hexdigest()}")
|
||||
print(f"item sha256 = {hashlib.sha256(di).hexdigest()}")
|
||||
print(f"ANCHOR altered (MUST be 0): block={alt_b} item={alt_i}")
|
||||
print(f"dup ids: block={len(bids)-len(set(bids))} item={len(iids)-len(set(iids))}"
|
||||
f" | dup names: block={len(bnm)-len(set(bnm))} item={len(inm)-len(set(inm))}")
|
||||
print(f"complete (universe names all present): block={compl_b} item={compl_i}")
|
||||
ok = (alt_b == 0 and alt_i == 0
|
||||
and len(bids) == len(set(bids)) and len(iids) == len(set(iids))
|
||||
and len(bnm) == len(set(bnm)) and len(inm) == len(set(inm))
|
||||
and compl_b and compl_i and max(bids) < 65536 and max(iids) < 65536)
|
||||
print("GATE PASS:", ok)
|
||||
sys.exit(0 if ok else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,109 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Remap 7DTD decoration.7dt / multiblocks.7dt block ids from a world's NATIVE
|
||||
table to the cluster CANONICAL table, BY NAME. Preserves all non-id bits.
|
||||
|
||||
7dt format (little-endian, no trailing bytes):
|
||||
[0] u8 version (=6)
|
||||
[1:5] u32 count
|
||||
then count * 17-byte records; block id = u16 @ record offset +12,
|
||||
low 15 bits (& 0x7FFF) = id, high bit + bytes +14/+16 = rotation/placement.
|
||||
Rewrite rule: new = (orig & ~0x7FFF) | (canon_id & 0x7FFF).
|
||||
"""
|
||||
import sys, struct
|
||||
sys.path.insert(0, '/home/refuge/nimcanon/tool')
|
||||
from nimtool import read_nim # -> (records[list of (id,flag,name)], version)
|
||||
|
||||
REC = 17; HDR = 5; IDOFF = 12
|
||||
|
||||
def load_id2name(path):
|
||||
if path.endswith('.nim'):
|
||||
recs, _ = read_nim(path)
|
||||
return {i: n for i, _, n in recs}
|
||||
d = {}
|
||||
for line in open(path, encoding='utf-8', errors='replace'):
|
||||
line = line.rstrip('\n')
|
||||
if not line: continue
|
||||
p = line.split('\t')
|
||||
if len(p) >= 2 and p[0].isdigit():
|
||||
d[int(p[0])] = p[1]
|
||||
return d
|
||||
|
||||
def load_name2id(path):
|
||||
recs, _ = read_nim(path)
|
||||
return {n: i for i, _, n in recs}
|
||||
|
||||
def canon_idset(path):
|
||||
recs, _ = read_nim(path)
|
||||
return {i for i, _, n in recs}
|
||||
|
||||
def decode_ids(b):
|
||||
ver = b[0]; count = struct.unpack_from('<I', b, 1)[0]
|
||||
assert len(b) == HDR + REC*count, f"BAD LEN {len(b)} != {HDR+REC*count} (ver={ver} count={count})"
|
||||
out = []
|
||||
for r in range(count):
|
||||
pos = HDR + r*REC + IDOFF
|
||||
orig = struct.unpack_from('<H', b, pos)[0]
|
||||
out.append((pos, orig, orig & 0x7FFF, orig & 0x8000))
|
||||
return ver, count, out
|
||||
|
||||
def analyze(native, canon, infile):
|
||||
b = open(infile, 'rb').read()
|
||||
n2 = load_id2name(native); c2 = load_name2id(canon)
|
||||
ver, count, recs = decode_ids(b)
|
||||
from collections import Counter
|
||||
dist = Counter(r[2] for r in recs)
|
||||
unmap_id = sorted(i for i in dist if i not in n2)
|
||||
nm_for = {i: n2.get(i) for i in dist}
|
||||
unmap_name = sorted(set(nm_for[i] for i in dist if nm_for[i] is not None and nm_for[i] not in c2))
|
||||
inst_unmap_id = sum(dist[i] for i in unmap_id)
|
||||
inst_unmap_nm = sum(dist[i] for i in dist if nm_for[i] in (None,) or (nm_for[i] is not None and nm_for[i] not in c2))
|
||||
print(f" file={infile.split('/')[-1]} ver={ver} count={count} distinct_ids={len(dist)}")
|
||||
print(f" distinct ids NOT in native table: {len(unmap_id)} (instances {inst_unmap_id}/{count} = {100*inst_unmap_id/count:.1f}%)")
|
||||
if unmap_id[:8]: print(f" e.g. {unmap_id[:8]}")
|
||||
print(f" distinct names NOT in canon: {len(unmap_name)}")
|
||||
if unmap_name[:8]: print(f" e.g. {unmap_name[:8]}")
|
||||
nchg = sum(dist[i] for i in dist if i in n2 and n2[i] in c2 and (c2[n2[i]] & 0x7FFF) != i)
|
||||
print(f" instances whose canon id DIFFERS from native (would be rewritten): {nchg} ({100*nchg/count:.1f}%)")
|
||||
ok = (len(unmap_id) == 0 and len(unmap_name) == 0)
|
||||
print(f" >>> FULLY REMAPPABLE: {ok}")
|
||||
return ok
|
||||
|
||||
def decodecheck(table, infile):
|
||||
b = open(infile, 'rb').read(); ids = canon_idset(table) if table.endswith('.nim') else set(load_id2name(table))
|
||||
ver, count, recs = decode_ids(b)
|
||||
from collections import Counter
|
||||
dist = Counter(r[2] for r in recs)
|
||||
miss = sum(dist[i] for i in dist if i not in ids)
|
||||
print(f" {infile.split('/')[-1]} vs {table.split('/')[-1]}: {count} instances, {len(dist)} distinct ids, MISSING-in-table {miss} ({100*miss/count:.1f}%) [clean world should be ~0%]")
|
||||
|
||||
def apply(native, canon, infile, outfile):
|
||||
src = open(infile, 'rb').read(); b = bytearray(src)
|
||||
n2 = load_id2name(native); c2 = load_name2id(canon); cset = canon_idset(canon)
|
||||
ver, count, recs = decode_ids(b)
|
||||
changed = 0; bad = []
|
||||
for (pos, orig, bid, high) in recs:
|
||||
name = n2.get(bid)
|
||||
if name is None: bad.append(('noid', bid)); continue
|
||||
cid = c2.get(name)
|
||||
if cid is None: bad.append(('noname', name)); continue
|
||||
new = high | (cid & 0x7FFF)
|
||||
if new != orig:
|
||||
struct.pack_into('<H', b, pos, new); changed += 1
|
||||
if bad:
|
||||
print(f" !! {len(bad)} unmappable records, ABORT (e.g. {bad[:5]})"); return False
|
||||
open(outfile, 'wb').write(b)
|
||||
# self-verify byte safety: same length, only +12/+13 differ, all new ids in canon
|
||||
assert len(b) == len(src), "LENGTH CHANGED"
|
||||
diffbytes = [k for k in range(len(src)) if src[k] != b[k]]
|
||||
badpos = [k for k in diffbytes if (k - HDR) % REC not in (IDOFF, IDOFF+1)]
|
||||
ver2, count2, recs2 = decode_ids(bytes(b))
|
||||
unresolved = sum(1 for (_,o,bid,_) in recs2 if bid not in cset)
|
||||
print(f" wrote {outfile.split('/')[-1]}: count={count}(unchanged) len={len(b)}(unchanged) records_rewritten={changed}")
|
||||
print(f" byte-diff outside id field: {len(badpos)} (MUST be 0) ids unresolved in canon after remap: {unresolved} (MUST be 0)")
|
||||
return len(badpos) == 0 and unresolved == 0
|
||||
|
||||
if __name__ == '__main__':
|
||||
cmd = sys.argv[1]
|
||||
if cmd == 'analyze': analyze(sys.argv[2], sys.argv[3], sys.argv[4])
|
||||
elif cmd == 'decodecheck': decodecheck(sys.argv[2], sys.argv[3])
|
||||
elif cmd == 'apply': sys.exit(0 if apply(sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5]) else 1)
|
||||
@@ -0,0 +1,142 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Remap 7DTD decoration.7dt / multiblocks.7dt block ids from a world's NATIVE
|
||||
table to the cluster CANONICAL table, BY NAME. Preserves all non-id bits.
|
||||
|
||||
VENDORED, SELF-CONTAINED copy of nim-freeze/remap_7dt.py with nimtool.read_nim
|
||||
inlined and the host sys.path dependency removed, so it runs inside the
|
||||
panel-7dtd container (python3-minimal, no extra packages). Keep behaviour
|
||||
byte-identical to remap_7dt.py — the entrypoint's auto-provision relies on
|
||||
apply()'s exit code (0 = remapped + self-verified safe, 1 = refuse to start).
|
||||
|
||||
7dt format (little-endian, no trailing bytes):
|
||||
[0] u8 version (=6)
|
||||
[1:5] u32 count
|
||||
then count * 17-byte records; block id = u16 @ record offset +12,
|
||||
low 15 bits (& 0x7FFF) = id, high bit + bytes +14/+16 = rotation/placement.
|
||||
Rewrite rule: new = (orig & ~0x7FFF) | (canon_id & 0x7FFF).
|
||||
"""
|
||||
import sys, struct
|
||||
|
||||
REC = 17; HDR = 5; IDOFF = 12
|
||||
|
||||
|
||||
# --- inlined from nimtool.py (read_nim / parse_nim_bytes), byte-faithful ---
|
||||
def read_nim(path):
|
||||
with open(path, "rb") as f:
|
||||
data = f.read()
|
||||
return parse_nim_bytes(data)
|
||||
|
||||
|
||||
def parse_nim_bytes(data):
|
||||
if len(data) < 8:
|
||||
raise ValueError("file too short for header (%d bytes)" % len(data))
|
||||
version, count = struct.unpack_from("<II", data, 0)
|
||||
off = 8
|
||||
records = []
|
||||
for i in range(count):
|
||||
if off + 5 > len(data):
|
||||
raise ValueError("record %d: truncated header at offset %d (len=%d)" % (i, off, len(data)))
|
||||
rid, flag = struct.unpack_from("<HH", data, off)
|
||||
namelen = data[off + 4]
|
||||
off += 5
|
||||
if off + namelen > len(data):
|
||||
raise ValueError("record %d (id=%d): name truncated at offset %d (len=%d)" % (i, rid, off, len(data)))
|
||||
name = data[off:off + namelen].decode("utf-8")
|
||||
off += namelen
|
||||
records.append((rid, flag, name))
|
||||
if off != len(data):
|
||||
raise ValueError("trailing bytes: consumed %d of %d" % (off, len(data)))
|
||||
return records, version
|
||||
# --- end inlined ---
|
||||
|
||||
|
||||
def load_id2name(path):
|
||||
if path.endswith('.nim'):
|
||||
recs, _ = read_nim(path)
|
||||
return {i: n for i, _, n in recs}
|
||||
d = {}
|
||||
for line in open(path, encoding='utf-8', errors='replace'):
|
||||
line = line.rstrip('\n')
|
||||
if not line: continue
|
||||
p = line.split('\t')
|
||||
if len(p) >= 2 and p[0].isdigit():
|
||||
d[int(p[0])] = p[1]
|
||||
return d
|
||||
|
||||
def load_name2id(path):
|
||||
recs, _ = read_nim(path)
|
||||
return {n: i for i, _, n in recs}
|
||||
|
||||
def canon_idset(path):
|
||||
recs, _ = read_nim(path)
|
||||
return {i for i, _, n in recs}
|
||||
|
||||
def decode_ids(b):
|
||||
ver = b[0]; count = struct.unpack_from('<I', b, 1)[0]
|
||||
assert len(b) == HDR + REC*count, f"BAD LEN {len(b)} != {HDR+REC*count} (ver={ver} count={count})"
|
||||
out = []
|
||||
for r in range(count):
|
||||
pos = HDR + r*REC + IDOFF
|
||||
orig = struct.unpack_from('<H', b, pos)[0]
|
||||
out.append((pos, orig, orig & 0x7FFF, orig & 0x8000))
|
||||
return ver, count, out
|
||||
|
||||
def analyze(native, canon, infile):
|
||||
b = open(infile, 'rb').read()
|
||||
n2 = load_id2name(native); c2 = load_name2id(canon)
|
||||
ver, count, recs = decode_ids(b)
|
||||
from collections import Counter
|
||||
dist = Counter(r[2] for r in recs)
|
||||
unmap_id = sorted(i for i in dist if i not in n2)
|
||||
nm_for = {i: n2.get(i) for i in dist}
|
||||
unmap_name = sorted(set(nm_for[i] for i in dist if nm_for[i] is not None and nm_for[i] not in c2))
|
||||
inst_unmap_id = sum(dist[i] for i in unmap_id)
|
||||
print(f" file={infile.split('/')[-1]} ver={ver} count={count} distinct_ids={len(dist)}")
|
||||
print(f" distinct ids NOT in native table: {len(unmap_id)} (instances {inst_unmap_id}/{count} = {100*inst_unmap_id/count:.1f}%)")
|
||||
if unmap_id[:8]: print(f" e.g. {unmap_id[:8]}")
|
||||
print(f" distinct names NOT in canon: {len(unmap_name)}")
|
||||
if unmap_name[:8]: print(f" e.g. {unmap_name[:8]}")
|
||||
nchg = sum(dist[i] for i in dist if i in n2 and n2[i] in c2 and (c2[n2[i]] & 0x7FFF) != i)
|
||||
print(f" instances whose canon id DIFFERS from native (would be rewritten): {nchg} ({100*nchg/count:.1f}%)")
|
||||
ok = (len(unmap_id) == 0 and len(unmap_name) == 0)
|
||||
print(f" >>> FULLY REMAPPABLE: {ok}")
|
||||
return ok
|
||||
|
||||
def decodecheck(table, infile):
|
||||
b = open(infile, 'rb').read(); ids = canon_idset(table) if table.endswith('.nim') else set(load_id2name(table))
|
||||
ver, count, recs = decode_ids(b)
|
||||
from collections import Counter
|
||||
dist = Counter(r[2] for r in recs)
|
||||
miss = sum(dist[i] for i in dist if i not in ids)
|
||||
print(f" {infile.split('/')[-1]} vs {table.split('/')[-1]}: {count} instances, {len(dist)} distinct ids, MISSING-in-table {miss} ({100*miss/count:.1f}%) [clean world should be ~0%]")
|
||||
|
||||
def apply(native, canon, infile, outfile):
|
||||
src = open(infile, 'rb').read(); b = bytearray(src)
|
||||
n2 = load_id2name(native); c2 = load_name2id(canon); cset = canon_idset(canon)
|
||||
ver, count, recs = decode_ids(b)
|
||||
changed = 0; bad = []
|
||||
for (pos, orig, bid, high) in recs:
|
||||
name = n2.get(bid)
|
||||
if name is None: bad.append(('noid', bid)); continue
|
||||
cid = c2.get(name)
|
||||
if cid is None: bad.append(('noname', name)); continue
|
||||
new = high | (cid & 0x7FFF)
|
||||
if new != orig:
|
||||
struct.pack_into('<H', b, pos, new); changed += 1
|
||||
if bad:
|
||||
print(f" !! {len(bad)} unmappable records, ABORT (e.g. {bad[:5]})"); return False
|
||||
open(outfile, 'wb').write(b)
|
||||
assert len(b) == len(src), "LENGTH CHANGED"
|
||||
diffbytes = [k for k in range(len(src)) if src[k] != b[k]]
|
||||
badpos = [k for k in diffbytes if (k - HDR) % REC not in (IDOFF, IDOFF+1)]
|
||||
ver2, count2, recs2 = decode_ids(bytes(b))
|
||||
unresolved = sum(1 for (_,o,bid,_) in recs2 if bid not in cset)
|
||||
print(f" wrote {outfile.split('/')[-1]}: count={count}(unchanged) len={len(b)}(unchanged) records_rewritten={changed}")
|
||||
print(f" byte-diff outside id field: {len(badpos)} (MUST be 0) ids unresolved in canon after remap: {unresolved} (MUST be 0)")
|
||||
return len(badpos) == 0 and unresolved == 0
|
||||
|
||||
if __name__ == '__main__':
|
||||
cmd = sys.argv[1]
|
||||
if cmd == 'analyze': analyze(sys.argv[2], sys.argv[3], sys.argv[4])
|
||||
elif cmd == 'decodecheck': decodecheck(sys.argv[2], sys.argv[3])
|
||||
elif cmd == 'apply': sys.exit(0 if apply(sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5]) else 1)
|
||||
Reference in New Issue
Block a user