sausage/docs/SAU-384-rework-1381.md

SAU-384 review rework — run 1381

Ready for review, not live execution. This changes only local documentation and disposable one-off tooling. No shipped host/plugin behavior, live boot/SDK/launcher, activation, restart, publication, release or push changed. Surma's go still precedes the mediator's live operation and every push.

Review findings addressed

1. No mediator needed during downtime. Replaced the disconnected procedures with one external CLI, driver.py start → arm → detached run → verify. start acknowledges a new OS session and blocks it on an arm FIFO. After arm the detached driver owns native freeze, immutable/resumable full backup, planning/frozen audit, offline apply, selectors, started, launch and readiness. It uses the exact pinned Herdr pane/shell, then publishes a new explicit default verification message, with a stable UUID, because the cut old mediator run is not recoverable. The last real-Core/Herdr test killed the invoking shell shim, then killed the detached controller at frozen, resumed it, and reached target in the same shell. Actual target run-start and mailbox-consumed records named that new verification message once. A real live SSH launch was not performed; the runbook makes the two finite foreground SSH launch/arm commands, and the required explicit go for that exception to SSH's background-work policy, part of the proposed operator plan.
2. Actual frozen-state gates. Kit.plan, apply, starting and pre-start rollback require the real freeze.json, phase frozen, current boot ID, matching world, dead pinned owners, empty old group, no subsequently running host, and unchanged checkpoint bytes. A caller-supplied stopped literal is gone. Captured/wrong-boot checkpoints, running owner, changed config/SQL projection and after-start resume refuse before mutation. Offline retries accept only their saved before/after images.
3. Persistent installation pins. Driver registers/verifies GC roots for exact old/target/loop packages and creates old/target selector and owner artifacts before freeze, so pre-start recovery already has the explicit old owner. Both wrappers export their own pinned selector. Cold restart in the same Herdr shell, without ambient SAUSAGE_FLAKE, ran the exact target; the explicit old wrapper ran the exact old package on a separate old-backup clone, never over new target work. Runbook prohibits bare loop/remote-default starts and explains that core.restart restarts the pin, not self-updates. Future host updates need an approved package/root/selector change; roots are not removed with migration scripts.
4. Real PTY job control. Used actual Herdr 0.9.0 and an interactive zsh job-control shell, not setsid+piped stdin alone. Loop was foreground process-group leader in the shell's session. Freeze sends TERM then CONT before pidfd wait. Separate injection left the real host in observed T state after outer-owner exit; TERM+CONT completed shutdown with no surviving group member. This exercises the stopped-state hazard; it does not claim every timing of naturally delivered SIGTTIN.
5. Planning, SQLite and scale. Incomplete backup resumes the same inventory/backup without cleanup or replacement. Streaming hashlib.file_digest avoids whole-file memory reads; os.walk/copy preserve directories and non-followed symlinks (including broken links). Complete DB sidecar families are backed up physically, recovered only in disposable copies, staged with SQLite backup in DELETE mode, and old sidecars removed before main-image replacement. Committed WAL and hot uncommitted rollback-journal fixtures passed interruption/rerun, exact losses, preserved job IDs/calendar/outcomes/generic JSON/unknown mail, and exact pre-start rollback. A full logical-size/count/bucket/directory-matched synthetic replica measured the whole-world backup path below; it was not an exact live-byte snapshot.

Evidence and exact gates

Evidence directory: /docs/symphony/artifacts/SAU-384/evidence/rework-1381/. It contains only JSON manifests/inventories/checkpoints/results, not migration executables. sandbox-operator-manifest.json records the tested invocation; do not execute it live. live-manifest.template.json deliberately requires fresh identities, typed caller review/guards and a new stable continuation UUID before approval. Frozen guard tests used the real cron table projection; the live table/file set must include the board/config/registration review described in the runbook.

All logs below are /logs/box/2026-10-31/:

Gate	Full stderr log / result
Final caller-killed + controller-killed/resumed + interrupted planning + same-Herdr-pane target chain	tc-asa3yr-exec-775822946.stderr — pass. 4,436 frozen regular files / 627,157,113 bytes in this session-size integration fixture; backup/hash stage 4.617 s. Earlier final chains also passed (…-2599916298, …-2393656781, …-2876439273).
Final process ownership/locked cred/new explicit verifier/preservation; stopped-PTY CONT; cold pinned target; explicit old package on old clone	tc-asa3yr-exec-3942858286.stderr — pass. post-target-processes.json, post-preservation.json, post-continuation-records.json, post-pty-stop-continue.json; only planned paths and enumerated new activity changed.
WAL + hot journal + interruption + exact rollback, repeated	tc-asa3yr-exec-1595763713.stderr — pass; first pass …-1628589419. db-sidecars.json.
Whole-world scale	tc-asa3yr-exec-3620999999.stderr — 138,444 files, 12,441,162,164 bytes, 45,310 directories. Inventory 70.555 s, copy plus double hash verification 135.385 s, combined 205.940 s. Exact functions used by Kit.plan; synthetic bytes/names, observed live size buckets/counts, warm-cache box. No cold-storage downtime SLA.
Current source equivalence + cargo test --workspace	tc-asa3yr-exec-430706202.stderr — pass; 121 Rust/Cargo inputs plus root Cargo/flake inputs matched current card tree. 208 host integration tests, lines 64–431, plus all workspace suites/doc tests. No Rust host source changed. The final cred main refresh/rebuild and repeated gates are below.
Negative config/frozen caller controls, selected bundle rehash, verify and deletion	tc-asa3yr-exec-3977808892.stderr — pass; negative-controls.json, bundle-recheck.json, tool-deletion.json, executed-test-deletion.json.
Source-as-log reconstruction/hash/AST check; no scripts/bytecode recreated	tc-asa3yr-exec-3884064136.stderr — pass; source-log-roundtrip.json.

Reconstructable, but deleted after success

The executed tool was three small files, not a plugin or host patch. Exact source survives as log only in tc-asa3yr-exec-3977808892.stderr:

• cutover_once.py: lines 7–152, SHA-256 202eb7e635d675bb3960317913ae154e4f699fb7f5fe25a53ef34c965d0869c5;
• freeze.py: lines 154–226, SHA-256 d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe;
• driver.py: lines 228–393, SHA-256 7a26cfba9873541404f9661d5b0117f566301d408448cbb509a965d7900037aa.

The verify CLI was actually executed, marked the rehearsal verified, logged/deleted these files, and removed caches. All executed test controllers were also logged/deleted; the named disposable Herdr server was stopped. Hash reconstruction was checked in memory, without recreating executable files. The live runbook mandates the same deletion. Permanent owner/selector artifacts are installation artifacts, represented as bytes/hashes in permanent-launcher-artifacts.json, not retained migration code.

Final main refresh and selected builds

The final required merge found cred main advanced to approved SAU-418, 0db401012cae99b4424670d5c07a346281e30e75 (plain anyone-may-use switch/shared SSH metadata and locked no-candidate fetch behavior). Inspected that diff and fast-forwarded the card worktree without discarding either side. No storage-schema or handover lifecycle delta was introduced. Rebuilt current cred: 1636 / 274 tests passed, /state/cargo/builds/1636/cred.wasm; full output /logs/cargo/2026-10-31/01a19281-a8ed-7127-914a-3ba4b5c15d17.{stdout,stderr}.log.

Final selection is now /docs/symphony/artifacts/SAU-384/cutover-targets-1381.tar.gz, 24,590,972 bytes, SHA-256 9295c5fddbf5d68cbfd525602ec67e7dd895bf6afb99324191a6c514282eac57. Only the cred payload/provenance changed; all other 21 payloads, native host and SDK selection remain as reviewed. The original bundle remains immutable prior evidence. Current evidence/manifest.json and evidence/rework-1381/main-refresh/selection.json pin the final bundle. Host remains /nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0; SDK-copy src/lib.rs only. Mail/cron/rooms remain 1600/1602/1603; approved web 3d760267 stays held until mediator/orchestrator release. No plugin implementation edits by this worker.

Reconstructed the same three tool sources from their hash-checked log solely for this required new-main rerun, then deleted them again after verified success. Repeated caller/controller death/resume and actual Herdr cutover with the new cred bundle passed (tc-asa3yr-exec-629237259.stderr, same log directory). The first preservation assertion additionally encountered normal core-journal WAL/SHM activity; explicitly classified those journal sidecars as activity rather than migration loss. Final same-pane ownership/locked cred/fresh verifier/preservation/stopped-PTY/cold target/old-clone gates, negative controls and full cargo test --workspace all passed tc-asa3yr-exec-4223877022.stderr (208 host integration tests). Sidecar/rollback tests also repeated successfully. Final verify/source-as-log/deletion proof: tc-asa3yr-exec-741401146.stderr, evidence/rework-1381/main-refresh/{tool-deletion,executed-test-deletion}.json. Tool source hashes are unchanged from the reconstruction list above; no executable/cache remains. Final live template and sandbox manifests are under that main-refresh/ evidence directory. Earlier replay/race/cron/watch/scale gates remain cumulative evidence; they were not silently relabeled as freshly rerun.

Failures, corrections and limits

• Whole-tree transfer through box.put_tree rejected the 64 MiB service cap for repos/plugins/logs. Next method: read-only host size histogram, then regular-file size-bucket replica and the exact backup routines. This measures whole-world scale, not original live contents/path topology. The original accepted session byte snapshot still supplies the real-history integration volume.
• Initial PTY fixtures lacked state/core for console history, then matched an http-prefixed address against a bare address. Corrected fixture and barrier. Typeahead from two early console commands lost the second; final driver queues one subscription command. An early Herdr multiline output matcher stalled; inspection showed Herdr wait-output itself polls, so final code removed it entirely in favor of inotify completion. The actual registration path is state/session/manager/registrations/subscriptions.json, now corrected.
• Shared-store foot gun: an early default-store root query pruned 1,144 stale auto-root links and 571 stale temporary-root files in shared Nix state. This was not an authorized live-config operation and no GC/package deletion was requested. Immediately stopped that method. Read-only SSH inspection on Citadel found none of those referenced targets and none of those PID numbers existing on the host, so no active roots required restoration. All subsequent root tests used a private local Nix state database, loaded from read-only closure metadata; no shared root query was repeated. Exact evidence/log: gc-root-incident.json, tc-asa3yr-exec-2216542428.stderr. This incidental stale-registration cleanup is disclosed rather than claiming literally zero host-state writes.
• One post-start assertion read history before the new verification message was consumed. Replaced it with arm-then-scan/history-write completion. In the final gate publication→run-start was 23.753 s, →consumption 24.530 s, queued behind startup reconciliation. This is not mail.send publication latency. Initial acknowledgement is not full scan completion.

Prior accepted replay/race/rollback/deadline/grace/crash-supervision/event-driven orchestrator evidence remains unchanged. Post-record manager ack 0.6078 s, catch-up 0.0128 s, mail publication 0.5376 s / integrated 0.5235 s remain below existing 3 s / 10 s publication budgets; integrated catch-up outlier 12.0419 s, ~0.10% idle duty, detached error propagation and 305.427 s real cron warning/stop remain disclosed in implementation.md. This rework does not replace those measurements with a claim that the initial scan is fast.

Still unverified/out of scope: actual approved live cut-over/frozen backup and complete production caller guards, live SSH detach/Herdr w7:p6 execution, orchestrator.release/publication/physical browser, cold-storage timing, power loss, provider cancellation/billing and independent recovery follow-ups. The live launch exception needs explicit Surma go as part of the proposed plan; no such approval is claimed or requested by executing anything here.