Rehearse shared-disk failures and forward-only cutover completion

Card: SAU-384

docs/SAU-384-rework-1392.md

@@ -0,0 +1,60 @@
+# SAU-384 round 4 — run 1392
+
+Ready for review; **not live execution approval**. Addresses all five findings in extra-round-4.md. No shipped Rust/plugin behavior changed. Current-main source and state contracts were rechecked; all work was in the card tree, review artifacts and disposable box tc-zg0j56.
+
+## Selection and operator plan
+
+The selected bundle is unchanged: `cutover-targets-1389.tar.gz`, **31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b**, 23 payloads. All payload sizes/hashes and the native closure hashes were freshly checked. Host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0` from e09c4f9f. Cred remains 0db40101 / build 1636 (274 tests); mail/cron/rooms remain 1600/1602/1603. Deferred git remains approved 9cc9fa6f, activated only after target cred is live and Surma unlocks. No new WASM/package build. SDK-copy list: **src/lib.rs only**. Approved web 3d760267 remains held until the operator releases SAU-375 after session/web and verification prerequisites.
+
+Exact commands and approval boundaries are in [runbook.md](runbook.md). Current non-executable operator template: `evidence/round4-1392/live-manifest.template.json`. It still requires fresh caller/owner identities, guards and explicit Surma go. The only executable source is reconstructable **as log**, not retained as a tool. Surma is the external human recovery operator. The old mediator is deliberately cut; the driver publishes a new explicit default verification input, not inferred recovery.
+
+## Five findings fixed
+
+1. **Shared-filesystem ENOSPC.** Space preflight measures POSIX allocated blocks for files/directories/root/non-followed links, and charges dense filesystem-fragment-rounded size where copy2 would expand a sparse file. It includes the concrete staging/prev/SDK/DB-copy roles; receipts save all measurements. Removed the false never-underestimates claim: metadata/normalization/growth/competing writers remain uncertainty. A separate sparse/tiny/directory fixture copied **4,599,808 allocated bytes** from **12,288 source-allocated bytes**, below its measured **4,603,904-byte** budget. No live margin/default was invented.
+
+   Mandatory foreground `driver.py monitor` receives memory-only Unix datagrams for inventory/backup progress and failure, and kernel pidfd controller-exit completion. Every start/resume waits for its own monitor; arm requires it. Failure JSON/log writes are best effort and cannot prevent CUTOVER FAILED or the independent Herdr notification attempt. A terminal still reports controller death on SIGKILL; no heartbeat/polling loop. Notification submission was tested, but returned shown:false/disabled; physical display is not claimed.
+
+   Actual ENOSPC used **one shared tmpfs for world, external root and driver.log**, device 83, **0 free bytes**, capacity 254,504,960 bytes derived from measured fixture/budget allocations. It failed during a real backup write after good preflight. Monitor showed failure and death despite unwritable disk evidence. Pre-plan abort removes only the reproducible incomplete backup subtree before any checkpoint write, retaining planning/inventory evidence. If the competing writer still fills the disk, it refuses before saving/launching. After that known test file was freed, the same abort restored coherent old ownership and readiness. Complete backups are never deleted. Every abort (with or without a completed plan) now checks space before its first checkpoint save; a second shared-filesystem zero-free-byte fixture proved the complete backup/checkpoint remained exact, then old relaunch worked after freeing only the known competing test file. Space is checked again before old relaunch. This is not proof against every filesystem's ENOSPC/power-loss behavior.
+2. **Post-start forward completion.** New `continue operator.json` locks, checks the bound manifest, exact pane/shell, exactly one target host and target plugin hashes, and performs only missing/read-retry-safe post-start RPCs. It never launches/applies/restores. Actual locked cred.status is saved durably before unlock; subscribe uses the stable cutover registration; send reuses the exact continuation UUID/text. It reaches awaiting_verification, then normal verify/delete is reachable. Real failures of **cred.status, subscribe, and send**, including send committed before its acknowledgement was lost, all recovered with one run-start/consumption. A real **120 s** absolute registration timeout recovered the same way without another owner. The value remains cron/src/types.rs's existing registration/publication bound. Runbook gives exact continue command, RPC payload derivation, locked-proof/unlock ordering and local Surma fallback if live SSH signing is unavailable.
+3. **Atomic temporary cleanup.** A child was SIGKILLed inside atomic after writing a real partial plugin replacement temporary, before rename. Abort removes only a saved operation's newly created .cutover-tmp, absent in frozen inventory and prefix-matching an approved before/after image; unknown bytes/links refuse. Full frozen inventory equality passed inside rollback before old launch. Resume still overwrites/completes its own temporary. No arbitrary world cleanup.
+4. **Converter pre-freeze dry run.** check/start, arm and immediate pre-freeze checks use the same pure cron converter as plan, on SQLite backup-API memory copies. interrupt:true, invalid text and non-object params refuse without freeze/mutation. Negative control: round-3 checker accepted the same invalid object payload, new checker refused. No compatibility code in a plugin.
+5. **Roots/wake-up documentation.** Every controller resume repeats all three roots queries and their possible stale-registration pruning; this remains part of the proposed Surma-approved live plan. This round queried roots only against a private rehearsal Nix DB. No cron wake-up is needed: the stable explicit driver continuation owns handoff. A due-during-downtime job is skipped; a just-before-freeze input can enter an old pending queue that is intentionally dropped. Neither is replayed. The runbook disallows relying on guessed backup duration as readiness.
+
+Testing also exposed a real **Nix Python wrapper exec identity race** in parent-side process capture. The launcher now receives the child's actual post-wrapper identity over a pipe-completion handshake. Controller kill/resume proved exact identity, a fresh monitor requirement and successful same-pane continuation. No liveness/endpoint polling was added.
+
+## Full logs and evidence
+
+All log names below have prefix `/logs/box/2026-10-31/tc-zg0j56-exec-`. Evidence is `evidence/round4-1392/`.
+
+| Gate | Full stderr / outcome |
+|---|---|
+| Final shared world/root/log ENOSPC, pre-save incomplete-backup deletion, low-space refusal, old readiness; partial atomic SIGKILL/abort; all three post-start RPC failures; real registration timeout | **166388571.stderr**, clean combined pass on final tool source. Evidence under `final/`, including shared-enospc.json, atomic-abort.json, *-reviewed.json and registration-timeout.json. |
+| Complete-plan ENOSPC on one shared filesystem: pre-write refusal, checkpoint/full-backup equality, old readiness after known competing-data cleanup | **4094731865.stderr**, pass; `final/complete-plan-enospc.json`. |
+| Controller wrapper handshake, killed-controller pidfd, resume waits for fresh monitor, target completion | **537908850.stderr**, pass; `resume-handshake.json`, both monitor logs. |
+| Converter pre-freeze refusals | **1954198187.stderr**, pass; `dryrun.json`. |
+| Negative control + sparse allocation + all payload/native closure hashes + **133** final source/build inputs | **1314968883.stderr**, pass; `converter-negative-control.json`, `allocated-space.json`, `source-and-bundle.json`. |
+| Committed WAL/hot rollback journal; exact pending/cancel/mail losses; preserved mail/jobs/IDs/calendar/outcomes/non-target JSON; apply interruption/rerun and physical rollback | **2622539348.stderr**, pass; `sqlite-sidecars.json`. |
+| cargo test --workspace, unchanged final production source | **3315145279.stderr**, pass; **208 host integration tests**, result at line **433**, plus all workspace suites/doc tests. |
+| Fresh target one-manager/locked-cred/original-byte preservation/pre-cut-prev verification; actual verify CLI and complete tool/test deletion | **3030415406.stderr**, pass; `final/` final-ownership-preservation.json, verification.json, final-verified-driver.json, tool-deletion.json, test-deletion.json. Earlier verify/deletion pass: 1403220230.stderr. |
+| In-memory source-log SHA/AST reconstruction; executables/bytecode still absent | **2019074395.stderr**, pass; `final/source-log-roundtrip.json`. Earlier docs/JSON/absence checks: **2041020851.stderr**; final docs/JSON/absence/23-payload/133-source-input validation: **1011452644.stderr**, `final/final-validation.json`. |
+
+### Executed source deleted after verified success
+
+Reconstruct only these three files from **3030415406.stderr** for a later approved live operation; do not reconstruct tests:
+
+- cutover_once.py: source lines **2–189**, SHA-256 **84b13644bd80c6b331dcd8df9bf79339f62fedb99f7dd376535b2bca7b0afd96**;
+- freeze.py: source lines **192–263**, SHA-256 **d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe**;
+- driver.py: source lines **266–669**, SHA-256 **5c3cba1a69b8b5e8f82ea83caa0697af494bf93dd0297f0c1b5231192cbb5fad**.
+
+After generalizing the no-write space gate to complete-plan and untouched abort too, the source was reconstructed for the final affected rerun, then the real verify command again marked that rehearsal verified, logged/deleted the three files and caches; all executed/reconstructed tests/patches/wrapper were logged/deleted too. Permanent owner/selector installation artifacts are distinct, not migration code. Live runbook still mandates deletion after verified success.
+
+## Failed approaches and limits
+
+- First interpreter alias linked only bin/, losing Nix Python's site-package context. Linked the complete existing interpreter environment; imports and gates then passed.
+- Cold workspace cache made six offline guest builds miss async-task; populated the disposable cache and reran the **whole workspace**, 208/208 host tests, unchanged source. Initial full log **2236066973.stderr**, missing dependency at lines 755–785; final pass above. No retry-without-diagnosis claim.
+- First shared filesystem test refused **before freeze** because actual source growth exceeded its provisioned measurement. Recorded that valid negative control, then provisioned the test from measured fixture/budget roles before inducing deliberate post-preflight pressure. A later test filled before the backup had copied any data: abort correctly needed a no-write space check before its first save; fixed and repeated with a before-backup event gate. Final shared test reached zero free bytes during copy and the final combined gate passed cleanly. Earlier fixture checkpoint-name collisions, an existing evidence-export directory and a busy mount affected cleanup/export only; used fresh identities/paths and explicit disposable-pane cleanup, never edited migration checkpoints. Mounts were left for disposable-box teardown, not live cleanup.
+- First post-start fixture omitted session prompt files, so it correctly recorded run-end error rather than mailbox consumption. Added synthetic prompts and repeated all three failure modes from fresh worlds; each then produced exactly one consumed continuation.
+
+This round's failure fixtures are **small synthetic real-core/Herdr worlds**, not another live-size benchmark. Prior accepted live-size race/replay/core-watch/upgrade/deadline/idle/grace/crash-supervision, scale and Git compatibility/unlocked activation evidence remain cumulative. Startup 0.6078 s, catch-up 0.0128 s, mail publication 0.5376 s vs callers' 3 s/10 s limits, catch-up outlier 12.0419 s and initial-scan delay remain disclosed in earlier reports; not remeasured or upgraded to guarantees here.
+
+Unverified/out of scope: actual approved live cut-over/frozen caller audit, live detached SSH/Herdr w7:p6 execution, live SSH signing through target cred, Surma's real unlock/new-git external networking, physical notifications/browser, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. Surma's external terminal has exact continue/verify commands, so final completion is not dependent on the untested live signer. No push, live activation/restart/config/owner change, release, deployment or /www publication. Surma go still precedes mediator core/session/cred operation, the proposed detached-SSH exception/root side effects, and **every push**.