sausage/docs/SAU-384-rework-1389.md

SAU-384 round 3 — run 1389

Ready for review; not authorization for live execution. Addresses /docs/symphony/artifacts/SAU-384/extra-round-3.md and review 1388. Only documentation and disposable tooling changed; no transition code in any shipped plugin/host.

Reviewed selection and proposed plan

• Selected bundle: cutover-targets-1389.tar.gz, 27,271,968 bytes, SHA-256 31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b. Embedded manifest SHA-256 b48f2dacda3f348a67574767c6accfae1f1102227f94f86868ec1436bd4df394; all 23 payload sizes/hashes checked. Prior bundles are immutable historical evidence.
• New deferred payload: approved git 9cc9fa6f1c9f01ce1a53d8042623bde60e026637, /state/orchestrator/builds/git/9cc9fa6f1c9f01ce1a53d8042623bde60e026637.wasm (177 tests, not rebuilt here). Staged world copy: /docs/symphony/artifacts/SAU-384/git-9cc9fa6f.wasm. Both manifest deferred_plugins and operator deferred_artifacts are hash-checked before freeze. Not installed offline: mediator activates it only after target cred is live and Surma unlocks through the UI. The runbook uses the real core.activate wasm argument and includes the activation check before web release/final deletion.
• Cred remains approved 0db40101, build 1636 / 274 tests. Mail/cron/rooms remain 1600/1602/1603. No new plugin build/source edit this run. Native host remains /nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0, source e09c4f9f; old host mggvibk8 and loop pz2b94a2 remain pinned. SDK-copy list: src/lib.rs only. Web remains approved 3d760267, release held until session/web/verification prerequisites.
• Exact external commands, operator/approval boundaries, checkpoint archive/retry and rollback rules: runbook.md. Live manifest template: evidence/round3-1389/live-manifest.template.json. Resolve fresh identities/caller guards before go; sandbox manifests are evidence, not live instructions.
• Expected live Nix side effect requiring inclusion in Surma's go: three nix-store --query --roots calls, for old host, target host and loop, may prune stale auto-root and temporary-root registrations. They do not delete store packages. Five root registrations (those three plus controller/Herdr) remain in the proposed operation. This round used only the private rehearsal Nix DB; no shared/live roots query was repeated. Prior incident remains disclosed in rework-1381.md.

Six findings fixed

1. Git and interim compatibility. Actual live old WASM (reported source f089b56d; SHA-256 4b460bce9b67552e223bb209591009056d409b77290182f6cf142adc8c4c97f4) ran against real new cred in a separate synthetic world. Explicit-grant SSH clone authenticated with a real loopback sshd, recorded exactly one sign use and fetched the synthetic commit. Locked use returned vault_locked. Old grant-free shared SSH still fails requesting a local key; old grant-free HTTPS retains its old config/anonymous behavior, not the new sharing switch (source checked). Old granted HTTPS reached/was accepted by new cred and reserved one use, then the deliberately closed loopback destination returned network_error plus capture; successful HTTPS transfer is not claimed. After actual synthetic unlock and new-git activation, shared grant-free SSH clone succeeded. Do not schedule Git network work in the locked/old-git interval, introduce key/config-secret workarounds, or blindly retry an already-started caller's errors. Local status/commit/main merges are unaffected. Live secret-backed operations/pushes remain untested and separately approval-gated.
2. Pre-start abort and human operator. check/start, arm and immediate pre-freeze gates inspect consistent live SQLite backup-API copies, special file types, caller/file/artifact guards and free space. Space budget derives from current regular-file logical bytes plus the actual staged plugin/prev/SDK and recovery/stage/audit DB copies; no arbitrary multiplier/headroom/default. Growth or later ENOSPC is still possible. Driver now has a real abort CLI: exclusively lock, finish freeze if begun, restore only approved before/after paths from the complete immutable backup, verify full frozen inventory, then checkpoint-before-launch the explicit old owner in the same pane. No completed plan means apply never began: do not restore an incomplete backup. Restore is resumable; after old_start_attempted inspect instead of launching twice. Surma is the named external operator, with independent terminal log monitoring, failure.json, and best-effort Herdr notification (submission result retained; physical display not claimed). Runbook gives exact resume/abort and evidence-archive/re-resolution commands. After target started, automatic resume/restore/relaunch stays forbidden. Missing registration now exits on one absolute 120 s deadline from cron/src/types.rs's existing registration/publication budget and the earlier real-core fixture bound; Herdr/RPC calls are finite too. No polling for quiescence or availability.
3. Executable-only SQL guards. Cron projects id/method/params_json/cron/at/timezone/detach/call_timeout_secs, not next/last/result/occurrence bookkeeping. Board excludes last_sent/ended and batch IDs/lines/due; unions active subscription config with pending config snapshots so creating/delivering the same active configuration is not drift. New/changed executable configuration still refuses. Final automatic subscription removal changes the executable set and requires refresh or pre-start abort, explicitly documented. Add any planned mediator cron wake-up before resolving/approving operator.json. No caller/boot/plugin/SDK edits between resolution and freeze; arm checks again. No in-progress manifest edits.
4. prev contract. Every offline-swapped plugin's prev is now its exact frozen pre-cut current, under before/after guards and restore. Tests checked all nine, including paths where an older prev existed. Git waits for normal later activation/prev behavior. This makes the fallback image meaningful, not a license for blind incompatible individual core.revert; coordinated rollback and unsubscribe-before-session-revert remain mandatory.
5. Roots side effect. Listed above and in the live runbook/approval plan, rather than describing roots queries as mutation-free. Subsequent tests stayed private.
6. Arm race. Replaced reader-dependent FIFO with atomic, manifest-bound arm.json and an arm-then-scan inotify watch. Idempotent arm returns with no reader. resume accepts waiting_for_arm; a kill after marker observation but before saving armed reuses the same marker. No manual FIFO writer/cleanup is needed.

Executed gates and full logs

Prefix: /logs/box/2026-10-31/tc-asa3yr-exec-.

Gate	Log / outcome
Final exact tool, caller killed + controller killed/resumed, incomplete backup resume, real Herdr job-control stop/swap/start, same shell, post-start preservation/new verifier, prev hashes, pinned cold restart and old clone	3699755631.stderr, pass. Final backup 4,436 files / 627,157,113 bytes, 4.559 s (session-size integration, not full-world downtime). No old mail owner; one manager; cred locked.
Live special-file/zero-space negative, mutable cron vs changed payload, pre-arm drift, frozen SQL mismatch -> real-driver abort/old relaunch, consumed-arm/save kill/resume, post-start abort refusal	1485773773.stderr, pass. Zero-space gate used a test-only mocked disk-usage response; actual ENOSPC tested below.
Real-driver partial swap -> abort/restore/old relaunch; actual backup ENOSPC on a private tmpfs introduced after good preflight -> untouched-old relaunch; lost console subscription -> bounded clean exit	4142682240.stderr, pass. Registration wait 120 s; elapsed 132.493 s including backup/launch. Started checkpoint refused both resume and abort.
Real old/new Git + new cred, loopback synthetic OpenSSH/HTTP negative, unlock then deferred activation	99781693.stderr, pass. No live key/grant/remote used.
SQLite WAL + genuinely hot rollback journal, exact losses, preserved job IDs/calendar/outcomes/generic JSON/uncertain mail, apply interruption/rerun and exact physical restore	1727732043.stderr, pass.
Board executable projections and old-select-star negative control; template/deferred-artifact checks; final driver's actual Herdr failure notification + abort	1276236846.stderr, pass.
cargo test --workspace against unchanged production source	2259546796.stderr, pass: 208 host integration tests, lines 62–431, plus all workspace suites/doc tests. 133 Rust/Cargo/flake inputs matched card tree. No Rust implementation change.
Executed verify mode, complete source-as-log and deletion	2013747101.stderr, pass. See hashes/ranges below.

Prior accepted race/replay/deadline/core-watch/upgrade/idle/grace/crash-supervision evidence is cumulative, not silently rerun or claimed fixed here. Prior live-size scale sample remains 138,444 files / 12,441,162,164 bytes / 45,310 directories, synthetic size-bucket match, 205.940 s inventory+copy/hash, warm cache. Post-record manager ack 0.6078 s, registration catch-up 0.0128 s, mail publication 0.5376 s (integrated 0.5235 s) remain under caller publication budgets 3 s orchestrator / 10 s cred; catch-up outlier 12.0419 s and initial-verifier startup scan delay 23.753/24.530 s remain disclosed. No production downtime/latency guarantee.

Deleted executable and reconstruction receipt

Only source-as-log remains in 2013747101.stderr. Reconstruct exactly three files for the later approved operation, hash-check before use; do not reconstruct test controllers:

• cutover_once.py: lines 2–164, SHA-256 3adf65ebb0b3401a0d1976524f35f7450e907ad32f907dcbb1cd2f7e0a7e5d3a;
• freeze.py: lines 166–238, SHA-256 d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe;
• driver.py: lines 240–503, SHA-256 686167318c84a32c1010daf4517f036bcb654384cadab987fa47fb13deecd2bf.

verify actually logged/deleted the executable after successful rehearsal. Executed test/patch/controllers and test launchers were logged/deleted too, with bytecode caches removed. tool-deletion.json and executed-test-deletion.json are under evidence/round3-1389/. Permanent pinned owner/selector configuration is not migration code; it stays rooted through the recovery window. The live runbook still requires mediator deletion after verified success.

Failures and limits

Tried next methods, not blockers: initial generated SQL fixture had quoting syntax wrong (corrected); generated lockfiles made whole-directory diff reject (compared actual Rust/Cargo/flake input bytes, then ran full suite); final-sidecar extraction included a trailing log status (trimmed); a post-check initially read the previous round's stopped-world manifest (corrected to this round, then passed). Synthetic SSH setup needed explicit typed key metadata, a sandbox-only sshd account and an unlocked sandbox login account; default root was locked. A failed fixture left two loopback sshd listeners, later explicitly identified/stopped. Initial sandbox activation used path; core required wasm, corrected and the complete final Git gate passed. No failed test was relabeled successful; full errors remain in logs.

Still unverified/out of scope: approved live cut-over/frozen caller snapshot, actual live SSH-detach/pane w7:p6 execution, Surma's real unlock/new-git live networking, physical Herdr notification/browser display, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. No live activation/restart/config/owner change, push, release, deploy or /www publication this run. Surma's go is still required before mediator core/session/cred operation, detached SSH exception, roots side effects and every push.