surma/sausage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Close cut-over recovery, executable-caller, arm and deferred Git gaps

Browse Source 
Card: SAU-384
This commit is contained in:
SAU-384 implementer
2026-10-31 15:44:19 +00:00
parent 5870a1b225
commit 30df27f310
3 changed files with 122 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/SAU-384-rework-1389.md
+53
View File
@@ -0,0 +1,53 @@
# SAU-384 round 3 — run 1389
Ready for review; **not authorization for live execution**. Addresses `/docs/symphony/artifacts/SAU-384/extra-round-3.md` and review 1388. Only documentation and disposable tooling changed; no transition code in any shipped plugin/host.
## Reviewed selection and proposed plan
- Selected bundle: `cutover-targets-1389.tar.gz`, **27,271,968 bytes**, SHA-256 **31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b**. Embedded manifest SHA-256 **b48f2dacda3f348a67574767c6accfae1f1102227f94f86868ec1436bd4df394**; all **23** payload sizes/hashes checked. Prior bundles are immutable historical evidence.
- New deferred payload: approved git **9cc9fa6f1c9f01ce1a53d8042623bde60e026637**, `/state/orchestrator/builds/git/9cc9fa6f1c9f01ce1a53d8042623bde60e026637.wasm` (177 tests, not rebuilt here). Staged world copy: `/docs/symphony/artifacts/SAU-384/git-9cc9fa6f.wasm`. Both manifest `deferred_plugins` and operator `deferred_artifacts` are hash-checked before freeze. **Not installed offline**: mediator activates it only after target cred is live and Surma unlocks through the UI. The runbook uses the real `core.activate` **wasm** argument and includes the activation check before web release/final deletion.
- Cred remains approved **0db40101**, build **1636 / 274 tests**. Mail/cron/rooms remain **1600/1602/1603**. No new plugin build/source edit this run. Native host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, source e09c4f9f; old host mggvibk8 and loop pz2b94a2 remain pinned. SDK-copy list: **src/lib.rs only**. Web remains approved **3d760267**, release held until session/web/verification prerequisites.
- Exact external commands, operator/approval boundaries, checkpoint archive/retry and rollback rules: [runbook.md](runbook.md). Live manifest template: `evidence/round3-1389/live-manifest.template.json`. Resolve fresh identities/caller guards before go; sandbox manifests are evidence, not live instructions.
- **Expected live Nix side effect requiring inclusion in Surma's go:** three `nix-store --query --roots` calls, for old host, target host and loop, may prune stale auto-root and temporary-root registrations. They do not delete store packages. Five root registrations (those three plus controller/Herdr) remain in the proposed operation. This round used only the **private rehearsal Nix DB**; no shared/live roots query was repeated. Prior incident remains disclosed in rework-1381.md.
## Six findings fixed
1. **Git and interim compatibility.** Actual live old WASM (reported source f089b56d; SHA-256 4b460bce9b67552e223bb209591009056d409b77290182f6cf142adc8c4c97f4) ran against real new cred in a separate synthetic world. Explicit-grant SSH clone authenticated with a real loopback sshd, recorded exactly one sign use and fetched the synthetic commit. Locked use returned `vault_locked`. Old grant-free shared SSH still fails requesting a local key; old grant-free HTTPS retains its old config/anonymous behavior, not the new sharing switch (source checked). Old granted HTTPS reached/was accepted by new cred and reserved one use, then the deliberately closed loopback destination returned `network_error` plus capture; **successful HTTPS transfer is not claimed**. After actual synthetic unlock and new-git activation, shared grant-free SSH clone succeeded. Do not schedule Git network work in the locked/old-git interval, introduce key/config-secret workarounds, or blindly retry an already-started caller's errors. Local status/commit/main merges are unaffected. Live secret-backed operations/pushes remain untested and separately approval-gated.
2. **Pre-start abort and human operator.** `check`/start, arm and immediate pre-freeze gates inspect consistent live SQLite backup-API copies, special file types, caller/file/artifact guards and free space. Space budget derives from current regular-file logical bytes plus the actual staged plugin/prev/SDK and recovery/stage/audit DB copies; no arbitrary multiplier/headroom/default. Growth or later ENOSPC is still possible. Driver now has a real `abort` CLI: exclusively lock, finish freeze if begun, restore only approved before/after paths from the complete immutable backup, verify full frozen inventory, then checkpoint-before-launch the explicit old owner in the same pane. No completed plan means apply never began: do not restore an incomplete backup. Restore is resumable; after `old_start_attempted` inspect instead of launching twice. Surma is the named external operator, with independent terminal log monitoring, `failure.json`, and best-effort Herdr notification (submission result retained; physical display not claimed). Runbook gives exact resume/abort and evidence-archive/re-resolution commands. After target `started`, automatic resume/restore/relaunch stays forbidden. Missing registration now exits on one absolute **120 s** deadline from cron/src/types.rs's existing registration/publication budget and the earlier real-core fixture bound; Herdr/RPC calls are finite too. No polling for quiescence or availability.
3. **Executable-only SQL guards.** Cron projects id/method/params_json/cron/at/timezone/detach/call_timeout_secs, not next/last/result/occurrence bookkeeping. Board excludes last_sent/ended and batch IDs/lines/due; unions active subscription config with pending config snapshots so creating/delivering the same active configuration is not drift. New/changed executable configuration still refuses. Final automatic subscription removal changes the executable set and requires refresh or pre-start abort, explicitly documented. Add any planned mediator cron wake-up **before** resolving/approving operator.json. No caller/boot/plugin/SDK edits between resolution and freeze; arm checks again. No in-progress manifest edits.
4. **prev contract.** Every offline-swapped plugin's prev is now its exact **frozen pre-cut current**, under before/after guards and restore. Tests checked all nine, including paths where an older prev existed. Git waits for normal later activation/prev behavior. This makes the fallback image meaningful, **not** a license for blind incompatible individual `core.revert`; coordinated rollback and unsubscribe-before-session-revert remain mandatory.
5. **Roots side effect.** Listed above and in the live runbook/approval plan, rather than describing roots queries as mutation-free. Subsequent tests stayed private.
6. **Arm race.** Replaced reader-dependent FIFO with atomic, manifest-bound **arm.json** and an arm-then-scan inotify watch. Idempotent arm returns with no reader. `resume` accepts waiting_for_arm; a kill after marker observation but before saving armed reuses the same marker. No manual FIFO writer/cleanup is needed.
## Executed gates and full logs
Prefix: `/logs/box/2026-10-31/tc-asa3yr-exec-`.
| Gate | Log / outcome |
|---|---|
| Final exact tool, caller killed + controller killed/resumed, incomplete backup resume, real Herdr job-control stop/swap/start, same shell, post-start preservation/new verifier, prev hashes, pinned cold restart and old clone | **3699755631.stderr**, pass. Final backup 4,436 files / 627,157,113 bytes, 4.559 s (session-size integration, not full-world downtime). No old mail owner; one manager; cred locked. |
| Live special-file/zero-space negative, mutable cron vs changed payload, pre-arm drift, frozen SQL mismatch -> real-driver abort/old relaunch, consumed-arm/save kill/resume, post-start abort refusal | **1485773773.stderr**, pass. Zero-space gate used a test-only mocked disk-usage response; actual ENOSPC tested below. |
| Real-driver partial swap -> abort/restore/old relaunch; actual backup ENOSPC on a private tmpfs introduced after good preflight -> untouched-old relaunch; lost console subscription -> bounded clean exit | **4142682240.stderr**, pass. Registration wait 120 s; elapsed 132.493 s including backup/launch. Started checkpoint refused both resume and abort. |
| Real old/new Git + new cred, loopback synthetic OpenSSH/HTTP negative, unlock then deferred activation | **99781693.stderr**, pass. No live key/grant/remote used. |
| SQLite WAL + genuinely hot rollback journal, exact losses, preserved job IDs/calendar/outcomes/generic JSON/uncertain mail, apply interruption/rerun and exact physical restore | **1727732043.stderr**, pass. |
| Board executable projections and old-select-star negative control; template/deferred-artifact checks; final driver's actual Herdr failure notification + abort | **1276236846.stderr**, pass. |
| `cargo test --workspace` against unchanged production source | **2259546796.stderr**, pass: **208 host integration tests, lines 62–431**, plus all workspace suites/doc tests. 133 Rust/Cargo/flake inputs matched card tree. No Rust implementation change. |
| Executed verify mode, complete source-as-log and deletion | **2013747101.stderr**, pass. See hashes/ranges below. |
Prior accepted race/replay/deadline/core-watch/upgrade/idle/grace/crash-supervision evidence is cumulative, not silently rerun or claimed fixed here. Prior live-size scale sample remains **138,444 files / 12,441,162,164 bytes / 45,310 directories**, synthetic size-bucket match, 205.940 s inventory+copy/hash, warm cache. Post-record manager ack 0.6078 s, registration catch-up 0.0128 s, mail publication 0.5376 s (integrated 0.5235 s) remain under caller publication budgets **3 s orchestrator / 10 s cred**; catch-up outlier 12.0419 s and initial-verifier startup scan delay 23.753/24.530 s remain disclosed. No production downtime/latency guarantee.
## Deleted executable and reconstruction receipt
Only source-as-log remains in **2013747101.stderr**. Reconstruct exactly three files for the later approved operation, hash-check before use; do not reconstruct test controllers:
- cutover_once.py: lines **2–164**, SHA-256 **3adf65ebb0b3401a0d1976524f35f7450e907ad32f907dcbb1cd2f7e0a7e5d3a**;
- freeze.py: lines **166–238**, SHA-256 **d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe**;
- driver.py: lines **240–503**, SHA-256 **686167318c84a32c1010daf4517f036bcb654384cadab987fa47fb13deecd2bf**.
`verify` actually logged/deleted the executable after successful rehearsal. Executed test/patch/controllers and test launchers were logged/deleted too, with bytecode caches removed. `tool-deletion.json` and `executed-test-deletion.json` are under `evidence/round3-1389/`. Permanent pinned owner/selector configuration is not migration code; it stays rooted through the recovery window. The live runbook still requires mediator deletion after verified success.
## Failures and limits
Tried next methods, not blockers: initial generated SQL fixture had quoting syntax wrong (corrected); generated lockfiles made whole-directory diff reject (compared actual Rust/Cargo/flake input bytes, then ran full suite); final-sidecar extraction included a trailing log status (trimmed); a post-check initially read the previous round's stopped-world manifest (corrected to this round, then passed). Synthetic SSH setup needed explicit typed key metadata, a sandbox-only sshd account and an unlocked sandbox login account; default root was locked. A failed fixture left two loopback sshd listeners, later explicitly identified/stopped. Initial sandbox activation used `path`; core required `wasm`, corrected and the complete final Git gate passed. No failed test was relabeled successful; full errors remain in logs.
Still unverified/out of scope: approved live cut-over/frozen caller snapshot, actual live SSH-detach/pane w7:p6 execution, Surma's real unlock/new-git live networking, physical Herdr notification/browser display, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. No live activation/restart/config/owner change, push, release, deploy or /www publication this run. Surma's go is still required before mediator core/session/cred operation, detached SSH exception, roots side effects and **every push**.