Rehearse detached cutover driver and checkpoint-safe operator runbook
Card: SAU-384
This commit is contained in:
SAU-384 implementer
@@ -6,6 +6,7 @@ Authority: [/docs/design/plugin-handover/design.md](/docs/design/plugin-handover
|
||||
|
||||
## Closed integration gaps
|
||||
|
||||
- **Review rework, run 1381:** one detached external CLI now owns freeze→backup/plan/audit→apply→same-Herdr-pane launch and publishes a new explicit mediator verification message. Caller/controller death and interrupted planning resume passed. Plan/apply check the actual current-boot frozen checkpoint; old/target GC roots and permanent pinned owner wrappers, stopped-PTY TERM+CONT, SQLite sidecars and whole-world scale were tested. Tool/controllers were logged/deleted after verification; current workspace tests passed again. See [rework-1381.md](rework-1381.md) for exact evidence, limitations and disclosure of an early shared-Nix stale-root cleanup (all referenced targets/PIDs were absent on the host; subsequent tests used private Nix state). [Runbook](runbook.md) now assigns all downtime work to that external controller, not the cut old mediator. Live launch/release still require the explicit operator approval described there.
|
||||
- **Run 1373:** after mediator installed the observed public host key, approved read-only SSH inspection identified the actual Herdr pane, outer `sausage-loop`, world path, native architecture, old package/derivation/source and runtime closure. The actual old Nix package/loop → stop-all → offline swap → exact new Nix release package/loop passed in the disposable live-size world. A non-parent external controller separately proved loop-first stop, interruption/resumption, stale-PID refusal and safe frozen rerun. A real core.restart after loop exit could not respawn the host. See [native evidence](resume-1373.md).
|
||||
- **Run 1369:** armed/exit-race delay-gate tests passed on real core with actual target WASM; replacement orchestrator readiness/one-owner/all-watch arming passed. Hooks existed only in a disposable host copy, never the selected host artifact or committed code.
|
||||
- **Rollback:** actual old-manager/gateway tests prove **unsubscribe works; ID-only re-key does not**. The old binary ignores best_effort and replays history to a renamed ID. The runbook mandates the card-authorized unsubscribe option and prevents web re-registration during a session revert. A physical browser/SSE display is not claimed.
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# SAU-384 review rework — run 1381
|
||||
|
||||
**Ready for review, not live execution.** This changes only local documentation and disposable one-off tooling. No shipped host/plugin behavior, live boot/SDK/launcher, activation, restart, publication, release or push changed. Surma's go still precedes the mediator's live operation and every push.
|
||||
|
||||
## Review findings addressed
|
||||
|
||||
1. **No mediator needed during downtime.** Replaced the disconnected procedures with one external CLI, `driver.py start → arm → detached run → verify`. `start` acknowledges a new OS session and blocks it on an arm FIFO. After arm the detached driver owns native freeze, immutable/resumable full backup, planning/frozen audit, offline apply, selectors, `started`, launch and readiness. It uses the exact pinned Herdr pane/shell, then publishes a **new explicit default verification message**, with a stable UUID, because the cut old mediator run is not recoverable. The last real-Core/Herdr test killed the invoking shell shim, then killed the detached controller at frozen, resumed it, and reached target in the same shell. Actual target run-start and mailbox-consumed records named that new verification message once. A real live SSH launch was not performed; the runbook makes the two finite foreground SSH launch/arm commands, and the required explicit go for that exception to SSH's background-work policy, part of the proposed operator plan.
|
||||
2. **Actual frozen-state gates.** `Kit.plan`, apply, starting and pre-start rollback require the real `freeze.json`, phase frozen, current boot ID, matching world, dead pinned owners, empty old group, no subsequently running host, and unchanged checkpoint bytes. A caller-supplied stopped literal is gone. Captured/wrong-boot checkpoints, running owner, changed config/SQL projection and after-start resume refuse before mutation. Offline retries accept only their saved before/after images.
|
||||
3. **Persistent installation pins.** Driver registers/verifies GC roots for exact old/target/loop packages and creates old/target selector and owner artifacts before freeze, so pre-start recovery already has the explicit old owner. Both wrappers export their own pinned selector. Cold restart in the same Herdr shell, without ambient SAUSAGE_FLAKE, ran the exact target; the explicit old wrapper ran the exact old package on a separate old-backup clone, never over new target work. Runbook prohibits bare loop/remote-default starts and explains that core.restart restarts the pin, not self-updates. Future host updates need an approved package/root/selector change; roots are not removed with migration scripts.
|
||||
4. **Real PTY job control.** Used actual Herdr 0.9.0 and an interactive zsh job-control shell, not setsid+piped stdin alone. Loop was foreground process-group leader in the shell's session. Freeze sends TERM then CONT before pidfd wait. Separate injection left the real host in observed **T** state after outer-owner exit; TERM+CONT completed shutdown with no surviving group member. This exercises the stopped-state hazard; it does not claim every timing of naturally delivered SIGTTIN.
|
||||
5. **Planning, SQLite and scale.** Incomplete backup resumes the same inventory/backup without cleanup or replacement. Streaming hashlib.file_digest avoids whole-file memory reads; os.walk/copy preserve directories and non-followed symlinks (including broken links). Complete DB sidecar families are backed up physically, recovered only in disposable copies, staged with SQLite backup in DELETE mode, and old sidecars removed before main-image replacement. Committed WAL and hot uncommitted rollback-journal fixtures passed interruption/rerun, exact losses, preserved job IDs/calendar/outcomes/generic JSON/unknown mail, and exact pre-start rollback. A full logical-size/count/bucket/directory-matched **synthetic** replica measured the whole-world backup path below; it was not an exact live-byte snapshot.
|
||||
|
||||
## Evidence and exact gates
|
||||
|
||||
Evidence directory: `/docs/symphony/artifacts/SAU-384/evidence/rework-1381/`. It contains only JSON manifests/inventories/checkpoints/results, not migration executables. `sandbox-operator-manifest.json` records the tested invocation; **do not execute it live**. `live-manifest.template.json` deliberately requires fresh identities, typed caller review/guards and a new stable continuation UUID before approval. Frozen guard tests used the real cron table projection; the live table/file set must include the board/config/registration review described in the runbook.
|
||||
|
||||
All logs below are `/logs/box/2026-10-31/`:
|
||||
|
||||
| Gate | Full stderr log / result |
|
||||
|---|---|
|
||||
| Final caller-killed + controller-killed/resumed + interrupted planning + same-Herdr-pane target chain | `tc-asa3yr-exec-775822946.stderr` — pass. 4,436 frozen regular files / 627,157,113 bytes in this session-size integration fixture; backup/hash stage 4.617 s. Earlier final chains also passed (`…-2599916298`, `…-2393656781`, `…-2876439273`). |
|
||||
| Final process ownership/locked cred/new explicit verifier/preservation; stopped-PTY CONT; cold pinned target; explicit old package on old clone | `tc-asa3yr-exec-3942858286.stderr` — pass. `post-target-processes.json`, `post-preservation.json`, `post-continuation-records.json`, `post-pty-stop-continue.json`; only planned paths and enumerated new activity changed. |
|
||||
| WAL + hot journal + interruption + exact rollback, repeated | `tc-asa3yr-exec-1595763713.stderr` — pass; first pass `…-1628589419`. `db-sidecars.json`. |
|
||||
| Whole-world scale | `tc-asa3yr-exec-3620999999.stderr` — **138,444 files, 12,441,162,164 bytes, 45,310 directories**. Inventory **70.555 s**, copy plus double hash verification **135.385 s**, combined **205.940 s**. Exact functions used by Kit.plan; synthetic bytes/names, observed live size buckets/counts, warm-cache box. No cold-storage downtime SLA. |
|
||||
| Current source equivalence + cargo test --workspace | `tc-asa3yr-exec-430706202.stderr` — pass; 121 Rust/Cargo inputs plus root Cargo/flake inputs matched current card tree. **208 host integration tests**, lines **64–431**, plus all workspace suites/doc tests. No Rust source changed or fresh plugin build needed. |
|
||||
| Negative config/frozen caller controls, selected bundle rehash, verify and deletion | `tc-asa3yr-exec-3977808892.stderr` — pass; `negative-controls.json`, `bundle-recheck.json`, `tool-deletion.json`, `executed-test-deletion.json`. |
|
||||
| Source-as-log reconstruction/hash/AST check; no scripts/bytecode recreated | `tc-asa3yr-exec-3884064136.stderr` — pass; `source-log-roundtrip.json`. |
|
||||
|
||||
### Reconstructable, but deleted after success
|
||||
|
||||
The executed tool was three small files, not a plugin or host patch. Exact source survives **as log only** in `tc-asa3yr-exec-3977808892.stderr`:
|
||||
|
||||
- `cutover_once.py`: lines **7–152**, SHA-256 `202eb7e635d675bb3960317913ae154e4f699fb7f5fe25a53ef34c965d0869c5`;
|
||||
- `freeze.py`: lines **154–226**, SHA-256 `d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe`;
|
||||
- `driver.py`: lines **228–393**, SHA-256 `7a26cfba9873541404f9661d5b0117f566301d408448cbb509a965d7900037aa`.
|
||||
|
||||
The verify CLI was actually executed, marked the rehearsal verified, logged/deleted these files, and removed caches. All executed test controllers were also logged/deleted; the named disposable Herdr server was stopped. Hash reconstruction was checked **in memory**, without recreating executable files. The live runbook mandates the same deletion. Permanent owner/selector artifacts are installation artifacts, represented as bytes/hashes in `permanent-launcher-artifacts.json`, not retained migration code.
|
||||
|
||||
## Selected builds unchanged
|
||||
|
||||
Selected bundle remains `/docs/symphony/artifacts/SAU-384/cutover-targets.tar.gz`, SHA-256 `1bbcae4ef615ca8394b0c53b9a408441a49f8a618f059335c3bc248d958977c4`; all **22** payload hashes matched again. Host `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`; SDK-copy **src/lib.rs only**. Mail/cred/cron/rooms builds **1600/1601/1602/1603** unchanged. Web artifact remains approved **3d760267**, held until target session/web verification and mediator/orchestrator release. No artifact repack or source change to a shipped plugin.
|
||||
|
||||
## Failures, corrections and limits
|
||||
|
||||
- Whole-tree transfer through box.put_tree rejected the 64 MiB service cap for repos/plugins/logs. Next method: read-only host size histogram, then regular-file size-bucket replica and the exact backup routines. This measures whole-world **scale**, not original live contents/path topology. The original accepted session byte snapshot still supplies the real-history integration volume.
|
||||
- Initial PTY fixtures lacked state/core for console history, then matched an http-prefixed address against a bare address. Corrected fixture and barrier. Typeahead from two early console commands lost the second; final driver queues **one** subscription command. An early Herdr multiline output matcher stalled; inspection showed Herdr wait-output itself polls, so final code removed it entirely in favor of inotify completion. The actual registration path is `state/session/manager/registrations/subscriptions.json`, now corrected.
|
||||
- **Shared-store foot gun:** an early default-store root query pruned **1,144 stale auto-root links and 571 stale temporary-root files** in shared Nix state. This was not an authorized live-config operation and no GC/package deletion was requested. Immediately stopped that method. Read-only SSH inspection on Citadel found **none** of those referenced targets and **none** of those PID numbers existing on the host, so no active roots required restoration. All subsequent root tests used a **private local Nix state database**, loaded from read-only closure metadata; no shared root query was repeated. Exact evidence/log: `gc-root-incident.json`, `tc-asa3yr-exec-2216542428.stderr`. This incidental stale-registration cleanup is disclosed rather than claiming literally zero host-state writes.
|
||||
- One post-start assertion read history before the new verification message was consumed. Replaced it with arm-then-scan/history-write completion. In the final gate publication→run-start was **23.753 s**, →consumption **24.530 s**, queued behind startup reconciliation. This is **not** mail.send publication latency. Initial acknowledgement is not full scan completion.
|
||||
|
||||
Prior accepted replay/race/rollback/deadline/grace/crash-supervision/event-driven orchestrator evidence remains unchanged. Post-record manager ack **0.6078 s**, catch-up **0.0128 s**, mail publication **0.5376 s** / integrated **0.5235 s** remain below existing **3 s / 10 s** publication budgets; integrated catch-up outlier **12.0419 s**, ~0.10% idle duty, detached error propagation and **305.427 s** real cron warning/stop remain disclosed in implementation.md. This rework does not replace those measurements with a claim that the initial scan is fast.
|
||||
|
||||
Still unverified/out of scope: actual approved live cut-over/frozen backup and complete production caller guards, live SSH detach/Herdr w7:p6 execution, orchestrator.release/publication/physical browser, cold-storage timing, power loss, provider cancellation/billing and independent recovery follow-ups. The live launch exception needs explicit Surma go as part of the proposed plan; no such approval is claimed or requested by executing anything here.
|
||||
+114
-58
@@ -1,47 +1,91 @@
|
||||
# One-off handover operator runbook — rehearsed, approval required
|
||||
# One-off handover runbook — rehearsed; live approval required
|
||||
|
||||
Read [rehearsal/results](implementation.md), [race/rollback verification](resume-1369.md), [native owner/package rehearsal](resume-1373.md), the [authority](/docs/design/plugin-handover/design.md) and [approval gates](/docs/symphony/artifacts/SAU-356/plan.md). This is **not** authorization to execute. Workers must not perform these live steps. Mediator/orchestrator alone lands and executes.
|
||||
Authority: [design / Rollout](/docs/design/plugin-handover/design.md), [SAU-356 gates](/docs/symphony/artifacts/SAU-356/plan.md). Results: [initial](implementation.md), [races/rollback](resume-1369.md), [native package](resume-1373.md), [external driver rework](rework-1381.md). This is a proposed live plan, **not authorization**. Workers never execute it. Mediator/orchestrator lands and executes only after Surma's go; every push separately needs a fresh go.
|
||||
|
||||
## 0. Verify and freeze the installation manifest (read-only first)
|
||||
## 0. Prepare everything while the old world is still available
|
||||
|
||||
Evidence root: `/docs/symphony/artifacts/SAU-384/evidence/`.
|
||||
Selected coherent bundle: `/docs/symphony/artifacts/SAU-384/cutover-targets.tar.gz`.
|
||||
The older `target-artifacts.tar.gz` remains prior rehearsal evidence, **not** the native installation selection.
|
||||
Do not begin downtime with scripts, caller decisions or launch instructions still to be assembled.
|
||||
|
||||
- Verify the new bundle hash against `evidence/cutover-targets-sha256.txt`, then every extracted file against `evidence/manifest.json`. It contains reviewed plugins/static assets, SDK `src/lib.rs`, and `native-package.nar.gz` with its closure manifest, **not** a migration executable. It excludes the old diagnostic debug host.
|
||||
- Native selection is the Nix release package `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, aarch64-linux, built from this card's tree with unchanged host source `e09c4f9f`. `evidence/native-host-manifest.json` pins its NAR/file hashes and every runtime closure path/hash. All dependency paths match the observed old package's closure; only the host root differs. Recheck closure availability/hashes before execution. Mediator may import the exact reviewed NAR if needed (`gzip -dc native-package.nar.gz | nix-store --import`); never substitute the debug binary. Import/deployment is not a worker action.
|
||||
- Owner is **Herdr workspace w7, pane w7:p6 (tab w7:t5)** running `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`, not a systemd Sausage unit. Re-identify its current loop/host PIDs using read-only `herdr pane process-info --pane w7:p6`; observed PIDs in `resume-1373-inspection.json` are evidence, never reusable constants. The loop refreshes a remote flake after any nonzero exit, including core.restart (75). Its existing 2-second restart sleep is not a migration quiescence mechanism.
|
||||
- Pin all old plugin hashes, full boot bytes, current SDK source hashes, caller configuration, and `/proc/*/info.json` inventory. Record active agent/tool processes and any external effects whose result is uncertain. Do not infer settlement from the last assistant text.
|
||||
- Inventory executable caller payloads using predecessor schemas, not substring rewriting. Run 1373 read-only audit covered cron jobs; board hooks, invariant target, attempt snapshots, subscriptions/batches; room/workflow events; orchestrator runs, skill reviews and host jobs; manager registrations and configuration. `evidence/resume-1373-inspection.json` records six legacy cron templates requiring conversion, an already-target-valid board hook **and invariant target**, eight historical running hook snapshots that must NOT replay, zero subscription batches, one abandoned room with no pending work, and no live session manager registrations. The only observed conversion beyond boot is cron `jobs.data` where `method == session.send`. Repeat against the frozen bytes: this observation is not a transactional final snapshot. Prompt strings/diagnostics are not executable templates; preserve them verbatim. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
|
||||
- Source references for predecessor formats: session `f9122585`, `src/main.rs` Paths (`pending.jsonl`, `cancel`) and `src/queue.rs`; cron `ba75c102`; target cron README's one-off conversion procedure. Active old component hashes are separately captured; do not equate a source reference with its compiled active binary without checking.
|
||||
- Confirm the frontend artifact is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`; keep `publish:hold` until step 5. Do not publish a worktree snapshot or release it early.
|
||||
- Rollback uses the card's explicit **unsubscribe** option. ID-only re-key is **unsafe**, proven again with the actual old manager and gateway in run 1369. Prevent the gateway from re-registering `web-chat` against an old session binary; never call `web.start` in that state. A different proposed “re-key” needs its own defined semantics/rehearsal.
|
||||
### Reviewed selection
|
||||
|
||||
## 1. Obtain approvals, then close ingress/ownership
|
||||
- Bundle: `/docs/symphony/artifacts/SAU-384/cutover-targets.tar.gz`, SHA-256 `1bbcae4ef615ca8394b0c53b9a408441a49f8a618f059335c3bc248d958977c4`. Verify all 22 payloads against its embedded `manifest.json`. `target-artifacts.tar.gz` is older debug-host evidence, **not** this installation.
|
||||
- Native host: `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, aarch64-linux Nix release package from unchanged host source `e09c4f9f`. The embedded `native-host-manifest.json` pins the NAR, binary and complete runtime closure. Import the reviewed NAR if needed; never overwrite immutable store files or substitute a diagnostic host. Import/deployment is an operator action, never a worker action.
|
||||
- Old package for this observed baseline: `/nix/store/mggvibk8lcgv2qw28jimrl60mi0qm41p-sausage-0.1.0`. Re-identify it and the owner before go; observed PIDs are not constants.
|
||||
- Actual owner: Herdr **w7:p6**, tab w7:t5, not a systemd Sausage unit. Its loop is `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`. Read `herdr pane process-info --pane w7:p6` once to resolve current shell/loop/host identities. Do not close Herdr or unrelated panes.
|
||||
- SDK copy list is **`src/lib.rs` only**. Preserve world-owned Cargo manifests, relocated `interface`, imports and all other SDK bytes. Resolve a changed baseline before proceeding.
|
||||
- Web is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f` (SAU-375), held by `publish:hold`. Do not release it before target session and web are active.
|
||||
|
||||
Surma's go must precede the live core/session/cred operation. Every push requires its own fresh go. A successful rehearsal grants neither. Record the approved manifest and any accepted first-update cuts before stopping anything.
|
||||
### External driver and resolved operator manifest
|
||||
|
||||
Run the one-off controller **outside the world and outside the service/process group being stopped**. A plugin or agent in this world cannot be the controller: stop-all must stop it too. Recreate the narrowly reviewed temporary executable only for this approved operation, from the logged implementation with the resolved manifest; never add compatibility/transition code to a plugin.
|
||||
Use external root `/home/surma/sau-384-cutover`, outside the world and stopped process group. Stage the extracted bundle under `targets/`. Reconstruct **only** `cutover_once.py`, `freeze.py`, and `driver.py` under `tool/` from the source-as-log headings in `/logs/box/2026-10-31/tc-asa3yr-exec-3977808892.stderr`; check their hashes against `evidence/rework-1381/tool-deletion.json`. These are the executed/reviewed versions, not the earlier scripts from run 1373. They have a real CLI entry point. Do not reconstruct rehearsal/test controllers.
|
||||
|
||||
Close ingress before stopping ownership. The tested exact native boundary is the deleted one-off `native_freeze.py` (source-as-log linked in resume-1373): invoke its reconstructed, reviewed copy **outside the world/group**, with current loop PID, current host PID and an external checkpoint path. Use external root `/home/surma/sau-384-cutover`, with the reconstructed scripts in `tool/` and checkpoint `evidence/freeze.json`. The observed Python interpreter is `/nix/store/33i3kijbra0cffncn3rmi9bdrav7a3y3-python3-3.13.15-env/bin/python3`. After Surma's go and fresh PID/identity inventory, the exact stop invocation is `PYTHONDONTWRITEBYTECODE=1 <that-python> tool/native_freeze.py "$LOOP_PID" "$HOST_PID" evidence/freeze.json` from the external root. Do not pass the rehearsal-only failure-injection argument. It pins boot ID/start ticks/executable/group and pidfds, sends SIGTERM to the **loop first**, awaits loop exit, checkpoints `owner_stopped`, then sends SIGTERM to the exact host and awaits its exit. Host SIGTERM awaits real `Host.stop`; its old source was inspected at the actual Nix derivation. After both exits, inspect the saved group once; any unexpected survivor blocks without blindly killing it. No process polling, endpoint retry, sleep or service-stop-request counting. Do not close the Herdr server or unrelated panes.
|
||||
The tested interpreter is `/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3`, with **inotify-simple 2.0.1**. Ensure its closure is available before go. Pin its package and the inspected Herdr package with host-visible GC roots for the operation. The driver itself registers and verifies **old host, target host and loop** roots at `<root>/gc-{old,target,loop}` before freeze. Use the **host namespace**, not a box talking to shared Nix state: `/rework` roots in a box are not host-visible. Do not run GC as a test. Keep old/target/loop roots and both selectors for recovery after tool deletion.
|
||||
|
||||
The non-parent-controller rehearsal proves loop exit does not kill the migration controller or old host prematurely; the host survives its parent and drains on its own signal. Interrupting after loop exit is resumable from the same pinned checkpoint; stale PID identities refuse. Even a real core.restart after loop exit cannot respawn the host. `core.restart` by itself is **not** stop-all. Core's shutdown suppresses service restart admission and stops all trees in parallel. Design budgets remain grace 5 seconds and separate EOF-stop 5 seconds; the controller awaits completion rather than inventing a live timeout. Never edit world state before the `frozen` checkpoint.
|
||||
Start from `evidence/rework-1381/live-manifest.template.json`, a **non-executable template**, and resolve every placeholder before approval:
|
||||
|
||||
## 2. Back up frozen state and plan changes
|
||||
- native paths, Herdr binary/context, pane and current shell/loop/host PIDs;
|
||||
- external root, world, bundle, embedded manifest SHA, artifact paths, SDK path, exact target boot list;
|
||||
- complete old in-flight `/proc/*/info.json` and native-owner inventory, plus uncertain external effects; a finished-looking assistant tail is not settlement;
|
||||
- a new continuation UUID, generated **once**, and the explicit new mediator verification instruction. Keep that UUID on all retries;
|
||||
- typed executable-caller review and its frozen guards below.
|
||||
|
||||
Only after shutdown completion:
|
||||
Do not copy sandbox PIDs, `gc_store`, or `test_gate_after_freeze` into a live manifest. The template deliberately omits those test-only fields. Record/hash the fully resolved `operator.json`; driver checkpoints bind to its exact bytes. Resolve a schema/selection discrepancy while the old world is available, not during downtime.
|
||||
|
||||
1. Make a complete frozen backup outside the target world's mutable tree, including plugin/previous binaries, boot/caller config, SDK, session histories/settings, mail, cron, cred/grants and journals. Preserve any SQLite sidecars; do not rename a live DB/WAL trio. Inventory paths/sizes/hashes and copy source hashes; retain removed rows/files explicitly. Record the old in-flight work inventory alongside it.
|
||||
2. Produce a manifest of each proposed change with before/after hashes and staged target bytes. Never regenerate the backup over an earlier backup on retry. Unknown file images refuse rather than overwrite. The tested file replacements sync the temporary file, rename and sync its parent directory.
|
||||
3. Select only the approved queue losses: old session `pending.jsonl` and `cancel`, plus unclaimed pending session delivery rows (`state='pending'`, recipient not `surma`) in `/state/mail/mail.db`. Back them up before removal. Preserve original messages, Surma copies, done rows and uncertain claimed `unknown` rows as diagnostics; the target has no deliverer to replay them. The tested cut does not infer an unknown delivery was safe to repeat.
|
||||
4. Convert each old cron `session.send` template once: require a JSON object/nonempty text; refuse `interrupt:true`; remove false/absent legacy `interrupt`, inner `detach`, `notify`, permanent `id`; set absent kind to `follow-up`; preserve all other template metadata/settings verbatim; set outer detach false. Preserve SQL IDs and every schedule/last-run/outcome field. Update all affected jobs in one `BEGIN IMMEDIATE` transaction, validate readback/integrity, and preserve non-session job JSON bytes. Record removed notify targets for review—do not replace them with an invented answer-forwarder.
|
||||
5. Stage target host and all affected plugin components from the resolved manifest as one offline cut. Do not call live `core.activate` for piecemeal contract upgrades. Unaffected plugin state/binaries stay intact. Copy only SDK `src/lib.rs` for this selected baseline; preserve relocated world Cargo manifests, imports and `interface`. If hashes differ from this baseline, resolve the coupled source-copy set rather than applying this one-file assumption blindly.
|
||||
### Executable caller review and frozen guards
|
||||
|
||||
Native host installation is an **outer launcher selection**, never an overwrite of immutable `/nix/store`. While stopped, install the static selector flake recorded in the manifest at `/home/surma/sau-384-cutover/target-selector/flake.nix` (outside the world). The proposed parent was absent at read-only inspection; refuse an unexpected existing image rather than overwrite it. Its sole app points to the exact target package's `bin/sausage`. Hash-guard and back up any existing selector; do not overwrite an unknown image. The selector is target launcher configuration, not a migration executable. The actual old Nix package/loop → offline plugin/SDK/boot swap → new Nix package/loop was rehearsed; neither public remote refresh nor a source-only host was substituted.
|
||||
Use predecessor formats: session source `f9122585` Paths/queue; cron `ba75c102` and current cron README one-off procedure. Do not substring-rewrite prompts or diagnostics. Review current stores as in `resume-1373.md`, then encode the **approved** file hashes and deterministic SQL projections in `caller_file_guards` and `caller_sql_guards`:
|
||||
|
||||
## 3. Install boot/caller configuration while stopped
|
||||
- cron `state/cron/jobs.sqlite`: `select id,data from jobs order by id`;
|
||||
- board `state/board/board.sqlite`: deterministic projections of `hooks`, `invariant_target`, `subscriptions`, and `subscription_batches`, using the actual current schema. Review hook/invariant templates. Historical hook-attempt snapshots remain diagnostics and **must not replay**;
|
||||
- boot and other discovered executable configuration/manager registrations. Review typed room/workflow/orchestrator stores too; preserve their histories/settings and non-executable records. Do not mistake prompt text for an RPC template.
|
||||
|
||||
Write the exact reviewed target boot list (JSON is valid YAML):
|
||||
A SQL guard hashes `sha256(json.dumps(rows,separators=(',',':')).encode())`, where `rows = connection.execute(sql).fetchall()` in the approved ordered projection. JSON strings inside rows retain their exact bytes. The observed live baseline had **six cron templates to convert**, already-target-valid board hook/invariant calls, no subscription batches and no live manager registrations. This is an observation, not a future frozen snapshot. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
|
||||
|
||||
After freeze the driver backs up all bytes, checks these projections on **disposable complete DB copies**, and verifies file guards against the frozen backup **before apply**. It never recovers/opens immutable backup evidence in SQLite. A changed projection stops offline with evidence; there is no in-world human gate after stop-all. Include any newly discovered executable store in the approved manifest before arm.
|
||||
|
||||
### Approval and external launch surface
|
||||
|
||||
Surma's go must explicitly cover this resolved manifest, the stop-all core/session/cred cut, the detached external controller launch/arm, pinned launcher installation and the post-start verification/release/deletion plan. This card's outside-world-controller scope requires a controller that survives the mediator; it does **not** authorize executing it now.
|
||||
|
||||
The proposed concrete execution surface is **two finite foreground `ssh.exec` calls as surma on citadel.vpn.surma.technology**, using the existing approved signing grant, with commands `cd /home/surma/sau-384-cutover && <the start command below>` and then the same prefix with `<the arm command below>`. Each command terminates; the reviewed native CLI, not a shell `&`, creates the detached OS child. **Ask Surma's go to explicitly authorize this planned exception to the ssh-exec skill's usual prohibition on creating background work through exec strings. Until that explicit authorization, do not launch.** Do not improvise a tunnel, SSH PTY, new worker session or an in-world controller. This is a fully prepared proposed live operation, not a claim that the current worker has permission or has tested a live SSH launch. The same finite foreground surface executes the final verify/delete command after target verification. After arm, loss of the old mediator/SSH RPC result is expected and does not terminate the already-detached controller; inspect the external checkpoint instead of retrying start.
|
||||
|
||||
## 1. Start, acknowledge, then arm the external controller
|
||||
|
||||
Close ingress and record accepted first-update cuts before arm. From the external root, on the approved host execution surface, mediator executes:
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py start operator.json
|
||||
```
|
||||
|
||||
`start` launches a new OS session with stdin closed and output in `<root>/driver.log`, records its PID/start identity, returns an acknowledgement, and leaves the controller blocked on an **arm FIFO**. Check that receipt and checkpoint once, then:
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py arm operator.json
|
||||
```
|
||||
|
||||
From here **the external driver, not the old mediator**, owns steps 2–4. The old default run and its tool calls are deliberately cut. They are not recovered from legacy history. The driver uses an external advisory lock to exclude another controller. Rehearsal killed both the invoking caller shim and, separately, the detached controller after freeze; checkpoint resume completed the chain.
|
||||
|
||||
## 2. Driver: freeze → full backup → plan/audit → offline apply
|
||||
|
||||
The driver verifies reviewed bundle/plugin/SDK bytes and native closure hashes, registers real GC roots, and creates hash-guarded **old and target selector/owner artifacts before freeze**. It then:
|
||||
|
||||
1. Pins boot ID, start ticks, executable and group for both native processes; confirms controller is outside their group. Signals the **outer loop first** and awaits its pidfd exit. Saves `owner_stopped`.
|
||||
2. Signals the exact host **SIGTERM then SIGCONT**, so a job stopped by background-PTY SIGTTIN can handle termination, and awaits host exit. Checks the saved group once for unexpected survivors; never blindly kills them. Saves `freeze.json` phase **`frozen`**, world path, boot ID and empty group inventory. No process/endpoint polling, sleeps or stop-request counting. Host shutdown stops all plugin/agent trees and suppresses restart admission. `core.restart` alone is not stop-all; after the loop is gone, exit 75 cannot respawn it.
|
||||
3. Makes the complete frozen backup outside the world. Streaming SHA-256 does not read whole files into memory. Directory inventory and copy preserve empty directories and symlinks **without following links**. An interrupted plan resumes the same immutable input inventory and partially completed backup; it neither overwrites an earlier backup nor requires deleting the plan directory. Unknown images refuse.
|
||||
4. Stages the approved transformations, then checks the frozen caller guards. **Both plan and apply independently require the actual frozen checkpoint**, same world/current boot, dead pinned owners, empty old group and no subsequently running host for this world. A caller-supplied `{remaining:0}` literal is not accepted. Changed checkpoint bytes also refuse.
|
||||
5. Applies hash-guarded before/after operations offline. File replacements fsync temporary bytes, rename, and fsync the parent. A third image refuses. `migration/plan.json` reaches `installed` only after all operations complete.
|
||||
|
||||
Exact logical losses are **only** old session `pending.jsonl`, `cancel`, and unclaimed `mail.deliveries` rows with `state='pending'` and recipient not `surma`. Original messages, Surma copies, done rows and uncertain `unknown` deliveries stay; do not replay them. Loss records and original bytes/rows remain in backup/evidence. Histories/settings, scheduled SQL IDs/calendar/outcome fields, grants/requests and unrelated files remain intact.
|
||||
|
||||
Cron conversion: require an object and nonempty text, refuse `interrupt:true`, remove false/absent legacy interrupt, inner detach/notify/permanent ID, set absent kind `follow-up`, outer detach false. Preserve other fields and non-session JSON bytes. Review removed notify destinations; no invented answer-forwarder.
|
||||
|
||||
**SQLite sidecars:** copy the complete DB plus `-journal/-wal/-shm` family into immutable backup first. Recover only another disposable copy; use SQLite's backup API to stage a consistent DELETE-journal image. Plan removal of old sidecars **before** replacing the main DB, retaining the original physical family. This is normalization, not additional logical data loss. Tests covered committed WAL data, a genuinely hot uncommitted rollback journal, interruption and exact pre-start restore.
|
||||
|
||||
Full-world backup is not instantaneous: the 138,444-file / 12,441,162,164-byte / 45,310-directory **size-bucket-matched synthetic** replica took **70.55 s inventory + 135.39 s copy/hash verification = 205.94 s** in the box. This includes repos/plugins/logs, unlike the earlier session-sized gate. It is warm-cache scale evidence, not an exact live-byte or cold-storage downtime guarantee. Allow that uncertainty in the approved maintenance plan; do not invent a timeout and kill the controller mid-copy. Power-loss recovery is not claimed.
|
||||
|
||||
## 3. Driver: ordered boot installation and pinned owner selection
|
||||
|
||||
The offline target boot list is:
|
||||
|
||||
```json
|
||||
[
|
||||
@@ -53,44 +97,56 @@ Write the exact reviewed target boot list (JSON is valid YAML):
|
||||
]
|
||||
```
|
||||
|
||||
This starts the new session manager before web, so `web-chat` becomes best-effort against the new contract. `mail.start` is removed; no session deliverer remains. Check any additional actual boot entries against the manifest rather than deleting them accidentally. Install the previously inventoried caller-config conversions at the same boundary. Keep ingress closed.
|
||||
Manager precedes web, so `web-chat` becomes best-effort. No `mail.start` deliverer remains. Resolve any extra actual boot entry before approval rather than deleting it accidentally.
|
||||
|
||||
Resume an interrupted **offline** swap only from its saved plan: each file must have either its before or after hash. A third image is a conflict. After all operations validate, checkpoint installed. Checkpoint **started before starting target**, conservatively disabling automatic whole-world rollback even if boot later fails.
|
||||
The permanent `<root>/target-owner` exports `SAUSAGE_FLAKE=path:<root>/target-selector` itself, then execs the inspected loop with the exact world path. The selector's sole app is the pinned target package. `<root>/old-owner` similarly selects the **explicit old package**, never the remote default. These are permanent installation/launcher artifacts, **not migration code**. Their bytes/hashes are retained as manifest evidence; do not delete them with `tool/`.
|
||||
|
||||
## 4. Start/reconcile and verify
|
||||
**Every later pane/cold restart uses `/home/surma/sau-384-cutover/target-owner`, with no caller environment required. Never invoke bare sausage-loop or the mutable gitea default for this world.** Keep this command in the operator's pane-restart procedure. Rehearsal cold-launched it in the same Herdr shell without ambient SAUSAGE_FLAKE and observed the target executable. The explicit old wrapper was executed only against a separate old-backup clone.
|
||||
|
||||
Checkpoint `started` **before any launch attempt**. In the identified Herdr shell, mediator starts the same inspected loop exactly once with an explicit pinned selector:
|
||||
`core.restart` now restarts the pinned reviewed host; it does **not** self-update from gitea. Future host updates require a separately approved package/closure/root/selector change. A remote push alone must never silently choose an old host against target state. Keep both GC roots through the rollback window; retire the old root only after an explicit decision, not tool deletion.
|
||||
|
||||
## 4. Driver: started → launch in the same pane → fresh verifier
|
||||
|
||||
The driver checks the pinned shell still owns the selected pane and no unexpected foreground process exists. It checkpoints **`started` before any launch attempt**, then submits the permanent target-owner path once via `herdr pane run` to the manifest's pane (live **w7:p6**). No shell reconstruction or in-world mediator action is needed during downtime.
|
||||
|
||||
Readiness is event driven: arm a kernel inotify watch on `state/session/manager/registrations`, queue one `session.subscribe` command in the console, and await its unique registration file publication. Then await actual gateway RPC completion (`cred.status` locked and registration acknowledgement). **Herdr wait-output is internally polling and is not used by the final driver.** It was an unsuccessful preliminary approach only.
|
||||
|
||||
Finally the driver publishes the manifest's **new explicit default verification message** with its stable UUID and records `awaiting_verification`. The manager, not the driver, runs it. This names the new mediator as verifier without claiming the cut default run was recovered. The test produced one run-start and mailbox-consumed pair for that message. Publication is not immediate model execution: in the last gate it waited **23.753 s to run-start / 24.530 s to consumption** behind initial reconciliation. Do not confuse that with publication latency or a completed startup scan.
|
||||
|
||||
No automatic relaunch, whole-world restore or reapply is allowed after `started`. If launch/acknowledgement is lost, inspect the pane/checkpoint once before any repair. Do not launch a second owner blindly.
|
||||
|
||||
## 5. New mediator: verify → release web → delete tool
|
||||
|
||||
The explicit new verification run reads the manifest, `driver.json`, migration/loss/audit evidence and this runbook. Keep ingress closed until checks pass:
|
||||
|
||||
- one target manager and one runner ownership lock per running session; no old owners, mail deliverer or old orchestrator service;
|
||||
- cred **locked**, no vault/key transfer; grants/requests/settings preserved. Interrupted callers reuse their existing stable credential-request keys, not automatic retries of secret-backed mutations;
|
||||
- explicit target run-start without run-end is the only automatic cut-run recovery; old finished-looking history alone is neither settlement nor permission to resume;
|
||||
- durable event/mailbox catch-up and same-ID default/advisor POST replay evidence holds; only intended loss/conversion/schema/new-activity differences exist. Surface uncertain mail/tool outcomes;
|
||||
- scheduled jobs and non-target state match the inventory; real new mediator/runner activity is distinguished from migration changes;
|
||||
- design effective grace 5 s and separate EOF-stop 5 s remain; crash supervision is enabled. Deliberate core.stop/kill disables service registration and is **not** a crash-supervision test.
|
||||
|
||||
Prior post-record live-size samples remain manager ack **0.6078 s**, registration catch-up **0.0128 s**, mail.send-to-default publication **0.5376 s** (integrated 0.5235 s), below existing orchestrator **3 s** / cred **10 s** caller budgets. Disclose integrated catch-up **12.0419 s** and the new initial-scan delay above. They are synthetic/warm-cache observations, not production latency guarantees. Prior real-core watch/upgrade/all-watch readiness, ~0.10% whole-host idle duty, detached error data and real cron warning/stop evidence remain linked in the result docs.
|
||||
|
||||
Only now use **`orchestrator.release` for SAU-375** to release its approved commit. Inspect that method's current typed arguments; do not invent them. Verify published asset hashes and ingress. No worker publishes anything. The rework tested approved asset hashes in temporary output, **not an actual live release or physical browser**.
|
||||
|
||||
Write external `verification.json` with evidence links and the required true checks named by the driver's `verify` mode: `one_manager`, `no_old_owner`, `locked_cred`, `non_target_preserved`, `losses_exact`, `catchup_and_replay`, `web_released_and_hashes`, `ingress_verified`. They are operator attestations after the checks, not substitutes for them. From the external root, mediator runs the finite command:
|
||||
|
||||
```sh
|
||||
SAUSAGE_FLAKE=path:/home/surma/sau-384-cutover/target-selector \
|
||||
/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py verify operator.json verification.json 2>> tool-source-and-delete.log
|
||||
```
|
||||
|
||||
The selector directory must contain exactly the recorded installation bytes from step 2, not a mutable public flake. This pins crash restarts to the same reviewed host instead of refreshing unknown future code. Keep that selector/closure reachable for the target owner after deleting the migration tool. Observe boot/RPC/watch completion events, not process/endpoint polling. If shell/launch acknowledgement is lost, inspect once before retrying; never launch a second owner blindly. Confirm:
|
||||
|
||||
- one manager, one ownership lock per running session; no old mail deliverer, old orchestrator service or old session agent owners;
|
||||
- cred starts **locked**, with no master-key/vault handover; stored grants/requests/settings remain intact; interrupted credential request callers retry their same stable key, not a secret-backed external mutation;
|
||||
- only explicit target run-start-without-run-end work is recovered under its existing agent-run UUID; finished-looking old history alone is not resumed;
|
||||
- pending new mailbox work and durable events catch up, callbacks tolerate repeats, fresh default/advisor POST retries use the same ID, no old pending queue/undelivered mail is replayed;
|
||||
- scheduled jobs/grants/non-target state match the inventory except recorded intended schema/activity additions; retained `unknown` mail/tool outcomes remain visible;
|
||||
- effective grace is 5 seconds regardless of legacy metadata, and crash supervision has not disabled a service. Deliberate `core.stop`/`core.kill` disables that service registration; do not use it as a crash-supervision test;
|
||||
- first-update cuts are surfaced: old runs without target records cannot be recovered automatically; tools may already have external effects; matching histories do not prove an effect did or did not happen.
|
||||
|
||||
Repeat live-size post-record publication/catch-up measurements if production timing is a release gate; compare `mail.send` to the existing 3-second orchestrator / 10-second cred limits. Fast manager boot acknowledgement is not a completed initial scan. Do not turn catch-up into a timer/poll loop.
|
||||
|
||||
## 5. Release web, reopen ingress, delete the one-off executable
|
||||
|
||||
Only after session and web plugins are active and verified, mediator/orchestrator uses the existing **`orchestrator.release` workflow for SAU-375** to clear its `publish:hold` and publish its approved commit. Check that method's current typed arguments before execution; this draft intentionally does not invent them. No worker performs this action.
|
||||
|
||||
Verify the published approved asset hashes and ingress once, mark the migration verified, then **delete its executable and any executable cached/controller copies**. The rehearsal did this; live mediator must too. Retain only the manifest, backup/inventory/loss/caller-conversion records and logs as migration evidence, plus separately reviewed target artifacts. Do not retain an enabled migration plugin or boot entry.
|
||||
It marks the saved plan/driver verified, logs source hashes/source-as-log, and **deletes the three migration executables and bytecode caches**. Confirm `tool-deletion.json` and absence once. Delete any other reconstructed executable/controller copies too. Retain only manifests, backup/inventory/loss/caller-audit records and logs as migration evidence, plus the separately reviewed target/permanent launcher artifacts. Rehearsal deleted all executed migration/test controllers before exporting evidence. If verification/deletion is interrupted, inspect the verified checkpoint and delete only remaining hash-matched tool copies; do not reconstruct and rerun migration effects.
|
||||
|
||||
## Failure/rollback boundaries
|
||||
|
||||
| Boundary | Action |
|
||||
| Boundary | Exact recovery rule |
|
||||
|---|---|
|
||||
| Before offline mutation | Stop with evidence; old frozen world is untouched. |
|
||||
| During swap, target never started | Resume the saved hash-guarded plan, or restore the full frozen backup and old external owner/configuration coherently. Do not partially start mixed contracts. |
|
||||
| Any target start attempted | No automatic full restore/reapply. New histories, mail, jobs, grants or external effects may already exist. Preserve new state; diagnose and plan a narrow forward repair/revert. Never blindly rerun an external effect or undo new work. |
|
||||
| Session revert after target activity | A new approved operation: close gateway/ingress, `session.unsubscribe {id:"web-chat"}` while the new manager is still available, verify removal, prevent gateway re-registration while old session runs, then coordinate the stopped version change. Do not restore a registry backup that reintroduces the registration. ID-only rename is proven unsafe. Run 1369 verified this with the actual gateway left running and no re-registration: zero historical callbacks after unsubscribe; ID-only re-key replayed history. Physical browser/SSE display is not claimed. |
|
||||
| Waiting for arm | World untouched. Inspect the saved controller identity before cancelling; do not launch another start against existing checkpoints. |
|
||||
| Armed/captured/owner_stopped/frozen; no target attempt | External operator can use `... driver.py resume operator.json` **only after the saved controller has exited**. It detaches a replacement; same boot/identities/manifest/backup/plan must pass. No agent needs to exist. Both caller-killed and controller-killed resume were rehearsed. |
|
||||
| Interrupted planning/swap, no start attempt | Resume the same immutable backup/plan with before/after hashes. A caller/config after-image is accepted only if that exact saved operation approved it; third images refuse. Alternatively restore the full frozen backup coherently and launch the explicit old-owner wrapper. Never partially start mixed contracts. A frozen audit discrepancy is an offline stop, not permission to improvise a conversion. |
|
||||
| `started` or later, even if boot seems failed | No automatic whole-world rollback/reapply/relaunch. New histories, mail, jobs, grants or external effects may exist. Preserve them and plan a narrow forward repair or coordinated revert. |
|
||||
| Session revert after target activity | Close gateway/ingress. While new manager is available, **unsubscribe web-chat**, verify removal, and prevent gateway re-registration while the old session binary runs. Never call web.start against that old manager or restore a registry backup that reintroduces it. Actual old-manager/gateway tests proved unsubscribe safe and ID-only re-key unsafe: the old binary ignores best_effort and replays history once. |
|
||||
| Host reboot / changed PID identity / unexpected survivor | Refuse saved native identities; re-inventory externally. Never signal a reused PID, blindly kill a group, or infer quiescence from a stop request. |
|
||||
|
||||
Independent plugin recovery follow-ups remain independent. Neither uniform grace nor this one-off migration proves their external side effects, usage accounting, crash recovery or provider cancellation are solved.
|
||||
Independent plugin recovery/accounting/provider-cancellation follow-ups remain independent. Accepted first-update cuts and uncertain external effects are not silently solved by this kit, grace=5, or a successful rehearsal.
|
||||
|
||||
Reference in New Issue
Block a user