Compare commits
6
Commits
466654e744
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ee88adf6c6 | ||
|
|
6dc21cd8dc | ||
|
|
5be5b3b397 | ||
|
|
a95b5287cf | ||
|
|
959ddf46ff | ||
|
|
4da8a28392 |
@@ -222,7 +222,6 @@ these areas fails.
|
|||||||
/proc/
|
/proc/
|
||||||
core/
|
core/
|
||||||
PROTOCOL.md this file
|
PROTOCOL.md this file
|
||||||
discover.json core.rpc.discover document
|
|
||||||
graph.json plugins, imports, and the result of each check
|
graph.json plugins, imports, and the result of each check
|
||||||
graph.dot the same graph for Graphviz
|
graph.dot the same graph for Graphviz
|
||||||
locks.json held locks and waiting processes
|
locks.json held locks and waiting processes
|
||||||
@@ -390,7 +389,6 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
|
|||||||
|
|
||||||
| Method | Params | Result |
|
| Method | Params | Result |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `core.rpc.discover` | none | OpenRPC core API document (also `/proc/core/discover.json`) |
|
|
||||||
| `core.call` | `method`, `params`, `detach`? | the result of the method |
|
| `core.call` | `method`, `params`, `detach`? | the result of the method |
|
||||||
| `core.watch` | `path` | stays open; `armed`, then `changed` notifications (section 4) |
|
| `core.watch` | `path` | stays open; `armed`, then `changed` notifications (section 4) |
|
||||||
| `core.notify` | `name`, `params` | none (a notification) |
|
| `core.notify` | `name`, `params` | none (a notification) |
|
||||||
|
|||||||
@@ -1,19 +0,0 @@
|
|||||||
{
|
|
||||||
"openrpc": "1.2.6",
|
|
||||||
"info": {"title":"core", "version":"0.1.0"},
|
|
||||||
"methods": [
|
|
||||||
{"name": "rpc.discover", "summary": "Describe the native core API.", "description": "Describe the native core API.", "paramStructure": "by-name", "params": [], "result": {"name": "result", "schema": {"type": "object"}}},
|
|
||||||
{"name": "watch", "summary": "Keep this call open until the caller process ends. First core.notify is armed, then changed after successful plugin-originated file mutations. Scan only after armed; rescan after changed. Events may coalesce; files are the truth. No polling or cancellation API. See PROTOCOL section 4.", "description": "Keep this call open until the caller process ends. First core.notify is armed, then changed after successful plugin-originated file mutations. Scan only after armed; rescan after changed. Events may coalesce; files are the truth. No polling or cancellation API. See PROTOCOL section 4.", "paramStructure": "by-name", "params": [{"name": "path", "required": true, "description": "Virtual file or directory path, including descendants; need not exist. /self is relative to the caller.", "schema": {"type": "string"}}], "result": {"name": "result", "schema": {"type": "null"}}},
|
|
||||||
{"name": "call", "summary": "Make a call and return its result; detach lets the new process outlive its caller.", "description": "Make a call and return its result; detach lets the new process outlive its caller.", "paramStructure": "by-name", "params": [{"name": "method", "required": true, "description": "Qualified method.", "schema": {"type": "string"}}, {"name": "params", "required": true, "description": "Named parameters.", "schema": {"type": "object"}}, {"name": "detach", "required": false, "description": "Detach the new process.", "schema": {"type": "boolean"}}], "result": {"name": "result", "schema": {}}},
|
|
||||||
{"name": "notify", "summary": "Notify this process's caller while its call is open.", "description": "Notify this process's caller while its call is open.", "paramStructure": "by-name", "params": [{"name": "name", "required": true, "description": "Event name.", "schema": {"type": "string"}}, {"name": "params", "required": false, "description": "Event payload.", "schema": {}}], "result": {"name": "result", "schema": {"type": "null"}}},
|
|
||||||
{"name": "stop", "summary": "End a process and await cleanup.", "description": "End a process and await cleanup.", "paramStructure": "by-name", "params": [{"name": "pid", "required": true, "description": "Process ID.", "schema": {"type": "string"}}], "result": {"name": "result", "schema": {"type": "object"}}},
|
|
||||||
{"name": "kill", "summary": "End a process and await cleanup.", "description": "End a process and await cleanup.", "paramStructure": "by-name", "params": [{"name": "pid", "required": true, "description": "Process ID.", "schema": {"type": "string"}}], "result": {"name": "result", "schema": {"type": "object"}}},
|
|
||||||
{"name": "lock", "summary": "Acquire a FIFO process-lifetime lock.", "description": "Acquire a FIFO process-lifetime lock.", "paramStructure": "by-name", "params": [{"name": "path", "required": true, "description": "Lock key.", "schema": {"type": "string"}}, {"name": "wait", "required": false, "description": "Wait for acquisition (default true).", "schema": {"type": "boolean"}}], "result": {"name": "result", "schema": {"type": "object", "properties": {"held": {"type": "boolean"}}, "required": ["held"]}}},
|
|
||||||
{"name": "unlock", "summary": "Release an owned lock.", "description": "Release an owned lock.", "paramStructure": "by-name", "params": [{"name": "path", "required": true, "description": "Lock key.", "schema": {"type": "string"}}], "result": {"name": "result", "schema": {"type": "null"}}},
|
|
||||||
{"name": "activate", "summary": "Install one or more checked plugin versions. See PROTOCOL section 10.", "description": "Install one or more checked plugin versions. See PROTOCOL section 10.", "paramStructure": "by-name", "params": [{"name": "plugin", "required": false, "description": "Single plugin name.", "schema": {"type": "string"}}, {"name": "wasm", "required": false, "description": "Candidate path.", "schema": {"type": "string"}}, {"name": "plugins", "required": false, "description": "Batch candidates.", "schema": {"type": "array", "items": {"type": "object", "properties": {"plugin": {"type": "string"}, "wasm": {"type": "string"}}, "required": ["plugin", "wasm"]}}}, {"name": "force", "required": false, "description": "Skip compatibility rejection.", "schema": {"type": "boolean"}}], "result": {"name": "result", "schema": {"type": "object"}}},
|
|
||||||
{"name": "revert", "summary": "Restore the previous plugin version.", "description": "Restore the previous plugin version.", "paramStructure": "by-name", "params": [{"name": "plugin", "required": true, "description": "Plugin name.", "schema": {"type": "string"}}, {"name": "force", "required": false, "description": "Skip compatibility rejection.", "schema": {"type": "boolean"}}], "result": {"name": "result", "schema": {"type": "object"}}},
|
|
||||||
{"name": "log", "summary": "Read the central journal (PROTOCOL section 7).", "description": "Read the central journal (PROTOCOL section 7).", "paramStructure": "by-name", "params": [{"name": "pid", "required": false, "description": "Process ID.", "schema": {"type": "string"}}, {"name": "plugin", "required": false, "description": "Plugin name.", "schema": {"type": "string"}}, {"name": "since", "required": false, "description": "ISO 8601 start time.", "schema": {"type": "string"}}, {"name": "until", "required": false, "description": "ISO 8601 end time.", "schema": {"type": "string"}}, {"name": "grep", "required": false, "description": "Line substring.", "schema": {"type": "string"}}, {"name": "limit", "required": false, "description": "Maximum entries (default 100).", "schema": {"type": "integer"}}], "result": {"name": "result", "schema": {"type": "array", "items": {"type": "object"}}}},
|
|
||||||
{"name": "restart", "summary": "Request clean host restart through the loop supervisor; needs explicit operator approval.", "description": "Request clean host restart through the loop supervisor; needs explicit operator approval.", "paramStructure": "by-name", "params": [], "result": {"name": "result", "schema": {"type": "object"}}},
|
|
||||||
{"name": "exit", "summary": "Request clean host restart through the loop supervisor; needs explicit operator approval.", "description": "Request clean host restart through the loop supervisor; needs explicit operator approval.", "paramStructure": "by-name", "params": [], "result": {"name": "result", "schema": {"type": "object"}}}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
@@ -12,7 +12,6 @@ impl Host {
|
|||||||
p: Value,
|
p: Value,
|
||||||
) -> RpcResult {
|
) -> RpcResult {
|
||||||
match method {
|
match method {
|
||||||
"rpc.discover" => Ok(serde_json::from_str(include_str!("core.json")).unwrap()),
|
|
||||||
"watch" => self.watch(caller, id, required_str(&p, "path")?).await,
|
"watch" => self.watch(caller, id, required_str(&p, "path")?).await,
|
||||||
"call" => {
|
"call" => {
|
||||||
self.route(
|
self.route(
|
||||||
|
|||||||
@@ -85,10 +85,6 @@ impl Host {
|
|||||||
}
|
}
|
||||||
fn core_files(&self) -> Generated {
|
fn core_files(&self) -> Generated {
|
||||||
let mut files = Generated::default();
|
let mut files = Generated::default();
|
||||||
files.insert(
|
|
||||||
"/discover.json",
|
|
||||||
|| Ok(include_bytes!("core.json").to_vec()),
|
|
||||||
);
|
|
||||||
let protocol = include_bytes!("../../PROTOCOL.md");
|
let protocol = include_bytes!("../../PROTOCOL.md");
|
||||||
files.insert("/PROTOCOL.md", move || Ok(protocol.to_vec()));
|
files.insert("/PROTOCOL.md", move || Ok(protocol.to_vec()));
|
||||||
let graph = self.graph_json();
|
let graph = self.graph_json();
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ use serde_json::{Value, json};
|
|||||||
use std::{collections::VecDeque, sync::atomic::Ordering, time::Duration};
|
use std::{collections::VecDeque, sync::atomic::Ordering, time::Duration};
|
||||||
use tokio::time::Instant;
|
use tokio::time::Instant;
|
||||||
|
|
||||||
// Agreed systemd-derived values: /docs/design/plugin-handover/design.md,
|
// Systemd-derived values:
|
||||||
// https://www.man7.org/linux/man-pages/man5/systemd-system.conf.5.html.
|
// https://www.man7.org/linux/man-pages/man5/systemd-system.conf.5.html.
|
||||||
const CRASH_LIMIT: usize = 5;
|
const CRASH_LIMIT: usize = 5;
|
||||||
const CRASH_WINDOW: Duration = Duration::from_secs(10);
|
const CRASH_WINDOW: Duration = Duration::from_secs(10);
|
||||||
|
|||||||
@@ -741,7 +741,7 @@ async fn standalone_builder() {
|
|||||||
1
|
1
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
/// SAU-126: a named CPU-bound WASI test times out, while other instances still run.
|
/// a named CPU-bound WASI test times out, while other instances still run.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn named_test_timeout_and_mutex_release() {
|
async fn named_test_timeout_and_mutex_release() {
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
@@ -815,7 +815,7 @@ async fn child_ended(child: u32) {
|
|||||||
.expect("cargo descendant survived cancellation/timeout");
|
.expect("cargo descendant survived cancellation/timeout");
|
||||||
}
|
}
|
||||||
|
|
||||||
/// SAU-126: an explicit small total budget kills build.rs and unblocks a queued build.
|
/// an explicit small total budget kills build.rs and unblocks a queued build.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn build_timeout_and_queued_build() {
|
async fn build_timeout_and_queued_build() {
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
@@ -853,7 +853,7 @@ async fn build_timeout_and_queued_build() {
|
|||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// SAU-126: the same total deadline covers release compilation and retains test results.
|
/// the same total deadline covers release compilation and retains test results.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn release_build_timeout_retains_tests() {
|
async fn release_build_timeout_retains_tests() {
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
@@ -903,7 +903,7 @@ fn main() {
|
|||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// SAU-126: omission selects the real 900-second deadline, without a 15-minute test.
|
/// omission selects the real 900-second deadline, without a 15-minute test.
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn omitted_build_timeout_uses_default() {
|
async fn omitted_build_timeout_uses_default() {
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
@@ -928,7 +928,7 @@ async fn omitted_build_timeout_uses_default() {
|
|||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// SAU-126: a caller ending stops its attached native build and releases the mutex.
|
/// a caller ending stops its attached native build and releases the mutex.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn ended_caller_releases_build_mutex() {
|
async fn ended_caller_releases_build_mutex() {
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
@@ -970,7 +970,7 @@ async fn ended_caller_releases_build_mutex() {
|
|||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// SAU-126: discovery exposes non-null defaults; invalid budgets cannot disable limits.
|
/// discovery exposes non-null defaults; invalid budgets cannot disable limits.
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn timeout_params_discovery_and_validation() {
|
async fn timeout_params_discovery_and_validation() {
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
|
|||||||
@@ -172,24 +172,3 @@ async fn rename_delete_and_mounts() {
|
|||||||
assert!(task.await.unwrap().is_err());
|
assert!(task.await.unwrap().is_err());
|
||||||
h.stop().await;
|
h.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// PROTOCOL 9: core discovery includes the armed watch and matches its proc projection.
|
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
|
||||||
async fn discovery() {
|
|
||||||
let (_dir, h) = start().await;
|
|
||||||
let doc = call(&h, "core.rpc.discover", json!({})).await;
|
|
||||||
let projected = call(&h, "probe.read", json!({"path":"/proc/core/discover.json"})).await;
|
|
||||||
assert_eq!(
|
|
||||||
doc,
|
|
||||||
serde_json::from_str::<Value>(projected.as_str().unwrap()).unwrap()
|
|
||||||
);
|
|
||||||
let watch = doc["methods"]
|
|
||||||
.as_array()
|
|
||||||
.unwrap()
|
|
||||||
.iter()
|
|
||||||
.find(|m| m["name"] == "watch")
|
|
||||||
.unwrap();
|
|
||||||
assert_eq!(watch["params"][0]["name"], "path");
|
|
||||||
assert!(watch["description"].as_str().unwrap().contains("armed"));
|
|
||||||
h.stop().await;
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,69 +0,0 @@
|
|||||||
# Disposable handover rehearsal — SAU-384
|
|
||||||
|
|
||||||
Authority: [/docs/design/plugin-handover/design.md](/docs/design/plugin-handover/design.md), Rollout; [/docs/symphony/artifacts/SAU-356/plan.md](/docs/symphony/artifacts/SAU-356/plan.md), Approval and operator gates.
|
|
||||||
|
|
||||||
**Status: rehearsal complete, ready for review; not a live execution approval.** Rehearsal evidence and the reviewed target bundle are under [/docs/symphony/artifacts/SAU-384](/docs/symphony/artifacts/SAU-384). No shipped plugin or host behavior changes on this branch. The temporary migration implementation and native controller were executed only in a throwaway box and deleted after the successful migration verification. Their hashes/deletion record and source-as-log remain, not an executable migration tool.
|
|
||||||
|
|
||||||
## Closed integration gaps
|
|
||||||
|
|
||||||
- **Round 4, run 1392:** allocated-block/directory/sparse-copy space budgeting; disk-independent foreground progress/failure/pidfd-death monitor; incomplete pre-plan backup cleanup before abort writes plus low-space refusal; forward-only post-start continue with durable locked proof and stable continuation ID; guarded atomic-temporary cleanup; converter dry-run before freeze. Shared-filesystem ENOSPC (world/root/log at zero free bytes), each post-start RPC failure including lost committed send reply, real 120 s registration timeout, partial atomic crash/abort, wrapper-exec handshake/resume, sidecars and full workspace passed. Tools deleted after verified success. Source/logs, fixture failures and limits: [rework-1392.md](rework-1392.md). Live plan is still approval-gated.
|
|
||||||
|
|
||||||
- **Round 3, run 1389:** deferred approved git 9cc9fa6f is bundled/preflighted, activated only after target cred is live and Surma unlocks. Real old/new Git + new cred loopback tests covered granted SSH, locked denial and new shared SSH. Added real-driver pre-start abort/restore/old-owner relaunch, live consistent-copy/executable-only caller guards, special-file/free-space gates, finite registration deadline/failure notification, exact pre-cut prev images and durable arm-marker crash recovery. Real partial-swap/ENOSPC/audit failures, arm interruption, final Herdr cutover/preservation/sidecars and full workspace (208 host tests) passed. Tool/controllers deleted after verification. Exact logs, selection and limits: [rework-1389.md](rework-1389.md); proposed live recovery/roots-side-effect plan: [runbook](runbook.md).
|
|
||||||
|
|
||||||
- **Review rework, run 1381:** one detached external CLI now owns freeze→backup/plan/audit→apply→same-Herdr-pane launch and publishes a new explicit mediator verification message. Caller/controller death and interrupted planning resume passed. Plan/apply check the actual current-boot frozen checkpoint; old/target GC roots and permanent pinned owner wrappers, stopped-PTY TERM+CONT, SQLite sidecars and whole-world scale were tested. Tool/controllers were logged/deleted after verification; current workspace tests passed again. See [rework-1381.md](rework-1381.md) for exact evidence, limitations and disclosure of an early shared-Nix stale-root cleanup (all referenced targets/PIDs were absent on the host; subsequent tests used private Nix state). [Runbook](runbook.md) now assigns all downtime work to that external controller, not the cut old mediator. Live launch/release still require the explicit operator approval described there.
|
|
||||||
- **Run 1373:** after mediator installed the observed public host key, approved read-only SSH inspection identified the actual Herdr pane, outer `sausage-loop`, world path, native architecture, old package/derivation/source and runtime closure. The actual old Nix package/loop → stop-all → offline swap → exact new Nix release package/loop passed in the disposable live-size world. A non-parent external controller separately proved loop-first stop, interruption/resumption, stale-PID refusal and safe frozen rerun. A real core.restart after loop exit could not respawn the host. See [native evidence](resume-1373.md).
|
|
||||||
- **Run 1369:** armed/exit-race delay-gate tests passed on real core with actual target WASM; replacement orchestrator readiness/one-owner/all-watch arming passed. Hooks existed only in a disposable host copy, never the selected host artifact or committed code.
|
|
||||||
- **Rollback:** actual old-manager/gateway tests prove **unsubscribe works; ID-only re-key does not**. The old binary ignores best_effort and replays history to a renamed ID. The runbook mandates the card-authorized unsubscribe option and prevents web re-registration during a session revert. A physical browser/SSE display is not claimed.
|
|
||||||
- Final selected coherent bundle is **`cutover-targets-1389.tar.gz`** (SHA-256 `31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b`), **23 payloads**, including deferred approved **git 9cc9fa6f / 177 tests**. Same Nix native package/closure, approved cred main `0db40101` build **1636 / 274 tests**, other reviewed plugins/web assets and SDK `src/lib.rs`; no diagnostic debug host. Git is not swapped offline; Surma unlock precedes its mediator activation. Earlier bundles and runs remain historical evidence. Final exact-driver native/preservation/sidecar/failure and full-workspace gates passed again; executable/controller deletion and source-as-log are in rework-1389.md.
|
|
||||||
|
|
||||||
The final frozen inventory/caller-schema/hash check remains an operator gate at the actual approved shutdown, not something a read-only rehearsal can perform in advance. Live execution still requires Surma's go and mediator/orchestrator action; no approval request is pending.
|
|
||||||
|
|
||||||
## What was built and exercised
|
|
||||||
|
|
||||||
- Host source `e09c4f9f0ef6b5833d6be066771eecd9e0a1f572`. Original debug native controller exercised real `sausage::Host`, VFS, watches, locks, service supervisor and WASM with synthetic/loopback effects. Run 1373 additionally built/rehearsed the actual **Nix release wrapper/package** `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`; its empty-caller-PATH runtime check built/tested a real WASI+C plugin. Workspace suite passed again; all 105 Rust source files matched the tested tree. Neither host was released/deployed or selected by the live launcher. Native artifact/closure hashes are in `evidence/native-host-manifest.json`.
|
|
||||||
- Reviewed session `cb844151` / SAU-409 final; orchestrator `c80a5854` / SAU-408; web `d883fbf2` / SAU-403; agent/board retained reviewed components. Fresh card-worktree builds: mail **1600 / 53 tests**, cred **1601 / 272**, cron **1602 / 24**, rooms **1603 / 59**, all passed. No activation. Component/source provenance and every bundled file hash are in `evidence/manifest.json`.
|
|
||||||
- Frontend is exactly approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`, retained production build from SAU-375 rework-1318. It remains `publish:hold`; no publication was attempted.
|
|
||||||
- World SDK runtime sources already matched the selected host SDK. The only needed source-copy delta is **`src/lib.rs`**, documenting effective grace. Preserve world-owned Cargo manifests, relocated `interface`, and imports. A first broad SDK-copy attempt incorrectly included repository-relative Cargo paths; comparison caught this before any live change, and the final migration used only the one-file patch. Never copy the repository's SDK Cargo manifests over the world SDK.
|
|
||||||
- Accepted SAU-362 byte snapshot: **1,491 sessions / 580,355,405 bytes**, including **70,689,796-byte default history**. Pristine snapshot remained unpreopened. Executable clones used synthetic model settings; their historical byte volume was retained. Source snapshot consistency is per-file, not a transactional live snapshot.
|
|
||||||
|
|
||||||
## Successful checks and limits
|
|
||||||
|
|
||||||
| Check | Evidence/result |
|
|
||||||
|---|---|
|
|
||||||
| Old ownership → stop-all → offline swap → target boot | Old `mail.start`, old session `send` and detached ledger processes inventoried. Native external controller awaited `Host.stop`, verified zero remaining processes and host exit, then backed up/transformed/swapped. Final shutdown **0.0958 s** in this fixture. One target manager, no mail deliverer, cred locked. |
|
|
||||||
| Partial swap + rerun | Injected interruption after operation 3; hash-guarded completion and repeated installed-phase apply preserved all **4,374 non-target inventoried files**. No live/new work reset. |
|
|
||||||
| Exact losses/preservation | Only `pending.jsonl`, `cancel`, and unclaimed pending non-Surma delivery rows selected for removal. Original mail messages, done/Surma copies and uncertain `unknown` deliveries retained without replay; scheduled SQL IDs, calendar/outcome fields and non-session JSON bytes retained. Synthetic grant DB bytes preserved. Loss inventory backs every removed file/row. |
|
|
||||||
| Offline failure boundaries | Before-start rollback restored exact bytes; conflicting outside edits refused; ambiguous legacy `interrupt:true` refused without mutation. After-start apply and whole-world rollback both refused. |
|
|
||||||
| Default/advisor HTTP replay | Three identical same-ID POSTs each, before/after consumption; exactly one answered run per session. Default returned null on repeats; advisor returned its same receipt. |
|
|
||||||
| Explicit open target run | A run-start/consumed record with a finished-looking assistant tail, but no run-end, recovered under the **same agent-run UUID**, with one start. Legacy finished-looking history alone produced no target event/run. |
|
|
||||||
| Real VFS/runner-boundary work | Actual SDK `armed` then plugin-write `changed`; five callback-boundary runs. Run 1369 additionally forced publication before runner `armed` and after its empty-mailbox exit decision while detached completion was held; no lost/duplicate inputs (disposable host delay gates, actual target WASM). |
|
|
||||||
| Real core detached errors | Full nested error `data` survived a detached `core.call`. |
|
|
||||||
| Real cron research deadline | **305.427 s** send→stopped in the final gate; real cron invoked warning and Stop. The 305-second test deadline is the existing 300-second warning interval plus design grace 5 seconds, not a new default. Held inference was cut, no follow-up run. Original configured research default remains 1,200 seconds. |
|
|
||||||
| Effective grace / crash supervision | Passing native host suite exercises real WASI uniform-grace and before/after-readiness crash-supervision fixtures. Deliberate `core.kill` is NOT a crash: it disables that boot service; an initial rehearsal expecting its restart failed, correctly, and was replaced by the existing crash tests. |
|
|
||||||
| Orchestrator upgrade/idle | Real file-watcher upgrade, FIFO singleton release, old exit before one replacement start event. Paused empty-board/live-size-world host CPU **0.02 s / 20.0002 s ≈ 0.10%**. This is whole-host idle CPU, not per-plugin CPU or live active-work load. Run 1369 additionally verified target acknowledgement, one owner and all 13 core watches armed after old exit. |
|
|
||||||
|
|
||||||
### Live-size latency after real records exist
|
|
||||||
|
|
||||||
After producing actual run-start/run-end/mailbox-consumed records, subsequent full boot measured **0.544 s** from manager service-start to the next boot service-start (includes discovery/ack overhead). Narrow registration catch-up samples ranged **0.0239 s to 12.0419 s** in the integrated world: the long sample coincided with other synthetic recovery/surfacing work, and the untagged sink cannot isolate attribution. Do not present this as a bounded manager-only benchmark.
|
|
||||||
|
|
||||||
The clean final live-size end-to-end gate measured manager acknowledgement **0.6078 s**, registration catch-up **0.0128 s**, and `mail.send` publication to default **0.5376 s**. The integrated post-record mail sample was **0.5235 s**. Both publication samples are below the existing orchestrator **3 s** and cred **10 s** caller budgets (SAU-400 report cites the originating SAU-366 review). These warm-cache synthetic-loopback real-core observations are not live latency guarantees. First scanning still delays first new input after the fast boot acknowledgement.
|
|
||||||
|
|
||||||
## Resume verification
|
|
||||||
|
|
||||||
Run 1369 forced recompilation and repeated the complete workspace suite on the restored original host source: pass, including **208 host integration tests**. All **144 source files** matched the selected source snapshot. Full log `/logs/box/2026-10-31/tc-asa3yr-exec-3838117964.stderr` (host tests lines 64–434; equivalence line 785). New tests and their limitations are detailed in `resume-1369.md`; new test controllers and instrumentation were deleted after successful checks. Original bundle and production source are unchanged.
|
|
||||||
|
|
||||||
## Evidence and failures
|
|
||||||
|
|
||||||
All following logs are under `/logs/box/2026-10-31/`:
|
|
||||||
|
|
||||||
- Full workspace pass: `tc-asa3yr-exec-2320283969.stderr`. First cold-cache/full-load failure: `tc-asa3yr-exec-158345583.stderr`, missing offline async-task and a queued-build timing test. Populated the separate SDK-probe cache; entire suite then passed unchanged.
|
|
||||||
- Final stop/swap/start: `tc-asa3yr-exec-1922130515.stderr`; offline fault/grant preservation: `tc-asa3yr-exec-616022371.stderr`.
|
|
||||||
- Final live-size HTTP/deadline gate: `tc-asa3yr-exec-1243993190.stderr`. Earlier malformed mail `to` scalar, 30-second startup observation bound and blank JSONL parser mistakes were harness errors; corrected to a sequence, existing 120-second test budget and blank-line skipping. Final gate passed.
|
|
||||||
- Open-run recovery and boundary runs: `tc-asa3yr-exec-903419814.stderr` (subsequent console-watch attempt failed). A console is not a running watch caller; the final SDK guest watch passed in `tc-asa3yr-exec-3261552495.stderr`. Same log retains the real upgrade trace and idle sample. An identical already-acknowledged registration correctly produced no second catch-up; the final probe used a fresh registration identity.
|
|
||||||
- Actual old-manager unsubscribe/re-key comparison: `tc-asa3yr-exec-3307393394.stderr`.
|
|
||||||
- Native artifact architecture/runtime paths: `tc-asa3yr-exec-899103518.stderr`; bundle build/CLI: `tc-asa3yr-exec-1075679130.stderr`.
|
|
||||||
- Executed temporary tool and controller **as log only**, followed by deletion proof: `tc-asa3yr-exec-2902726782.stderr` (31,475 bytes). `evidence/tool-deletion.json` lists removed executables. No transition code was put in a shipped plugin.
|
|
||||||
|
|
||||||
The initial broad-copy fixture and earlier failed tests are not the final kit. `evidence/migration-plan.json` identifies the final verified migration. Provider billing/cancellation, power loss, hung host filesystem behavior, physical browsers and independent per-plugin recovery follow-ups are not claimed complete. No live DB was opened; the accepted session snapshot was reused. No live config/SDK change, stop, activation, release, restart, push, deployment or `/www` write occurred.
|
|
||||||
|
|
||||||
See [operator runbook](runbook.md) for exact owner/package/selector steps, frozen-audit and approval gates. [Run 1373](resume-1373.md) records the final native rehearsal and remaining unverified live-only boundaries.
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# SAU-384 resume — run 1369
|
|
||||||
|
|
||||||
Authority remains the card, design Rollout and SAU-356 approval gates. No live cut-over has been authorized or executed.
|
|
||||||
|
|
||||||
## Results added
|
|
||||||
|
|
||||||
- **Armed race passed:** actual target session WASM on real core, held immediately after core registered the runner's watch but before `armed`; a second real `session.send` published during the hold. Both IDs were consumed once in one run.
|
|
||||||
- **Exit race passed:** held only delivery of the runner's initial result after the target's empty-mailbox exit decision. A second real publication arrived while the manager still considered that runner busy. Releasing detached completion produced exactly two runs, each consuming its own one ID. At publication the original runner had already exited; only the manager remained. No new timer/polling was added.
|
|
||||||
- **Orchestrator upgrade readiness passed:** actual old SAU-370 component → selected SAU-408 component through the native plugin file watcher, old exit at 13:23:42.625Z before target start at 13:23:42.747Z, successful target acknowledgement and exactly one new owner. All 13 paths in `service.rs::WATCHES` armed in the target process. Prior whole-host idle duty measurement remains ~0.10% over 20 s in the paused, empty-board/live-size fixture; not a live workload guarantee.
|
|
||||||
- **Actual gateway rollback passed:** new gateway first made its real `web-chat` best-effort registration; without another `web.start`, the session binary reverted to the actual pre-best-effort manager. Explicit unsubscribe caused zero `web.session_event` calls through the old manager's catch-up barrier. ID-only re-key caused one historical gateway callback. Therefore use the card's **unsubscribe** option; do not offer ID-only re-key as safe. This checks the actual callback/endpoint, not a physical browser/SSE display.
|
|
||||||
- Restored production host source after disposable observation/delay hooks, forced recompilation, and reran `cargo test --workspace`: pass, including 208 native host integration tests and grace/supervision tests. All 144 source files matched the selected source snapshot. No source behavior changed in any committed plugin or host.
|
|
||||||
|
|
||||||
## Evidence
|
|
||||||
|
|
||||||
Under `/docs/symphony/artifacts/SAU-384/evidence/`:
|
|
||||||
`resume-races-summary.json`, `resume-upgrade-summary.json`, `resume-upgrade-wire.json`, `resume-rollback-summary.json`, `resume-tool-deletion.json`, `resume-source-equivalence.json`.
|
|
||||||
|
|
||||||
Full logs under `/logs/box/2026-10-31/`:
|
|
||||||
- `tc-asa3yr-exec-1457078935.stderr`: race passes, followed by the initial upgrade-observation failure. Immediate swap raced initial asynchronous graph discovery; no old-version discovery baseline existed yet. Next method waited once for discovery completion, not a liveness loop.
|
|
||||||
- `tc-asa3yr-exec-870753431.stderr`: successful upgrade readiness/order.
|
|
||||||
- `tc-asa3yr-exec-4006214778.stderr`: actual gateway unsubscribe/re-key comparison.
|
|
||||||
- `tc-asa3yr-exec-621695466.stderr`: executed tests/controller and host-only instrumentation as log, then deletion. No executable one-off/controller copies remain.
|
|
||||||
- `tc-asa3yr-exec-3838117964.stderr`: forced-recompile full workspace pass; host tests lines 64–434, source equivalence line 785. Earlier unforced repeat also passed but is not used to establish recompilation.
|
|
||||||
|
|
||||||
The original migration tool remains deleted. Recreated only disposable test controllers to close the above gaps; deleted those too after verification. Retained target bundle hash/artifacts are unchanged. Source-as-log is evidence, not a retained executable migration kit.
|
|
||||||
|
|
||||||
## Remaining work / approval wait
|
|
||||||
|
|
||||||
Only the actual external launcher/native installation manifest is still unresolved. Source `/repos/nixenv/machines/citadel/default.nix:22–27` disables `services.sausage`; `/repos/nixenv/machines/nexus/service-sausage.nix:10–11` describes the current Herdr panel. These sources do not pin the live panel, world path, launcher command, restart owner, stop completion API, executable/runtime closure, or architecture. Do not substitute `systemctl stop sausage` or `core.restart`.
|
|
||||||
|
|
||||||
Read-only systemd-web GETs confirmed no active system `sausage.service`, and user-manager journal entries mention `/home/surma/.config/herdr/herdr.sock`; they still do not identify the currently running panel. Sources used:
|
|
||||||
- `http://citadel-systemd.nexus.hosts.10.0.0.2.nip.io:8081/api/units?type=service`
|
|
||||||
- `http://citadel-systemd.nexus.hosts.10.0.0.2.nip.io:8081/api/logs?unit=user%401000.service&lines=100&boot=0`
|
|
||||||
|
|
||||||
No attached grant existed. One credential request is pending: **055995a51608c57dd25b323bea731fcb**, `id_scout`, `sub:card:SAU-384`, card duration, for **read-only** Citadel launcher inspection. The initial attempt supplied the immutable item ID, but live request expects a name and rejected it before filing; retry with `id_scout` filed the sole request. Do not request again; reuse it on resume. A credential approval is not permission to stop/restart/deploy.
|
|
||||||
|
|
||||||
After approval: inspect actual process/parent/Herdr configuration read-only; pin owner, world path, synchronous stop/start/completion mechanism and correct native package/closure. Finish/rehearse that specific outer host swap or report a precise unresolved interface. Do not silently claim that the existing target-host/native-controller rehearsal replaced the old native service package. The current local host bundle is an aarch64-linux debug-profile stripped binary, not a validated live installation package.
|
|
||||||
|
|
||||||
The surviving throwaway box is `tc-asa3yr`; temporary build/state copies remain there until its TTL. It contains no executable migration controller. If it expires, restore from retained target bundle/evidence and read the source logs, not a shipped plugin compatibility path. No pushes, activations, restarts, releases, config/SDK mutation, or `/www` writes occurred in the live world.
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
# SAU-384 resume — run 1372
|
|
||||||
|
|
||||||
No live operation is authorized or performed. The existing rehearsal and target bundle are unchanged.
|
|
||||||
|
|
||||||
## Read-only launcher inspection: trust prerequisite
|
|
||||||
|
|
||||||
The existing card-duration SSH grant was approved; no new credential request was filed. `ssh.exec` to `citadel.vpn.surma.technology`, login `surma`, with the read-only command `uname -m; ps -eo pid,ppid,user,args | grep -E '([h]erdr|[s]ausage)'` was refused **before authentication/command execution** because the host is unknown. The refusal reported:
|
|
||||||
|
|
||||||
```text
|
|
||||||
SHA256:Fj3jS6DuV9SAo4PVQl/hTLI9SZPqWDwEd7E5hemfFkE
|
|
||||||
citadel.vpn.surma.technology ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB8gP37Pnb3EyyQnkR2VdeJNzGU3OAuyJc3E3a8XpDaV
|
|
||||||
```
|
|
||||||
|
|
||||||
The SSH skill permits installing this observed key without another person approval, but this card explicitly forbids worker live-configuration changes. Therefore the **mediator** must add the observed line to `/config/known_hosts`, preserving existing entries, then resume this worker. Surma does nothing. No private key was requested or fetched. `ssh.hosts_status` confirmed there is no already-trusted Citadel alias. The preceding run already tried the preferred read-only systemd-web route: it cannot identify the Herdr panel/native installation boundary. Do not substitute Nexus credentials, forward another host's SSH agent, bypass host checking, or treat this as permission to stop anything.
|
|
||||||
|
|
||||||
After trust is installed: reuse the attached grant, inspect launcher/process parents/Herdr config read-only; pin actual world path, architecture, executable/runtime closure and event/completion stop/start interface; then build/rehearse that specific outer host swap in a disposable box. No live stop, restart, config change or deployment is part of the resume.
|
|
||||||
|
|
||||||
## Caller-store inspection (not a frozen live audit)
|
|
||||||
|
|
||||||
`evidence/resume-1372-caller-inventory.json` records a read-only observation via `cron.list`, `board.hook_list`, boot-file inspection and current-main schema inspection. All eight cron jobs currently have `next_run:null`; six are historical `session.send` jobs with the legacy inner `detach:true`. Convert their executable JSON fields once with the already-rehearsed cron procedure, preserving SQL IDs, schedule/outcome fields and prompt bytes. References to old RPCs *inside their prompt strings* are history/instructions, not nested executable configuration; do not rewrite or replay them.
|
|
||||||
|
|
||||||
Five board hooks were observed. Only `mediator-done` calls `session.send`; its `{session,text}` payload already fits the target contract without legacy detach/notify/interrupt fields. Target board supplies a fresh per-attempt input ID and follow-up kind. No rewrite is needed for that observed hook. Other hooks target mail/orchestrator and must remain intact.
|
|
||||||
|
|
||||||
The frozen operator audit must additionally include board `invariant_target.data` (an arbitrary RPC template without a read-only target-get API), hooks and historical `hook_attempts`, board subscriptions/batches, room workflow/action records, orchestrator run/skill-review/host-job records and target session event registrations if present. Distinguish resumable target inputs from old diagnostic snapshots and never infer safe replay from a stored call-looking string. Current schemas are pointers, not substitutes for predecessor decoding or the actual frozen bytes. This session does **not** claim that final audit complete.
|
|
||||||
|
|
||||||
## Evidence/checks
|
|
||||||
|
|
||||||
- SSH refusal/public trust metadata: `evidence/resume-1372-ssh-trust.json`. No secret/JWT is copied to artifacts.
|
|
||||||
- Existing box `tc-asa3yr` was inspected, not restarted. Executed controllers remain absent, including Python cache contents. Inspection log: `/logs/box/2026-10-31/tc-asa3yr-exec-803562803.stderr`, lines 1–40; the initial directory-list command returned 2 only because optional guessed directories did not exist, next inspection succeeded.
|
|
||||||
- Host worktree fetched origin and merged local `main`: already current at documentation commit `8bf2a3c661ead774689e508ed45f609b40f60dd1`. Initial merge used `target` instead of the typed `rev` argument and was rejected without changes; corrected call succeeded.
|
|
||||||
- JSON/caller/trust/manifest assertions and controller/cache-absence checks passed: `/logs/box/2026-10-31/tc-asa3yr-exec-666665822.stderr`. All 20 retained target file sizes/SHA-256 matched the updated manifest: `/logs/box/2026-10-31/tc-asa3yr-exec-3598104138.stderr`. No executable test/controller was written for these inline checks.
|
|
||||||
- No Rust/plugin behavior changed or rebuilt. Run 1369's forced-recompile workspace pass and builds 1600–1603 remain the relevant gates. Independent plugin recovery follow-ups remain independent.
|
|
||||||
@@ -1,74 +0,0 @@
|
|||||||
# SAU-384 resume — run 1373
|
|
||||||
|
|
||||||
**Ready for review, not live execution.** Authority remains the card/design Rollout and SAU-356 approval gates. No host/session/cred operation, push or web publication was authorized or performed. No shipped host/plugin behavior changed.
|
|
||||||
|
|
||||||
## Actual owner and native package resolved
|
|
||||||
|
|
||||||
After mediator installed the observed host key, reused the approved SSH grant for **read-only** inspection as `surma`. The actual owner is Herdr workspace **w7**, tab **w7:t5**, pane **w7:p6**. Its shell runs `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`. This is not a systemd Sausage unit. The loop uses a mutable remote flake, exits on zero and refreshes/restarts after any nonzero exit (its existing sleep is 2 s). Therefore stopping plugins or requesting core.restart is not a deliberate native stop boundary.
|
|
||||||
|
|
||||||
Citadel is **aarch64-linux**. Observed old package, derivation, original source and wrapper/binary hashes are pinned in `evidence/resume-1373-inspection.json`. Read the old derivation's actual CLI source: SIGTERM awaits `Host.stop`; an already-requested restart can still return 75. Do not reuse observed PIDs as constants.
|
|
||||||
|
|
||||||
Built the exact Nix **release** package from the unchanged card host source:
|
|
||||||
|
|
||||||
- `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`
|
|
||||||
- `bin/.sausage-wrapped` SHA-256 `683623c4c77dbcfff590c9f0aa78f045d07935a466836a099ccc53d7336853cb`
|
|
||||||
- Package NAR and every closure path/hash: `evidence/native-host-manifest.json`. ELF is AArch64 with the pinned Nix glibc interpreter. The closure equals the observed old package's closure except for the host root itself.
|
|
||||||
- Empty caller PATH runtime check **passed**, building/testing an actual WASI plugin with C and the packaged toolchain; it was rebuilt with visible output, not inferred from `--version`.
|
|
||||||
|
|
||||||
The box shares Citadel's Nix store: this build made an inert store path visible there, **not** selected by the live launcher. No live package/configuration/boot/SDK was changed.
|
|
||||||
|
|
||||||
## Native end-to-end and interruption tests
|
|
||||||
|
|
||||||
The disposable live-size world reused the accepted SAU-362 snapshot (1,491 sessions / 580,355,405 bytes). Executed the actual old Nix package and inspected loop, actual old mail/session components and actual target gateway; then:
|
|
||||||
|
|
||||||
1. Recorded old plugin ownership through the real gateway/proc projection.
|
|
||||||
2. External controller terminated the **outer loop first**, awaited its pidfd exit, then SIGTERM'd the exact old host and awaited its exit. The host survived its parent until its own graceful shutdown; controller survived outside the group. One final group inventory was empty; no process polling.
|
|
||||||
3. Backed up frozen state, applied the approved queue/mail losses and cron conversion, injected interruption after operation 3, completed and repeated the hash-guarded offline plan.
|
|
||||||
4. Changed a static external selector from the old to the exact new package. Marked started before launch. Started the same actual loop with explicit `SAUSAGE_FLAKE=path:<selector>`.
|
|
||||||
5. Verified the running OS executable was the new Nix binary; exactly one target manager, no old mail deliverer, locked cred, real subscriber-registration completion. After-start apply/whole-world rollback refused. Marked verified, then stopped cleanly.
|
|
||||||
|
|
||||||
Separate **non-parent controller** test used the reusable one-off `native_freeze.py` positional interface, not Python's parent-only wait(): pidfds pin the old loop/host, boot ID/start ticks/executable/group guard identities, owner exit precedes host stop. It proved interruption after owner exit then resume, stale-PID refusal without stopping the host, successful freeze and harmless frozen rerun. Another real-core test requested **core.restart after loop exit**: host exited and no owner/process in the old group could restart it. These tests close the external-owner gap; earlier run 1369 remains the evidence for armed/exit races, all-watch orchestrator upgrade ordering, and actual gateway rollback.
|
|
||||||
|
|
||||||
Native gate inventory contained 4,434 files. Of the 4,372 non-operation files, 4,370 stayed byte-identical after target activity; the only changes were the expected manager registration file and core journal. Histories/settings/mail/jobs/grants outside intended operations remained equal. Prior offline tests additionally proved exact losses, unknown mail retention, SQL ID/calendar/outcome preservation, grant preservation and rollback before start.
|
|
||||||
|
|
||||||
## Caller-store observation
|
|
||||||
|
|
||||||
Read-only SQLite connections (`mode=ro`, query_only, read transactions) inspected board, cron and rooms; JSON workflow/config stores were also inventoried without printing prompts or grants. See `evidence/resume-1373-inspection.json`:
|
|
||||||
|
|
||||||
- Six legacy cron session templates need the already-tested one-time conversion. Do not execute historical jobs or rewrite RPC words inside their prompts.
|
|
||||||
- Both board's mediator hook **and invariant target** are already target-valid `{session,text}` calls. Preserve their bytes.
|
|
||||||
- Eight historical running board session-attempt snapshots are diagnostic/uncertain, **never replayed** (`hooks.rs` explicitly documents this). Zero board subscriptions/batches were observed.
|
|
||||||
- One abandoned room has no pending turn/review/split. Orchestrator run/skill-review records have no embedded old session/agent call objects; zero host jobs and no live session-manager registrations were observed. Preserve typed workflow records.
|
|
||||||
- Three board runs had no outcome at observation, including this worker. This is old in-flight work, not proof of settlement or permission to replay it.
|
|
||||||
|
|
||||||
The operator must repeat this audit on the **frozen** backup at the approved shutdown. This changing live observation cannot be that future snapshot. Unknown executable references block preflight; no shipped compatibility path was added.
|
|
||||||
|
|
||||||
## Selected review artifacts and live steps
|
|
||||||
|
|
||||||
Selected coherent bundle: **`/docs/symphony/artifacts/SAU-384/cutover-targets.tar.gz`** (24,591,858 bytes), SHA-256 `1bbcae4ef615ca8394b0c53b9a408441a49f8a618f059335c3bc248d958977c4`. The 22 payloads are hash-pinned in `evidence/manifest.json`; bundle also carries that manifest. It contains the native NAR/closure manifest, reviewed plugins, approved web `3d760267`, and SDK **`src/lib.rs` only**. It excludes the earlier diagnostic debug host. Older `target-artifacts.tar.gz` remains immutable prior evidence, not the selected native install.
|
|
||||||
|
|
||||||
[Runbook](runbook.md) now gives actual owner/world/package paths, exact loop-first stop/checkpoint interface, proposed external root `/home/surma/sau-384-cutover` (absent at inspection), pinned selector and restart command, five-entry boot order (session before web; no mail.start), frozen caller conversion, web-release/rollback boundaries and mandatory executable deletion. Surma's go precedes mediator's live core/session/cred operation; every push separately needs go. Only mediator/orchestrator lands/executes; SAU-375 stays publish:hold until session and web are active, then orchestrator.release.
|
|
||||||
|
|
||||||
All five executed one-off/migration/test scripts were **deleted after verified success**; Python bytecode cache was empty. Hashes/deletion proof: `evidence/native-1373/native-tool-deletion.json`. Source is retained **as log only**, never as a shipped plugin or retained migration executable. The live mediator must reconstruct its reviewed temporary copy, execute only after go, and delete it plus executable caches after verification too. Static target selector configuration must remain for the pinned native loop; it is not a migration tool.
|
|
||||||
|
|
||||||
## Exact checks/logs
|
|
||||||
|
|
||||||
All full logs below are `/logs/box/2026-10-31/`:
|
|
||||||
|
|
||||||
- Nix package build/CLI: `tc-asa3yr-exec-1149169379.stderr`; packaged empty-PATH WASI+C gate: `tc-asa3yr-exec-395020656.stderr`.
|
|
||||||
- Actual native old stop/offline interruption/swap/new boot: `tc-asa3yr-exec-3989255528.stderr`.
|
|
||||||
- Non-parent controller interruption/stale-PID/frozen-rerun: `tc-asa3yr-exec-4245550011.stderr`; real core.restart after owner death: `tc-asa3yr-exec-2683606349.stderr`.
|
|
||||||
- Native architecture/closure equivalence/hashes: `tc-asa3yr-exec-1926991205.stderr`.
|
|
||||||
- Repeated `cargo test --workspace`: **pass**, including 208 host integration tests; `tc-asa3yr-exec-844023236.stderr`. All 105 Rust sources matched package/workspace copies. Run 1369's forced-recompile/source equivalence remains the earlier independent gate.
|
|
||||||
- Executed source-as-log and deletion: `tc-asa3yr-exec-2945036769.stderr`; 22 payload hashes/controller absence: `tc-asa3yr-exec-643070322.stderr`.
|
|
||||||
- New inventories/plan/losses/target process records/closure/source equivalence: `evidence/native-1373/`.
|
|
||||||
|
|
||||||
Prior builds remain mail **1600**, cred **1601**, cron **1602**, rooms **1603**; no plugin source changed. Their remote-less repos have no origin (fetch correctly rejected); merging shared local main was already-current for all four. Host fetch/merge was already current before work; final commit/fetch/merge is reported separately.
|
|
||||||
|
|
||||||
Failed approaches were corrected, not blockers: one incorrectly transcribed grant was rejected before any SSH command; the exact attached grant succeeded. zsh wildcard expansion and missing PATH tools were replaced by explicit installed Nix paths. `core.ps` does not exist; the native gate switched to the actual gateway `/fs/proc` projection and passed. Nix store logs were unavailable, so rebuilt the runtime check with visible `-L` output. One evidence-copy Path/string expression failed; corrected and resumed without re-executing migration. Full failure logs remain under /logs.
|
|
||||||
|
|
||||||
## Limits, not silently claimed complete
|
|
||||||
|
|
||||||
No new production-latency guarantee is claimed. Prior post-record real-core live-size samples remain: final manager acknowledgement **0.6078 s**, registration catch-up **0.0128 s**, mail.send-to-default publication **0.5376 s** (integrated **0.5235 s**), below callers' existing **3 s / 10 s** budgets. The integrated catch-up outlier **12.0419 s** remains disclosed. Orchestrator idle whole-host CPU was ~**0.10%** in the paused fixture; prior real cron warning/stop took **305.427 s**, and detached error data passed. These measurements and their synthetic/warm-cache limits are in implementation.md.
|
|
||||||
|
|
||||||
Live shutdown/frozen backup/push/activation/release/publication, physical browser display, power loss and provider cancellation/billing remain unverified/out of scope. Old unrecorded runs are not automatically inferred complete/resumed. Independent per-plugin recovery follow-ups remain independent; the design accepts first-update cuts until they land.
|
|
||||||
@@ -1,56 +0,0 @@
|
|||||||
# SAU-384 review rework — run 1381
|
|
||||||
|
|
||||||
**Ready for review, not live execution.** This changes only local documentation and disposable one-off tooling. No shipped host/plugin behavior, live boot/SDK/launcher, activation, restart, publication, release or push changed. Surma's go still precedes the mediator's live operation and every push.
|
|
||||||
|
|
||||||
## Review findings addressed
|
|
||||||
|
|
||||||
1. **No mediator needed during downtime.** Replaced the disconnected procedures with one external CLI, `driver.py start → arm → detached run → verify`. `start` acknowledges a new OS session and blocks it on an arm FIFO. After arm the detached driver owns native freeze, immutable/resumable full backup, planning/frozen audit, offline apply, selectors, `started`, launch and readiness. It uses the exact pinned Herdr pane/shell, then publishes a **new explicit default verification message**, with a stable UUID, because the cut old mediator run is not recoverable. The last real-Core/Herdr test killed the invoking shell shim, then killed the detached controller at frozen, resumed it, and reached target in the same shell. Actual target run-start and mailbox-consumed records named that new verification message once. A real live SSH launch was not performed; the runbook makes the two finite foreground SSH launch/arm commands, and the required explicit go for that exception to SSH's background-work policy, part of the proposed operator plan.
|
|
||||||
2. **Actual frozen-state gates.** `Kit.plan`, apply, starting and pre-start rollback require the real `freeze.json`, phase frozen, current boot ID, matching world, dead pinned owners, empty old group, no subsequently running host, and unchanged checkpoint bytes. A caller-supplied stopped literal is gone. Captured/wrong-boot checkpoints, running owner, changed config/SQL projection and after-start resume refuse before mutation. Offline retries accept only their saved before/after images.
|
|
||||||
3. **Persistent installation pins.** Driver registers/verifies GC roots for exact old/target/loop packages and creates old/target selector and owner artifacts before freeze, so pre-start recovery already has the explicit old owner. Both wrappers export their own pinned selector. Cold restart in the same Herdr shell, without ambient SAUSAGE_FLAKE, ran the exact target; the explicit old wrapper ran the exact old package on a separate old-backup clone, never over new target work. Runbook prohibits bare loop/remote-default starts and explains that core.restart restarts the pin, not self-updates. Future host updates need an approved package/root/selector change; roots are not removed with migration scripts.
|
|
||||||
4. **Real PTY job control.** Used actual Herdr 0.9.0 and an interactive zsh job-control shell, not setsid+piped stdin alone. Loop was foreground process-group leader in the shell's session. Freeze sends TERM then CONT before pidfd wait. Separate injection left the real host in observed **T** state after outer-owner exit; TERM+CONT completed shutdown with no surviving group member. This exercises the stopped-state hazard; it does not claim every timing of naturally delivered SIGTTIN.
|
|
||||||
5. **Planning, SQLite and scale.** Incomplete backup resumes the same inventory/backup without cleanup or replacement. Streaming hashlib.file_digest avoids whole-file memory reads; os.walk/copy preserve directories and non-followed symlinks (including broken links). Complete DB sidecar families are backed up physically, recovered only in disposable copies, staged with SQLite backup in DELETE mode, and old sidecars removed before main-image replacement. Committed WAL and hot uncommitted rollback-journal fixtures passed interruption/rerun, exact losses, preserved job IDs/calendar/outcomes/generic JSON/unknown mail, and exact pre-start rollback. A full logical-size/count/bucket/directory-matched **synthetic** replica measured the whole-world backup path below; it was not an exact live-byte snapshot.
|
|
||||||
|
|
||||||
## Evidence and exact gates
|
|
||||||
|
|
||||||
Evidence directory: `/docs/symphony/artifacts/SAU-384/evidence/rework-1381/`. It contains only JSON manifests/inventories/checkpoints/results, not migration executables. `sandbox-operator-manifest.json` records the tested invocation; **do not execute it live**. `live-manifest.template.json` deliberately requires fresh identities, typed caller review/guards and a new stable continuation UUID before approval. Frozen guard tests used the real cron table projection; the live table/file set must include the board/config/registration review described in the runbook.
|
|
||||||
|
|
||||||
All logs below are `/logs/box/2026-10-31/`:
|
|
||||||
|
|
||||||
| Gate | Full stderr log / result |
|
|
||||||
|---|---|
|
|
||||||
| Final caller-killed + controller-killed/resumed + interrupted planning + same-Herdr-pane target chain | `tc-asa3yr-exec-775822946.stderr` — pass. 4,436 frozen regular files / 627,157,113 bytes in this session-size integration fixture; backup/hash stage 4.617 s. Earlier final chains also passed (`…-2599916298`, `…-2393656781`, `…-2876439273`). |
|
|
||||||
| Final process ownership/locked cred/new explicit verifier/preservation; stopped-PTY CONT; cold pinned target; explicit old package on old clone | `tc-asa3yr-exec-3942858286.stderr` — pass. `post-target-processes.json`, `post-preservation.json`, `post-continuation-records.json`, `post-pty-stop-continue.json`; only planned paths and enumerated new activity changed. |
|
|
||||||
| WAL + hot journal + interruption + exact rollback, repeated | `tc-asa3yr-exec-1595763713.stderr` — pass; first pass `…-1628589419`. `db-sidecars.json`. |
|
|
||||||
| Whole-world scale | `tc-asa3yr-exec-3620999999.stderr` — **138,444 files, 12,441,162,164 bytes, 45,310 directories**. Inventory **70.555 s**, copy plus double hash verification **135.385 s**, combined **205.940 s**. Exact functions used by Kit.plan; synthetic bytes/names, observed live size buckets/counts, warm-cache box. No cold-storage downtime SLA. |
|
|
||||||
| Current source equivalence + cargo test --workspace | `tc-asa3yr-exec-430706202.stderr` — pass; 121 Rust/Cargo inputs plus root Cargo/flake inputs matched current card tree. **208 host integration tests**, lines **64–431**, plus all workspace suites/doc tests. No Rust host source changed. The final cred main refresh/rebuild and repeated gates are below. |
|
|
||||||
| Negative config/frozen caller controls, selected bundle rehash, verify and deletion | `tc-asa3yr-exec-3977808892.stderr` — pass; `negative-controls.json`, `bundle-recheck.json`, `tool-deletion.json`, `executed-test-deletion.json`. |
|
|
||||||
| Source-as-log reconstruction/hash/AST check; no scripts/bytecode recreated | `tc-asa3yr-exec-3884064136.stderr` — pass; `source-log-roundtrip.json`. |
|
|
||||||
|
|
||||||
### Reconstructable, but deleted after success
|
|
||||||
|
|
||||||
The executed tool was three small files, not a plugin or host patch. Exact source survives **as log only** in `tc-asa3yr-exec-3977808892.stderr`:
|
|
||||||
|
|
||||||
- `cutover_once.py`: lines **7–152**, SHA-256 `202eb7e635d675bb3960317913ae154e4f699fb7f5fe25a53ef34c965d0869c5`;
|
|
||||||
- `freeze.py`: lines **154–226**, SHA-256 `d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe`;
|
|
||||||
- `driver.py`: lines **228–393**, SHA-256 `7a26cfba9873541404f9661d5b0117f566301d408448cbb509a965d7900037aa`.
|
|
||||||
|
|
||||||
The verify CLI was actually executed, marked the rehearsal verified, logged/deleted these files, and removed caches. All executed test controllers were also logged/deleted; the named disposable Herdr server was stopped. Hash reconstruction was checked **in memory**, without recreating executable files. The live runbook mandates the same deletion. Permanent owner/selector artifacts are installation artifacts, represented as bytes/hashes in `permanent-launcher-artifacts.json`, not retained migration code.
|
|
||||||
|
|
||||||
## Final main refresh and selected builds
|
|
||||||
|
|
||||||
The final required merge found cred main advanced to approved **SAU-418, `0db401012cae99b4424670d5c07a346281e30e75`** (plain anyone-may-use switch/shared SSH metadata and locked no-candidate fetch behavior). Inspected that diff and fast-forwarded the card worktree without discarding either side. No storage-schema or handover lifecycle delta was introduced. Rebuilt current cred: **1636 / 274 tests passed**, `/state/cargo/builds/1636/cred.wasm`; full output `/logs/cargo/2026-10-31/01a19281-a8ed-7127-914a-3ba4b5c15d17.{stdout,stderr}.log`.
|
|
||||||
|
|
||||||
Final selection is now **`/docs/symphony/artifacts/SAU-384/cutover-targets-1381.tar.gz`**, **24,590,972 bytes**, SHA-256 **`9295c5fddbf5d68cbfd525602ec67e7dd895bf6afb99324191a6c514282eac57`**. Only the cred payload/provenance changed; all other 21 payloads, native host and SDK selection remain as reviewed. The original bundle remains immutable prior evidence. Current `evidence/manifest.json` and `evidence/rework-1381/main-refresh/selection.json` pin the final bundle. Host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`; SDK-copy **src/lib.rs only**. Mail/cron/rooms remain **1600/1602/1603**; approved web **3d760267** stays held until mediator/orchestrator release. No plugin implementation edits by this worker.
|
|
||||||
|
|
||||||
Reconstructed the same three tool sources from their hash-checked log solely for this required new-main rerun, then deleted them again after verified success. Repeated caller/controller death/resume and actual Herdr cutover with the new cred bundle passed (`tc-asa3yr-exec-629237259.stderr`, same log directory). The first preservation assertion additionally encountered normal core-journal WAL/SHM activity; explicitly classified those journal sidecars as activity rather than migration loss. Final same-pane ownership/locked cred/fresh verifier/preservation/stopped-PTY/cold target/old-clone gates, negative controls and full `cargo test --workspace` all passed **`tc-asa3yr-exec-4223877022.stderr`** (208 host integration tests). Sidecar/rollback tests also repeated successfully. Final verify/source-as-log/deletion proof: **`tc-asa3yr-exec-741401146.stderr`**, **`evidence/rework-1381/main-refresh/{tool-deletion,executed-test-deletion}.json`**. Tool source hashes are unchanged from the reconstruction list above; no executable/cache remains. Final live template and sandbox manifests are under that `main-refresh/` evidence directory. Earlier replay/race/cron/watch/scale gates remain cumulative evidence; they were not silently relabeled as freshly rerun.
|
|
||||||
|
|
||||||
## Failures, corrections and limits
|
|
||||||
|
|
||||||
- Whole-tree transfer through box.put_tree rejected the 64 MiB service cap for repos/plugins/logs. Next method: read-only host size histogram, then regular-file size-bucket replica and the exact backup routines. This measures whole-world **scale**, not original live contents/path topology. The original accepted session byte snapshot still supplies the real-history integration volume.
|
|
||||||
- Initial PTY fixtures lacked state/core for console history, then matched an http-prefixed address against a bare address. Corrected fixture and barrier. Typeahead from two early console commands lost the second; final driver queues **one** subscription command. An early Herdr multiline output matcher stalled; inspection showed Herdr wait-output itself polls, so final code removed it entirely in favor of inotify completion. The actual registration path is `state/session/manager/registrations/subscriptions.json`, now corrected.
|
|
||||||
- **Shared-store foot gun:** an early default-store root query pruned **1,144 stale auto-root links and 571 stale temporary-root files** in shared Nix state. This was not an authorized live-config operation and no GC/package deletion was requested. Immediately stopped that method. Read-only SSH inspection on Citadel found **none** of those referenced targets and **none** of those PID numbers existing on the host, so no active roots required restoration. All subsequent root tests used a **private local Nix state database**, loaded from read-only closure metadata; no shared root query was repeated. Exact evidence/log: `gc-root-incident.json`, `tc-asa3yr-exec-2216542428.stderr`. This incidental stale-registration cleanup is disclosed rather than claiming literally zero host-state writes.
|
|
||||||
- One post-start assertion read history before the new verification message was consumed. Replaced it with arm-then-scan/history-write completion. In the final gate publication→run-start was **23.753 s**, →consumption **24.530 s**, queued behind startup reconciliation. This is **not** mail.send publication latency. Initial acknowledgement is not full scan completion.
|
|
||||||
|
|
||||||
Prior accepted replay/race/rollback/deadline/grace/crash-supervision/event-driven orchestrator evidence remains unchanged. Post-record manager ack **0.6078 s**, catch-up **0.0128 s**, mail publication **0.5376 s** / integrated **0.5235 s** remain below existing **3 s / 10 s** publication budgets; integrated catch-up outlier **12.0419 s**, ~0.10% idle duty, detached error propagation and **305.427 s** real cron warning/stop remain disclosed in implementation.md. This rework does not replace those measurements with a claim that the initial scan is fast.
|
|
||||||
|
|
||||||
Still unverified/out of scope: actual approved live cut-over/frozen backup and complete production caller guards, live SSH detach/Herdr w7:p6 execution, orchestrator.release/publication/physical browser, cold-storage timing, power loss, provider cancellation/billing and independent recovery follow-ups. The live launch exception needs explicit Surma go as part of the proposed plan; no such approval is claimed or requested by executing anything here.
|
|
||||||
@@ -1,53 +0,0 @@
|
|||||||
# SAU-384 round 3 — run 1389
|
|
||||||
|
|
||||||
Ready for review; **not authorization for live execution**. Addresses `/docs/symphony/artifacts/SAU-384/extra-round-3.md` and review 1388. Only documentation and disposable tooling changed; no transition code in any shipped plugin/host.
|
|
||||||
|
|
||||||
## Reviewed selection and proposed plan
|
|
||||||
|
|
||||||
- Selected bundle: `cutover-targets-1389.tar.gz`, **27,271,968 bytes**, SHA-256 **31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b**. Embedded manifest SHA-256 **b48f2dacda3f348a67574767c6accfae1f1102227f94f86868ec1436bd4df394**; all **23** payload sizes/hashes checked. Prior bundles are immutable historical evidence.
|
|
||||||
- New deferred payload: approved git **9cc9fa6f1c9f01ce1a53d8042623bde60e026637**, `/state/orchestrator/builds/git/9cc9fa6f1c9f01ce1a53d8042623bde60e026637.wasm` (177 tests, not rebuilt here). Staged world copy: `/docs/symphony/artifacts/SAU-384/git-9cc9fa6f.wasm`. Both manifest `deferred_plugins` and operator `deferred_artifacts` are hash-checked before freeze. **Not installed offline**: mediator activates it only after target cred is live and Surma unlocks through the UI. The runbook uses the real `core.activate` **wasm** argument and includes the activation check before web release/final deletion.
|
|
||||||
- Cred remains approved **0db40101**, build **1636 / 274 tests**. Mail/cron/rooms remain **1600/1602/1603**. No new plugin build/source edit this run. Native host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, source e09c4f9f; old host mggvibk8 and loop pz2b94a2 remain pinned. SDK-copy list: **src/lib.rs only**. Web remains approved **3d760267**, release held until session/web/verification prerequisites.
|
|
||||||
- Exact external commands, operator/approval boundaries, checkpoint archive/retry and rollback rules: [runbook.md](runbook.md). Live manifest template: `evidence/round3-1389/live-manifest.template.json`. Resolve fresh identities/caller guards before go; sandbox manifests are evidence, not live instructions.
|
|
||||||
- **Expected live Nix side effect requiring inclusion in Surma's go:** three `nix-store --query --roots` calls, for old host, target host and loop, may prune stale auto-root and temporary-root registrations. They do not delete store packages. Five root registrations (those three plus controller/Herdr) remain in the proposed operation. This round used only the **private rehearsal Nix DB**; no shared/live roots query was repeated. Prior incident remains disclosed in rework-1381.md.
|
|
||||||
|
|
||||||
## Six findings fixed
|
|
||||||
|
|
||||||
1. **Git and interim compatibility.** Actual live old WASM (reported source f089b56d; SHA-256 4b460bce9b67552e223bb209591009056d409b77290182f6cf142adc8c4c97f4) ran against real new cred in a separate synthetic world. Explicit-grant SSH clone authenticated with a real loopback sshd, recorded exactly one sign use and fetched the synthetic commit. Locked use returned `vault_locked`. Old grant-free shared SSH still fails requesting a local key; old grant-free HTTPS retains its old config/anonymous behavior, not the new sharing switch (source checked). Old granted HTTPS reached/was accepted by new cred and reserved one use, then the deliberately closed loopback destination returned `network_error` plus capture; **successful HTTPS transfer is not claimed**. After actual synthetic unlock and new-git activation, shared grant-free SSH clone succeeded. Do not schedule Git network work in the locked/old-git interval, introduce key/config-secret workarounds, or blindly retry an already-started caller's errors. Local status/commit/main merges are unaffected. Live secret-backed operations/pushes remain untested and separately approval-gated.
|
|
||||||
2. **Pre-start abort and human operator.** `check`/start, arm and immediate pre-freeze gates inspect consistent live SQLite backup-API copies, special file types, caller/file/artifact guards and free space. Space budget derives from current regular-file logical bytes plus the actual staged plugin/prev/SDK and recovery/stage/audit DB copies; no arbitrary multiplier/headroom/default. Growth or later ENOSPC is still possible. Driver now has a real `abort` CLI: exclusively lock, finish freeze if begun, restore only approved before/after paths from the complete immutable backup, verify full frozen inventory, then checkpoint-before-launch the explicit old owner in the same pane. No completed plan means apply never began: do not restore an incomplete backup. Restore is resumable; after `old_start_attempted` inspect instead of launching twice. Surma is the named external operator, with independent terminal log monitoring, `failure.json`, and best-effort Herdr notification (submission result retained; physical display not claimed). Runbook gives exact resume/abort and evidence-archive/re-resolution commands. After target `started`, automatic resume/restore/relaunch stays forbidden. Missing registration now exits on one absolute **120 s** deadline from cron/src/types.rs's existing registration/publication budget and the earlier real-core fixture bound; Herdr/RPC calls are finite too. No polling for quiescence or availability.
|
|
||||||
3. **Executable-only SQL guards.** Cron projects id/method/params_json/cron/at/timezone/detach/call_timeout_secs, not next/last/result/occurrence bookkeeping. Board excludes last_sent/ended and batch IDs/lines/due; unions active subscription config with pending config snapshots so creating/delivering the same active configuration is not drift. New/changed executable configuration still refuses. Final automatic subscription removal changes the executable set and requires refresh or pre-start abort, explicitly documented. Add any planned mediator cron wake-up **before** resolving/approving operator.json. No caller/boot/plugin/SDK edits between resolution and freeze; arm checks again. No in-progress manifest edits.
|
|
||||||
4. **prev contract.** Every offline-swapped plugin's prev is now its exact **frozen pre-cut current**, under before/after guards and restore. Tests checked all nine, including paths where an older prev existed. Git waits for normal later activation/prev behavior. This makes the fallback image meaningful, **not** a license for blind incompatible individual `core.revert`; coordinated rollback and unsubscribe-before-session-revert remain mandatory.
|
|
||||||
5. **Roots side effect.** Listed above and in the live runbook/approval plan, rather than describing roots queries as mutation-free. Subsequent tests stayed private.
|
|
||||||
6. **Arm race.** Replaced reader-dependent FIFO with atomic, manifest-bound **arm.json** and an arm-then-scan inotify watch. Idempotent arm returns with no reader. `resume` accepts waiting_for_arm; a kill after marker observation but before saving armed reuses the same marker. No manual FIFO writer/cleanup is needed.
|
|
||||||
|
|
||||||
## Executed gates and full logs
|
|
||||||
|
|
||||||
Prefix: `/logs/box/2026-10-31/tc-asa3yr-exec-`.
|
|
||||||
|
|
||||||
| Gate | Log / outcome |
|
|
||||||
|---|---|
|
|
||||||
| Final exact tool, caller killed + controller killed/resumed, incomplete backup resume, real Herdr job-control stop/swap/start, same shell, post-start preservation/new verifier, prev hashes, pinned cold restart and old clone | **3699755631.stderr**, pass. Final backup 4,436 files / 627,157,113 bytes, 4.559 s (session-size integration, not full-world downtime). No old mail owner; one manager; cred locked. |
|
|
||||||
| Live special-file/zero-space negative, mutable cron vs changed payload, pre-arm drift, frozen SQL mismatch -> real-driver abort/old relaunch, consumed-arm/save kill/resume, post-start abort refusal | **1485773773.stderr**, pass. Zero-space gate used a test-only mocked disk-usage response; actual ENOSPC tested below. |
|
|
||||||
| Real-driver partial swap -> abort/restore/old relaunch; actual backup ENOSPC on a private tmpfs introduced after good preflight -> untouched-old relaunch; lost console subscription -> bounded clean exit | **4142682240.stderr**, pass. Registration wait 120 s; elapsed 132.493 s including backup/launch. Started checkpoint refused both resume and abort. |
|
|
||||||
| Real old/new Git + new cred, loopback synthetic OpenSSH/HTTP negative, unlock then deferred activation | **99781693.stderr**, pass. No live key/grant/remote used. |
|
|
||||||
| SQLite WAL + genuinely hot rollback journal, exact losses, preserved job IDs/calendar/outcomes/generic JSON/uncertain mail, apply interruption/rerun and exact physical restore | **1727732043.stderr**, pass. |
|
|
||||||
| Board executable projections and old-select-star negative control; template/deferred-artifact checks; final driver's actual Herdr failure notification + abort | **1276236846.stderr**, pass. |
|
|
||||||
| `cargo test --workspace` against unchanged production source | **2259546796.stderr**, pass: **208 host integration tests, lines 62–431**, plus all workspace suites/doc tests. 133 Rust/Cargo/flake inputs matched card tree. No Rust implementation change. |
|
|
||||||
| Executed verify mode, complete source-as-log and deletion | **2013747101.stderr**, pass. See hashes/ranges below. |
|
|
||||||
|
|
||||||
Prior accepted race/replay/deadline/core-watch/upgrade/idle/grace/crash-supervision evidence is cumulative, not silently rerun or claimed fixed here. Prior live-size scale sample remains **138,444 files / 12,441,162,164 bytes / 45,310 directories**, synthetic size-bucket match, 205.940 s inventory+copy/hash, warm cache. Post-record manager ack 0.6078 s, registration catch-up 0.0128 s, mail publication 0.5376 s (integrated 0.5235 s) remain under caller publication budgets **3 s orchestrator / 10 s cred**; catch-up outlier 12.0419 s and initial-verifier startup scan delay 23.753/24.530 s remain disclosed. No production downtime/latency guarantee.
|
|
||||||
|
|
||||||
## Deleted executable and reconstruction receipt
|
|
||||||
|
|
||||||
Only source-as-log remains in **2013747101.stderr**. Reconstruct exactly three files for the later approved operation, hash-check before use; do not reconstruct test controllers:
|
|
||||||
|
|
||||||
- cutover_once.py: lines **2–164**, SHA-256 **3adf65ebb0b3401a0d1976524f35f7450e907ad32f907dcbb1cd2f7e0a7e5d3a**;
|
|
||||||
- freeze.py: lines **166–238**, SHA-256 **d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe**;
|
|
||||||
- driver.py: lines **240–503**, SHA-256 **686167318c84a32c1010daf4517f036bcb654384cadab987fa47fb13deecd2bf**.
|
|
||||||
|
|
||||||
`verify` actually logged/deleted the executable after successful rehearsal. Executed test/patch/controllers and test launchers were logged/deleted too, with bytecode caches removed. `tool-deletion.json` and `executed-test-deletion.json` are under `evidence/round3-1389/`. Permanent pinned owner/selector configuration is not migration code; it stays rooted through the recovery window. The live runbook still requires mediator deletion after verified success.
|
|
||||||
|
|
||||||
## Failures and limits
|
|
||||||
|
|
||||||
Tried next methods, not blockers: initial generated SQL fixture had quoting syntax wrong (corrected); generated lockfiles made whole-directory diff reject (compared actual Rust/Cargo/flake input bytes, then ran full suite); final-sidecar extraction included a trailing log status (trimmed); a post-check initially read the previous round's stopped-world manifest (corrected to this round, then passed). Synthetic SSH setup needed explicit typed key metadata, a sandbox-only sshd account and an unlocked sandbox login account; default root was locked. A failed fixture left two loopback sshd listeners, later explicitly identified/stopped. Initial sandbox activation used `path`; core required `wasm`, corrected and the complete final Git gate passed. No failed test was relabeled successful; full errors remain in logs.
|
|
||||||
|
|
||||||
Still unverified/out of scope: approved live cut-over/frozen caller snapshot, actual live SSH-detach/pane w7:p6 execution, Surma's real unlock/new-git live networking, physical Herdr notification/browser display, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. No live activation/restart/config/owner change, push, release, deploy or /www publication this run. Surma's go is still required before mediator core/session/cred operation, detached SSH exception, roots side effects and **every push**.
|
|
||||||
@@ -1,60 +0,0 @@
|
|||||||
# SAU-384 round 4 — run 1392
|
|
||||||
|
|
||||||
Ready for review; **not live execution approval**. Addresses all five findings in extra-round-4.md. No shipped Rust/plugin behavior changed. Current-main source and state contracts were rechecked; all work was in the card tree, review artifacts and disposable box tc-zg0j56.
|
|
||||||
|
|
||||||
## Selection and operator plan
|
|
||||||
|
|
||||||
The selected bundle is unchanged: `cutover-targets-1389.tar.gz`, **31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b**, 23 payloads. All payload sizes/hashes and the native closure hashes were freshly checked. Host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0` from e09c4f9f. Cred remains 0db40101 / build 1636 (274 tests); mail/cron/rooms remain 1600/1602/1603. Deferred git remains approved 9cc9fa6f, activated only after target cred is live and Surma unlocks. No new WASM/package build. SDK-copy list: **src/lib.rs only**. Approved web 3d760267 remains held until the operator releases SAU-375 after session/web and verification prerequisites.
|
|
||||||
|
|
||||||
Exact commands and approval boundaries are in [runbook.md](runbook.md). Current non-executable operator template: `evidence/round4-1392/live-manifest.template.json`. It still requires fresh caller/owner identities, guards and explicit Surma go. The only executable source is reconstructable **as log**, not retained as a tool. Surma is the external human recovery operator. The old mediator is deliberately cut; the driver publishes a new explicit default verification input, not inferred recovery.
|
|
||||||
|
|
||||||
## Five findings fixed
|
|
||||||
|
|
||||||
1. **Shared-filesystem ENOSPC.** Space preflight measures POSIX allocated blocks for files/directories/root/non-followed links, and charges dense filesystem-fragment-rounded size where copy2 would expand a sparse file. It includes the concrete staging/prev/SDK/DB-copy roles; receipts save all measurements. Removed the false never-underestimates claim: metadata/normalization/growth/competing writers remain uncertainty. A separate sparse/tiny/directory fixture copied **4,599,808 allocated bytes** from **12,288 source-allocated bytes**, below its measured **4,603,904-byte** budget. No live margin/default was invented.
|
|
||||||
|
|
||||||
Mandatory foreground `driver.py monitor` receives memory-only Unix datagrams for inventory/backup progress and failure, and kernel pidfd controller-exit completion. Every start/resume waits for its own monitor; arm requires it. Failure JSON/log writes are best effort and cannot prevent CUTOVER FAILED or the independent Herdr notification attempt. A terminal still reports controller death on SIGKILL; no heartbeat/polling loop. Notification submission was tested, but returned shown:false/disabled; physical display is not claimed.
|
|
||||||
|
|
||||||
Actual ENOSPC used **one shared tmpfs for world, external root and driver.log**, device 83, **0 free bytes**, capacity 254,504,960 bytes derived from measured fixture/budget allocations. It failed during a real backup write after good preflight. Monitor showed failure and death despite unwritable disk evidence. Pre-plan abort removes only the reproducible incomplete backup subtree before any checkpoint write, retaining planning/inventory evidence. If the competing writer still fills the disk, it refuses before saving/launching. After that known test file was freed, the same abort restored coherent old ownership and readiness. Complete backups are never deleted. Every abort (with or without a completed plan) now checks space before its first checkpoint save; a second shared-filesystem zero-free-byte fixture proved the complete backup/checkpoint remained exact, then old relaunch worked after freeing only the known competing test file. Space is checked again before old relaunch. This is not proof against every filesystem's ENOSPC/power-loss behavior.
|
|
||||||
2. **Post-start forward completion.** New `continue operator.json` locks, checks the bound manifest, exact pane/shell, exactly one target host and target plugin hashes, and performs only missing/read-retry-safe post-start RPCs. It never launches/applies/restores. Actual locked cred.status is saved durably before unlock; subscribe uses the stable cutover registration; send reuses the exact continuation UUID/text. It reaches awaiting_verification, then normal verify/delete is reachable. Real failures of **cred.status, subscribe, and send**, including send committed before its acknowledgement was lost, all recovered with one run-start/consumption. A real **120 s** absolute registration timeout recovered the same way without another owner. The value remains cron/src/types.rs's existing registration/publication bound. Runbook gives exact continue command, RPC payload derivation, locked-proof/unlock ordering and local Surma fallback if live SSH signing is unavailable.
|
|
||||||
3. **Atomic temporary cleanup.** A child was SIGKILLed inside atomic after writing a real partial plugin replacement temporary, before rename. Abort removes only a saved operation's newly created .cutover-tmp, absent in frozen inventory and prefix-matching an approved before/after image; unknown bytes/links refuse. Full frozen inventory equality passed inside rollback before old launch. Resume still overwrites/completes its own temporary. No arbitrary world cleanup.
|
|
||||||
4. **Converter pre-freeze dry run.** check/start, arm and immediate pre-freeze checks use the same pure cron converter as plan, on SQLite backup-API memory copies. interrupt:true, invalid text and non-object params refuse without freeze/mutation. Negative control: round-3 checker accepted the same invalid object payload, new checker refused. No compatibility code in a plugin.
|
|
||||||
5. **Roots/wake-up documentation.** Every controller resume repeats all three roots queries and their possible stale-registration pruning; this remains part of the proposed Surma-approved live plan. This round queried roots only against a private rehearsal Nix DB. No cron wake-up is needed: the stable explicit driver continuation owns handoff. A due-during-downtime job is skipped; a just-before-freeze input can enter an old pending queue that is intentionally dropped. Neither is replayed. The runbook disallows relying on guessed backup duration as readiness.
|
|
||||||
|
|
||||||
Testing also exposed a real **Nix Python wrapper exec identity race** in parent-side process capture. The launcher now receives the child's actual post-wrapper identity over a pipe-completion handshake. Controller kill/resume proved exact identity, a fresh monitor requirement and successful same-pane continuation. No liveness/endpoint polling was added.
|
|
||||||
|
|
||||||
## Full logs and evidence
|
|
||||||
|
|
||||||
All log names below have prefix `/logs/box/2026-10-31/tc-zg0j56-exec-`. Evidence is `evidence/round4-1392/`.
|
|
||||||
|
|
||||||
| Gate | Full stderr / outcome |
|
|
||||||
|---|---|
|
|
||||||
| Final shared world/root/log ENOSPC, pre-save incomplete-backup deletion, low-space refusal, old readiness; partial atomic SIGKILL/abort; all three post-start RPC failures; real registration timeout | **166388571.stderr**, clean combined pass on final tool source. Evidence under `final/`, including shared-enospc.json, atomic-abort.json, *-reviewed.json and registration-timeout.json. |
|
|
||||||
| Complete-plan ENOSPC on one shared filesystem: pre-write refusal, checkpoint/full-backup equality, old readiness after known competing-data cleanup | **4094731865.stderr**, pass; `final/complete-plan-enospc.json`. |
|
|
||||||
| Controller wrapper handshake, killed-controller pidfd, resume waits for fresh monitor, target completion | **537908850.stderr**, pass; `resume-handshake.json`, both monitor logs. |
|
|
||||||
| Converter pre-freeze refusals | **1954198187.stderr**, pass; `dryrun.json`. |
|
|
||||||
| Negative control + sparse allocation + all payload/native closure hashes + **133** final source/build inputs | **1314968883.stderr**, pass; `converter-negative-control.json`, `allocated-space.json`, `source-and-bundle.json`. |
|
|
||||||
| Committed WAL/hot rollback journal; exact pending/cancel/mail losses; preserved mail/jobs/IDs/calendar/outcomes/non-target JSON; apply interruption/rerun and physical rollback | **2622539348.stderr**, pass; `sqlite-sidecars.json`. |
|
|
||||||
| cargo test --workspace, unchanged final production source | **3315145279.stderr**, pass; **208 host integration tests**, result at line **433**, plus all workspace suites/doc tests. |
|
|
||||||
| Fresh target one-manager/locked-cred/original-byte preservation/pre-cut-prev verification; actual verify CLI and complete tool/test deletion | **3030415406.stderr**, pass; `final/` final-ownership-preservation.json, verification.json, final-verified-driver.json, tool-deletion.json, test-deletion.json. Earlier verify/deletion pass: 1403220230.stderr. |
|
|
||||||
| In-memory source-log SHA/AST reconstruction; executables/bytecode still absent | **2019074395.stderr**, pass; `final/source-log-roundtrip.json`. Earlier docs/JSON/absence checks: **2041020851.stderr**; final docs/JSON/absence/23-payload/133-source-input validation: **1011452644.stderr**, `final/final-validation.json`. |
|
|
||||||
|
|
||||||
### Executed source deleted after verified success
|
|
||||||
|
|
||||||
Reconstruct only these three files from **3030415406.stderr** for a later approved live operation; do not reconstruct tests:
|
|
||||||
|
|
||||||
- cutover_once.py: source lines **2–189**, SHA-256 **84b13644bd80c6b331dcd8df9bf79339f62fedb99f7dd376535b2bca7b0afd96**;
|
|
||||||
- freeze.py: source lines **192–263**, SHA-256 **d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe**;
|
|
||||||
- driver.py: source lines **266–669**, SHA-256 **5c3cba1a69b8b5e8f82ea83caa0697af494bf93dd0297f0c1b5231192cbb5fad**.
|
|
||||||
|
|
||||||
After generalizing the no-write space gate to complete-plan and untouched abort too, the source was reconstructed for the final affected rerun, then the real verify command again marked that rehearsal verified, logged/deleted the three files and caches; all executed/reconstructed tests/patches/wrapper were logged/deleted too. Permanent owner/selector installation artifacts are distinct, not migration code. Live runbook still mandates deletion after verified success.
|
|
||||||
|
|
||||||
## Failed approaches and limits
|
|
||||||
|
|
||||||
- First interpreter alias linked only bin/, losing Nix Python's site-package context. Linked the complete existing interpreter environment; imports and gates then passed.
|
|
||||||
- Cold workspace cache made six offline guest builds miss async-task; populated the disposable cache and reran the **whole workspace**, 208/208 host tests, unchanged source. Initial full log **2236066973.stderr**, missing dependency at lines 755–785; final pass above. No retry-without-diagnosis claim.
|
|
||||||
- First shared filesystem test refused **before freeze** because actual source growth exceeded its provisioned measurement. Recorded that valid negative control, then provisioned the test from measured fixture/budget roles before inducing deliberate post-preflight pressure. A later test filled before the backup had copied any data: abort correctly needed a no-write space check before its first save; fixed and repeated with a before-backup event gate. Final shared test reached zero free bytes during copy and the final combined gate passed cleanly. Earlier fixture checkpoint-name collisions, an existing evidence-export directory and a busy mount affected cleanup/export only; used fresh identities/paths and explicit disposable-pane cleanup, never edited migration checkpoints. Mounts were left for disposable-box teardown, not live cleanup.
|
|
||||||
- First post-start fixture omitted session prompt files, so it correctly recorded run-end error rather than mailbox consumption. Added synthetic prompts and repeated all three failure modes from fresh worlds; each then produced exactly one consumed continuation.
|
|
||||||
|
|
||||||
This round's failure fixtures are **small synthetic real-core/Herdr worlds**, not another live-size benchmark. Prior accepted live-size race/replay/core-watch/upgrade/deadline/idle/grace/crash-supervision, scale and Git compatibility/unlocked activation evidence remain cumulative. Startup 0.6078 s, catch-up 0.0128 s, mail publication 0.5376 s vs callers' 3 s/10 s limits, catch-up outlier 12.0419 s and initial-scan delay remain disclosed in earlier reports; not remeasured or upgraded to guarantees here.
|
|
||||||
|
|
||||||
Unverified/out of scope: actual approved live cut-over/frozen caller audit, live detached SSH/Herdr w7:p6 execution, live SSH signing through target cred, Surma's real unlock/new-git external networking, physical notifications/browser, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. Surma's external terminal has exact continue/verify commands, so final completion is not dependent on the untested live signer. No push, live activation/restart/config/owner change, release, deployment or /www publication. Surma go still precedes mediator core/session/cred operation, the proposed detached-SSH exception/root side effects, and **every push**.
|
|
||||||
@@ -1,234 +0,0 @@
|
|||||||
# One-off handover runbook — rehearsed; live approval required
|
|
||||||
|
|
||||||
Authority: [design / Rollout](/docs/design/plugin-handover/design.md), [SAU-356 gates](/docs/symphony/artifacts/SAU-356/plan.md). Results: [initial](implementation.md), [races/rollback](resume-1369.md), [native package](resume-1373.md), [external driver rework](rework-1381.md), [round-3 recovery/git gates](rework-1389.md), [round-4 failure paths](rework-1392.md). This is a proposed live plan, **not authorization**. Workers never execute it. Mediator/orchestrator lands and executes only after Surma's go; every push separately needs a fresh go.
|
|
||||||
|
|
||||||
## 0. Prepare everything while the old world is still available
|
|
||||||
|
|
||||||
Do not begin downtime with scripts, caller decisions or launch instructions still to be assembled.
|
|
||||||
|
|
||||||
### Reviewed selection
|
|
||||||
|
|
||||||
- Bundle: `/docs/symphony/artifacts/SAU-384/cutover-targets-1389.tar.gz`, SHA-256 `31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b`. It includes final-main cred `0db40101`, build **1636 / 274 tests**, preserving approved SAU-418 instead of downgrading it. Git **9cc9fa6f1c9f01ce1a53d8042623bde60e026637**, approved SAU-418 artifact `/state/orchestrator/builds/git/9cc9fa6f1c9f01ce1a53d8042623bde60e026637.wasm` (177 tests), is included as **deferred_plugins/deferred_artifacts**, not an offline replacement. It activates only after target cred is live AND Surma unlocks it in the UI. Earlier bundles are immutable historical evidence, not this selection. Verify all **23** payloads (including deferred git) against its embedded `manifest.json`. `target-artifacts.tar.gz` is older debug-host evidence, **not** this installation.
|
|
||||||
- Native host: `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, aarch64-linux Nix release package from unchanged host source `e09c4f9f`. The embedded `native-host-manifest.json` pins the NAR, binary and complete runtime closure. Import the reviewed NAR if needed; never overwrite immutable store files or substitute a diagnostic host. Import/deployment is an operator action, never a worker action.
|
|
||||||
- Old package for this observed baseline: `/nix/store/mggvibk8lcgv2qw28jimrl60mi0qm41p-sausage-0.1.0`. Re-identify it and the owner before go; observed PIDs are not constants.
|
|
||||||
- Actual owner: Herdr **w7:p6**, tab w7:t5, not a systemd Sausage unit. Its loop is `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`. Read `herdr pane process-info --pane w7:p6` once to resolve current shell/loop/host identities. Do not close Herdr or unrelated panes.
|
|
||||||
- SDK copy list is **`src/lib.rs` only**. Preserve world-owned Cargo manifests, relocated `interface`, imports and all other SDK bytes. Resolve a changed baseline before proceeding.
|
|
||||||
- Web is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f` (SAU-375), held by `publish:hold`. Do not release it before target session and web are active.
|
|
||||||
|
|
||||||
### External driver and resolved operator manifest
|
|
||||||
|
|
||||||
Use external root `/home/surma/sau-384-cutover`, outside the world and stopped process group. Stage the extracted bundle under `targets/`. Reconstruct **only** `cutover_once.py`, `freeze.py`, and `driver.py` under `tool/` from the source-as-log headings in `/logs/box/2026-10-31/tc-zg0j56-exec-3030415406.stderr`; check their hashes against `evidence/round4-1392/final/tool-deletion.json`. These are the executed round-4 versions (source lines 2–189, 192–263, 266–669 respectively), not earlier scripts. Hashes/ranges are in evidence/round4-1392/final/source-log-roundtrip.json. They have a real CLI entry point. Do not reconstruct rehearsal/test controllers.
|
|
||||||
|
|
||||||
The tested interpreter is `/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3`, with **inotify-simple 2.0.1**. Resolve its closure availability before go. After go and before starting the driver, pin its package and the inspected Herdr package with host-visible GC roots for the operation (`nix-store --realise <exact-package> --add-root <external-root>/gc-controller --indirect`, and equivalently `gc-herdr`). These root writes/imports are included in the approved live plan, not preparation actions authorized to a worker. The driver itself registers and verifies **old host, target host and loop** roots at `<root>/gc-{old,target,loop}` before freeze. Use the **host namespace**, not a box talking to shared Nix state: `/rework` roots in a box are not host-visible. Do not run GC as a test. Keep old/target/loop roots and both selectors for recovery after tool deletion.
|
|
||||||
|
|
||||||
Start from `evidence/round4-1392/live-manifest.template.json`, a **non-executable template**, and resolve every placeholder before approval:
|
|
||||||
|
|
||||||
- native paths, Herdr binary/context, pane and current shell/loop/host PIDs;
|
|
||||||
- external root, world, bundle, embedded manifest SHA, offline artifact paths **and deferred git**, SDK path, exact target boot list; root and world are on the same filesystem in this plan; resolve a different storage topology before go;
|
|
||||||
- complete old in-flight `/proc/*/info.json` and native-owner inventory, plus uncertain external effects; a finished-looking assistant tail is not settlement;
|
|
||||||
- a new continuation UUID, generated **once**, and the explicit new mediator verification instruction. Keep that UUID on all retries;
|
|
||||||
- typed executable-caller review and its frozen guards below.
|
|
||||||
|
|
||||||
Do not copy sandbox PIDs, `gc_store`, or any `test_*` gate into a live manifest. The template deliberately omits those test-only fields. Record/hash the fully resolved `operator.json`; driver checkpoints bind to its exact bytes. Resolve a schema/selection discrepancy while the old world is available, not during downtime.
|
|
||||||
|
|
||||||
### Executable caller review and frozen guards
|
|
||||||
|
|
||||||
Use predecessor formats: session source `f9122585` Paths/queue; cron `ba75c102` and current cron README one-off procedure. Do not substring-rewrite prompts or diagnostics. Review current stores as in `resume-1373.md`, then encode the **approved** file hashes and deterministic SQL projections in `caller_file_guards` and `caller_sql_guards`:
|
|
||||||
|
|
||||||
- cron `state/cron/jobs.sqlite`: use the template's ordered `id, json_extract(data,...)` projection of **method, params_json, cron, at, timezone, detach, call_timeout_secs**. Do not hash next_run, last_run, last_result_json or last_occurrence_id;
|
|
||||||
- board `state/board/board.sqlite`: ordered `id,data` for hooks/invariant_target; `id,target,session,data` for subscriptions; and `select id,data from subscriptions union select subscription,data from subscription_batches order by 1,2` for executable subscription snapshots. Ignore last_sent/ended, transient batch IDs/lines/due; a pending delivery of the same active configuration does not change this union. Historical hook-attempt snapshots remain diagnostics and **must not replay**. Automatic final subscription removal changes the executable set; resolve/refresh before arm, or take the pre-start abort path if it happens after arm;
|
|
||||||
- boot and other discovered executable configuration/manager registrations. Review typed room/workflow/orchestrator stores too; preserve their histories/settings and non-executable records. Do not mistake prompt text for an RPC template.
|
|
||||||
|
|
||||||
A SQL guard hashes `sha256(json.dumps(rows,separators=(',',':')).encode())`, where `rows = connection.execute(sql).fetchall()` in the approved ordered projection. JSON strings inside rows retain their exact bytes. These projections, including all column names, live in the current template and were tested against mutable cron/board bookkeeping. Any genuinely new executable store/field still needs review; do not silently omit it. The observed live baseline had **six cron templates to convert**, already-target-valid board hook/invariant calls, no subscription batches and no live manager registrations. This is an observation, not a future frozen snapshot. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
|
|
||||||
|
|
||||||
**Before start, again before arm, and once immediately before freeze**, `check`/driver checks artifact and file guards, SQL projections on consistent live SQLite backup-API copies, special-file types (without hashing mutable bytes), space, and a **dry run of the same cron converter used by plan**. Object/text/interrupt refusals therefore occur before downtime, not just in plan. The dry run never rewrites the live DB. After freeze, guards run on disposable complete DB copies before apply; immutable backup evidence is never opened/recovered in SQLite.
|
|
||||||
|
|
||||||
**Space is an allocated-block measurement, not a guarantee.** Root/world must have the same filesystem device in this reviewed layout. Walk regular files, directories (including the root) and non-followed symlinks; charge each at the larger of POSIX `st_blocks × 512` and its logical size rounded up to that filesystem's `f_frsize`. This includes sparse-file expansion by copy2, not just logical file bytes. Add the concrete staged plugin/prev/SDK allocations and the three existing DB-copy roles (recovery, staging, audit). Save source allocation, backup allocation budget, directory/file counts, fragment size and required/free bytes in `preflight-live.json`; refuse before freeze unless free covers that budget. The fresh sparse/tiny-file/directory fixture measured 12,288 source-allocated bytes and 4,599,808 copied bytes against a 4,603,904-byte budget (see evidence/round4-1392/allocated-space.json for actual measured values). **No claim that this can never underestimate:** metadata, DB normalization, growth, changed mounts and competing writers can still exhaust space. Resolve storage uncertainty in the approved maintenance plan, not with an invented margin.
|
|
||||||
|
|
||||||
Add any genuinely new executable store to the manifest before arm. From resolution until freeze: no caller/boot/SDK/plugin changes; routine cron next/last/result and subscription delivery bookkeeping is permitted. If new intent is needed, cancel unarmed or abort pre-start, re-resolve, and approve the changed manifest; never edit a hash-bound in-progress manifest.
|
|
||||||
|
|
||||||
**No cron wake-up is required:** the external driver publishes the explicit new mediator input. Do not rely on a timed wake-up for controller recovery. If a fallback wake-up is explicitly wanted, add/review it **before** resolving operator.json, not between resolution and freeze. It is useful only after the target is running and can accept new input. One due during downtime is skipped by cron; one firing immediately before freeze can enter an old pending queue that this approved cut deliberately drops. Neither is migrated/replayed. A guessed wall-clock time after the estimated backup duration is not readiness proof; use the driver's event/completion evidence and stable continuation instead.
|
|
||||||
|
|
||||||
### Approval and external launch surface
|
|
||||||
|
|
||||||
Surma's go must explicitly cover this resolved manifest, the stop-all core/session/cred cut, the detached external controller launch/arm, pinned launcher installation and the post-start verification/release/deletion plan. This card's outside-world-controller scope requires a controller that survives the mediator; it does **not** authorize executing it now.
|
|
||||||
|
|
||||||
The proposed concrete execution surface is **two finite foreground `ssh.exec` calls as surma on citadel.vpn.surma.technology**, using the existing approved signing grant, with commands `cd /home/surma/sau-384-cutover && <the start command below>` and then the same prefix with `<the arm command below>`. Each command terminates; the reviewed native CLI, not a shell `&`, creates the detached OS child. **Ask Surma's go to explicitly authorize this planned exception to the ssh-exec skill's usual prohibition on creating background work through exec strings. Until that explicit authorization, do not launch.** Do not improvise a tunnel, SSH PTY, new worker session or an in-world controller. This is a fully prepared proposed live operation, not a claim that the current worker has permission or has tested a live SSH launch. The same finite foreground surface executes the final verify/delete command after target verification. After arm, loss of the old mediator/SSH RPC result is expected and does not terminate the already-detached controller; inspect the external checkpoint instead of retrying start.
|
|
||||||
|
|
||||||
## 1. Surma monitors; mediator starts/arms the external controller
|
|
||||||
|
|
||||||
Close ingress and record accepted first-update cuts before arm. **Surma is the external human recovery operator while the world is down.** He uses an independent Citadel terminal, never an in-world session/tool process. After mediator `start` acknowledges, Surma runs the foreground monitor below and leaves it visible. It binds a local Unix datagram socket and records the monitor/controller identities before arm. The controller waits for **its own** monitor on every start/resume; arm refuses without it. If an old monitor was killed, a new monitor removes its stale socket only after the saved exact process identity is dead; never unlink a live monitor's socket by hand.
|
|
||||||
|
|
||||||
The monitor receives inventory begin/completion and completed backup-entry progress **without writing disk**, plus `CUTOVER FAILED` and the notification outcome. It also waits on the controller's **pidfd**: normal exit, SIGKILL or crash prints `CONTROLLER EXITED`, even when there is no log/failure.json. Progress datagrams are best effort; the independent death event is not. No heartbeat/liveness/endpoint polling or backup-duration timeout. Read `driver.json` and inspect the pane once after exit; distinguish awaiting_verification from failure instead of blindly restarting.
|
|
||||||
|
|
||||||
Disk-backed `failure.json`, `failure-notification.json` and driver.log are best effort. Each may be unwritable at ENOSPC; none is a prerequisite for the failure signal or the Herdr notification attempt. One best-effort Herdr notification `Cut-over needs Surma` is submitted without first saving failure.json. Rehearsal submission returned shown:false/disabled; neither delivery nor a physical display is guaranteed. **Keep the external terminal monitor attached:** a silent tail of driver.log is not sufficient during backup. Whole-machine power loss can stop both controller and monitor and is not claimed covered.
|
|
||||||
|
|
||||||
From the external root, on the approved host execution surface, mediator first runs the finite preflight command, then starts:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py check operator.json
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py start operator.json
|
|
||||||
```
|
|
||||||
|
|
||||||
`start` launches a new OS session with stdin closed and output in `<root>/driver.log`. A pipe completion handshake records the child's **actual post-Nix-wrapper-exec identity**, rather than racing an interpreter exec. It returns an acknowledgement and leaves the child waiting on inotify for its own monitor, then **durable arm.json**. There is no arm FIFO or reader-dependent write. Check that receipt/checkpoint once. Surma now runs in his independent terminal:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cd /home/surma/sau-384-cutover
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py monitor operator.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Wait for `MONITOR attached; arm may now proceed` in that terminal. Then mediator runs the finite arm command:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py arm operator.json
|
|
||||||
```
|
|
||||||
|
|
||||||
`arm` requires the current monitor, repeats live preflight and atomically writes a manifest-bound marker. Re-arming is idempotent while waiting with that monitor; without it arm refuses finitely, never blocks on a missing FIFO reader. A crash after observing the marker but before saving `armed` resumes from `waiting_for_arm` and reads the same marker.
|
|
||||||
|
|
||||||
From here **the external driver, not the old mediator**, owns steps 2–4. The old default run and its tool calls are deliberately cut. They are not recovered from legacy history. The driver uses an external advisory lock to exclude another controller. Rehearsal killed both the invoking caller shim and, separately, the detached controller after freeze; checkpoint resume completed the chain.
|
|
||||||
|
|
||||||
## 2. Driver: freeze → full backup → plan/audit → offline apply
|
|
||||||
|
|
||||||
The driver verifies reviewed bundle/plugin/SDK bytes and native closure hashes, registers real GC roots, and creates hash-guarded **old and target selector/owner artifacts before freeze**. It then:
|
|
||||||
|
|
||||||
1. Pins boot ID, start ticks, executable and group for both native processes; confirms controller is outside their group. Signals the **outer loop first** and awaits its pidfd exit. Saves `owner_stopped`.
|
|
||||||
2. Signals the exact host **SIGTERM then SIGCONT**, so a job stopped by background-PTY SIGTTIN can handle termination, and awaits host exit. Checks the saved group once for unexpected survivors; never blindly kills them. Saves `freeze.json` phase **`frozen`**, world path, boot ID and empty group inventory. No process/endpoint polling, sleeps or stop-request counting. Host shutdown stops all plugin/agent trees and suppresses restart admission. `core.restart` alone is not stop-all; after the loop is gone, exit 75 cannot respawn it.
|
|
||||||
3. Makes the complete frozen backup outside the world. Streaming SHA-256 does not read whole files into memory. Directory inventory and copy preserve empty directories and symlinks **without following links**. An interrupted plan resumes the same immutable input inventory and partially completed backup; it neither overwrites an earlier backup nor requires deleting the plan directory. Progress is sent to the external terminal monitor. A pre-plan abort may discard only that reproducible incomplete backup, as described below. Unknown images refuse.
|
|
||||||
4. Stages the approved transformations, then checks the frozen caller guards. **Both plan and apply independently require the actual frozen checkpoint**, same world/current boot, dead pinned owners, empty old group and no subsequently running host for this world. A caller-supplied `{remaining:0}` literal is not accepted. Changed checkpoint bytes also refuse.
|
|
||||||
5. Stages **plugin.prev.wasm = that plugin's frozen pre-cut plugin.wasm** for every offline-swapped plugin, then target plugin.wasm. Both prev/current writes have before/after hashes and participate in restore. It never retains the unrelated older prev as the cut-over fallback. Git is deliberately untouched here; its later standard activation will save its own preceding binary. Applies hash-guarded before/after operations offline. File replacements fsync temporary bytes, rename, and fsync the parent. A third image refuses. If atomic replacement dies before rename, abort removes only that saved operation’s new `.cutover-tmp`, absent from the frozen inventory and byte-prefix-matching its approved before/after image; unknown temporaries refuse. Resume overwrites/completes its own temporary. `migration/plan.json` reaches `installed` only after all operations complete.
|
|
||||||
|
|
||||||
Exact logical losses are **only** old session `pending.jsonl`, `cancel`, and unclaimed `mail.deliveries` rows with `state='pending'` and recipient not `surma`. Original messages, Surma copies, done rows and uncertain `unknown` deliveries stay; do not replay them. Loss records and original bytes/rows remain in backup/evidence. Histories/settings, scheduled SQL IDs/calendar/outcome fields, grants/requests and unrelated files remain intact.
|
|
||||||
|
|
||||||
Cron conversion: require an object and nonempty text, refuse `interrupt:true`, remove false/absent legacy interrupt, inner detach/notify/permanent ID, set absent kind `follow-up`, outer detach false. Preserve other fields and non-session JSON bytes. Review removed notify destinations; no invented answer-forwarder.
|
|
||||||
|
|
||||||
**SQLite sidecars:** copy the complete DB plus `-journal/-wal/-shm` family into immutable backup first. Recover only another disposable copy; use SQLite's backup API to stage a consistent DELETE-journal image. Plan removal of old sidecars **before** replacing the main DB, retaining the original physical family. This is normalization, not additional logical data loss. Tests covered committed WAL data, a genuinely hot uncommitted rollback journal, interruption and exact pre-start restore.
|
|
||||||
|
|
||||||
Full-world backup is not instantaneous: the 138,444-file / 12,441,162,164-byte / 45,310-directory **size-bucket-matched synthetic** replica took **70.55 s inventory + 135.39 s copy/hash verification = 205.94 s** in the box. This includes repos/plugins/logs, unlike the earlier session-sized gate. It is warm-cache scale evidence, not an exact live-byte or cold-storage downtime guarantee. Allow that uncertainty in the approved maintenance plan; do not invent a timeout and kill the controller mid-copy. Power-loss recovery is not claimed.
|
|
||||||
|
|
||||||
## 3. Driver: ordered boot installation and pinned owner selection
|
|
||||||
|
|
||||||
The offline target boot list is:
|
|
||||||
|
|
||||||
```json
|
|
||||||
[
|
|
||||||
{"method":"session.manager","params":{}},
|
|
||||||
{"method":"web.start","params":{}},
|
|
||||||
{"method":"cred.serve","params":{}},
|
|
||||||
{"method":"orchestrator.start","params":{}},
|
|
||||||
{"method":"cron.start","params":{}}
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
Manager precedes web, so `web-chat` becomes best-effort. No `mail.start` deliverer remains. Resolve any extra actual boot entry before approval rather than deleting it accidentally.
|
|
||||||
|
|
||||||
The permanent `<root>/target-owner` exports `SAUSAGE_FLAKE=path:<root>/target-selector` itself, then execs the inspected loop with the exact world path. The selector's sole app is the pinned target package. `<root>/old-owner` similarly selects the **explicit old package**, never the remote default. These are permanent installation/launcher artifacts, **not migration code**. Their bytes/hashes are retained as manifest evidence; do not delete them with `tool/`.
|
|
||||||
|
|
||||||
**Every later pane/cold restart uses `/home/surma/sau-384-cutover/target-owner`, with no caller environment required. Never invoke bare sausage-loop or the mutable gitea default for this world.** Keep this command in the operator's pane-restart procedure. Rehearsal cold-launched it in the same Herdr shell without ambient SAUSAGE_FLAKE and observed the target executable. Earlier cold rollback exercised the explicit old wrapper against a separate old-backup clone; pre-start abort gates also exercised it against the exactly restored original frozen fixture, never over new target work.
|
|
||||||
|
|
||||||
`core.restart` now restarts the pinned reviewed host; it does **not** self-update from gitea. Future host updates require a separately approved package/closure/root/selector change. A remote push alone must never silently choose an old host against target state. Keep both GC roots through the rollback window; retire the old root only after an explicit decision, not tool deletion.
|
|
||||||
|
|
||||||
**Expected approved live Nix side effect:** each controller run makes three `nix-store --query --roots` calls, one each for old package, target package and loop, immediately after rooting those paths and before freeze/resume of offline work. **Every resume repeats all three queries and their possible pruning**, even if the roots already exist. Monitor/check/arm/continue/verify alone do not make those roots queries. Those queries can prune stale auto-root registrations and stale temporary-root files in Nix state, even though they do not delete store packages. Surma's go must include that side effect (and the five root registrations including controller/Herdr); the earlier shared-store incident is disclosed in rework-1381.md. Rehearsal now uses only a private Nix DB; no live root query was rerun.
|
|
||||||
|
|
||||||
## 4. Driver: started → launch in the same pane → fresh verifier
|
|
||||||
|
|
||||||
The driver checks the pinned shell still owns the selected pane and no unexpected foreground process exists. It checkpoints **`started` before any launch attempt**, then submits the permanent target-owner path once via `herdr pane run` to the manifest's pane (live **w7:p6**). No shell reconstruction or in-world mediator action is needed during downtime.
|
|
||||||
|
|
||||||
Readiness is event driven: arm a kernel inotify watch on `state/session/manager/registrations`, queue one `session.subscribe` command in the console, and await its unique registration file publication. The wait has **one absolute 120 s deadline**, from cron `src/types.rs` AddParams' existing registration/publication budget (also the earlier test_driver completion bound); it is not a periodic scan. Herdr/RPC calls use that existing finite bound. Missing registration or a failed post-start `cred.status`/subscribe/send produces the disk-independent failure/death signal and exits, keeping **`started`**. Never resume, restore or launch again automatically. Herdr wait-output is internally polling and is not used.
|
|
||||||
|
|
||||||
### Forward completion after a post-start failure (Surma's independent terminal)
|
|
||||||
|
|
||||||
1. Read checkpoint/monitor and inspect Herdr pane/native identities once. If target is not actually running, preserve all new work and obtain a narrow forward repair plan; this mode will refuse and **does not launch it**. Do not relaunch blindly, restore an old snapshot or hand-edit the checkpoint. If only registration/publication failed but exactly one target is running, use:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cd /home/surma/sau-384-cutover
|
|
||||||
PY=/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py continue operator.json
|
|
||||||
```
|
|
||||||
|
|
||||||
2. `continue` exclusively locks, binds the unchanged manifest, inspects that exact shell/pane, exactly one target host for this world and all offline target plugin hashes. It never freezes, applies, restores or launches. It then performs the **skipped exact RPCs**, through operator.json's gateway: `cred.status {}` must report locked and is durably saved as `driver.json.locked_cred_status`/`locked_cred:true` **before Surma unlocks**; `session.subscribe` uses id `cutover:<continuation_id>`, session `cutover-readiness`, method `web.session_event`, best_effort true; `session.send` uses session default, the **same** continuation_id, kind follow-up and exact continuation_text. It saves the receipt and reaches `awaiting_verification`. A recorded locked-startup proof is reused if a later retry occurs after UI unlock; absent that proof, unlocking first is a discrepancy, not permission to fake the check.
|
|
||||||
3. If a method/reply fails again, keep target work, inspect the error and fix only that cause; repeat the **same continue command**, not migration/start. Stable subscribe/message IDs make this safe even when send committed and only its reply was lost. Both started and awaiting_verification are accepted. Tests injected each RPC failure and committed-then-lost send acknowledgement; one continuation run-start/consumption remained. The real 120 s registration timeout also recovered through this command without a second owner.
|
|
||||||
4. `verify` still requires awaiting_verification. Run continue first when phase is started; there is no direct started→verified override. The fresh mediator can then verify/ask for unlock/release as §5; if locked cred prevents SSH signing, Surma performs the same approved local external commands. Live SSH/new-cred signing is not claimed tested. No hand-edited checkpoint or new continuation UUID is needed.
|
|
||||||
|
|
||||||
Finally the driver publishes the manifest's **new explicit default verification message** with its stable UUID and records `awaiting_verification`. The manager, not the driver, runs it. This names the new mediator as verifier without claiming the cut default run was recovered. The test produced one run-start and mailbox-consumed pair for that message. Publication is not immediate model execution: in the last gate it waited **23.753 s to run-start / 24.530 s to consumption** behind initial reconciliation. Do not confuse that with publication latency or a completed startup scan.
|
|
||||||
|
|
||||||
No automatic relaunch, whole-world restore or reapply is allowed after `started`. If launch/acknowledgement is lost, inspect the pane/checkpoint once before any repair. Do not launch a second owner blindly.
|
|
||||||
|
|
||||||
## 5. New mediator: locked proof → Surma unlock → git → release web → delete tool
|
|
||||||
|
|
||||||
The explicit new verification run reads this runbook and in-world ownership/cred state. While cred is locked, Surma checks the external manifest/checkpoint/migration evidence in his independent terminal; the new mediator can read it through approved SSH only after UI unlock if signing works. Surma can also perform the final local verify command, so no vault-key transfer or untested signer is required to finish. Keep ingress closed until checks pass:
|
|
||||||
|
|
||||||
- one target manager and one runner ownership lock per running session; no old owners, mail deliverer or old orchestrator service;
|
|
||||||
- cred **locked**, no vault/key transfer; grants/requests/settings preserved. Interrupted callers reuse their existing stable credential-request keys, not automatic retries of secret-backed mutations;
|
|
||||||
- explicit target run-start without run-end is the only automatic cut-run recovery; old finished-looking history alone is neither settlement nor permission to resume;
|
|
||||||
- durable event/mailbox catch-up and same-ID default/advisor POST replay evidence holds; only intended loss/conversion/schema/new-activity differences exist. Surface uncertain mail/tool outcomes;
|
|
||||||
- scheduled jobs and non-target state match the inventory; real new mediator/runner activity is distinguished from migration changes;
|
|
||||||
- design effective grace 5 s and separate EOF-stop 5 s remain; crash supervision is enabled. Deliberate core.stop/kill disables service registration and is **not** a crash-supervision test.
|
|
||||||
|
|
||||||
Prior post-record live-size samples remain manager ack **0.6078 s**, registration catch-up **0.0128 s**, mail.send-to-default publication **0.5376 s** (integrated 0.5235 s), below existing orchestrator **3 s** / cred **10 s** caller budgets. Disclose integrated catch-up **12.0419 s** and the new initial-scan delay above. They are synthetic/warm-cache observations, not production latency guarantees. Prior real-core watch/upgrade/all-watch readiness, ~0.10% whole-host idle duty, detached error data and real cron warning/stop evidence remain linked in the result docs.
|
|
||||||
|
|
||||||
Before any Git network work, record locked-startup proof, ask **Surma to unlock the target vault through the existing UI**, then check `cred.status` reports unlocked for that same target service. Wait here if Surma is unavailable; never transfer a key/password. **Only after that**, mediator/orchestrator activates the reviewed git artifact:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"method":"core.activate","params":{"plugin":"git","wasm":"/docs/symphony/artifacts/SAU-384/git-9cc9fa6f.wasm"}}
|
|
||||||
```
|
|
||||||
|
|
||||||
This is a proposed live operator step covered by the go, not a worker action. The permanent world artifact above must be staged/hash-checked from the selected bundle before go; the driver preflights its external `deferred_artifacts.git` copy too. Verify active git hash equals manifest `files["git.wasm"].sha256` before releasing Git callers. See `/plugins/git/source/{VAULT-HTTPS.md,VAULT-SSH.md}`; both require cred first, live and unlocked. No push is part of verification; **every** later push still needs a separate fresh Surma go. Final proof includes `git_activated_after_unlock`.
|
|
||||||
|
|
||||||
Interim compatibility: observed live Git **f089b56d** still speaks new cred's explicit-grant fetch/item_get/public_key/sign contracts. Actual old-WASM + new-cred loopback SSH clone signed/authenticated successfully with a synthetic explicit grant after unlock; locked use returned vault_locked. Granted HTTPS reached new cred, reserved exactly one use, then the deliberately closed loopback port returned network_error with a capture (no successful HTTPS transfer claimed). Old grant-free shared SSH fails asking for a local key; old grant-free HTTPS retains its old config/anonymous path, not new anyone_may_use selection. Therefore do not schedule Git network work during the locked/old-git interval, do not introduce local key/config-secret workarounds, and surface any already-started caller's lock/missing-key error rather than blindly retrying it. Local git status/commit/authoritative-main merges remain available. New Git shared SSH clone was tested after real synthetic unlock/activation.
|
|
||||||
|
|
||||||
Only now use **`orchestrator.release` for SAU-375** to release its approved commit. Inspect that method's current typed arguments; do not invent them. Verify published asset hashes and ingress. No worker publishes anything. The rework tested approved asset hashes in temporary output, **not an actual live release or physical browser**.
|
|
||||||
|
|
||||||
The Surma unlock above is a prerequisite for using the existing approved signing grant for the final external SSH verification/deletion command (live signing interoperability is still unverified); it is not a second vault handover. Never ask for Surma's password or transfer the old vault key. Read-only gateway checks and checkpoint inspection do not justify bypassing the lock. If unlock is not available, remain `awaiting_verification`, keep git activation/ingress/release gated, and do not reconstruct or rerun the migration.
|
|
||||||
|
|
||||||
Write external `verification.json` with evidence links and the required true checks named by the driver's `verify` mode: `one_manager`, `no_old_owner`, `locked_cred`, `non_target_preserved`, `losses_exact`, `catchup_and_replay`, `web_released_and_hashes`, `ingress_verified`, `git_activated_after_unlock`. They are operator attestations after the checks, not substitutes for them. From the external root, mediator runs the finite command below after target cred is unlocked and signing works; **Surma may instead run the identical command in his independent Citadel terminal**, avoiding any dependency on in-world SSH signing:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py verify operator.json verification.json 2>> tool-source-and-delete.log
|
|
||||||
```
|
|
||||||
|
|
||||||
It marks the saved plan/driver verified, logs source hashes/source-as-log, and **deletes the three migration executables and bytecode caches**. Confirm `tool-deletion.json` and absence once. Delete any other reconstructed executable/controller copies too. Retain only manifests, backup/inventory/loss/caller-audit records and logs as migration evidence, plus the separately reviewed target/permanent launcher artifacts. Rehearsal deleted all executed migration/test controllers before exporting evidence. If verification/deletion is interrupted, inspect the verified checkpoint and delete only remaining hash-matched tool copies; do not reconstruct and rerun migration effects.
|
|
||||||
|
|
||||||
## Failure/rollback boundaries
|
|
||||||
|
|
||||||
| Boundary | Exact recovery rule |
|
|
||||||
|---|---|
|
|
||||||
| Waiting for arm, including consumed-marker/save race | World untouched. Repeated arm is finite even with no reader. After the saved controller exits, `resume` accepts waiting_for_arm and the durable marker; attach a new `monitor` for that replacement before it continues. Or run abort (untouched) after obtaining the external controller lock. |
|
|
||||||
| Armed/captured/owner_stopped/frozen; no target attempt | External operator can use `... driver.py resume operator.json` **only after the saved controller has exited**. It detaches a replacement and returns; Surma then runs the same foreground monitor command. The replacement waits for its own monitor receipt before effects. Same boot/identities/manifest/backup/plan must pass. No agent needs to exist. Both caller-killed and controller-killed resume were rehearsed. |
|
|
||||||
| Interrupted planning/swap, no start attempt | Resume the same immutable backup/plan with before/after hashes. A caller/config after-image is accepted only if that exact saved operation approved it; third images refuse. Alternatively Surma runs the real driver's **abort** mode below: it restores every approved changed path from the immutable complete backup, verifies the full frozen inventory, and launches the explicit old-owner in the selected pane. It never restores an incomplete backup. If planning never completed, apply cannot have started: abort first discards only `migration/backup` (including its partial copy temporaries), retaining planning/inventory/log evidence, then checks space **before any checkpoint save**. If enough space is available, it verifies any saved inventory and relaunches the untouched old world. If a competing writer still fills the disk it refuses before writes/launch; Surma must free only identified, separately approved non-world data and repeat abort. Never delete a complete backup/plan or unrelated world data to make space. Restore is before/after-hash guarded, resumable, and writes main DBs before their old sidecars without opening them. Never partially start mixed contracts. A frozen audit discrepancy is an offline stop, not permission to improvise a conversion. |
|
|
||||||
| `started` or later, even if boot seems failed | No automatic whole-world rollback/reapply/relaunch. New histories, mail, jobs, grants or external effects may exist. Preserve them. Use the forward-only continue command above for readiness/post-start RPC failure with one healthy target; otherwise plan a narrow forward repair or coordinated revert. |
|
|
||||||
| Any plugin core.revert | Offline-swapped prev now means exactly pre-cut current (hash-tested), not pre-pre-cut. **Do not use blind individual core.revert across incompatible contracts.** Coordinate providers/callers and preserve new work; the session prerequisite below still applies. Later git activation uses normal core prev semantics. |
|
|
||||||
| Session revert after target activity | Close gateway/ingress. While new manager is available, **unsubscribe web-chat**, verify removal, and prevent gateway re-registration while the old session binary runs. Never call web.start against that old manager or restore a registry backup that reintroduces it. Actual old-manager/gateway tests proved unsubscribe safe and ID-only re-key unsafe: the old binary ignores best_effort and replays history once. |
|
|
||||||
| Host reboot / changed PID identity / unexpected survivor | Refuse saved native identities; re-inventory externally. Never signal a reused PID, blindly kill a group, or infer quiescence from a stop request. |
|
|
||||||
|
|
||||||
### Exact external failure commands (Surma, not an in-world mediator)
|
|
||||||
|
|
||||||
In the independent Citadel terminal, `cd /home/surma/sau-384-cutover`; read `driver.json`, monitor output, any available `failure.json`/`driver.log`, and the pane once. At ENOSPC those disk-backed diagnostics may be absent or truncated; the terminal failure/death signal is authoritative that the controller ended, not that rollback is safe. Commands use the pinned interpreter:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
PY=/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3
|
|
||||||
# Same manifest/boot/owners, known exited controller, pre-start only:
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py resume operator.json
|
|
||||||
# After resume acknowledges, attach its new foreground monitor before effects:
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py monitor operator.json
|
|
||||||
# Or abandon pre-start attempt, restore coherent old bytes, submit old-owner once:
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py abort operator.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Choose **one**, not both. Both use a nonblocking exclusive controller lock; do not delete the lock file or bypass it. Abort accepts waiting/armed/frozen/installed/restoring/restored only. It checkpoints `old_start_attempted` **before** pane launch, then `old_launched`. If interrupted during restoring/restored, repeat abort on the same manifest/checkpoints; if `old_start_attempted`/`old_launched`, inspect the pane instead of repeating it. Abort is forbidden after any target `started` attempt. Never call Kit.rollback by hand or relaunch a bare loop. If the pinned shell/pane changed, stop for explicit identity resolution; do not guess a replacement.
|
|
||||||
|
|
||||||
After abort, Surma verifies exactly one old package owner in the pane and old world readiness before retry. Preserve the frozen evidence; the sole permitted deletion of backup evidence before verified success is the **reproducible incomplete pre-plan backup subtree** in the guarded abort path above. Complete backups are never deleted by abort. Every abort, including one with a completed plan, checks space before its first checkpoint save; a full disk is a deliberate pre-write refusal, not a failed checkpoint write. Abort rechecks the same measured space budget again immediately before old launch; if it refuses, free only identified/approved competing non-world data, not histories, jobs or complete backup evidence, then repeat abort on the same checkpoint. For a clean retry **only after old_launched and verification**, archive checkpoints (not owners/selectors/GC roots/tool):
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# Pick a fresh, unused archive path and record it; this example is this run.
|
|
||||||
ARCHIVE=attempt-before-retry
|
|
||||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B -c 'import json; assert json.load(open("driver.json"))["phase"] == "old_launched"'
|
|
||||||
mkdir -- "$ARCHIVE"
|
|
||||||
for p in driver.json freeze.json arm.json controller-process.json migration driver.log failure.json failure-notification.json preflight-live.json monitor.json poststart-inspection.json; do
|
|
||||||
if [ -e "$p" ]; then mv -- "$p" "$ARCHIVE/"; fi
|
|
||||||
done
|
|
||||||
cp -- operator.json "$ARCHIVE/operator.json"
|
|
||||||
```
|
|
||||||
|
|
||||||
Re-identify the now-running old loop/host and pane shell; refresh old-work/caller guards (including any wake-up) and approval for the new resolved `operator.json`, retaining the stable continuation UUID if no target start was attempted. Then check/start/arm as §1. Do not merely edit a live checkpoint or delete the backup. For aborted_untouched, no old relaunch is necessary; archive only after verifying the original owners are still running. No such reset/archive is allowed for started/awaiting_verification/verified or unknown owners: retain new work and plan a forward repair.
|
|
||||||
|
|
||||||
Independent plugin recovery/accounting/provider-cancellation follow-ups remain independent. Accepted first-update cuts and uncertain external effects are not silently solved by this kit, grace=5, or a successful rehearsal.
|
|
||||||
Reference in New Issue
Block a user