Compare commits

..
17 Commits
Author SHA1 Message Date
Sausage ee88adf6c6 Remove private-world pointer from service timing comment
ci / check (push) In progress
2026-11-01 22:17:04 +00:00
Sausage 6dc21cd8dc Remove card labels from reusable cargo deadline tests 2026-11-01 22:17:04 +00:00
Sausage 5be5b3b397 Clean up dropped SDK calls and make drain assertions order-independent 2026-11-01 22:17:04 +00:00
Sausage a95b5287cf Add armed file watches without serializing unrelated IO 2026-11-01 22:17:04 +00:00
Sausage 959ddf46ff Supervise boot services with uniform five-second upgrade grace 2026-11-01 20:51:08 +00:00
Sausage 4da8a28392 Notify superseded plugin processes and support SDK draining 2026-11-01 20:51:08 +00:00
SAU-317 implementer c54edea596 Name the received JSON type in non-object params errors
ci / check (push) Failing after 2m38s
Keep -32602 and existing validation order; tell callers to send the object itself.
Cover host routing, stdout fast paths, SDK calls and generated dispatch.

Card: SAU-317
2026-10-28 19:16:39 +00:00
Sausage Worker 9ef73e6bc7 Use the world SDK for cargo builds and seed it only when missing 2026-10-28 08:39:36 +00:00
SAU-249 implementer dc81f78c20 Add shared SQLite open macro with WASI crash recovery and in-memory temp storage
ci / check (push) Failing after 3m53s
Card: SAU-249
2026-10-28 00:33:33 +00:00
SAU-158 implementer 9a8c143856 Return compact activation summaries instead of discovery schemas
ci / check (push) Failing after 2m40s
Card: SAU-158
2026-10-26 22:24:43 +00:00
surma a14f051f97 Merge 'e26260d36eb8e7bf9e0b7edfed90c0dcff77fc19'
ci / check (push) Failing after 11m35s
2026-10-26 13:59:16 +00:00
surma e26260d36e Spool native cargo build and test streams to persistent world logs 2026-10-26 13:54:15 +00:00
SAU-143 implementer 037377fd27 host: preserve journals, full traps and embedded agent output
Write new journal entries under logs/core and merge read-only legacy
history without moving SQLite/WAL files. Atomically spool failed batches,
retaining unspoolable entries for event-driven retries. Journal full trap
diagnostics before UTF-8-safe status display. Port session overflow archives
and followable line/byte excerpts to the still-shipped embedded agent.
Document retention/recovery and cover the contracts with regressions.
Include one rustfmt-only normalization in the existing SDK WASI test.

Card: SAU-143
2026-10-26 13:40:15 +00:00
surma 4be9ae41de Merge 'card/sau-126' 2026-10-26 12:49:01 +00:00
SAU-135 implementer 168795b408 seed: carry restart approval policy into new worlds
Make legacy core.exit share core.restart's supervisor gate and pending
request. Seed the stable agent prompt and boot comments with the approval
and shutdown contract, and check real init output under a deadline.
Cover both aliases' gate, response grace, coalescing and exit status.

Card: SAU-135
2026-10-26 10:29:12 +00:00
SAU-134 implementer 8c46d9b9d0 host: add supervisor-gated clean core.restart RPC
Return success before scheduling the existing shutdown path, coalesce pending
requests, and mark #loop invocations with SAUSAGE_LOOP=1. Keep core.exit
compatibility and document explicit per-call approval for all agents.

Card: SAU-134
2026-10-26 09:16:10 +00:00
surma c15e50d752 Bound native cargo builds and individual WASI tests with cancellable deadlines 2026-10-25 20:14:32 +00:00
54 changed files with 5296 additions and 394 deletions
+24
View File
@@ -61,6 +61,20 @@ a plugin, or a change that the agent makes to itself.
## Small choices ## Small choices
- **Restart is a clean stop supervised by `#loop`.** `core.restart` requires
the host startup marker `SAUSAGE_LOOP=1`, which the loop exports for its
invocation. Without a supervisor, restarting would only stop the world.
A single pending request answers first, allows 250 ms for delivery, then
signals the existing shutdown path; it does not use `process::exit` or
bypass detached-process cleanup, lock release, or journal draining. The
executable returns dedicated restart code **75** after cleanup. `#loop`
refreshes and starts again on 75 (and other nonzero exits); 0 stops it.
`core.exit` is a compatibility alias with the same supervisor gate,
response grace, and coalescing; new callers use `core.restart`.
**HARD RULE: the mediator and all agents NEVER call `core.restart` without
the person's explicit approval each time.** This
approval rule also covers legacy `core.exit`; it must not bypass the gate.
- **YAML for configuration, JSON for data.** A person edits configuration, so - **YAML for configuration, JSON for data.** A person edits configuration, so
it needs comments. `init` writes commented templates instead of asking it needs comments. `init` writes commented templates instead of asking
questions, so there is one way to configure a world. questions, so there is one way to configure a world.
@@ -70,6 +84,16 @@ a plugin, or a change that the agent makes to itself.
that occur only in WASI, for example a missing WASI function or a panic at that occur only in WASI, for example a missing WASI function or a panic at
start. The test runner uses the same Wasmtime and the same mounts as the start. The test runner uses the same Wasmtime and the same mounts as the
core. core.
- **Tests never run without a timeout.** `cargo.build` serializes its work
folder, so one hung test or build script must not block all later builds.
`timeout_secs` defaults to 900 seconds after mutex acquisition, excluding
queue time; `test_timeout_secs` defaults to 120 seconds per WASI instance.
Both are optional positive integers, never null/unlimited. Test timeouts name
the failing test and let other instances finish. Build deadlines and
cancellation kill cargo's process group before dropping the mutex; timeout
errors retain the phase, elapsed time, and recent output. Source preparation
and publication retain serialization while their blocking filesystem work
finishes, even if cancelled, to avoid overlapping writes to the work folder.
- **The SDK is not published.** `cargo` points `sausage-plugin = "0.1"` at - **The SDK is not published.** `cargo` points `sausage-plugin = "0.1"` at
its own copy, which it also shows in `/plugins/cargo/sdk/`. its own copy, which it also shows in `/plugins/cargo/sdk/`.
- **Compiled code is a cache, not state.** It lives outside the world, in - **Compiled code is a cache, not state.** It lives outside the world, in
+194 -17
View File
@@ -19,7 +19,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. | | call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. | | interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. | | endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
| service | A call in `/config/boot.yaml`. The core starts it at boot and again after an upgrade of its plugin. | | service | A call in `/config/boot.yaml`. The core starts it at boot, restarts it after a crash, and starts it again after an upgrade of its plugin. |
| import | An interface of another plugin that a plugin uses. | | import | An interface of another plugin that a plugin uses. |
| provider, consumer | The plugin that has an interface, and a plugin that imports it. | | provider, consumer | The plugin that has an interface, and a plugin that imports it. |
| journal | The central log of the world. | | journal | The central log of the world. |
@@ -34,7 +34,8 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
source/ the source. `cargo.build` builds it from here. source/ the source. `cargo.build` builds it from here.
imports/<plugin>.json descriptions of the imported plugins imports/<plugin>.json descriptions of the imported plugins
state/<plugin>/ persistent state of a plugin, by convention state/<plugin>/ persistent state of a plugin, by convention
state/core/journal.sqlite the journal logs/core/journal.sqlite the journal
state/core/ console history (state, not logs)
config/ global configuration config/ global configuration
boot.yaml the services boot.yaml the services
agent.yaml the API, base URL, model, and key of the agent agent.yaml the API, base URL, model, and key of the agent
@@ -69,7 +70,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
`notes.rpc.discover` sends it `rpc.discover`. `notes.rpc.discover` sends it `rpc.discover`.
- `core.<method>` is a method of the core (section 9). - `core.<method>` is a method of the core (section 9).
- Method names that start with `rpc.` are reserved: `rpc.discover`, - Method names that start with `rpc.` are reserved: `rpc.discover`,
`rpc.retired`, and `rpc.endpoint`. `rpc.superseded`, `rpc.retired`, and `rpc.endpoint`.
### 3.3 What a call does ### 3.3 What a call does
@@ -152,9 +153,17 @@ A process is an instance of one plugin version. When a new version becomes
active (section 10.3): active (section 10.3):
1. New calls get the new version at once. 1. New calls get the new version at once.
2. Each process of an older version gets a grace period to finish its work. 2. At grace start, the core sends each process of an older version the
The default is 60 seconds. A plugin can declare its own value in JSON-RPC notification `rpc.superseded {"current": "<sha256 of the new file>"}`
`rpc.discover` (`x-sausage-grace`). Then the core stops the process. on stdin, without an ID. The process is still running: calls and responses
continue to work. It can stop accepting work, finish its in-flight calls,
and exit early. Ignoring the notice changes nothing. The grace period is
the upper bound before the core stops the process (section 3.6); its
duration is 5 seconds for every plugin. Discovery metadata, including an
explicit `x-sausage-grace`, does not change it. After grace, stdin closes
and the separate 5-second stop budget applies (section 3.6): at most two
5-second windows. Discovery-only processes do not receive this notice or
enter upgrade grace; they follow the discovery lifetime (section 10.1).
3. When the old process of a service has ended, the core starts the service 3. When the old process of a service has ended, the core starts the service
again with the new version. again with the new version.
4. When the last process of an older version has ended, the core starts a 4. When the last process of an older version has ended, the core starts a
@@ -240,6 +249,40 @@ Rules:
Principle: **inspect through files, act through calls.** Principle: **inspect through files, act through calls.**
### File watching
`core.watch {"path": "/state/session/mailbox"}` stays open for the caller
process's lifetime. It needs a request ID and a running plugin caller.
Its first `core.notify` is `"armed"`, with empty params. Only then scan the
files. Writes before arming are visible in that scan; writes after arming
produce `"changed"` notifications (also with empty params). Rescan after
each notification; files, not event counts, are the truth.
Notifications follow successful plugin-originated file mutations: creation,
truncation, descriptor/stream writes, rename (both source and destination),
and deletion, including mounted `/run` and `/self` files. Paths are normalized
in the virtual namespace; `/self` means the caller's `/proc/<pid>` folder.
A watch matches itself and descendants, not sibling prefixes. Moving or
removing an ancestor also invalidates it. Paths need not exist when armed.
Open file writes follow plugin renames; unlinked/replaced handles no longer
name visible files. External host writes are outside this contract.
Changes may coalesce, including a change already visible in the initial
scan, but a change after the last notification always produces another.
Delivery holds no filesystem locks. Observation does not serialize descriptor
IO; only overlapping opens and rename/removal wait for one another.
Watch the narrowest useful path: delivery is not rate-limited, so watching a
busy parent can cause a rescan for every write.
The host-side watch ends only when its caller process exits, not on a change
notification; there is no cancellation API or polling/retry loop. SDK callers
should poll the watch future alongside their handler's work and drop it on
return, not keep it in a background task that prevents an idle process from
exiting. Dropping the future removes the SDK's pending entry and callback, so
late notifications and responses are ignored; it does not cancel the host
watch. A long-lived service keeps one watch per purpose for its lifetime,
rather than repeatedly opening and dropping watches.
## 5. Endpoints ## 5. Endpoints
An endpoint is a JSON-RPC interface that a running process serves over TCP, An endpoint is a JSON-RPC interface that a running process serves over TCP,
@@ -279,9 +322,23 @@ for example the control interface of a web server.
## 7. The journal ## 7. The journal
- An SQLite file at `/state/core/journal.sqlite`. Only the core writes it. - New entries go to `/logs/core/journal.sqlite`. Only the core writes it.
An existing `/state/core/journal.sqlite` stays in place and is opened
read-only; queries merge both databases by entry ID before applying the
limit. The host never renames a live SQLite/WAL trio. Console history stays
in `/state/core`. No journal or log files are automatically pruned.
- If a journal transaction fails, the complete batch (including original
IDs and timestamps) is saved in an exclusive, synced JSON file under
`/logs/core/failed-batches/`; the host reports its path on stderr. These
files are recovery artifacts, not query results: recovery/import is
deliberate, using IDs to avoid duplicates, and never deletes the source.
If spooling also fails, entries stay in memory and retry on the next
command (no polling); at writer shutdown their full text is reported to
host stderr. Storage failure plus host loss can still lose unsaved data.
- Each entry has a UUIDv7 (it sorts by time), the process ID, the plugin, the - Each entry has a UUIDv7 (it sorts by time), the process ID, the plugin, the
stream (`stderr` or `core`), and the line. stream (`stderr` or `core`), and the line.
- Traps retain their full diagnostic text in the journal; short exit-status
displays are UTF-8-boundary-safe.
- The core writes its own events too: start (with method and params), exit - The core writes its own events too: start (with method and params), exit
status, stop, kill, upgrades, and the warning about discarded stdout. status, stop, kill, upgrades, and the warning about discarded stdout.
- `core.log {plugin?, pid?, since?, until?, grep?, limit?}` queries the - `core.log {plugin?, pid?, since?, until?, grep?, limit?}` queries the
@@ -302,10 +359,29 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
- A service is a detached call with the parent `core`. The core waits for - A service is a detached call with the parent `core`. The core waits for
its result before it starts the next one. The result shows that the its result before it starts the next one. The result shows that the
service is ready. service is ready.
- If a service fails, the core writes the error to the journal and continues. - If a service fails before answering, the core writes the error to the
- After an upgrade of the plugin of a service, the core starts the service journal and continues booting. A failure to admit a process (for example,
again with the new version (section 3.7). a missing plugin or invalid parameters) is logged, not retried in a loop.
- The core does not restart other detached processes. - The core supervises the process lifetime, not its readiness response. A
nonzero exit, trap or failed host task restarts the same service with its
original method and parameters, including a crash before readiness. A
readiness response alone never triggers a restart; a clean exit does not
count as a crash or trigger an immediate restart.
- More than 5 crashes within 10 seconds gives up on that service and writes
`service gave up` to the journal. Each service has an independent sliding
window; crashes exactly 10 seconds old expire. There is no retry delay or
periodic liveness check. Giving up is latched for this host lifetime:
neither elapsed time nor an upgrade re-arms it.
- After an upgrade of the plugin of a service, the same supervisor starts
it again with the current version after its old process ends (section
3.7). Superseded exits are retirement, not crashes. Admission and cleanup
reserve a single owner/attempt, so an upgrade racing a crash cannot start
two owners. A matching boot-service start from `rpc.retired` is ignored
(returns `null`), even while restart is pending or after giving up; older
plugins' fallback starts do not bypass supervision.
- An explicit `core.stop` or `core.kill` disables automatic attempts for
that service, including during an upgrade. Host shutdown starts no new
processes. The core does not restart other detached processes.
- A singleton is a service with a fixed endpoint in `/run`. Only the core - A singleton is a service with a fixed endpoint in `/run`. Only the core
starts it, so two instances cannot start at the same time. starts it, so two instances cannot start at the same time.
@@ -314,15 +390,46 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
| Method | Params | Result | | Method | Params | Result |
| --- | --- | --- | | --- | --- | --- |
| `core.call` | `method`, `params`, `detach`? | the result of the method | | `core.call` | `method`, `params`, `detach`? | the result of the method |
| `core.watch` | `path` | stays open; `armed`, then `changed` notifications (section 4) |
| `core.notify` | `name`, `params` | none (a notification) | | `core.notify` | `name`, `params` | none (a notification) |
| `core.stop` | `pid` | `{}` when the process has ended | | `core.stop` | `pid` | `{}` when the process has ended |
| `core.kill` | `pid` | `{}` | | `core.kill` | `pid` | `{}` |
| `core.lock` | `path`, `wait`? | `{"held": bool}` | | `core.lock` | `path`, `wait`? | `{"held": bool}` |
| `core.unlock` | `path` | none (a notification) | | `core.unlock` | `path` | none (a notification) |
| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents | | `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | short activation summaries, keyed by plugin |
| `core.revert` | `plugin`, `force`? | like `core.activate` | | `core.revert` | `plugin`, `force`? | like `core.activate` |
| `core.log` | filters (section 7) | journal entries | | `core.log` | filters (section 7) | journal entries |
| `core.exit` | none | `{}`. Then the host stops with exit code 75. | | `core.restart` | none | `{}`. With `SAUSAGE_LOOP=1`, schedule clean shutdown and exit code 75. |
| `core.exit` | none | Alias of `core.restart`: same supervisor gate, response, coalescing, and exit code 75. |
### Host restart
**HARD RULE: the mediator and all agents NEVER call `core.restart` without
the person's explicit approval each time.** The same approval rule applies
to the legacy alias `core.exit`; it uses the same gate and pending request.
`core.restart {}` is enabled only when the host starts with `SAUSAGE_LOOP=1`.
The host captures the marker at startup. An absent marker (or any value other
than `1`) returns error -32600: `restart needs the loop supervisor
(SAUSAGE_LOOP not set); start with SAUSAGE_LOOP=1`, without scheduling an exit.
This is a supervisor marker, not a permission system.
The method returns `{}` immediately and schedules the shutdown request after
250 milliseconds to let the response reach the caller before stdin closes.
Repeated requests while a restart is pending return `{}` without scheduling
another restart or extending that delay. The executable then uses the same
`Host::stop` path as a normal stop: reject new processes, stop all processes
(including detached ones) in parallel with the normal five-second grace,
wait for outstanding file operations, release locks, and drain journal writes.
Only after that does the executable return the dedicated restart exit code
**75**. The library signals this path; it never kills its embedding process.
`nix run .#loop -- <world>` sets `SAUSAGE_LOOP=1` on the host invocation and
starts the newest host again after exit code 75 (and, as before, other nonzero
exits). Exit code 0 ends the loop. If using an older loop script, the person
must set `SAUSAGE_LOOP=1` in that supervisor's environment; do not set it for
an unsupervised host. The NixEnv service invokes this flake's `#loop`, so a
new loop script supplies the marker without changes to NixEnv.
Everything else is a file. Everything else is a file.
@@ -341,7 +448,8 @@ Params are by name. The document can have these extension fields:
`methods`. `methods`.
- `x-sausage-imports`: for each provider, the part of its document that the - `x-sausage-imports`: for each provider, the part of its document that the
plugin uses. plugin uses.
- `x-sausage-grace`: the grace period in seconds. - `x-sausage-grace`: legacy grace declaration in seconds. The core ignores
this field; upgrade grace is always 5 seconds (section 3.7).
The schema of params describes what the provider can read. The schema of a The schema of params describes what the provider can read. The schema of a
result describes what the provider writes. Descriptions are part of the result describes what the provider writes. Descriptions are part of the
@@ -389,6 +497,26 @@ imported method:
If a check fails, nothing changes, and the error -32003 lists each problem, If a check fails, nothing changes, and the error -32003 lists each problem,
with a counterexample where possible. with a counterexample where possible.
The result is an object keyed by plugin name. Each value contains:
- `plugin`: the plugin name.
- `wasm`: the installed world path (`/plugins/<name>/plugin.wasm`).
- `hash`: the SHA-256 of the installed copy.
- `previous`: the replaced version's SHA-256, or `null` on first installation.
For an identical-hash no-op, this is the current hash.
- `method_count`: the number of methods in the new main interface.
- `added`, `removed`, `changed`: sorted method-name arrays compared with the
previous discovery document. A change to a method description or any
transitively referenced local schema counts as changed. Unrelated schemas
and endpoint interfaces do not affect these arrays. With no previous
document, all methods are added.
- `interface_check`: `"passed"` after successful checks, or `"skipped"` with
`force: true` (not a claim of compatibility).
The result does not include schemas. The full document remains available
through `<plugin>.rpc.discover` and `/proc/core/plugins/<name>/discover.json`.
An identical-hash no-op and `core.revert` return the same summary shape.
- The core runs one activation at a time. - The core runs one activation at a time.
- `plugins: [...]` activates several plugins together. The checks then use - `plugins: [...]` activates several plugins together. The checks then use
the new set. the new set.
@@ -418,13 +546,46 @@ builds `/plugins/notes/source/`:
3. It builds the release `.wasm` for `wasm32-wasip2`. The tests and the build 3. It builds the release `.wasm` for `wasm32-wasip2`. The tests and the build
share the compiled dependencies. share the compiled dependencies.
4. It returns `{"wasm": "/state/cargo/builds/<n>/notes.wasm", "tests": 4. It returns `{"wasm": "/state/cargo/builds/<n>/notes.wasm", "tests":
{"passed", "failed", "output"}, "warnings"}`. If a step fails, it returns {"passed", "failed", "output"}, "warnings", "logs"}`. If a step fails, it returns
an error with the compiler or test output. an error with the compiler or test output.
- Before running commands, it creates `/logs/cargo/<YYYY-MM-DD>/<build-id>.stdout.log`
and `.stderr.log` (UTC date, unique build ID). Both test and release phases
append their raw bytes directly to these files, before any parsing, progress
filtering or UTF-8 conversion. This includes successful JSON build output.
`/logs` is a top-level world folder, created if missing, not under `/state`.
- Success results and error -32004 `data` include `logs`:
`{"stdout":{"path":"/logs/cargo/...stdout.log","bytes":123},
"stderr":{"path":"/logs/cargo/...stderr.log","bytes":45},"total_bytes":168}`.
Counts are raw bytes, not characters. `logs` is null only if log creation
itself fails. Preparation failures otherwise include empty log files.
Inline diagnostics may be filtered or shortened; the files are the full truth.
- Logs remain after failure, total/test timeout, and cancellation. Cancellation
need not return a result; its files remain discoverable in `/logs/cargo/`.
Pruning numeric `/state/cargo/builds/` artifact folders does not prune logs.
- Optional `timeout_secs` is an integer of at least 1, default **900** (not
null or unlimited). The wall-clock budget starts when the build acquires its
mutex, excluding queue time, and covers tests and release compilation together.
On expiry, cargo and its descendant process group are killed and the mutex is
released. Error -32004 has reason/message `timeout`, with `data.reason`,
`data.elapsed_secs`, `data.step` (`tests` or `release build`), `data.tests`,
and `data.output` containing the last 50 output lines.
- Optional `test_timeout_secs` is an integer of at least 1, default **120**.
Each WASI test instance has this wall-clock budget. On expiry, the instance
is killed, its named test fails with `timed out after N s`, and remaining
tests continue. The overall build budget still applies.
- Cancellation kills cargo's process group, including build scripts and test
runners, before releasing the build mutex. An attached build also stops when
its caller ends (under the normal process stopping deadline).
- A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal - A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal
entries, like other calls. entries, like other calls.
- `/plugins/cargo/sdk/` shows the source of the SDK. The dependency - `/plugins/cargo/sdk/` is the SDK used by builds. The dependency
`sausage-plugin = "0.1"` points to the same files in the host's cache. `sausage-plugin = "0.1"` points directly to this folder, including its macros
and interface crate. Edits apply to the next build without a host restart.
- The host seeds a missing SDK folder from its embedded copy at startup or
before a build. An existing folder is never overwritten or repaired, even
after a host upgrade. To reset to the host's bundled SDK, remove the whole
folder; the next startup or build re-seeds it. Back up local edits first.
- Compiled crates stay in the host's cache, outside the world, and all worlds - Compiled crates stay in the host's cache, outside the world, and all worlds
share them. So a second build compiles only what changed. share them. So a second build compiles only what changed.
- Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2` - Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2`
@@ -443,3 +604,19 @@ Tests:
files through `env!("CARGO_MANIFEST_DIR")`. files through `env!("CARGO_MANIFEST_DIR")`.
- The test runner answers `core.lock`, `core.unlock`, `core.notify`, and - The test runner answers `core.lock`, `core.unlock`, `core.notify`, and
`core.log`. A call to a plugin fails: a test has no other plugins. `core.log`. A call to a plugin fails: a test has no other plugins.
## 12. Embedded agent tool output
The agent shipped by `sausage init` uses a context-size guard of 50 KiB or
2000 lines, not a data-retention limit. Oversized tool results, including
errors, are saved in full under `/logs/agent/<date>/default/` before history
and provider delivery receive an excerpt. Overflow JSON string values are
rendered as physical text lines so the reference's one-based inclusive line
ranges can be followed with `read_file {path, line_offset, line_limit}`
(zero-based offset; default 200 lines). Huge single lines give exact byte
ranges and continuation offsets. Command/log/build/test results show the
tail; other tools show the head. Reading an existing `/logs` file excerpts
that same file without making another archive. Archives are exclusive and
never automatically deleted. If archival fails, the agent reports the error
to stderr and keeps the complete result instead of discarding it. Existing
history formats and paths are unchanged.
+26 -5
View File
@@ -45,9 +45,18 @@ nix run git+https://gitea.surma.technology/surma/sausage#loop # ./
nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world
``` ```
When the host crashes or a plugin calls `core.exit`, the loop fetches the When the host crashes or a plugin calls `core.restart`, the loop fetches the
newest commit and starts it again. `quit` ends the loop. Set `SAUSAGE_FLAKE` newest commit and starts it again. The loop sets `SAUSAGE_LOOP=1` to enable
to run another flake URL instead, for example your fork. that RPC; without it, restart returns an error and leaves the host running.
Restart answers first, then cleanly stops all processes and exits with code
75. `quit` (code 0) ends the loop. Set `SAUSAGE_FLAKE` to run another flake
URL instead, for example your fork.
**The mediator and all agents NEVER call `core.restart` without the person's
explicit approval each time.** This also applies to legacy `core.exit`, an
alias with the same supervisor gate, response-before-exit behavior, and
request coalescing.
See [the protocol](PROTOCOL.md#host-restart) for the shutdown contract.
From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the
same commands are `nix run`, `nix run -- init`, `nix run -- run`, and same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
@@ -57,7 +66,7 @@ same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
`--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host `--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host
runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls
`core.exit` (exit code 75, so the loop starts the newest version): `core.restart` (requires `SAUSAGE_LOOP=1`; exit code 75 restarts the loop):
```sh ```sh
nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world
@@ -84,10 +93,22 @@ plugins/<name>/plugin.wasm the current version of a plugin
plugins/<name>/source/ its source plugins/<name>/source/ its source
state/<plugin>/ the state of a plugin, by convention state/<plugin>/ the state of a plugin, by convention
state/core/journal.sqlite the journal: stderr of all processes, and core events state/core/journal.sqlite the journal: stderr of all processes, and core events
config/boot.yaml the services that start with the world config/boot.yaml ordered boot services, supervised for crashes (PROTOCOL §8)
config/agent.yaml the configuration of the agent config/agent.yaml the configuration of the agent
``` ```
Boot services restart on process crashes, not on readiness responses. More
than 5 crashes within 10 seconds stops automatic attempts for that service
and journals `service gave up`; an upgrade does not reset that decision.
Explicitly stopping or killing a service also leaves it stopped for this
host lifetime. See [Boot and services](PROTOCOL.md#8-boot-and-services).
Rust builds use the SDK in `plugins/cargo/sdk/` directly. The host seeds this
folder only when it is missing; restarts and host upgrades preserve existing
files. SDK edits take effect on the next `cargo.build`, without a restart.
To reset to the host's bundled SDK, back up any edits and remove the whole
folder; the next startup or build re-seeds it.
Each process also sees `/proc` (the processes and the state of the core), Each process also sees `/proc` (the processes and the state of the core),
`/self` (its own folder in `/proc`), and `/run` (shared files in memory). `/self` (its own folder in `/proc`), and `/run` (shared files in memory).
+28 -9
View File
@@ -94,19 +94,38 @@ impl Agent {
return Ok(final_text); return Ok(final_text);
} }
for (id, name, arguments) in calls { for (id, name, arguments) in calls {
let output = match serde_json::from_str(&arguments) { let args = serde_json::from_str::<serde_json::Value>(&arguments);
Ok(args) => tools::run(&name, &args) let output = match &args {
Ok(args) => tools::run(&name, args)
.await .await
.unwrap_or_else(|error| json!({"error":error})), .unwrap_or_else(|error| json!({"error":error})),
Err(error) => json!({"error":format!("arguments are not valid JSON: {error}")}), Err(error) => json!({"error":format!("arguments are not valid JSON: {error}")}),
}; };
append( let output = if name == "read_file"
&mut entries, && args
Entry::ToolResult { .as_ref()
id, .ok()
output: tools::limit_output(output.to_string()), .and_then(|a| a["path"].as_str())
}, .is_some_and(|p| p.starts_with("/logs/"))
)?; {
output["content"]
.as_str()
.map(str::to_owned)
.unwrap_or_else(|| output.to_string())
} else {
output.to_string()
};
let tail = args
.as_ref()
.ok()
.and_then(|a| a["method"].as_str())
.is_some_and(|m| {
[".exec", ".log", ".build", ".test"]
.iter()
.any(|suffix| m.ends_with(suffix))
});
let output = tools::overflow::guard(output, "default", &id, tail);
append(&mut entries, Entry::ToolResult { id, output })?;
} }
} }
} }
+258
View File
@@ -0,0 +1,258 @@
//! Context-size guard; full output is in the referenced file. Never a data limit.
use std::io::{self, Write};
use std::sync::atomic::{AtomicU64, Ordering};
// One policy point: Pi's 50 KiB / 2000 lines, including room for the reference.
const CONTEXT_LIMIT: (usize, usize) = (50 * 1024, 2000);
static NEXT: AtomicU64 = AtomicU64::new(0);
fn component(s: &str) -> String {
s.bytes()
.take(80)
.map(|b| {
if b.is_ascii_alphanumeric() || b == b'-' || b == b'_' {
b as char
} else {
'_'
}
})
.collect()
}
pub fn exceeds_limit(output: &str) -> bool {
output.len() > CONTEXT_LIMIT.0 || output.lines().count() > CONTEXT_LIMIT.1
}
/// Save before excerpting. On storage failure return all data, not a false reference.
pub fn guard(output: String, session: &str, id: &str, tail: bool) -> String {
if !exceeds_limit(&output) {
return output;
}
// Decode only results that crossed the context guard. Archive physical
// string lines, not JSON escapes, even if decoding makes the text smaller.
let output = serde_json::from_str::<serde_json::Value>(&output)
.map(|value| super::render(&value))
.unwrap_or(output);
let pid = std::env::var("SAUSAGE_PID").unwrap_or_else(|_| "test".into());
let dir = format!("/logs/agent/{}/{}", date(), component(session));
let saved = (|| -> io::Result<String> {
std::fs::create_dir_all(&dir)?;
// create_new prevents overwriting even if an ID is reused or a process restarts.
loop {
let path = format!(
"{dir}/{}-{}-{}.txt",
component(id),
component(&pid),
NEXT.fetch_add(1, Ordering::Relaxed)
);
match std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(&path)
{
Ok(mut f) => {
f.write_all(output.as_bytes())?;
return Ok(path);
}
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
})();
match saved {
Ok(path) => excerpt(&output, &path, tail, 0, output.len()),
Err(e) => {
eprintln!("overflow archive failed: {e}; retaining complete tool result");
output
}
}
}
fn date() -> String {
let days = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
/ 86400;
let z = days as i64 + 719468;
let era = z / 146097;
let doe = z - era * 146097;
let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365;
let y = yoe + era * 400;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let d = doy - (153 * mp + 2) / 5 + 1;
let m = mp + if mp < 10 { 3 } else { -9 };
format!("{:04}-{m:02}-{d:02}", y + i64::from(m <= 2))
}
/// Excerpt an existing archive in place. Byte/line ranges refer to that file,
/// including when a caller requests an oversized line or the entire archive.
pub fn existing(bytes: &[u8], path: &str, args: &serde_json::Value) -> String {
let text = String::from_utf8_lossy(bytes);
let (mut from, mut to) = if let Some(line) = args["line_offset"].as_u64() {
let from = text
.split_inclusive('\n')
.take(line as usize)
.map(str::len)
.sum::<usize>();
let len = text[from..]
.split_inclusive('\n')
.take(args["line_limit"].as_u64().unwrap_or(200) as usize)
.map(str::len)
.sum::<usize>();
(from, from + len)
} else {
let from = (args["offset"].as_u64().unwrap_or(0) as usize).min(text.len());
(
from,
args["max_bytes"].as_u64().map_or(text.len(), |n| {
from.saturating_add(n as usize).min(text.len())
}),
)
};
while !text.is_char_boundary(from) {
from += 1;
}
while to > from && !text.is_char_boundary(to) {
to -= 1;
}
to = to.max(from);
excerpt(&text, path, false, from, to)
}
fn excerpt(output: &str, path: &str, tail: bool, from: usize, to: usize) -> String {
let selected = &output[from..to];
let lines: Vec<&str> = selected.split_inclusive('\n').collect();
let budget = CONTEXT_LIMIT.0.saturating_sub(1024);
let mut start = 0;
let mut end = lines.len();
let mut bytes = 0;
if tail {
start = end;
while start > 0
&& end - start < CONTEXT_LIMIT.1.saturating_sub(2)
&& bytes + lines[start - 1].len() <= budget
{
start -= 1;
bytes += lines[start].len();
}
} else {
end = 0;
while end < lines.len()
&& end < CONTEXT_LIMIT.1.saturating_sub(2)
&& bytes + lines[end].len() <= budget
{
bytes += lines[end].len();
end += 1;
}
}
let mut shown = lines[start..end].concat();
// A single huge line: preserve valid UTF-8 and explicitly label a partial line.
let partial = start == end;
if partial && !lines.is_empty() {
if tail {
start = lines.len() - 1;
let line = lines[start];
let mut from = line.len().saturating_sub(budget);
while !line.is_char_boundary(from) {
from += 1;
}
shown.push_str(&line[from..]);
} else {
let mut to = budget.min(lines[0].len());
while !lines[0].is_char_boundary(to) {
to -= 1;
}
shown.push_str(&lines[0][..to]);
}
}
let mut byte_start = from + lines[..start].iter().map(|s| s.len()).sum::<usize>();
if partial && tail {
byte_start = to - shown.len();
}
let byte_end = byte_start + shown.len();
let line_start = output[..byte_start].bytes().filter(|b| *b == b'\n').count();
let line_end = line_start + shown.split_inclusive('\n').count();
let total_lines = output.split_inclusive('\n').count();
let partial = partial
|| (from > 0 && !output[..from].ends_with('\n'))
|| (to < output.len() && !output[..to].ends_with('\n'));
let read = if partial {
format!(
"Shown byte range [{byte_start},{byte_end}); continue with read_file path={path}, offset={byte_end}, max_bytes={budget} (or choose an earlier byte offset)"
)
} else {
format!(
"Read with read_file path={path}, line_offset={line_start}, line_limit={}; next line_offset={line_end}",
line_end - line_start
)
};
format!(
"{shown}\n[Showing lines {}-{} of {}{}; total {} bytes. Full output: {path}. {read}]",
line_start + 1,
line_end,
total_lines,
if partial { " (partial line)" } else { "" },
output.len()
)
}
/// Zero-based line offsets; byte-based reads remain unchanged when absent.
pub fn read_lines(bytes: &[u8], offset: usize, limit: usize) -> String {
String::from_utf8_lossy(bytes)
.split_inclusive('\n')
.skip(offset)
.take(limit)
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn full_archive_and_context_guard() {
let original = "😀 line\n".repeat(5400);
for tail in [false, true] {
let shown = guard(original.clone(), "guard-test", "call", tail);
assert!(shown.len() <= CONTEXT_LIMIT.0);
assert!(shown.lines().count() <= CONTEXT_LIMIT.1);
assert!(shown.contains(if tail {
"Showing lines 3403-5400"
} else {
"Showing lines 1-1998"
}));
let path = shown
.split("Full output: ")
.nth(1)
.unwrap()
.split(". Read with")
.next()
.unwrap();
assert_eq!(std::fs::read_to_string(path).unwrap(), original);
}
assert_eq!(guard("small".into(), "test", "id", false), "small");
}
#[test]
fn archive_failure_does_not_discard_data() {
std::fs::create_dir_all(format!("/logs/agent/{}", date())).unwrap();
std::fs::write(format!("/logs/agent/{}/not-a-directory", date()), "block").unwrap();
let output = "no loss\n".repeat(6000);
assert_eq!(
guard(output.clone(), "not-a-directory", "test", false),
output
);
}
#[test]
fn huge_line_and_exact_boundaries() {
for tail in [false, true] {
let result = guard("😀".repeat(40000), "test", "utf8", tail);
assert!(result.len() <= CONTEXT_LIMIT.0);
assert!(result.contains("partial line"));
}
let exact = "a\n".repeat(CONTEXT_LIMIT.1);
assert_eq!(guard(exact.clone(), "test", "exact", false), exact);
assert_eq!(read_lines(b"one\ntwo\nthree", 1, 1), "two\n");
assert_eq!(read_lines(b"one", 3, 1), "");
}
}
+2
View File
@@ -3,6 +3,8 @@ You are the Sausage agent. Answer the person. Use tools to inspect or change the
Read the contract at `/proc/core/PROTOCOL.md`. The Rust plugin SDK and its documentation are at `/plugins/cargo/sdk/`. Read the contract at `/proc/core/PROTOCOL.md`. The Rust plugin SDK and its documentation are at `/plugins/cargo/sdk/`.
Build a plugin with `cargo.build`. Activate its WASM file with `core.activate`. Build a plugin with `cargo.build`. Activate its WASM file with `core.activate`.
Your own source is at `/plugins/agent/source/`. You can change and rebuild it like any other plugin. Your own source is at `/plugins/agent/source/`. You can change and rebuild it like any other plugin.
HARD RULE: the mediator and all agents NEVER call `core.restart` (or its alias `core.exit`) without the person's explicit approval each time.
Restart requires the host to run under `#loop` (`SAUSAGE_LOOP=1`); it answers before clean shutdown with exit code 75.
Never call `agent.chat`. It would wait on the history lock that your current turn holds. Never call `agent.chat`. It would wait on the history lock that your current turn holds.
Write logs to stderr. Use call results and notifications for communication. Write logs to stderr. Use call results and notifications for communication.
The interface check does not cover persistent state. Preserve state compatibility when you change a plugin. The interface check does not cover persistent state. Preserve state compatibility when you change a plugin.
+85 -23
View File
@@ -4,21 +4,47 @@ use crate::llm::Tool;
use serde_json::{Value, json}; use serde_json::{Value, json};
use std::io; use std::io;
/// Maximum UTF-8 bytes in a serialized tool result, including its truncation marker. #[path = "overflow.rs"]
pub const MAX_OUTPUT_BYTES: usize = 65_536; pub mod overflow;
/// Bound every tool result before it enters history or a provider request. /// Human-readable, lossless text representation of tool JSON. String values
pub fn limit_output(mut output: String) -> String { /// retain real newlines, with JSON-quoted field/index labels for structure.
const MARKER: &str = "\n[tool output truncated at 65536 bytes]"; fn render(value: &Value) -> String {
if output.len() > MAX_OUTPUT_BYTES { fn visit(value: &Value, label: &str, out: &mut String) {
let mut end = MAX_OUTPUT_BYTES - MARKER.len(); match value {
while !output.is_char_boundary(end) { Value::Object(fields) => {
end -= 1; if fields.is_empty() {
out.push_str(&format!("{label}: {{}}\n"));
}
for (key, value) in fields {
visit(
value,
&format!("{label}[{}]", serde_json::to_string(key).unwrap()),
out,
);
}
}
Value::Array(items) => {
if items.is_empty() {
out.push_str(&format!("{label}: []\n"));
}
for (i, value) in items.iter().enumerate() {
visit(value, &format!("{label}[{i}]"), out);
}
}
Value::String(text) => {
out.push_str(&format!("{label} (string, {} bytes):\n", text.len()));
out.push_str(text);
out.push('\n');
}
_ => {
out.push_str(&format!("{label}: {value}\n"));
}
} }
output.truncate(end);
output.push_str(MARKER);
} }
output let mut out = String::new();
visit(value, "$", &mut out);
out
} }
/// Describe the five tools sent to either inference API. /// Describe the five tools sent to either inference API.
@@ -26,7 +52,7 @@ pub fn definitions() -> Vec<Tool> {
vec![ vec![
Tool { Tool {
name: "call", name: "call",
description: "Call a plugin method through the core. Await its result. Set detach to let its process survive this caller. Output is capped at 65536 bytes with a truncation marker.", description: "Call a plugin method through the core. Await its result. Set detach to let its process survive this caller. Large results show an excerpt with a full-output /logs reference.",
parameters: schema( parameters: schema(
json!({ json!({
"method": {"type": "string", "description": "Qualified plugin method"}, "method": {"type": "string", "description": "Qualified plugin method"},
@@ -38,19 +64,21 @@ pub fn definitions() -> Vec<Tool> {
}, },
Tool { Tool {
name: "read_file", name: "read_file",
description: "Read a text file in the world, including `/proc`. `offset` and `max_bytes` are optional. Output is capped at 65536 bytes with a truncation marker.", description: "Read a text file in the world, including `/proc`. `offset`/`max_bytes` select bytes; `line_offset`/`line_limit` select zero-based lines (default 200). Large results show an excerpt with a full-output /logs reference.",
parameters: schema( parameters: schema(
json!({ json!({
"path": {"type": "string"}, "path": {"type": "string"},
"offset": {"type": "integer", "description": "Byte offset (default 0)"}, "offset": {"type": "integer", "description": "Byte offset (default 0)"},
"max_bytes": {"type": "integer", "description": "Optional maximum number of bytes to read"} "max_bytes": {"type": "integer", "description": "Optional maximum number of bytes to read"},
"line_offset": {"type": "integer", "description": "Zero-based starting line; overrides bytes"},
"line_limit": {"type": "integer", "description": "Number of lines; default 200"}
}), }),
&["path"], &["path"],
), ),
}, },
Tool { Tool {
name: "write_file", name: "write_file",
description: "Write a text file in the world. Parent folders are created. Output is capped at 65536 bytes with a truncation marker.", description: "Write a text file in the world. Parent folders are created. Large results show an excerpt with a full-output /logs reference.",
parameters: schema( parameters: schema(
json!({"path": {"type": "string"}, "content": {"type": "string"}}), json!({"path": {"type": "string"}, "content": {"type": "string"}}),
&["path", "content"], &["path", "content"],
@@ -58,12 +86,12 @@ pub fn definitions() -> Vec<Tool> {
}, },
Tool { Tool {
name: "list_dir", name: "list_dir",
description: "List entries in a world directory. Returns each name, type, and size. Output is capped at 65536 bytes with a truncation marker.", description: "List entries in a world directory. Returns each name, type, and size. Large results show an excerpt with a full-output /logs reference.",
parameters: schema(json!({"path": {"type": "string"}}), &["path"]), parameters: schema(json!({"path": {"type": "string"}}), &["path"]),
}, },
Tool { Tool {
name: "copy_dir", name: "copy_dir",
description: "Copy a directory tree, for example a plugin source to a draft folder. Skips `target/` folders. Output is capped at 65536 bytes with a truncation marker.", description: "Copy a directory tree, for example a plugin source to a draft folder. Skips `target/` folders. Large results show an excerpt with a full-output /logs reference.",
parameters: schema( parameters: schema(
json!({"from": {"type": "string"}, "to": {"type": "string"}}), json!({"from": {"type": "string"}, "to": {"type": "string"}}),
&["from", "to"], &["from", "to"],
@@ -103,6 +131,16 @@ pub async fn run(name: &str, args: &Value) -> sausage_plugin::Result<Value> {
fn read_file(args: &Value) -> io::Result<Value> { fn read_file(args: &Value) -> io::Result<Value> {
let path = args["path"].as_str().unwrap_or_default(); let path = args["path"].as_str().unwrap_or_default();
let bytes = std::fs::read(path)?; let bytes = std::fs::read(path)?;
if path.starts_with("/logs/") {
return Ok(
json!({"path":path,"size":bytes.len(),"content":overflow::existing(&bytes, path, args)}),
);
}
if let Some(line) = args["line_offset"].as_u64() {
return Ok(
json!({"path":path,"size":bytes.len(),"line_offset":line,"content":overflow::read_lines(&bytes, line as usize, args["line_limit"].as_u64().unwrap_or(200) as usize)}),
);
}
let offset = (args["offset"].as_u64().unwrap_or(0) as usize).min(bytes.len()); let offset = (args["offset"].as_u64().unwrap_or(0) as usize).min(bytes.len());
let end = args["max_bytes"].as_u64().map_or(bytes.len(), |max| { let end = args["max_bytes"].as_u64().map_or(bytes.len(), |max| {
offset.saturating_add(max as usize).min(bytes.len()) offset.saturating_add(max as usize).min(bytes.len())
@@ -185,11 +223,35 @@ fn copy_tree(from: &str, to: &str) -> io::Result<u64> {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
#[test] #[test]
fn tool_output_limit_preserves_utf8_and_marks_truncation() { fn overflow_json_can_be_followed_by_line_without_rearchiving() {
let limited = super::limit_output("😀".repeat(40_000)); let value = serde_json::json!({"stdout":"hello 😀\n".repeat(4000),"exit":1,"error":{"text":"failure\n".repeat(4000)}});
assert!(limited.len() <= super::MAX_OUTPUT_BYTES); let expected = super::render(&value);
assert!(limited.ends_with("[tool output truncated at 65536 bytes]")); let shown = super::overflow::guard(value.to_string(), "json-test", "call", true);
assert_eq!(super::limit_output("small".into()), "small"); assert!(shown.len() <= 50 * 1024 && shown.lines().count() <= 2000);
let path = shown
.split("Full output: ")
.nth(1)
.unwrap()
.split(". Read with")
.next()
.unwrap();
assert_eq!(std::fs::read_to_string(path).unwrap(), expected);
let parent = std::path::Path::new(path).parent().unwrap();
let count = std::fs::read_dir(parent).unwrap().count();
let args = serde_json::json!({"path":path,"line_offset":30,"line_limit":5});
let read = super::read_file(&args).unwrap();
let read = read["content"].as_str().unwrap();
assert!(read.contains("Showing lines 31-35"));
assert!(read.contains(path));
assert_eq!(std::fs::read_dir(parent).unwrap().count(), count);
assert!(
super::overflow::existing(
expected.as_bytes(),
path,
&serde_json::json!({"line_offset":u64::MAX})
)
.contains("Full output:")
);
} }
#[test] #[test]
fn copy_rejects_a_destination_inside_the_source() { fn copy_rejects_a_destination_inside_the_source() {
+1
View File
@@ -36,6 +36,7 @@ chrono = { version = "0.4.42", default-features = false, features = ["clock"] }
tempfile = "3.27.0" tempfile = "3.27.0"
[dev-dependencies] [dev-dependencies]
tokio = { version = "1", features = ["test-util"] }
hyper = { version = "1.12.0", features = ["server", "http1"] } hyper = { version = "1.12.0", features = ["server", "http1"] }
hyper-util = { version = "0.1.21", features = ["tokio"] } hyper-util = { version = "0.1.21", features = ["tokio"] }
sausage-interface = { path = "../interface" } sausage-interface = { path = "../interface" }
+94 -13
View File
@@ -6,6 +6,44 @@ use crate::{
use serde_json::{Value, json}; use serde_json::{Value, json};
use std::{collections::BTreeMap, path::PathBuf}; use std::{collections::BTreeMap, path::PathBuf};
/// Method descriptions plus their reachable local references, without expanding cycles.
fn methods(document: &Value) -> BTreeMap<String, Value> {
fn references(value: &Value, document: &Value, found: &mut BTreeMap<String, Value>) {
match value {
Value::Object(object) => {
if let Some(reference) = object.get("$ref").and_then(Value::as_str)
&& let Some(pointer) = reference.strip_prefix('#')
&& !found.contains_key(reference)
&& let Some(target) = document.pointer(pointer)
{
found.insert(reference.into(), target.clone());
references(target, document, found);
}
for child in object.values() {
references(child, document, found);
}
}
Value::Array(array) => {
for child in array {
references(child, document, found);
}
}
_ => {}
}
}
document["methods"]
.as_array()
.into_iter()
.flatten()
.filter_map(|method| {
let name = method["name"].as_str()?;
let mut found = BTreeMap::new();
references(method, document, &mut found);
Some((name.into(), json!({"method":method,"references":found})))
})
.collect()
}
struct Candidate { struct Candidate {
plugin: String, plugin: String,
copy: PathBuf, copy: PathBuf,
@@ -113,7 +151,31 @@ impl Host {
.map(|candidate| candidate.plugin.clone()) .map(|candidate| candidate.plugin.clone())
.collect(); .collect();
self.refresh_graph_except(&names).await?; self.refresh_graph_except(&names).await?;
let previous = self.0.graph.lock().unwrap().plugins.clone(); let mut previous = self.0.graph.lock().unwrap().plugins.clone();
// Startup discovery may not yet have visited a replaced plugin. Discover
// its immutable old copy for the diff, but never prevent repairing it.
for candidate in candidates.iter() {
if let Some(old) = previous.get_mut(&candidate.plugin)
&& old.document.is_null()
&& old.error.is_none()
{
if old.hash == candidate.interface.hash {
old.document = candidate.interface.document.clone();
continue;
}
let directory = self.0.world.plugin_dir(&candidate.plugin);
if let Ok((bytes, hash)) = read_hashed(directory.join("plugin.wasm")).await {
let copy = directory.join(format!("discover-{}.wasm", uuid::Uuid::now_v7()));
if tokio::fs::write(&copy, bytes).await.is_ok()
&& let Ok(interface) =
self.discover_copy(&candidate.plugin, &copy, hash).await
{
*old = interface;
}
let _ = tokio::fs::remove_file(copy).await;
}
}
}
let mut proposed = previous.clone(); let mut proposed = previous.clone();
for candidate in candidates.iter() { for candidate in candidates.iter() {
proposed.insert(candidate.plugin.clone(), candidate.interface.clone()); proposed.insert(candidate.plugin.clone(), candidate.interface.clone());
@@ -148,7 +210,7 @@ impl Host {
); );
} }
} }
self.install_candidates(candidates, previous, proposed) self.install_candidates(candidates, previous, proposed, params["force"] != true)
.await .await
} }
/// Installs staged files and snapshots all old admissions under the shared write gate. /// Installs staged files and snapshots all old admissions under the shared write gate.
@@ -157,6 +219,7 @@ impl Host {
candidates: &[Candidate], candidates: &[Candidate],
previous: BTreeMap<String, Interface>, previous: BTreeMap<String, Interface>,
proposed: BTreeMap<String, Interface>, proposed: BTreeMap<String, Interface>,
checked: bool,
) -> RpcResult { ) -> RpcResult {
let admission = self.0.admission.write().await; let admission = self.0.admission.write().await;
// Every candidate already has a temporary name before any installed file changes. // Every candidate already has a temporary name before any installed file changes.
@@ -183,25 +246,43 @@ impl Host {
.map_err(io_error)?; .map_err(io_error)?;
} }
let mut upgrades = Vec::new(); let mut upgrades = Vec::new();
let mut documents = serde_json::Map::new(); let mut summaries = serde_json::Map::new();
for candidate in candidates { for candidate in candidates {
documents.insert( let old = previous.get(&candidate.plugin);
candidate.plugin.clone(), let before = methods(old.map(|old| &old.document).unwrap_or(&Value::Null));
candidate.interface.document.clone(), let after = methods(&candidate.interface.document);
); let added: Vec<_> = after
.keys()
.filter(|name| !before.contains_key(*name))
.collect();
let removed: Vec<_> = before
.keys()
.filter(|name| !after.contains_key(*name))
.collect();
let changed: Vec<_> = after
.iter()
.filter(|(name, method)| before.get(*name).is_some_and(|old| old != *method))
.map(|(name, _)| name)
.collect();
summaries.insert(candidate.plugin.clone(), json!({
"plugin": candidate.plugin,
"wasm": format!("/plugins/{}/plugin.wasm", candidate.plugin),
"hash": candidate.interface.hash,
"previous": old.map(|old| &old.hash).filter(|hash| !hash.is_empty()),
"method_count": candidate.interface.document["methods"].as_array().map_or(0, Vec::len),
"added": added,
"removed": removed,
"changed": changed,
"interface_check": if checked { "passed" } else { "skipped" },
}));
if let Some(old) = previous if let Some(old) = previous
.get(&candidate.plugin) .get(&candidate.plugin)
.filter(|old| !old.hash.is_empty() && old.hash != candidate.interface.hash) .filter(|old| !old.hash.is_empty() && old.hash != candidate.interface.hash)
{ {
let grace = old.document["x-sausage-grace"]
.as_f64()
.filter(|value| value.is_finite() && *value >= 0.0)
.unwrap_or(60.0);
upgrades.push(self.snapshot_upgrade( upgrades.push(self.snapshot_upgrade(
&candidate.plugin, &candidate.plugin,
old.hash.clone(), old.hash.clone(),
&candidate.interface.hash, &candidate.interface.hash,
grace,
)); ));
} }
self.0.journal.write( self.0.journal.write(
@@ -216,6 +297,6 @@ impl Host {
for upgrade in upgrades { for upgrade in upgrades {
self.begin_upgrade(upgrade); self.begin_upgrade(upgrade);
} }
Ok(Value::Object(documents)) Ok(Value::Object(summaries))
} }
} }
+3
View File
@@ -2,6 +2,9 @@
# Each method is a plugin.method name. params contains its named arguments. # Each method is a plugin.method name. params contains its named arguments.
# The core waits for each result before it starts the next service. # The core waits for each result before it starts the next service.
# After a plugin upgrade, the core restarts that plugin's services. # After a plugin upgrade, the core restarts that plugin's services.
# Host restart is separate: #loop sets SAUSAGE_LOOP=1 to enable core.restart
# (and its alias core.exit). It answers before clean shutdown with exit code 75.
# Agents need the person's explicit approval each time; never add it as a service.
# Example (replace [] with this list): # Example (replace [] with this list):
# - method: www.start # - method: www.start
# params: # params:
+454 -77
View File
@@ -3,12 +3,44 @@ use crate::{Host, world::World};
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
use include_dir::{Dir, include_dir}; use include_dir::{Dir, include_dir};
use serde_json::{Value, json}; use serde_json::{Value, json};
use sha2::{Digest, Sha256}; use std::time::Duration;
use std::{ use std::{
path::{Path, PathBuf}, path::{Path, PathBuf},
sync::Arc, sync::Arc,
}; };
use tokio::process::Command; use tokio::{process::Command, time::Instant};
/// Default wall-clock budget after acquiring the build mutex (not queue time).
pub const DEFAULT_TIMEOUT_SECS: u64 = 900;
/// Default wall-clock budget for each WASI test instance.
pub const DEFAULT_TEST_TIMEOUT_SECS: u64 = 120;
/// Wall-clock budgets for a build and its individual WASI tests.
#[derive(Clone, Copy, Debug)]
pub struct BuildOptions {
pub timeout_secs: u64,
pub test_timeout_secs: u64,
}
impl Default for BuildOptions {
fn default() -> Self {
Self {
timeout_secs: DEFAULT_TIMEOUT_SECS,
test_timeout_secs: DEFAULT_TEST_TIMEOUT_SECS,
}
}
}
/// Optional RPC seconds are positive integers; null does not disable a deadline.
fn seconds(params: &Value, name: &str, default: u64) -> Result<u64, Value> {
match params.get(name) {
None => Ok(default),
Some(value) => value
.as_u64()
.filter(|seconds| *seconds >= 1)
.ok_or_else(|| {
crate::error(-32602, &format!("{name} must be an integer of at least 1"))
}),
}
}
/// The native build plugin. /// The native build plugin.
pub const ID: &str = "cargo"; pub const ID: &str = "cargo";
@@ -25,9 +57,78 @@ pub struct TestResults {
pub failed: u64, pub failed: u64,
pub output: String, pub output: String,
} }
/// Raw, unfiltered output shared by the test and release phases of one build.
#[derive(Clone, Debug, serde::Serialize)]
pub struct BuildLogs {
pub stdout: StreamLog,
pub stderr: StreamLog,
pub total_bytes: u64,
}
#[derive(Clone, Debug, serde::Serialize)]
pub struct StreamLog {
pub path: String,
pub bytes: u64,
}
/// Persistent descriptors are inherited by cargo, so cancellation cannot lose
/// bytes already written by cargo or its descendants. No buffered writer is used.
struct LogFiles {
stdout: std::fs::File,
stderr: std::fs::File,
stdout_path: PathBuf,
stderr_path: PathBuf,
}
impl LogFiles {
fn create(world: &Path) -> Result<Self> {
let directory = world
.join("logs/cargo")
.join(chrono::Utc::now().format("%Y-%m-%d").to_string());
std::fs::create_dir_all(&directory).context("create cargo log directory")?;
let id = uuid::Uuid::now_v7();
let stdout_path = directory.join(format!("{id}.stdout.log"));
let stderr_path = directory.join(format!("{id}.stderr.log"));
let create = |path: &Path| {
std::fs::OpenOptions::new()
.create_new(true)
.append(true)
.open(path)
.with_context(|| format!("create {}", path.display()))
};
Ok(Self {
stdout: create(&stdout_path)?,
stderr: create(&stderr_path)?,
stdout_path,
stderr_path,
})
}
fn info(&self, world: &Path) -> Result<BuildLogs> {
let stream = |file: &std::fs::File, path: &Path| -> Result<StreamLog> {
Ok(StreamLog {
path: format!("/{}", path.strip_prefix(world)?.display()),
bytes: file.metadata()?.len(),
})
};
let stdout = stream(&self.stdout, &self.stdout_path)?;
let stderr = stream(&self.stderr, &self.stderr_path)?;
Ok(BuildLogs {
total_bytes: stdout.bytes + stderr.bytes,
stdout,
stderr,
})
}
fn read(&self, path: &Path, offset: u64) -> Result<Vec<u8>> {
use std::io::{Read, Seek, SeekFrom};
let mut file =
std::fs::File::open(path).with_context(|| format!("read {}", path.display()))?;
file.seek(SeekFrom::Start(offset))?;
let mut bytes = Vec::new();
file.read_to_end(&mut bytes)?;
Ok(bytes)
}
}
/// A successful build, usable without a running host. /// A successful build, usable without a running host.
#[derive(Debug)] #[derive(Debug)]
pub struct BuildOutput { pub struct BuildOutput {
pub logs: Option<BuildLogs>,
pub wasm: PathBuf, pub wasm: PathBuf,
pub warnings: String, pub warnings: String,
pub tests: TestResults, pub tests: TestResults,
@@ -35,9 +136,12 @@ pub struct BuildOutput {
/// A failed compiler or test phase, with its complete diagnostics. /// A failed compiler or test phase, with its complete diagnostics.
#[derive(Debug)] #[derive(Debug)]
pub struct BuildFailure { pub struct BuildFailure {
pub logs: Option<BuildLogs>,
pub reason: String, pub reason: String,
pub tests: TestResults, pub tests: TestResults,
pub output: String, pub output: String,
pub elapsed_secs: Option<f64>,
pub step: Option<&'static str>,
} }
impl std::fmt::Display for BuildFailure { impl std::fmt::Display for BuildFailure {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
@@ -48,9 +152,12 @@ impl std::error::Error for BuildFailure {}
impl From<anyhow::Error> for BuildFailure { impl From<anyhow::Error> for BuildFailure {
fn from(error: anyhow::Error) -> Self { fn from(error: anyhow::Error) -> Self {
Self { Self {
logs: None,
reason: "build preparation failed".into(), reason: "build preparation failed".into(),
output: format!("{error:#}"), output: format!("{error:#}"),
tests: TestResults::default(), tests: TestResults::default(),
elapsed_secs: None,
step: None,
} }
} }
} }
@@ -61,10 +168,9 @@ pub struct Cargo {
runner: PathBuf, runner: PathBuf,
target: PathBuf, target: PathBuf,
sdk: PathBuf, sdk: PathBuf,
sdk_files: Arc<std::collections::BTreeMap<PathBuf, Vec<u8>>>,
} }
impl Cargo { impl Cargo {
/// Installs changed SDK files and selects the current sausage executable as runner. /// Seeds a missing world SDK and selects the current sausage executable as runner.
pub fn new(world: World, cache: &Path) -> Result<Self> { pub fn new(world: World, cache: &Path) -> Result<Self> {
let mut runner = std::env::current_exe()?; let mut runner = std::env::current_exe()?;
if runner if runner
@@ -75,9 +181,23 @@ impl Cargo {
} }
Self::with_runner(world, runner, cache) Self::with_runner(world, runner, cache)
} }
/// Installs the SDK and accepts an explicit sausage binary, including for init integrations. /// Seeds a missing world SDK and accepts an explicit sausage binary, including for init integrations.
pub fn with_runner(world: World, runner: PathBuf, cache: &Path) -> Result<Self> { pub fn with_runner(world: World, runner: PathBuf, cache: &Path) -> Result<Self> {
let sdk = world.plugin_dir(ID).join("sdk"); let sdk = world.plugin_dir(ID).join("sdk");
Self::seed_sdk(&sdk)?;
Ok(Self {
sdk,
target: std::path::absolute(cache)?.join("cargo-target"),
world,
runner,
builds: Arc::new(tokio::sync::Mutex::new(())),
})
}
/// Seeds only an absent SDK; an existing world owns all of its SDK files.
fn seed_sdk(sdk: &Path) -> Result<()> {
if sdk.try_exists()? {
return Ok(());
}
let mut files = std::collections::BTreeMap::new(); let mut files = std::collections::BTreeMap::new();
collect_files(&SDK, Path::new("src"), &mut files); collect_files(&SDK, Path::new("src"), &mut files);
collect_files(&MACROS, Path::new("macros/src"), &mut files); collect_files(&MACROS, Path::new("macros/src"), &mut files);
@@ -115,26 +235,23 @@ impl Cargo {
} }
files.insert(destination, document.to_string().into_bytes()); files.insert(destination, document.to_string().into_bytes());
} }
let mut hash = Sha256::new(); let parent = sdk.parent().context("SDK has no parent")?;
std::fs::create_dir_all(parent).with_context(|| format!("create {}", parent.display()))?;
let staging = tempfile::Builder::new()
.prefix("seed-sdk-")
.tempdir_in(parent)
.with_context(|| format!("stage SDK in {}", parent.display()))?;
for (path, bytes) in &files { for (path, bytes) in &files {
let name = path.to_str().unwrap().as_bytes(); write_changed(&staging.path().join(path), bytes)?;
hash.update((name.len() as u64).to_le_bytes());
hash.update(name);
hash.update((bytes.len() as u64).to_le_bytes());
hash.update(bytes);
write_changed(&sdk.join(path), bytes)?;
} }
let cached = std::path::absolute(cache)? // Another builder may have seeded the world while we prepared the files.
.join("sdk") if !sdk.try_exists()?
.join(format!("{:x}", hash.finalize())); && let Err(error) = std::fs::rename(staging.path(), sdk)
Ok(Self { && !sdk.try_exists()?
sdk: cached, {
sdk_files: Arc::new(files), return Err(error).with_context(|| format!("seed {}", sdk.display()));
target: std::path::absolute(cache)?.join("cargo-target"), }
world, Ok(())
runner,
builds: Arc::new(tokio::sync::Mutex::new(())),
})
} }
/// Registers cargo.build as a cancellable native process. /// Registers cargo.build as a cancellable native process.
pub fn register(self, host: &Host) { pub fn register(self, host: &Host) {
@@ -142,19 +259,59 @@ impl Cargo {
host.register_native(ID,Arc::new(move |_context,method,params| { host.register_native(ID,Arc::new(move |_context,method,params| {
let builder=builder.clone(); let builder=builder.clone();
Box::pin(async move { Box::pin(async move {
if method=="rpc.discover" { return Ok(json!({"openrpc":"1.2.6","info":{"title":"cargo","version":"native"},"methods":[{"name":"build","paramStructure":"by-name","params":[{"name":"plugin","required":true,"schema":{"type":"string"}}],"result":{"name":"build","schema":{"type":"object","required":["wasm","tests","warnings"],"properties":{"wasm":{"type":"string"},"warnings":{"type":"string"},"tests":{"type":"object","required":["passed","failed","output"],"properties":{"passed":{"type":"integer","minimum":0},"failed":{"type":"integer","minimum":0},"output":{"type":"string"}}}}}},"errors":[{"code":-32004,"message":"Build failed"}]}]})); } if method=="rpc.discover" { return Ok(json!({"openrpc":"1.2.6","info":{"title":"cargo","version":"native"},"methods":[{"name":"build","paramStructure":"by-name","description":"Copies source, runs bounded WASI tests, then builds a release component. Deadlines kill cargo and its descendants; cancellation releases the build mutex.","params":[{"name":"plugin","required":true,"schema":{"type":"string"}},{"name":"timeout_secs","required":false,"description":"Total wall-clock limit in seconds for tests and release build after acquiring the build mutex (queue time excluded). Default 900; minimum 1. Cannot be disabled.","schema":{"type":"integer","minimum":1,"default":DEFAULT_TIMEOUT_SECS}},{"name":"test_timeout_secs","required":false,"description":"Wall-clock limit in seconds for each WASI test instance. Default 120; minimum 1. A timed-out test is named and fails; remaining tests continue.","schema":{"type":"integer","minimum":1,"default":DEFAULT_TEST_TIMEOUT_SECS}}],"result":{"name":"build","schema":{"type":"object","required":["wasm","tests","warnings","logs"],"properties":{"logs":{"type":"object","required":["stdout","stderr","total_bytes"],"properties":{"stdout":{"type":"object","required":["path","bytes"],"properties":{"path":{"type":"string"},"bytes":{"type":"integer","minimum":0}}},"stderr":{"type":"object","required":["path","bytes"],"properties":{"path":{"type":"string"},"bytes":{"type":"integer","minimum":0}}},"total_bytes":{"type":"integer","minimum":0}}},"wasm":{"type":"string"},"warnings":{"type":"string"},"tests":{"type":"object","required":["passed","failed","output"],"properties":{"passed":{"type":"integer","minimum":0},"failed":{"type":"integer","minimum":0},"output":{"type":"string"}}}}}},"errors":[{"code":-32004,"message":"Build failed"}]}]})); }
if method!="build" { return Err(crate::error(-32601,"cargo method not found")); } if method!="build" { return Err(crate::error(-32601,"cargo method not found")); }
let plugin=params["plugin"].as_str().ok_or_else(||crate::error(-32602,"plugin must be a string"))?; let plugin=params["plugin"].as_str().ok_or_else(||crate::error(-32602,"plugin must be a string"))?;
match builder.build(&builder.world.plugin_dir(plugin).join("source")).await { let options = BuildOptions {
Ok(output)=>Ok(json!({"wasm":format!("/{}",output.wasm.strip_prefix(&builder.world.root).unwrap().display()),"tests":output.tests,"warnings":output.warnings})), timeout_secs: seconds(&params, "timeout_secs", DEFAULT_TIMEOUT_SECS)?,
Err(failure)=>Err(json!({"code":-32004,"message":failure.reason,"data":{"tests":failure.tests,"output":failure.output}})), test_timeout_secs: seconds(&params, "test_timeout_secs", DEFAULT_TEST_TIMEOUT_SECS)?,
};
match builder.build_with_options(&builder.world.plugin_dir(plugin).join("source"), options).await {
Ok(output)=>Ok(json!({"wasm":format!("/{}",output.wasm.strip_prefix(&builder.world.root).unwrap().display()),"tests":output.tests,"warnings":output.warnings,"logs":output.logs})),
Err(failure)=>Err(json!({"code":-32004,"message":failure.reason,"data":{"reason":failure.reason,"tests":failure.tests,"output":failure.output,"elapsed_secs":failure.elapsed_secs,"step":failure.step,"logs":failure.logs}})),
} }
}) })
})); }));
} }
/// Copies source once, runs WASI tests, then publishes a release component under state/cargo/builds. /// Copies source once, runs WASI tests, then publishes a release component under state/cargo/builds.
pub async fn build(&self, source: &Path) -> std::result::Result<BuildOutput, BuildFailure> { pub async fn build(&self, source: &Path) -> std::result::Result<BuildOutput, BuildFailure> {
self.build_with_options(source, BuildOptions::default())
.await
}
/// Like build, with explicit positive wall-clock budgets. Waiting for the mutex is unbounded.
pub async fn build_with_options(
&self,
source: &Path,
options: BuildOptions,
) -> std::result::Result<BuildOutput, BuildFailure> {
let logs = LogFiles::create(&self.world.root)?;
let result = self.build_logged(source, options, &logs).await;
let info = logs.info(&self.world.root)?;
match result {
Ok(mut output) => {
output.logs = Some(info);
Ok(output)
}
Err(mut failure) => {
failure.logs = Some(info);
Err(failure)
}
}
}
async fn build_logged(
&self,
source: &Path,
options: BuildOptions,
logs: &LogFiles,
) -> std::result::Result<BuildOutput, BuildFailure> {
if options.timeout_secs == 0 || options.test_timeout_secs == 0 {
return Err(anyhow::anyhow!("timeouts must be at least 1 second").into());
}
let build_guard = self.builds.clone().lock_owned().await; let build_guard = self.builds.clone().lock_owned().await;
let started = Instant::now();
let deadline = started
.checked_add(Duration::from_secs(options.timeout_secs))
.ok_or_else(|| anyhow::anyhow!("timeout_secs is too large"))?;
let work = self.world.state_dir(ID).join("work"); let work = self.world.state_dir(ID).join("work");
let target = self.target.clone(); let target = self.target.clone();
let temporary = self.world.state_dir(ID).join("tmp"); let temporary = self.world.state_dir(ID).join("tmp");
@@ -162,27 +319,8 @@ impl Cargo {
let source = source.to_path_buf(); let source = source.to_path_buf();
let copy = work.clone(); let copy = work.clone();
let sdk = self.sdk.clone(); let sdk = self.sdk.clone();
let sdk_files = self.sdk_files.clone();
let build_guard = tokio::task::spawn_blocking(move || -> Result<_> { let build_guard = tokio::task::spawn_blocking(move || -> Result<_> {
// Host startup installs only the readable world copy. Builds populate the cache. Self::seed_sdk(&sdk)?;
if !sdk.is_dir() {
let parent = sdk.parent().unwrap();
std::fs::create_dir_all(parent)
.with_context(|| format!("create {}", parent.display()))?;
let staging = tempfile::Builder::new()
.prefix("install-")
.tempdir_in(parent)
.with_context(|| format!("stage SDK in {}", parent.display()))?;
for (path, bytes) in sdk_files.iter() {
write_changed(&staging.path().join(path), bytes)?;
}
// Another builder can publish the same complete SDK first.
if let Err(error) = std::fs::rename(staging.path(), &sdk)
&& !sdk.is_dir()
{
return Err(error).with_context(|| format!("install {}", sdk.display()));
}
}
if cleanup.exists() { if cleanup.exists() {
std::fs::remove_dir_all(&cleanup) std::fs::remove_dir_all(&cleanup)
.with_context(|| format!("remove {}", cleanup.display()))?; .with_context(|| format!("remove {}", cleanup.display()))?;
@@ -208,16 +346,35 @@ impl Cargo {
.await .await
.map_err(anyhow::Error::from)??; .map_err(anyhow::Error::from)??;
let mut test = Command::new("cargo"); let mut test = Command::new("cargo");
test.args(["test", "--release", "--target", TARGET, "--target-dir"]) test.args([
.arg(&target) "test",
.current_dir(&work) "--no-fail-fast",
.env("TMPDIR", &temporary) "--release",
.arg("--config") "--target",
.arg(format!( TARGET,
"target.wasm32-wasip2.runner={}", "--target-dir",
json!([self.runner, "test-runner"]) ])
)); .arg(&target)
let output = run_command(test).await?; .current_dir(&work)
.env("TMPDIR", &temporary)
.env(
"SAUSAGE_TEST_TIMEOUT_SECS",
options.test_timeout_secs.to_string(),
)
.arg("--config")
.arg(format!(
"target.wasm32-wasip2.runner={}",
json!([self.runner, "test-runner"])
));
let output = run_command(
test,
started,
deadline,
"tests",
TestResults::default(),
logs,
)
.await?;
let stdout = String::from_utf8_lossy(&output.stdout); let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr); let stderr = String::from_utf8_lossy(&output.stderr);
let tests = TestResults { let tests = TestResults {
@@ -231,25 +388,36 @@ impl Cargo {
}; };
if !output.status.success() { if !output.status.success() {
return Err(BuildFailure { return Err(BuildFailure {
logs: None,
reason: "WASI tests failed".into(), reason: "WASI tests failed".into(),
output: tests.output.clone(), output: tests.output.clone(),
tests, tests,
elapsed_secs: None,
step: None,
}); });
} }
let mut build = Command::new("cargo");
build
.args([
"build",
"--release",
"--target",
TARGET,
"--message-format=json-render-diagnostics",
"--target-dir",
])
.arg(&target)
.current_dir(&work);
let output = run_command(
build,
started,
deadline,
"release build",
tests.clone(),
logs,
)
.await?;
let result = async { let result = async {
let mut build = Command::new("cargo");
build
.args([
"build",
"--release",
"--target",
TARGET,
"--message-format=json-render-diagnostics",
"--target-dir",
])
.arg(&target)
.current_dir(&work);
let output = run_command(build).await?;
let stdout = String::from_utf8_lossy(&output.stdout); let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr); let stderr = String::from_utf8_lossy(&output.stderr);
let diagnostics = stderr.to_string(); let diagnostics = stderr.to_string();
@@ -282,6 +450,7 @@ impl Cargo {
}) })
.await??; .await??;
Ok(BuildOutput { Ok(BuildOutput {
logs: None,
wasm: destination, wasm: destination,
warnings: strip_progress(&diagnostics), warnings: strip_progress(&diagnostics),
tests: tests.clone(), tests: tests.clone(),
@@ -289,9 +458,12 @@ impl Cargo {
} }
.await; .await;
result.map_err(|error: anyhow::Error| BuildFailure { result.map_err(|error: anyhow::Error| BuildFailure {
logs: None,
reason: "release build failed".into(), reason: "release build failed".into(),
output: format!("{error:#}"), output: format!("{error:#}"),
tests, tests,
elapsed_secs: None,
step: None,
}) })
} }
} }
@@ -306,16 +478,81 @@ impl Drop for ProcessGroup {
} }
} }
/// Starts one process group and retains its guard across all output collection. /// Starts one process group and retains its guard across all output collection.
async fn run_command(mut command: Command) -> Result<std::process::Output> { async fn run_command(
mut command: Command,
started: Instant,
deadline: Instant,
step: &'static str,
mut tests: TestResults,
logs: &LogFiles,
) -> std::result::Result<std::process::Output, BuildFailure> {
command command
.process_group(0) .process_group(0)
.kill_on_drop(true) .kill_on_drop(true)
.stdin(std::process::Stdio::null()) .stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::piped()) // Child writes go straight to persistent files: no pipe buffers or reader
.stderr(std::process::Stdio::piped()); // tasks can be dropped on cancellation. Parsing happens only afterwards.
let child = command.spawn().context("start cargo")?; .stdout(logs.stdout.try_clone().context("clone stdout log")?)
let _group = ProcessGroup(child.id().unwrap()); .stderr(logs.stderr.try_clone().context("clone stderr log")?);
Ok(child.wait_with_output().await?) let out_start = logs.stdout.metadata().context("stat stdout log")?.len();
let err_start = logs.stderr.metadata().context("stat stderr log")?.len();
let mut child = command.spawn().context("start cargo")?;
let group = ProcessGroup(child.id().unwrap());
let (status, timed_out) = {
let collect = child.wait();
tokio::pin!(collect);
tokio::select! {
biased;
_ = tokio::time::sleep_until(deadline) => {
// Kill the entire tree before waiting for EOF/reaping cargo. The guard also
// covers cancellation while collecting, and runs before the mutex drops.
let _ = nix::sys::signal::killpg(nix::unistd::Pid::from_raw(group.0 as i32), nix::sys::signal::Signal::SIGKILL);
(collect.await, true)
}
status = &mut collect => (status, false),
}
};
let status = status.map_err(anyhow::Error::from)?;
// Also stop any descendants that kept the streams open after cargo exited.
drop(group);
let out = logs.read(&logs.stdout_path, out_start)?;
let err = logs.read(&logs.stderr_path, err_start)?;
if timed_out {
let output = format!(
"{}\n{}",
String::from_utf8_lossy(&out),
String::from_utf8_lossy(&err)
);
let output = output
.trim_end()
.lines()
.rev()
.take(50)
.collect::<Vec<_>>()
.into_iter()
.rev()
.collect::<Vec<_>>()
.join("\n");
if step == "tests" {
tests.passed = test_count(&String::from_utf8_lossy(&out), "passed");
tests.failed = test_count(&String::from_utf8_lossy(&out), "failed");
tests.output = output.clone();
}
return Err(BuildFailure {
logs: None,
reason: "timeout".into(),
tests,
output,
elapsed_secs: Some(started.elapsed().as_secs_f64()),
step: Some(step),
});
}
Ok(std::process::Output {
status,
stdout: out,
stderr: err,
})
} }
/// Copies only source, excluding build outputs and git internals at every level. /// Copies only source, excluding build outputs and git internals at every level.
fn copy_source(source: &Path, destination: &Path, root: bool) -> Result<()> { fn copy_source(source: &Path, destination: &Path, root: bool) -> Result<()> {
@@ -412,7 +649,7 @@ fn write_changed(path: &Path, bytes: &[u8]) -> Result<()> {
} }
Ok(()) Ok(())
} }
/// Collects embedded source files for deterministic hashing and installation. /// Collects embedded source files for seeding a missing world SDK.
fn collect_files( fn collect_files(
directory: &Dir<'_>, directory: &Dir<'_>,
destination: &Path, destination: &Path,
@@ -457,3 +694,143 @@ fn strip_progress(output: &str) -> String {
.collect::<Vec<_>>() .collect::<Vec<_>>()
.join("\n") .join("\n")
} }
#[cfg(test)]
mod timeout_tests {
use super::*;
/// Streams retain exact bytes, including non-UTF8, progress and JSON; phases append.
#[tokio::test]
async fn raw_streams_append_before_parsing() {
let directory = tempfile::tempdir().unwrap();
let logs = LogFiles::create(directory.path()).unwrap();
for _ in 0..2 {
let mut command = Command::new("sh");
command.args(["-c", "printf '\\377raw\\r'; printf ' Compiling progress\\n{\"reason\":\"compiler-artifact\"}\\n' >&2; exit 3"]);
let now = Instant::now();
let output = tokio::time::timeout(
Duration::from_secs(5),
run_command(
command,
now,
now + Duration::from_secs(3),
"tests",
TestResults::default(),
&logs,
),
)
.await
.unwrap()
.unwrap();
assert!(!output.status.success());
assert_eq!(output.stdout, b"\xffraw\r");
assert!(String::from_utf8_lossy(&output.stderr).contains("Compiling progress"));
}
assert_eq!(
std::fs::read(&logs.stdout_path).unwrap(),
b"\xffraw\r\xffraw\r"
);
let info = logs.info(directory.path()).unwrap();
assert_eq!(info.stdout.bytes, 10);
assert_eq!(info.total_bytes, info.stdout.bytes + info.stderr.bytes);
assert!(info.stdout.path.starts_with("/logs/cargo/"));
assert!(!directory.path().join("state/cargo").exists());
}
/// Cancellation leaves kernel-written data intact, even without a newline.
#[tokio::test]
async fn cancelled_command_keeps_logs() {
use notify::Watcher;
let directory = tempfile::tempdir().unwrap();
let logs = LogFiles::create(directory.path()).unwrap();
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel();
let mut watcher =
notify::recommended_watcher(move |event: notify::Result<notify::Event>| {
if let Ok(event) = event {
if event
.paths
.iter()
.any(|path| path.file_name().is_some_and(|name| name == "ready"))
{
let _ = tx.send(());
}
}
})
.unwrap();
watcher
.watch(directory.path(), notify::RecursiveMode::NonRecursive)
.unwrap();
let mut command = Command::new("sh");
command.args(["-c", "printf 'partial\\377'; printf 'stderr\\r' >&2; touch \"$1/ready\"; exec sleep 1000", "sh"]).arg(directory.path());
let now = Instant::now();
let task = tokio::spawn(async move {
run_command(
command,
now,
now + Duration::from_secs(60),
"tests",
TestResults::default(),
&logs,
)
.await
});
tokio::time::timeout(Duration::from_secs(5), rx.recv())
.await
.unwrap()
.unwrap();
task.abort();
assert!(task.await.unwrap_err().is_cancelled());
let paths: Vec<_> = std::fs::read_dir(directory.path().join("logs/cargo"))
.unwrap()
.flat_map(|day| std::fs::read_dir(day.unwrap().path()).unwrap())
.map(|file| file.unwrap().path())
.collect();
assert_eq!(paths.len(), 2);
for path in paths {
let expected: &[u8] = if path.to_string_lossy().ends_with(".stdout.log") {
b"partial\xff"
} else {
b"stderr\r"
};
assert_eq!(std::fs::read(path).unwrap(), expected);
}
}
/// Deadline errors retain the shared start time, phase, test totals, and output tail.
#[tokio::test]
async fn timeout_diagnostics() {
let mut command = Command::new("sh");
command.args([
"-c",
"i=0; while [ $i -lt 100 ]; do echo diagnostic-$i; i=$((i+1)); done; exec sleep 1000",
]);
let directory = tempfile::tempdir().unwrap();
let logs = LogFiles::create(directory.path()).unwrap();
let now = Instant::now();
let failure = run_command(
command,
now - Duration::from_secs(10),
now + Duration::from_secs(1),
"release build",
TestResults {
passed: 7,
failed: 0,
output: "successful tests".into(),
},
&logs,
)
.await
.unwrap_err();
assert_eq!(failure.reason, "timeout");
assert_eq!(failure.step, Some("release build"));
assert!(failure.elapsed_secs.unwrap() >= 11.0);
assert_eq!(failure.tests.passed, 7);
assert_eq!(failure.tests.output, "successful tests");
assert_eq!(failure.output.lines().count(), 50);
assert!(failure.output.contains("diagnostic-99"));
assert!(!failure.output.contains("diagnostic-0\n"));
let raw = std::fs::read_to_string(&logs.stdout_path).unwrap();
assert_eq!(raw.lines().count(), 100);
assert!(raw.starts_with("diagnostic-0\n"));
}
}
+2 -4
View File
@@ -12,6 +12,7 @@ impl Host {
p: Value, p: Value,
) -> RpcResult { ) -> RpcResult {
match method { match method {
"watch" => self.watch(caller, id, required_str(&p, "path")?).await,
"call" => { "call" => {
self.route( self.route(
caller.into(), caller.into(),
@@ -62,10 +63,7 @@ impl Host {
} }
"log" => Ok(self.0.journal.query(p).await), "log" => Ok(self.0.journal.query(p).await),
"activate" | "revert" => self.activation(caller, method, p).await, "activate" | "revert" => self.activation(caller, method, p).await,
"exit" => { "restart" | "exit" => self.request_restart(),
self.0.exit.send_replace(true);
Ok(json!({}))
}
_ => Err(error(-32601, "method not found")), _ => Err(error(-32601, "method not found")),
} }
} }
+24 -7
View File
@@ -47,23 +47,40 @@ impl Host {
tracker: Process, tracker: Process,
) -> anyhow::Result<Mounts> { ) -> anyhow::Result<Mounts> {
let root = self.0.world.root.clone(); let root = self.0.world.root.clone();
let operations = tracker.clone();
let host = tracker let host = tracker
.clone() .clone()
.blocking(move || sausage_vfs::Host::new(root, tracker)) .blocking(move || sausage_vfs::Host::new(root, tracker))
.await?; .await?;
let mut mounts = Mounts::default(); let mut mounts = Mounts::default();
mounts.mount("/", Arc::new(host), false); mounts.mount(
"/",
self.0
.observer
.layer(Arc::new(host), "/", operations.clone()),
false,
);
mounts.mount("/proc", Arc::new(Proc(self.clone())), true); mounts.mount("/proc", Arc::new(Proc(self.clone())), true);
mounts.mount( mounts.mount(
"/self", "/self",
Arc::new(SelfFolder { self.0.observer.layer(
host: self.clone(), Arc::new(SelfFolder {
pid, host: self.clone(),
memory, pid: pid.clone(),
}), memory,
}),
&format!("/proc/{}", pid.as_deref().unwrap_or("console")),
operations.clone(),
),
false,
);
mounts.mount(
"/run",
self.0
.observer
.layer(Arc::new(self.0.run.clone()), "/run", operations),
false, false,
); );
mounts.mount("/run", Arc::new(self.0.run.clone()), false);
Ok(mounts) Ok(mounts)
} }
fn core_files(&self) -> Generated { fn core_files(&self) -> Generated {
+1 -10
View File
@@ -223,16 +223,7 @@ impl Host {
if let Some(old) = old_plugins.get(plugin).filter(|old| { if let Some(old) = old_plugins.get(plugin).filter(|old| {
!old.hash.is_empty() && old.hash != current.hash && current.error.is_none() !old.hash.is_empty() && old.hash != current.hash && current.error.is_none()
}) { }) {
let grace = old.document["x-sausage-grace"] upgrades.push(self.snapshot_upgrade(plugin, old.hash.clone(), &current.hash));
.as_f64()
.filter(|value| value.is_finite() && *value >= 0.0)
.unwrap_or(60.0);
upgrades.push(self.snapshot_upgrade(
plugin,
old.hash.clone(),
&current.hash,
grace,
));
} }
} }
self.publish_graph(plugins); self.publish_graph(plugins);
+19 -4
View File
@@ -26,6 +26,8 @@ pub(crate) struct Inner {
pub(crate) journal: Arc<journal::Journal>, pub(crate) journal: Arc<journal::Journal>,
pub(crate) notifications: tokio::sync::broadcast::Sender<Value>, pub(crate) notifications: tokio::sync::broadcast::Sender<Value>,
pub(crate) exit: watch::Sender<bool>, pub(crate) exit: watch::Sender<bool>,
pub(crate) loop_supervised: bool,
pub(crate) restart_pending: std::sync::atomic::AtomicBool,
pub(crate) components: components::Components, pub(crate) components: components::Components,
/// Calls hold it for reading from the choice of a version until their /// Calls hold it for reading from the choice of a version until their
/// process is in the table. Activation holds it for writing while it /// process is in the table. Activation holds it for writing while it
@@ -35,6 +37,8 @@ pub(crate) struct Inner {
/// processes, and upgrade tasks do nothing. /// processes, and upgrade tasks do nothing.
pub(crate) shutdown: std::sync::atomic::AtomicBool, pub(crate) shutdown: std::sync::atomic::AtomicBool,
pub(crate) run: sausage_vfs::Memory, pub(crate) run: sausage_vfs::Memory,
pub(crate) watches: crate::watches::Watches,
pub(crate) observer: sausage_vfs::observe::Observer,
pub(crate) graph: Mutex<crate::graph::Graph>, pub(crate) graph: Mutex<crate::graph::Graph>,
pub(crate) activations: tokio::sync::Mutex<()>, pub(crate) activations: tokio::sync::Mutex<()>,
pub(crate) watcher: Mutex<Option<notify::RecommendedWatcher>>, pub(crate) watcher: Mutex<Option<notify::RecommendedWatcher>>,
@@ -64,8 +68,11 @@ impl Host {
Ok((World::open(&path)?, Runtime::new(&cache)?)) Ok((World::open(&path)?, Runtime::new(&cache)?))
}) })
.await??; .await??;
let journal = let journal = journal::Journal::open_with_legacy(
journal::Journal::open(world.state_dir("core").join("journal.sqlite")).await?; world.root.join("logs/core/journal.sqlite"),
Some(world.state_dir("core").join("journal.sqlite")),
)
.await?;
let host = Self::with_parts(world, runtime, journal); let host = Self::with_parts(world, runtime, journal);
let cargo_world = host.0.world.clone(); let cargo_world = host.0.world.clone();
let cargo = tokio::task::spawn_blocking(move || { let cargo = tokio::task::spawn_blocking(move || {
@@ -86,6 +93,10 @@ impl Host {
)) ))
} }
fn with_parts(world: World, runtime: Arc<Runtime>, journal: journal::Journal) -> Self { fn with_parts(world: World, runtime: Arc<Runtime>, journal: journal::Journal) -> Self {
let watches = crate::watches::Watches::default();
let events = watches.clone();
let observer =
sausage_vfs::observe::Observer::new(move |path| crate::watches::changed(&events, path));
Self(Arc::new(Inner { Self(Arc::new(Inner {
world, world,
runtime, runtime,
@@ -98,10 +109,14 @@ impl Host {
}), }),
notifications: tokio::sync::broadcast::channel(1024).0, notifications: tokio::sync::broadcast::channel(1024).0,
exit: watch::channel(false).0, exit: watch::channel(false).0,
loop_supervised: crate::restart::enabled(std::env::var_os("SAUSAGE_LOOP").as_deref()),
restart_pending: std::sync::atomic::AtomicBool::new(false),
components: components::Components::default(), components: components::Components::default(),
admission: tokio::sync::RwLock::new(()), admission: tokio::sync::RwLock::new(()),
shutdown: std::sync::atomic::AtomicBool::new(false), shutdown: std::sync::atomic::AtomicBool::new(false),
run: sausage_vfs::Memory::default(), run: sausage_vfs::Memory::default(),
watches,
observer,
graph: Mutex::new(crate::graph::Graph::default()), graph: Mutex::new(crate::graph::Graph::default()),
activations: tokio::sync::Mutex::new(()), activations: tokio::sync::Mutex::new(()),
watcher: Mutex::new(None), watcher: Mutex::new(None),
@@ -143,12 +158,12 @@ impl Host {
) )
.await .await
} }
/// Wait until a core.exit handler requests a restart. /// Wait until core.restart (or its alias core.exit) requests shutdown after its response grace.
pub async fn exit_requested(&self) { pub async fn exit_requested(&self) {
let mut exit = self.0.exit.subscribe(); let mut exit = self.0.exit.subscribe();
let _ = exit.wait_for(|requested| *requested).await; let _ = exit.wait_for(|requested| *requested).await;
} }
/// Return whether a core.exit handler requested a restart. /// Return whether a restart has reached the normal shutdown path.
pub fn restart_requested(&self) -> bool { pub fn restart_requested(&self) -> bool {
*self.0.exit.borrow() *self.0.exit.borrow()
} }
+274 -31
View File
@@ -1,13 +1,17 @@
//! A dedicated SQLite writer batches entries and serves ordered queries without propagating disk failures into the host. //! A dedicated SQLite writer batches entries and serves ordered queries without propagating disk failures into the host.
use rusqlite::{Connection, params}; use rusqlite::{Connection, OpenFlags, params};
use serde::{Deserialize, Serialize};
use serde_json::{Value, json}; use serde_json::{Value, json};
use std::io::Write;
use std::{ use std::{
collections::HashSet, collections::HashSet,
path::PathBuf, path::PathBuf,
sync::{Mutex, mpsc}, sync::{Mutex, mpsc},
}; };
use tokio::sync::oneshot; use tokio::sync::oneshot;
#[derive(Debug, Serialize, Deserialize)]
struct Entry { struct Entry {
time: String,
id: String, id: String,
pid: String, pid: String,
plugin: String, plugin: String,
@@ -21,44 +25,83 @@ enum Command {
pub(crate) struct Journal { pub(crate) struct Journal {
sender: mpsc::Sender<Command>, sender: mpsc::Sender<Command>,
warned: Mutex<HashSet<String>>, warned: Mutex<HashSet<String>>,
spool: PathBuf,
} }
impl Journal { impl Journal {
pub(crate) async fn memory() -> anyhow::Result<Self> { pub(crate) async fn memory() -> anyhow::Result<Self> {
Self::open(PathBuf::from(":memory:")).await Self::open(PathBuf::from(":memory:")).await
} }
pub(crate) async fn open(path: PathBuf) -> anyhow::Result<Self> { pub(crate) async fn open(path: PathBuf) -> anyhow::Result<Self> {
Self::open_with_legacy(path, None).await
}
pub(crate) async fn open_with_legacy(
path: PathBuf,
legacy: Option<PathBuf>,
) -> anyhow::Result<Self> {
let spool = path
.parent()
.unwrap_or(std::path::Path::new("."))
.join("failed-batches");
let worker_spool = spool.clone();
let (sender, receiver) = mpsc::channel(); let (sender, receiver) = mpsc::channel();
let (ready, initialized) = oneshot::channel(); let (ready, initialized) = oneshot::channel();
std::thread::spawn(move || match open_database(path) { std::thread::spawn(move || {
Ok(mut db) => { match (|| -> anyhow::Result<_> {
let _ = ready.send(Ok(())); let db = open_database(path)?;
while let Ok(first) = receiver.recv() { // Never rename, checkpoint, or write the old SQLite/WAL files.
let mut batch = vec![first]; let old = legacy
batch.extend(receiver.try_iter().take(255)); .filter(|p| p.exists())
write_batch(&mut db, batch); .map(|p| Connection::open_with_flags(p, OpenFlags::SQLITE_OPEN_READ_ONLY))
.transpose()?;
Ok((db, old))
})() {
Ok((mut db, old)) => {
let _ = ready.send(Ok(()));
let mut pending = Vec::new();
while let Ok(first) = receiver.recv() {
let mut batch = vec![first];
batch.extend(receiver.try_iter().take(255));
pending = write_batch(&mut db, old.as_ref(), &worker_spool, pending, batch);
}
if !pending.is_empty() {
// Both storage locations failed. Last resort: retain full text in host stderr.
eprintln!(
"journal shutdown with unpersisted entries: {}",
serde_json::to_string(&pending).unwrap()
);
}
}
Err(error) => {
eprintln!("journal open failed: {error}");
let _ = ready.send(Err(error));
} }
}
Err(error) => {
eprintln!("journal open failed: {error}");
let _ = ready.send(Err(error));
} }
}); });
initialized.await??; initialized.await??;
Ok(Self { Ok(Self {
sender, sender,
warned: Mutex::new(HashSet::new()), warned: Mutex::new(HashSet::new()),
spool,
}) })
} }
pub(crate) fn write(&self, pid: &str, plugin: &str, stream: &str, line: &str) { pub(crate) fn write(&self, pid: &str, plugin: &str, stream: &str, line: &str) {
let entry = Entry { let entry = Entry {
time: chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Millis, true),
id: uuid::Uuid::now_v7().to_string(), id: uuid::Uuid::now_v7().to_string(),
pid: pid.into(), pid: pid.into(),
plugin: plugin.into(), plugin: plugin.into(),
stream: stream.into(), stream: stream.into(),
line: line.into(), line: line.into(),
}; };
if self.sender.send(Command::Write(entry)).is_err() { if let Err(mpsc::SendError(Command::Write(entry))) = self.sender.send(Command::Write(entry))
eprintln!("journal writer stopped"); {
eprintln!("journal writer stopped; saving entry outside SQLite");
if let Err(error) = spool_batch(&self.spool, std::slice::from_ref(&entry)) {
eprintln!(
"journal spool failed: {error}; unpersisted entry: {}",
serde_json::to_string(&entry).unwrap()
);
}
} }
} }
pub(crate) fn warning(&self, pid: &str, plugin: &str) { pub(crate) fn warning(&self, pid: &str, plugin: &str) {
@@ -94,28 +137,94 @@ fn open_database(path: PathBuf) -> anyhow::Result<Connection> {
CREATE INDEX IF NOT EXISTS journal_plugin ON journal(plugin);")?; CREATE INDEX IF NOT EXISTS journal_plugin ON journal(plugin);")?;
Ok(db) Ok(db)
} }
fn write_batch(db: &mut Connection, batch: Vec<Command>) { /// Exclusive, synced JSON files preserve failed transactions without a busy retry loop.
let write = (|| -> rusqlite::Result<()> { /// Recovery is deliberate: files are never automatically removed or replayed.
let transaction = db.transaction()?; fn spool_batch(dir: &std::path::Path, entries: &[Entry]) -> anyhow::Result<PathBuf> {
for command in &batch { std::fs::create_dir_all(dir)?;
if let Command::Write(e) = command { let path = dir.join(format!("{}.json", uuid::Uuid::now_v7()));
transaction.execute("INSERT INTO journal (id,time,pid,plugin,stream,line) VALUES (?1,strftime('%Y-%m-%dT%H:%M:%fZ','now'),?2,?3,?4,?5)",params![e.id,e.pid,e.plugin,e.stream,e.line])?; // A failed/partial write must not look like a complete recovery batch.
let mut file = tempfile::NamedTempFile::new_in(dir)?;
serde_json::to_writer(&mut file, entries)?;
file.write_all(b"\n")?;
file.as_file().sync_all()?;
file.persist_noclobber(&path)?;
std::fs::File::open(dir)?.sync_all()?;
eprintln!(
"journal failed batch preserved: {} ({} entries)",
path.display(),
entries.len()
);
Ok(path)
}
fn write_batch(
db: &mut Connection,
old: Option<&Connection>,
spool: &std::path::Path,
mut pending: Vec<Entry>,
batch: Vec<Command>,
) -> Vec<Entry> {
let mut queries = Vec::new();
for command in batch {
match command {
Command::Write(entry) => pending.push(entry),
Command::Query(filters, response) => queries.push((filters, response)),
}
}
if !pending.is_empty() {
let write = (|| -> rusqlite::Result<()> {
let transaction = db.transaction()?;
for e in &pending {
transaction.execute("INSERT OR IGNORE INTO journal (id,time,pid,plugin,stream,line) VALUES (?1,?2,?3,?4,?5,?6)",params![e.id,e.time,e.pid,e.plugin,e.stream,e.line])?;
}
transaction.commit()
})();
match write {
Ok(()) => pending.clear(),
Err(error) => {
eprintln!("journal write failed: {error}");
match spool_batch(spool, &pending) {
Ok(_) => pending.clear(),
Err(error) => eprintln!(
"journal spool failed: {error}; retaining {} entries in memory until next command",
pending.len()
),
}
} }
} }
transaction.commit()
})();
if let Err(error) = write {
eprintln!("journal write failed: {error}");
} }
for command in batch { for (filters, response) in queries {
if let Command::Query(filters, response) = command { let value = query_all(db, old, &filters).unwrap_or_else(|error| {
let value = query_database(db, &filters).unwrap_or_else(|error| { eprintln!("journal query failed: {error}");
eprintln!("journal query failed: {error}"); json!([])
json!([]) });
}); let _ = response.send(value);
let _ = response.send(value); }
pending
}
fn query_all(
db: &Connection,
old: Option<&Connection>,
filters: &Value,
) -> rusqlite::Result<Value> {
let mut entries = query_database(db, filters)?.as_array().unwrap().clone();
if let Some(old) = old {
entries.extend(
query_database(old, filters)?
.as_array()
.unwrap()
.iter()
.cloned(),
);
entries.sort_by(|a, b| a["id"].as_str().cmp(&b["id"].as_str()));
entries.dedup_by(|a, b| a["id"] == b["id"]);
let limit = filters["limit"].as_i64().unwrap_or(100);
if limit >= 0 {
entries.drain(..entries.len().saturating_sub(limit as usize));
} }
} }
Ok(json!(entries))
} }
fn query_database(db: &Connection, filters: &Value) -> rusqlite::Result<Value> { fn query_database(db: &Connection, filters: &Value) -> rusqlite::Result<Value> {
let mut statement = db.prepare( let mut statement = db.prepare(
@@ -135,6 +244,140 @@ fn query_database(db: &Connection, filters: &Value) -> rusqlite::Result<Value> {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
fn entry(line: &str) -> Entry {
Entry {
id: uuid::Uuid::now_v7().to_string(),
time: "2026-10-26T12:00:00.000Z".into(),
pid: "pid".into(),
plugin: "test".into(),
stream: "core".into(),
line: line.into(),
}
}
#[test]
fn failed_transaction_is_spooled_in_full_and_queries_finish() {
let dir = tempfile::tempdir().unwrap();
let mut db = open_database(PathBuf::from(":memory:")).unwrap();
db.execute_batch("PRAGMA query_only=ON").unwrap();
let original = "😀 full failure\n".repeat(6000);
let (tx, mut rx) = oneshot::channel();
let pending = write_batch(
&mut db,
None,
dir.path(),
vec![],
vec![
Command::Write(entry(&original)),
Command::Query(json!({}), tx),
],
);
assert!(pending.is_empty());
assert_eq!(rx.try_recv().unwrap(), json!([]));
let path = std::fs::read_dir(dir.path())
.unwrap()
.next()
.unwrap()
.unwrap()
.path();
let saved: Vec<Entry> = serde_json::from_slice(&std::fs::read(path).unwrap()).unwrap();
assert_eq!(saved.len(), 1);
assert_eq!(saved[0].line, original);
assert_eq!(saved[0].time, "2026-10-26T12:00:00.000Z");
}
#[test]
fn failed_spool_retries_on_next_command_without_duplicates() {
let dir = tempfile::tempdir().unwrap();
let blocked = dir.path().join("blocked");
std::fs::write(&blocked, "not a directory").unwrap();
let mut db = open_database(PathBuf::from(":memory:")).unwrap();
db.execute_batch("PRAGMA query_only=ON").unwrap();
let pending = write_batch(
&mut db,
None,
&blocked,
vec![],
vec![Command::Write(entry("retained"))],
);
assert_eq!(pending.len(), 1);
db.execute_batch("PRAGMA query_only=OFF").unwrap();
let (tx, mut rx) = oneshot::channel();
let pending = write_batch(
&mut db,
None,
&blocked,
pending,
vec![Command::Query(json!({}), tx)],
);
assert!(pending.is_empty());
assert_eq!(rx.try_recv().unwrap()[0]["line"], "retained");
assert_eq!(
query_all(&db, None, &json!({}))
.unwrap()
.as_array()
.unwrap()
.len(),
1
);
}
#[tokio::test]
async fn legacy_wal_and_new_journal_share_filters_order_and_limit() {
let dir = tempfile::tempdir().unwrap();
let legacy = dir.path().join("state/core/journal.sqlite");
let mut old = open_database(legacy.clone()).unwrap();
write_batch(
&mut old,
None,
dir.path(),
vec![],
vec![Command::Write(entry("old match"))],
);
// Leave the WAL connection open: opening the new journal must not move its files.
let wal = legacy.with_extension("sqlite-wal");
let before = std::fs::read(&wal).unwrap();
let new = dir.path().join("logs/core/journal.sqlite");
let journal = Journal::open_with_legacy(new.clone(), Some(legacy.clone()))
.await
.unwrap();
journal.write("pid", "test", "core", "new match");
let entries = tokio::time::timeout(
std::time::Duration::from_secs(5),
journal.query(json!({"pid":"pid","grep":"match","limit":-1})),
)
.await
.unwrap();
assert_eq!(entries.as_array().unwrap().len(), 2);
assert_eq!(entries[0]["line"], "old match");
assert_eq!(entries[1]["line"], "new match");
assert_eq!(
journal.query(json!({"limit":1})).await[0]["line"],
"new match"
);
assert_eq!(
journal.query(json!({"until":entries[0]["id"]})).await[0]["line"],
"old match"
);
assert_eq!(
journal
.query(json!({"since":entries[1]["id"],"plugin":"test"}))
.await[0]["line"],
"new match"
);
assert_eq!(journal.query(json!({"grep":"absent"})).await, json!([]));
assert_eq!(std::fs::read(wal).unwrap(), before);
assert!(legacy.exists() && new.exists());
assert_eq!(
query_database(&old, &json!({}))
.unwrap()
.as_array()
.unwrap()
.len(),
1
);
}
/// PROTOCOL 7: completed processes do not leave warning identities in memory. /// PROTOCOL 7: completed processes do not leave warning identities in memory.
#[tokio::test] #[tokio::test]
async fn completion_forgets_warning_identity() { async fn completion_forgets_warning_identity() {
+2
View File
@@ -12,11 +12,13 @@ mod journal;
mod locks; mod locks;
mod native; mod native;
mod process; mod process;
mod restart;
mod routing; mod routing;
pub mod runtime; pub mod runtime;
mod services; mod services;
mod upgrades; mod upgrades;
mod watcher; mod watcher;
mod watches;
pub mod world; pub mod world;
pub use host::Host; pub use host::Host;
pub use native::{NativeContext, NativeFuture, NativePlugin}; pub use native::{NativeContext, NativeFuture, NativePlugin};
+22 -1
View File
@@ -33,6 +33,8 @@ pub(crate) struct Process {
pub(crate) id: Option<Value>, pub(crate) id: Option<Value>,
pub(crate) answer: Option<oneshot::Sender<RpcResult>>, pub(crate) answer: Option<oneshot::Sender<RpcResult>>,
pub(crate) ended: watch::Sender<bool>, pub(crate) ended: watch::Sender<bool>,
pub(crate) stopped: bool,
pub(crate) superseded: bool,
} }
impl Host { impl Host {
/// Called only after the execution task ends. Wait for the store and file operations before releasing locks. /// Called only after the execution task ends. Wait for the store and file operations before releasing locks.
@@ -57,6 +59,7 @@ impl Host {
.map(|_| self.0.journal.query_later(json!({"pid":pid,"limit":20}))); .map(|_| self.0.journal.query_later(json!({"pid":pid,"limit":20})));
self.0.journal.forget(pid); self.0.journal.forget(pid);
Self::release_locks(&mut state, pid); Self::release_locks(&mut state, pid);
self.service_ended(&mut state, pid, status, p.stopped, p.superseded);
let status = status.to_string(); let status = status.to_string();
tokio::spawn(async move { tokio::spawn(async move {
if let Some(answer) = p.answer { if let Some(answer) = p.answer {
@@ -84,13 +87,24 @@ impl Host {
} }
/// Close stdin and start the same five-second deadline for each attached child. /// Close stdin and start the same five-second deadline for each attached child.
pub(crate) fn begin_stop(&self, pid: &str) -> Option<watch::Receiver<bool>> { pub(crate) fn begin_stop(&self, pid: &str) -> Option<watch::Receiver<bool>> {
self.stop_process(pid, false)
}
/// Grace expiry is retirement, not an operator stopping the service.
pub(crate) fn begin_retire(&self, pid: &str) -> Option<watch::Receiver<bool>> {
self.stop_process(pid, true)
}
fn stop_process(&self, pid: &str, retirement: bool) -> Option<watch::Receiver<bool>> {
let mut state = self.0.state.lock().unwrap(); let mut state = self.0.state.lock().unwrap();
let p = state.processes.get_mut(pid)?; let p = state.processes.get_mut(pid)?;
if !retirement {
p.superseded = false;
}
let ended = p.ended.subscribe(); let ended = p.ended.subscribe();
if p.info.state == "stopping" { if p.info.state == "stopping" {
return Some(ended); return Some(ended);
} }
p.info.state = "stopping".into(); p.info.state = "stopping".into();
p.stopped = true;
p.input.take(); p.input.take();
self.0.journal.write(pid, &p.info.plugin, "core", "stop"); self.0.journal.write(pid, &p.info.plugin, "core", "stop");
for lock in state.locks.values_mut() { for lock in state.locks.values_mut() {
@@ -110,17 +124,24 @@ impl Host {
let pid = pid.to_string(); let pid = pid.to_string();
tokio::spawn(async move { tokio::spawn(async move {
tokio::time::sleep(std::time::Duration::from_secs(5)).await; tokio::time::sleep(std::time::Duration::from_secs(5)).await;
h.kill(&pid); h.kill_process(&pid, retirement);
}); });
Some(ended) Some(ended)
} }
/// Abort a process task and perform its ended-state cleanup. /// Abort a process task and perform its ended-state cleanup.
pub(crate) fn kill(&self, pid: &str) { pub(crate) fn kill(&self, pid: &str) {
self.kill_process(pid, false);
}
fn kill_process(&self, pid: &str, retirement: bool) {
let mut state = self.0.state.lock().unwrap(); let mut state = self.0.state.lock().unwrap();
for lock in state.locks.values_mut() { for lock in state.locks.values_mut() {
lock.queue.retain(|(queued, _)| queued != pid); lock.queue.retain(|(queued, _)| queued != pid);
} }
if let Some(p) = state.processes.get_mut(pid) { if let Some(p) = state.processes.get_mut(pid) {
if !retirement {
p.superseded = false;
}
p.stopped = true;
p.info.state = "stopping".into(); p.info.state = "stopping".into();
p.input.take(); p.input.take();
self.0.journal.write(pid, &p.info.plugin, "core", "kill"); self.0.journal.write(pid, &p.info.plugin, "core", "kill");
+129
View File
@@ -0,0 +1,129 @@
//! Supervisor-gated restart requests; the executable still owns normal shutdown.
use crate::{Host, RpcResult, error};
use serde_json::json;
use std::{ffi::OsStr, sync::atomic::Ordering, time::Duration};
pub(crate) fn enabled(value: Option<&OsStr>) -> bool {
value == Some(OsStr::new("1"))
}
impl Host {
pub(crate) fn request_restart(&self) -> RpcResult {
if !self.0.loop_supervised {
return Err(error(
-32600,
"restart needs the loop supervisor (SAUSAGE_LOOP not set); start with SAUSAGE_LOOP=1",
));
}
if !self.0.restart_pending.swap(true, Ordering::SeqCst) {
let host = self.clone();
tokio::spawn(async move {
// Let the RPC response reach the caller before closing any stdin,
// including that of a detached gateway carrying this response.
tokio::time::sleep(Duration::from_millis(250)).await;
host.0.exit.send_replace(true);
});
}
Ok(json!({}))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{runtime::Runtime, world::World};
use std::sync::Arc;
use tokio::time::timeout;
#[test]
fn only_explicit_loop_marker_enables_restart() {
assert!(!enabled(None));
for value in ["", "0", "true", "yes", "2"] {
assert!(!enabled(Some(OsStr::new(value))));
}
assert!(enabled(Some(OsStr::new("1"))));
}
async fn host(supervised: bool) -> (tempfile::TempDir, Host) {
let dir = tempfile::tempdir().unwrap();
World::create(&dir.path().join("world")).unwrap();
let runtime = Runtime::new(&dir.path().join("cache")).unwrap();
let mut host = Host::test_world(&dir.path().join("world"), runtime)
.await
.unwrap();
// No global environment mutation: parallel tests have independent gates.
Arc::get_mut(&mut host.0).unwrap().loop_supervised = supervised;
(dir, host)
}
#[tokio::test]
async fn restart_without_supervisor_never_triggers_exit() {
timeout(Duration::from_secs(10), async {
let (_dir, host) = host(false).await;
for method in ["core.restart", "core.exit"] {
let error = host.call(method, json!({})).await.unwrap_err();
assert_eq!(error["code"], -32600);
assert!(
error["message"]
.as_str()
.unwrap()
.contains("SAUSAGE_LOOP not set")
);
let indirect = host
.call("core.call", json!({"method":method}))
.await
.unwrap_err();
assert_eq!(indirect, error);
}
assert!(!host.0.restart_pending.load(Ordering::SeqCst));
assert!(
timeout(Duration::from_millis(300), host.exit_requested())
.await
.is_err()
);
host.stop().await;
})
.await
.unwrap();
}
#[tokio::test]
async fn restart_and_exit_share_response_grace_and_coalescing() {
timeout(Duration::from_secs(10), async {
for first in ["core.restart", "core.exit"] {
let (_dir, host) = host(true).await;
// Same dispatch as a plugin's core.call, without killing this runner.
assert_eq!(
host.call("core.call", json!({"method":first}))
.await
.unwrap(),
json!({})
);
assert!(!host.restart_requested());
assert!(host.0.restart_pending.load(Ordering::SeqCst));
assert_eq!(
host.call("core.restart", json!({})).await.unwrap(),
json!({})
);
timeout(Duration::from_secs(1), host.exit_requested())
.await
.unwrap();
// A duplicate after the trigger must not schedule a second task.
host.0.exit.send_replace(false);
assert_eq!(
host.call("core.restart", json!({})).await.unwrap(),
json!({})
);
assert_eq!(host.call("core.exit", json!({})).await.unwrap(), json!({}));
assert!(
timeout(Duration::from_millis(300), host.exit_requested())
.await
.is_err()
);
host.stop().await;
}
})
.await
.unwrap();
}
}
+60 -9
View File
@@ -7,6 +7,23 @@ use serde_json::{Value, json};
use tokio::sync::{mpsc, oneshot, watch}; use tokio::sync::{mpsc, oneshot, watch};
use uuid::Uuid; use uuid::Uuid;
fn params_not_object(params: &Value) -> Value {
let kind = match params {
Value::String(_) => "a string",
Value::Array(_) => "an array",
Value::Number(_) => "a number",
Value::Null => "null",
Value::Bool(_) => "a boolean",
Value::Object(_) => "an object",
};
error(
-32602,
&format!(
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
),
)
}
impl Host { impl Host {
/// Queue an incoming response or notification for its destination. /// Queue an incoming response or notification for its destination.
pub(crate) fn send(&self, pid: &str, value: Value) { pub(crate) fn send(&self, pid: &str, value: Value) {
@@ -34,15 +51,20 @@ impl Host {
.processes .processes
.get(pid) .get(pid)
.and_then(|p| p.answer.as_ref().map(|_| (p.caller.clone(), p.id.clone()))); .and_then(|p| p.answer.as_ref().map(|_| (p.caller.clone(), p.id.clone())));
if let Some((caller, id)) = target { if let Some((caller, Some(id))) = target {
let notification = json!({ self.call_notify(&caller, &id, name, params);
"jsonrpc": "2.0",
"method": "core.notify",
"params": { "call": id, "name": name, "params": params }
});
self.send(&caller, notification);
} }
} }
/// Deliver a notification on an open core call using the same envelope as plugin progress.
pub(crate) fn call_notify(&self, caller: &str, id: &Value, name: &str, params: Value) {
self.send(
caller,
json!({
"jsonrpc":"2.0", "method":"core.notify",
"params": {"call":id, "name":name, "params":params}
}),
);
}
/// Validate and route a request, then start its native or WASI process. /// Validate and route a request, then start its native or WASI process.
pub(crate) fn route( pub(crate) fn route(
&self, &self,
@@ -69,7 +91,7 @@ impl Host {
let host = self.clone(); let host = self.clone();
Box::pin(async move { Box::pin(async move {
if !params.is_object() { if !params.is_object() {
return Err(error(-32602, "params must be an object")); return Err(params_not_object(&params));
} }
if caller != "console" && caller != "core" { if caller != "console" && caller != "core" {
let state = host.0.state.lock().unwrap(); let state = host.0.state.lock().unwrap();
@@ -149,6 +171,33 @@ impl Host {
{ {
return Err(error(-32600, "caller stopped or ended")); return Err(error(-32600, "caller stopped or ended"));
} }
// Older plugins may still try to start their boot service from
// rpc.retired. The supervisor owns it even while a restart is
// pending, or after it has given up. Do not create a second owner.
if state
.processes
.get(&caller)
.is_some_and(|p| p.info.method == "rpc.retired")
&& state
.services
.iter()
.any(|s| s.method == format!("{plugin}.{method}") && s.params == params)
{
return Ok(Value::Null);
}
if caller == "core"
&& let Some(service) = state
.services
.iter_mut()
.find(|s| Some(&s.request) == id.as_ref())
{
if !service.enabled || service.owner.is_some() {
return Err(error(-32600, "service attempt is no longer current"));
}
service.owner = Some(pid.clone());
service.pending = false;
service.version = Some(info.version.clone());
}
host.0 host.0
.journal .journal
.write(&pid, plugin, "core", &format!("start {method} {params}")); .write(&pid, plugin, "core", &format!("start {method} {params}"));
@@ -165,6 +214,8 @@ impl Host {
id: id.clone(), id: id.clone(),
answer: if id.is_some() { Some(tx) } else { None }, answer: if id.is_some() { Some(tx) } else { None },
ended: watch::channel(false).0, ended: watch::channel(false).0,
stopped: false,
superseded: false,
}, },
); );
} }
@@ -238,7 +289,7 @@ impl Host {
// Deliver progress before a following response removes its destination. // Deliver progress before a following response removes its destination.
let params = v.get("params").cloned().unwrap_or(json!({})); let params = v.get("params").cloned().unwrap_or(json!({}));
let result = if !params.is_object() { let result = if !params.is_object() {
Err(error(-32602, "params must be an object")) Err(params_not_object(&params))
} else if method == "core.unlock" { } else if method == "core.unlock" {
if let Some(path) = params["path"].as_str() { if let Some(path) = params["path"].as_str() {
self.unlock(pid, path); self.unlock(pid, path);
+23 -1
View File
@@ -212,7 +212,11 @@ fn short_trap(full: &str) -> String {
.find_map(|l| l.find("wasm trap:").map(|i| l[i..].to_string())) .find_map(|l| l.find("wasm trap:").map(|i| l[i..].to_string()))
.unwrap_or_else(|| full.lines().next().unwrap_or("trap").to_string()); .unwrap_or_else(|| full.lines().next().unwrap_or("trap").to_string());
let mut line = line.trim().to_string(); let mut line = line.trim().to_string();
line.truncate(300); let mut end = line.len().min(300);
while !line.is_char_boundary(end) {
end -= 1;
}
line.truncate(end);
line line
} }
@@ -361,6 +365,7 @@ impl Runtime {
}); });
let (err_tx, err_rx) = mpsc::channel(1024); let (err_tx, err_rx) = mpsc::channel(1024);
let trap_log = log.clone();
let stderr_task = tokio::spawn(async move { let stderr_task = tokio::spawn(async move {
let mut reader = BufReader::new(host_err); let mut reader = BufReader::new(host_err);
let mut bytes = Vec::new(); let mut bytes = Vec::new();
@@ -400,6 +405,9 @@ impl Runtime {
Some(I32Exit(c)) => Exit::Code(*c), Some(I32Exit(c)) => Exit::Code(*c),
None => { None => {
let full = format!("{e:?}"); let full = format!("{e:?}");
if let Some((journal, pid, plugin)) = &trap_log {
journal.write(pid, plugin, "core", &full);
}
Exit::Trap(short_trap(&full)) Exit::Trap(short_trap(&full))
} }
}, },
@@ -416,3 +424,17 @@ impl Runtime {
}) })
} }
} }
#[cfg(test)]
mod trap_text_tests {
use super::*;
#[test]
fn short_trap_display_is_utf8_safe() {
let full = format!("backtrace\nwasm trap: {}\nmore detail", "😀".repeat(1000));
let status = short_trap(&full);
assert!(status.len() <= 300);
assert!(status.starts_with("wasm trap:"));
assert_eq!(short_trap(""), "trap");
assert_eq!(short_trap("first line\nsecond"), "first line");
}
}
+132 -19
View File
@@ -1,12 +1,36 @@
//! Ordered boot services and their identities for upgrade restarts (PROTOCOL.md section 8). //! Ordered boot calls and the single restart owner for each service (PROTOCOL.md section 8).
use crate::Host; use crate::{Host, host::State};
use serde_json::{Value, json}; use serde_json::{Value, json};
#[derive(Clone)] use std::{collections::VecDeque, sync::atomic::Ordering, time::Duration};
use tokio::time::Instant;
// Systemd-derived values:
// https://www.man7.org/linux/man-pages/man5/systemd-system.conf.5.html.
const CRASH_LIMIT: usize = 5;
const CRASH_WINDOW: Duration = Duration::from_secs(10);
pub(crate) struct Service { pub(crate) struct Service {
pub(crate) method: String, pub(crate) method: String,
pub(crate) params: Value, pub(crate) params: Value,
pub(crate) request: Value, pub(crate) request: Value,
pub(crate) starting: std::sync::Arc<std::sync::atomic::AtomicBool>, pub(crate) owner: Option<String>,
pub(crate) pending: bool,
pub(crate) version: Option<String>,
pub(crate) enabled: bool,
crashes: VecDeque<Instant>,
}
impl Service {
fn crashed(&mut self, now: Instant) -> bool {
while self
.crashes
.front()
.is_some_and(|t| now.duration_since(*t) >= CRASH_WINDOW)
{
self.crashes.pop_front();
}
self.crashes.push_back(now);
self.crashes.len() <= CRASH_LIMIT
}
} }
impl Host { impl Host {
/// Starts discovery in the background, then waits for each boot service in order. /// Starts discovery in the background, then waits for each boot service in order.
@@ -24,30 +48,119 @@ impl Host {
} }
Ok(()) Ok(())
} }
/// Records a unique request identity before the service can answer or exit. /// Register once, before admission or readiness. Restarts reuse this registration.
pub(crate) async fn start_service(&self, method: String, params: Value) { pub(crate) async fn start_service(&self, method: String, params: Value) {
if self.0.shutdown.load(std::sync::atomic::Ordering::SeqCst) {
return;
}
let request = json!(uuid::Uuid::now_v7().to_string()); let request = json!(uuid::Uuid::now_v7().to_string());
let starting = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(true)); {
self.0.state.lock().unwrap().services.push(Service { let mut state = self.0.state.lock().unwrap();
method: method.clone(), if self.0.shutdown.load(Ordering::SeqCst) {
params: params.clone(), return;
request: request.clone(), }
starting: starting.clone(), state.services.push(Service {
}); method,
params,
request: request.clone(),
owner: None,
enabled: true,
pending: true,
version: None,
crashes: VecDeque::new(),
});
}
self.run_service(request).await;
}
/// Await readiness only; process cleanup, not this response, decides whether to restart.
async fn run_service(&self, request: Value) {
let (method, params) = {
let state = self.0.state.lock().unwrap();
if self.0.shutdown.load(Ordering::SeqCst) {
return;
}
let Some(service) = state
.services
.iter()
.find(|s| s.request == request && s.enabled)
else {
return;
};
(service.method.clone(), service.params.clone())
};
self.0 self.0
.journal .journal
.write("core", "core", "core", &format!("service start {method}")); .write("core", "core", "core", &format!("service start {method}"));
if let Err(error) = self let result = self
.route("core".into(), Some(request), method, params, true) .route("core".into(), Some(request.clone()), method, params, true)
.await .await;
{ {
let mut state = self.0.state.lock().unwrap();
if let Some(service) = state.services.iter_mut().find(|s| s.request == request) {
service.pending = false;
}
}
if let Err(error) = result {
// Admission failures have no process lifetime to supervise. A process crash
// before readiness is handled by finish just like one after readiness.
self.0 self.0
.journal .journal
.write("core", "core", "core", &format!("service failed {error}")); .write("core", "core", "core", &format!("service failed {error}"));
} }
starting.store(false, std::sync::atomic::Ordering::SeqCst); }
/// Called under the state lock after the old owner and its files/locks are gone.
pub(crate) fn service_ended(
&self,
state: &mut State,
pid: &str,
status: &str,
stopped: bool,
superseded: bool,
) {
let Some(service) = state
.services
.iter_mut()
.find(|s| s.owner.as_deref() == Some(pid))
else {
return;
};
service.owner = None;
if self.0.shutdown.load(Ordering::SeqCst) || !service.enabled {
return;
}
if !superseded {
if stopped {
service.enabled = false;
return;
}
if status == "exit 0" {
return;
}
if !service.crashed(Instant::now()) {
service.enabled = false;
self.0.journal.write(
"core",
"core",
"core",
&format!(
"service gave up {}: more than {CRASH_LIMIT} crashes within {} seconds",
service.method,
CRASH_WINDOW.as_secs(),
),
);
return;
}
}
self.queue_service(service);
}
/// Reserve the next attempt before spawning it; upgrades and crashes share this owner.
pub(crate) fn queue_service(&self, service: &mut Service) {
service.request = json!(uuid::Uuid::now_v7().to_string());
service.pending = true;
let request = service.request.clone();
let host = self.clone();
tokio::spawn(async move {
host.run_service(request).await;
});
} }
} }
#[cfg(test)]
mod tests;
+356
View File
@@ -0,0 +1,356 @@
//! Deterministic service supervision tests: native tasks stand in for guest lifetimes.
use super::*;
use crate::{NativePlugin, runtime::Runtime};
use std::sync::Arc;
use tokio::sync::{mpsc, oneshot, watch};
struct Rig {
host: Host,
starts: mpsc::UnboundedReceiver<(String, oneshot::Sender<()>)>,
_world: tempfile::TempDir,
}
impl Rig {
async fn new() -> Self {
let world = tempfile::tempdir().unwrap();
let runtime = Runtime::new(world.path()).unwrap();
let host = Host::test_world(world.path(), runtime).await.unwrap();
let (tx, starts) = mpsc::unbounded_channel();
let plugin: NativePlugin = Arc::new(move |ctx, method, _| {
let tx = tx.clone();
Box::pin(async move {
if method != "serve" {
return Ok(Value::Null);
}
let (end, ended) = oneshot::channel();
tx.send((ctx.pid, end)).unwrap();
let _ = ended.await;
Ok(Value::Null)
})
});
host.register_native("demo", plugin.clone());
host.register_native("other", plugin);
Self {
host,
starts,
_world: world,
}
}
async fn start(&mut self, method: &str) -> (String, oneshot::Sender<()>) {
let host = self.host.clone();
let method = method.to_string();
tokio::spawn(async move {
host.start_service(method, json!({"kept":true})).await;
});
self.next().await
}
async fn next(&mut self) -> (String, oneshot::Sender<()>) {
// Reuse the host integration call bound (activation.rs::call).
tokio::time::timeout(Duration::from_secs(60), self.starts.recv())
.await
.unwrap()
.unwrap()
}
fn ended(&self, pid: &str) -> watch::Receiver<bool> {
self.host.0.state.lock().unwrap().processes[pid]
.ended
.subscribe()
}
async fn crash(&self, pid: &str) {
let mut ended = self.ended(pid);
// Abort execution without asking the supervisor for an intentional stop.
self.host.0.state.lock().unwrap().processes[pid]
.abort
.as_ref()
.unwrap()
.abort();
ended.wait_for(|done| *done).await.unwrap();
}
async fn clean(&self, pid: &str, end: oneshot::Sender<()>) {
let mut ended = self.ended(pid);
end.send(()).unwrap();
ended.wait_for(|done| *done).await.unwrap();
}
fn assert_single(&self, pid: &str, registrations: usize) {
let state = self.host.0.state.lock().unwrap();
assert_eq!(state.services.len(), registrations);
let p = &state.processes[pid];
assert_eq!(p.info.params, json!({"kept":true}));
assert!(p.info.detached);
assert_eq!(p.caller, "core");
assert_eq!(
state
.services
.iter()
.filter(|s| s.owner.as_deref() == Some(pid))
.count(),
1
);
}
fn no_start(&mut self) {
assert!(self.starts.try_recv().is_err());
}
}
#[tokio::test(start_paused = true)]
async fn crash_before_and_after_readiness_observes_process_not_response() {
let mut rig = Rig::new().await;
let (pid, _end) = rig.start("demo.serve").await;
rig.assert_single(&pid, 1);
rig.crash(&pid).await; // No ready RPC response was sent.
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
rig.host.answer(&pid, Ok(json!({"ready":true})));
tokio::task::yield_now().await;
rig.no_start(); // Ready does not end the service lifetime.
rig.crash(&pid).await;
drop(end);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
rig.clean(&pid, end).await;
rig.no_start(); // A clean, intentional return is not a crash.
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn fifth_restarts_sixth_gives_up_and_journals_once() {
let mut rig = Rig::new().await;
let (mut pid, mut end) = rig.start("demo.serve").await;
for crash in 1..=CRASH_LIMIT {
rig.crash(&pid).await;
drop(end);
(pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
assert_eq!(
rig.host.0.state.lock().unwrap().services[0].crashes.len(),
crash
);
}
rig.crash(&pid).await;
drop(end);
rig.no_start();
let logs = rig
.host
.call("core.log", json!({"grep":"service gave up"}))
.await
.unwrap();
assert_eq!(logs.as_array().unwrap().len(), 1);
assert!(
logs[0]["line"]
.as_str()
.unwrap()
.contains("more than 5 crashes within 10 seconds")
);
assert!(!rig.host.0.state.lock().unwrap().services[0].enabled);
// A retired hook cannot bypass give-up while its supervisor has no owner.
rig.host.register_native(
"fallback",
Arc::new(move |ctx, _, _| {
Box::pin(async move {
ctx.call(
"core.call",
json!({"method":"demo.serve","params":{"kept":true},"detach":true}),
)
.await
})
}),
);
assert_eq!(
rig.host
.call("fallback.rpc.retired", json!({}))
.await
.unwrap(),
Value::Null
);
rig.no_start();
// Neither time passing nor an upgrade re-arms a given-up service.
tokio::time::advance(CRASH_WINDOW).await;
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
rig.host.begin_upgrade(upgrade);
tokio::task::yield_now().await;
rig.no_start();
rig.host.stop().await;
}
#[test]
fn sliding_window_boundary_and_progress() {
let now = Instant::now();
let service = || Service {
method: "demo.serve".into(),
params: json!({}),
request: json!("request"),
owner: None,
enabled: true,
pending: false,
version: None,
crashes: VecDeque::new(),
};
let mut before = service();
let mut exact = service();
let mut after = service();
for _ in 0..CRASH_LIMIT {
assert!(before.crashed(now));
assert!(exact.crashed(now));
assert!(after.crashed(now));
}
assert!(!before.crashed(now + CRASH_WINDOW - Duration::from_nanos(1)));
assert!(exact.crashed(now + CRASH_WINDOW));
assert_eq!(exact.crashes.len(), 1);
assert!(after.crashed(now + CRASH_WINDOW + Duration::from_nanos(1)));
// Repeated progress beyond the window never accumulates old failures.
for step in 2..=CRASH_LIMIT as u32 + 1 {
assert!(exact.crashed(now + CRASH_WINDOW * step));
assert_eq!(exact.crashes.len(), 1);
}
}
#[tokio::test(start_paused = true)]
async fn services_have_independent_crash_budgets() {
let mut rig = Rig::new().await;
let (mut a, mut a_end) = rig.start("demo.serve").await;
let (b, b_end) = rig.start("other.serve").await;
for _ in 0..CRASH_LIMIT {
rig.crash(&a).await;
drop(a_end);
(a, a_end) = rig.next().await;
}
rig.crash(&a).await;
drop(a_end);
rig.crash(&b).await;
drop(b_end);
let (b, b_end) = rig.next().await;
rig.assert_single(&b, 2);
rig.clean(&b, b_end).await;
rig.no_start();
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn upgrade_crash_race_has_one_restart_and_retired_fallback_cannot_duplicate_it() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
rig.crash(&pid).await; // Exits between snapshot and begin_upgrade.
drop(end);
rig.host.begin_upgrade(upgrade);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
assert!(
rig.host.0.state.lock().unwrap().services[0]
.crashes
.is_empty()
);
// A native retired hook exercises the same routing path as a guest fallback.
rig.host.register_native(
"fallback",
Arc::new(move |ctx, _, _| {
Box::pin(async move {
ctx.call(
"core.call",
json!({"method":"demo.serve","params":{"kept":true},"detach":true}),
)
.await
})
}),
);
assert_eq!(
rig.host
.call("fallback.rpc.retired", json!({}))
.await
.unwrap(),
Value::Null
);
rig.no_start();
rig.clean(&pid, end).await;
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn crash_before_upgrade_snapshot_also_has_one_restart() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
rig.crash(&pid).await;
drop(end);
// Restart may be pending admission or already admitted: snapshot handles both.
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "native");
rig.host.begin_upgrade(upgrade);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
rig.no_start();
rig.clean(&pid, end).await;
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn pending_admission_is_single_owned_across_upgrade() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
let host = rig.host.clone();
let admission = host.0.admission.write().await;
rig.crash(&pid).await;
drop(end);
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "native");
rig.host.begin_upgrade(upgrade);
assert!(rig.host.0.state.lock().unwrap().services[0].pending);
drop(admission);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
rig.no_start();
rig.clean(&pid, end).await;
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn shutdown_rejects_restart_already_waiting_for_admission() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
let host = rig.host.clone();
let admission = host.0.admission.write().await;
rig.crash(&pid).await;
drop(end);
rig.host.stop().await;
drop(admission);
tokio::task::yield_now().await;
rig.no_start();
assert!(rig.host.processes().is_empty());
}
#[tokio::test(start_paused = true)]
async fn clean_service_stays_dormant_until_upgrade_and_explicit_stop_stays_disabled() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
rig.clean(&pid, end).await;
rig.no_start();
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
rig.host.begin_upgrade(upgrade);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
// An explicit stop stays disabled even if the task subsequently exits cleanly.
let mut ended = rig.host.begin_stop(&pid).unwrap();
end.send(()).unwrap();
ended.wait_for(|done| *done).await.unwrap();
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "newer");
rig.host.begin_upgrade(upgrade);
tokio::task::yield_now().await;
rig.no_start();
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn intentional_stop_and_shutdown_never_restart_even_during_upgrade() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
// An explicit kill after supersession overrides the automatic upgrade restart.
rig.host
.call("core.kill", json!({"pid":pid}))
.await
.unwrap();
drop(end);
rig.host.begin_upgrade(upgrade);
tokio::task::yield_now().await;
rig.no_start();
let (_pid, end) = rig.start("other.serve").await;
rig.host.stop().await;
drop(end);
rig.no_start();
}
+19 -2
View File
@@ -64,12 +64,21 @@ async fn instance(
("SAUSAGE_PLUGIN".into(), "test".into()), ("SAUSAGE_PLUGIN".into(), "test".into()),
("SAUSAGE_PARENT".into(), "core".into()), ("SAUSAGE_PARENT".into(), "core".into()),
]); ]);
let timeout_secs = std::env::var("SAUSAGE_TEST_TIMEOUT_SECS")
.ok()
.and_then(|value| value.parse::<u64>().ok())
.filter(|value| *value >= 1)
.unwrap_or(sausage::cargo::DEFAULT_TEST_TIMEOUT_SECS);
let deadline = tokio::time::Instant::now()
.checked_add(std::time::Duration::from_secs(timeout_secs))
.ok_or_else(|| anyhow::anyhow!("test_timeout_secs is too large"))?;
let (ended, mut ended_rx) = tokio::sync::watch::channel(true);
let mut guest = runtime let mut guest = runtime
.start_with_args( .start_with_args(
&component, &component,
InstanceConfig { InstanceConfig {
filesystem, filesystem,
store_ended: tokio::sync::watch::channel(true).0, store_ended: ended,
network: true, network: true,
env, env,
}, },
@@ -88,6 +97,7 @@ async fn instance(
let mut stdout_open = true; let mut stdout_open = true;
let mut stderr_open = true; let mut stderr_open = true;
let mut handlers = tokio::task::JoinSet::new(); let mut handlers = tokio::task::JoinSet::new();
let result = tokio::time::timeout_at(deadline, async {
while stdout_open || stderr_open || !pending.is_empty() { while stdout_open || stderr_open || !pending.is_empty() {
tokio::select! { tokio::select! {
line=async { if let Some(line) = pending.pop_front() { Some(line) } else { guest.stdout.recv().await } }, if stdout_open || !pending.is_empty()=>match line { line=async { if let Some(line) = pending.pop_front() { Some(line) } else { guest.stdout.recv().await } }, if stdout_open || !pending.is_empty()=>match line {
@@ -129,11 +139,18 @@ async fn instance(
}, },
} }
} }
Ok::<_, anyhow::Error>(guest.wait().await)
}).await;
handlers.abort_all(); handlers.abort_all();
let exit = guest.wait().await; // Dropping the instance aborts even a CPU-bound WASI loop. Wait for the
// store and outstanding filesystem work before removing its test world.
drop(guest);
let _ = ended_rx.wait_for(|ended| *ended).await;
let _ = stderr_task.await; let _ = stderr_task.await;
tracker.idle().await; tracker.idle().await;
host.stop().await; host.stop().await;
let exit =
result.map_err(|_| anyhow::anyhow!("timed out after {timeout_secs} s\n{output}"))??;
Ok(Run { exit, output }) Ok(Run { exit, output })
} }
/// Extracts libtest names from its stable list format. /// Extracts libtest names from its stable list format.
+41 -46
View File
@@ -1,9 +1,15 @@
//! Grace periods, service restarts, and retirement of replaced versions (PROTOCOL.md section 3.7). //! Grace periods, service restarts, and retirement of replaced versions (PROTOCOL.md section 3.7).
use crate::{Host, services::Service}; use crate::Host;
use serde_json::json; use serde_json::json;
use std::{collections::BTreeMap, sync::atomic::Ordering, time::Duration}; use std::{collections::BTreeMap, sync::atomic::Ordering, time::Duration};
use tokio::sync::watch; use tokio::sync::watch;
/// Uniform host policy (PROTOCOL 3.7), independent of discovery metadata.
const UPGRADE_GRACE: Duration = Duration::from_secs(5);
#[cfg(test)]
mod tests;
pub(crate) struct OldProcess { pub(crate) struct OldProcess {
pid: String, pid: String,
version: String, version: String,
@@ -12,9 +18,8 @@ pub(crate) struct OldProcess {
pub(crate) struct Upgrade { pub(crate) struct Upgrade {
plugin: String, plugin: String,
previous: String, previous: String,
grace: Duration, current: String,
processes: Vec<OldProcess>, processes: Vec<OldProcess>,
services: Vec<(Service, Option<watch::Receiver<bool>>)>,
} }
impl Host { impl Host {
/// Snapshots old admissions while activation holds the admission write guard. /// Snapshots old admissions while activation holds the admission write guard.
@@ -23,57 +28,59 @@ impl Host {
plugin: &str, plugin: &str,
previous: String, previous: String,
current: &str, current: &str,
grace: f64,
) -> Upgrade { ) -> Upgrade {
let mut state = self.0.state.lock().unwrap(); let mut state = self.0.state.lock().unwrap();
let processes = state let processes = state
.processes .processes
.iter() .iter_mut()
.filter(|(_, process)| { .filter(|(_, process)| {
process.info.plugin == plugin process.info.plugin == plugin
&& process.info.version != current && process.info.version != current
&& process.info.method != "rpc.discover" && process.info.method != "rpc.discover"
}) })
.map(|(pid, process)| OldProcess { .map(|(pid, process)| {
pid: pid.clone(), // The service supervisor owns the restart, including an exit on
version: process.info.version.clone(), // the superseded notice before the grace timer is installed.
ended: process.ended.subscribe(), process.superseded = !process.stopped;
OldProcess {
pid: pid.clone(),
version: process.info.version.clone(),
ended: process.ended.subscribe(),
}
}) })
.collect(); .collect();
let mut services = Vec::new(); // A cleanly returned service is dormant until an upgrade. A pending
let mut retained = Vec::new(); // admission will choose the current version itself. Disabled services
for service in std::mem::take(&mut state.services) { // (explicit stop or crash-limit give-up) must never be re-armed here.
if service for service in &mut state.services {
.method if service.enabled
.split_once('.') && !service.pending
.is_some_and(|(name, _)| name == plugin) && service.owner.is_none()
&& service.version.as_deref() != Some(current)
&& service
.method
.split_once('.')
.is_some_and(|(name, _)| name == plugin)
{ {
let process = state self.queue_service(service);
.processes
.values()
.find(|process| process.id.as_ref() == Some(&service.request));
if process.is_some_and(|process| process.info.version == current)
|| (process.is_none() && service.starting.load(Ordering::SeqCst))
{
retained.push(service);
} else {
services.push((service, process.map(|process| process.ended.subscribe())));
}
} else {
retained.push(service);
} }
} }
state.services = retained;
Upgrade { Upgrade {
plugin: plugin.into(), plugin: plugin.into(),
previous, previous,
grace: Duration::from_secs_f64(grace), current: current.into(),
processes, processes,
services,
} }
} }
/// Lets old calls finish, stops them after grace, and starts each lifecycle hook once. /// Lets old calls finish, stops them after grace, and starts each lifecycle hook once.
pub(crate) fn begin_upgrade(&self, upgrade: Upgrade) { pub(crate) fn begin_upgrade(&self, upgrade: Upgrade) {
// Queue the notice before returning from activation, without waiting on guest code.
for process in &upgrade.processes {
self.send(
&process.pid,
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":upgrade.current}}),
);
}
let host = self.clone(); let host = self.clone();
tokio::spawn(async move { tokio::spawn(async move {
if host.0.shutdown.load(Ordering::SeqCst) { if host.0.shutdown.load(Ordering::SeqCst) {
@@ -83,7 +90,7 @@ impl Host {
"core", "core",
&upgrade.plugin, &upgrade.plugin,
"core", "core",
&format!("upgrade grace {} seconds", upgrade.grace.as_secs_f64()), &format!("upgrade grace {} seconds", UPGRADE_GRACE.as_secs()),
); );
let mut versions: BTreeMap<String, Vec<watch::Receiver<bool>>> = BTreeMap::new(); let mut versions: BTreeMap<String, Vec<watch::Receiver<bool>>> = BTreeMap::new();
versions.entry(upgrade.previous).or_default(); versions.entry(upgrade.previous).or_default();
@@ -93,23 +100,11 @@ impl Host {
.or_default() .or_default()
.push(process.ended.clone()); .push(process.ended.clone());
let host = host.clone(); let host = host.clone();
let grace = upgrade.grace;
tokio::spawn(async move { tokio::spawn(async move {
let mut ended = process.ended; let mut ended = process.ended;
tokio::select! { tokio::select! {
_ = ended.wait_for(|ended|*ended) => (), _ = ended.wait_for(|ended|*ended) => (),
_ = tokio::time::sleep(grace) => { host.begin_stop(&process.pid); } _ = tokio::time::sleep(UPGRADE_GRACE) => { host.begin_retire(&process.pid); }
}
});
}
for (service, ended) in upgrade.services {
let host = host.clone();
tokio::spawn(async move {
if let Some(mut ended) = ended {
let _ = ended.wait_for(|ended| *ended).await;
}
if !host.0.shutdown.load(Ordering::SeqCst) {
host.start_service(service.method, service.params).await;
} }
}); });
} }
+115
View File
@@ -0,0 +1,115 @@
//! Controlled-clock coverage of both upgrade windows and discovery exclusion.
use super::*;
use crate::{
process::{Process, ProcessInfo},
runtime::Runtime,
world::World,
};
use tokio::{
sync::mpsc,
task::JoinHandle,
time::{Instant, advance},
};
/// A process with active work that ignores both superseded and EOF.
fn process(
host: &Host,
pid: &str,
method: &str,
) -> (mpsc::UnboundedReceiver<serde_json::Value>, JoinHandle<()>) {
let (input, incoming) = mpsc::unbounded_channel();
let task = tokio::spawn(std::future::pending());
host.0.state.lock().unwrap().processes.insert(
pid.into(),
Process {
info: ProcessInfo {
plugin: "notes".into(),
version: "old".into(),
method: method.into(),
params: json!({}),
parent: None,
detached: true,
start_time: 0,
state: "running".into(),
},
memory: sausage_vfs::Memory::default(),
files: sausage_vfs::Process::default(),
store_ended: watch::channel(true).0,
input: Some(input),
abort: Some(task.abort_handle()),
caller: "console".into(),
id: None,
answer: None,
ended: watch::channel(false).0,
stopped: false,
superseded: false,
},
);
(incoming, task)
}
/// PROTOCOL 3.7/3.6: five seconds running, then five seconds stopping; discovery is untouched.
#[tokio::test]
async fn two_five_second_windows_and_discovery_exclusion() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path().join("world");
World::create(&root).unwrap();
let host = Host::test_world(&root, Runtime::new(&dir.path().join("cache")).unwrap())
.await
.unwrap();
let (mut input, work) = process(&host, "old-work", "work");
let (mut discovery_input, discovery) = process(&host, "old-discovery", "rpc.discover");
let upgrade = host.snapshot_upgrade("notes", "old".into(), "new");
assert_eq!(upgrade.processes.len(), 1);
tokio::time::pause();
let start = Instant::now();
host.begin_upgrade(upgrade);
assert_eq!(
input.recv().await.unwrap(),
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":"new"}})
);
// Run the upgrade dispatcher, then its per-process task, so the deadline is armed.
tokio::task::yield_now().await;
tokio::task::yield_now().await;
let window = Duration::from_secs(5);
assert_eq!(UPGRADE_GRACE, window);
// tokio::time::sleep documents millisecond granularity; the paused driver
// may round a deadline up by one tick. Check each window independently.
let tick = Duration::from_millis(1);
let before_deadline = window - tick;
advance(before_deadline).await;
tokio::task::yield_now().await;
assert_eq!(host.processes()["old-work"].state, "running");
assert!(!work.is_finished());
advance(tick).await;
assert!(
input.recv().await.is_none(),
"EOF at grace, even with active work"
);
let stopped = Instant::now();
assert!((window..=window + tick).contains(&(stopped - start)));
assert_eq!(host.processes()["old-work"].state, "stopping");
// begin_retire schedules the independent PROTOCOL 3.6 deadline.
tokio::task::yield_now().await;
advance(before_deadline).await;
tokio::task::yield_now().await;
assert!(
!work.is_finished(),
"on_stop still has its full five seconds"
);
advance(tick).await;
assert!(work.await.unwrap_err().is_cancelled());
assert!((window..=window + tick).contains(&(Instant::now() - stopped)));
assert_eq!(host.processes()["old-discovery"].state, "running");
assert!(matches!(
discovery_input.try_recv(),
Err(mpsc::error::TryRecvError::Empty)
));
assert!(!discovery.is_finished());
discovery.abort();
let _ = discovery.await;
// Synthetic processes have no execution cleanup task; discard only this test table.
host.0.shutdown.store(true, Ordering::SeqCst);
host.0.state.lock().unwrap().processes.clear();
host.stop().await;
}
+178
View File
@@ -0,0 +1,178 @@
//! Armed plugin-file watches. A Notify permit coalesces changes without losing a later change.
use crate::{Host, RpcResult, error};
use sausage_vfs::observe::under;
use serde_json::{Value, json};
use std::{
collections::HashMap,
sync::{Arc, Mutex},
};
use tokio::sync::Notify;
use uuid::Uuid;
pub(crate) struct Watch {
path: String,
changed: Arc<Notify>,
}
pub(crate) type Watches = Arc<Mutex<HashMap<Uuid, Watch>>>;
pub(crate) fn changed(watches: &Watches, path: &str) {
let targets: Vec<_> = watches
.lock()
.unwrap()
.values()
// Removing/moving an ancestor also invalidates a watch inside that subtree.
.filter(|w| under(path, &w.path) || under(&w.path, path))
.map(|w| w.changed.clone())
.collect();
for changed in targets {
changed.notify_one();
}
}
struct Registration {
watches: Watches,
key: Uuid,
}
impl Drop for Registration {
fn drop(&mut self) {
self.watches.lock().unwrap().remove(&self.key);
}
}
fn path(caller: &str, path: &str) -> String {
let path = sausage_vfs::resolve("/", path);
let path = path.trim_end_matches('/');
if path.is_empty() {
return "/".into();
}
if under(path, "/self") {
format!("/proc/{caller}{}", &path[5..])
} else {
path.into()
}
}
impl Host {
pub(crate) async fn watch(&self, caller: &str, id: Option<Value>, target: &str) -> RpcResult {
let id = id.ok_or_else(|| error(-32600, "watch needs a request ID"))?;
let mut ended = self
.0
.state
.lock()
.unwrap()
.processes
.get(caller)
.filter(|p| p.info.state == "running")
.map(|p| p.ended.subscribe())
.ok_or_else(|| error(-32600, "watch needs a running caller process"))?;
let changed = Arc::new(Notify::new());
let registration = Registration {
watches: self.0.watches.clone(),
key: Uuid::now_v7(),
};
self.0.watches.lock().unwrap().insert(
registration.key,
Watch {
path: path(caller, target),
changed: changed.clone(),
},
);
self.call_notify(caller, &id, "armed", json!({}));
loop {
tokio::select! {
biased;
_ = ended.wait_for(|ended| *ended) => return Ok(Value::Null),
_ = changed.notified() => self.call_notify(caller, &id, "changed", json!({})),
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Pending changes coalesce; a change after consuming the last permit is retained.
#[tokio::test]
async fn coalescing_and_cleanup() {
let watches = Watches::default();
let n = Arc::new(Notify::new());
let r = Registration {
watches: watches.clone(),
key: Uuid::now_v7(),
};
watches.lock().unwrap().insert(
r.key,
Watch {
path: "/mailbox".into(),
changed: n.clone(),
},
);
changed(&watches, "/mailboxes/a");
assert!(
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
.await
.is_err()
);
changed(&watches, "/mailbox/a");
changed(&watches, "/mailbox/b");
n.notified().await;
assert!(
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
.await
.is_err()
);
changed(&watches, "/mailbox/c");
n.notified().await;
drop(r);
assert!(watches.lock().unwrap().is_empty());
changed(&watches, "/mailbox/d");
assert!(
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
.await
.is_err()
);
}
#[test]
fn self_alias_and_normalization() {
assert_eq!(path("writer", "/self/../self/a/"), "/proc/writer/a");
assert_eq!(path("writer", "/selfish/a"), "/selfish/a");
assert_eq!(path("writer", "/"), "/");
}
}
#[cfg(test)]
mod lifetime_tests {
use super::*;
/// The real caller's ended event releases the registration, without a cancellation RPC.
#[tokio::test]
async fn caller_death_removes_watch() {
let temp = tempfile::tempdir().unwrap();
let runtime = crate::runtime::Runtime::new(&temp.path().join("cache")).unwrap();
let host = Host::test_world(&temp.path().join("world"), runtime)
.await
.unwrap();
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel();
host.register_native(
"holder",
Arc::new(move |ctx, _, _| {
let tx = tx.clone();
Box::pin(async move {
tx.send(ctx.pid.clone()).unwrap();
std::future::pending().await
})
}),
);
let h = host.clone();
let task = tokio::spawn(async move { h.call("holder.hold", json!({})).await });
let pid = rx.recv().await.unwrap();
let mut watch = Box::pin(host.watch(&pid, Some(json!("watch")), "/mailbox"));
assert!(
watch
.as_mut()
.poll(&mut std::task::Context::from_waker(std::task::Waker::noop()))
.is_pending()
);
assert_eq!(host.0.watches.lock().unwrap().len(), 1);
host.call("core.kill", json!({"pid":pid})).await.unwrap();
watch.await.unwrap();
assert!(host.0.watches.lock().unwrap().is_empty());
assert!(task.await.unwrap().is_err());
host.stop().await;
}
}
+1
View File
@@ -92,6 +92,7 @@ mod tests {
"/config", "/config",
"/state", "/state",
"/state/core/journal.sqlite", "/state/core/journal.sqlite",
"/logs/core/journal.sqlite",
] { ] {
assert!(!std::path::Path::new(path).exists(), "{path}"); assert!(!std::path::Path::new(path).exists(), "{path}");
} }
+311 -11
View File
@@ -43,6 +43,86 @@ fn fixture(version: usize) -> PathBuf {
})[version - 1] })[version - 1]
.clone() .clone()
} }
fn fixture_hash(version: usize) -> String {
use sha2::{Digest, Sha256};
format!(
"{:x}",
Sha256::digest(std::fs::read(fixture(version)).unwrap())
)
}
/// PROTOCOL 10.3: summaries omit even large schemas and report sorted main-method diffs.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn summarizes_method_changes() {
let mut old = document("unchanged");
old["methods"] = json!([
{"name":"z_removed"}, {"name":"a_removed"}, {"name":"direct", "params":[]},
{"name":"referenced", "result":{"schema":{"$ref":"#/components/schemas/Node"}}},
{"name":"unrelated", "result":{"schema":{"type":"string"}}}
]);
old["components"]["schemas"] = json!({
"Node":{"properties":{"next":{"$ref":"#/components/schemas/Node"}, "value":{"$ref":"#/components/schemas/Value"}}},
"Value":{"type":"string"}, "Unused":{"type":"string"}
});
let mut next = old.clone();
next["methods"].as_array_mut().unwrap().drain(..2);
next["methods"][0]["params"] = json!([{"name":"limit","schema":{"type":"integer"}}]);
next["methods"].as_array_mut().unwrap().extend(
json!([
{"name":"z_added"}, {"name":"a_added"}
])
.as_array()
.unwrap()
.iter()
.cloned(),
);
next["components"]["schemas"]["Value"]["type"] = json!("integer");
next["components"]["schemas"]["Unused"]["description"] = json!("x".repeat(220_000));
let world = TestWorld::new();
world.plugin("notes", old, next.clone());
let host = world.start().await;
let result = activate(&host, "notes").await.unwrap();
assert_eq!(result["notes"]["method_count"], 5);
assert_eq!(result["notes"]["added"], json!(["a_added", "z_added"]));
assert_eq!(
result["notes"]["removed"],
json!(["a_removed", "z_removed"])
);
assert_eq!(result["notes"]["changed"], json!(["direct", "referenced"]));
assert!(result.to_string().len() < 1024);
assert_eq!(call(&host, "notes.rpc.discover", json!({})).await, next);
assert_eq!(
call(
&host,
"notes.read",
json!({"path":"/proc/core/plugins/notes/discover.json"})
)
.await,
next
);
host.stop().await;
}
/// PROTOCOL 10.3: first installation has no previous build; a no-op has no method changes.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn summarizes_first_install_and_noop() {
let world = TestWorld::new();
world.plugin("notes", document("add"), document("add"));
std::fs::remove_file(world.0.join("plugins/notes/plugin.wasm")).unwrap();
let host = world.start().await;
let result = activate(&host, "notes").await.unwrap();
assert_eq!(result["notes"]["previous"], Value::Null);
assert_eq!(result["notes"]["added"], json!(["add"]));
assert_eq!(result["notes"]["hash"], fixture_hash(2));
let result = activate(&host, "notes").await.unwrap();
assert_eq!(result["notes"]["previous"], fixture_hash(2));
for field in ["added", "removed", "changed"] {
assert_eq!(result["notes"][field], json!([]));
}
assert!(!world.0.join("plugins/notes/plugin.prev.wasm").exists());
host.stop().await;
}
struct TestWorld(PathBuf); struct TestWorld(PathBuf);
impl TestWorld { impl TestWorld {
/// Creates an isolated world with two version documents per plugin. /// Creates an isolated world with two version documents per plugin.
@@ -80,9 +160,9 @@ impl Drop for TestWorld {
let _ = std::fs::remove_dir_all(&self.0); let _ = std::fs::remove_dir_all(&self.0);
} }
} }
/// Builds a method-only document without an upgrade delay. /// Builds a method-only document; all plugins use the host's five-second grace.
fn document(name: &str) -> Value { fn document(name: &str) -> Value {
json!({"methods":[{"name":name,"params":[],"result":{"schema":{"type":"string"}}}],"x-sausage-grace":0}) json!({"methods":[{"name":name,"params":[],"result":{"schema":{"type":"string"}}}]})
} }
/// Adds an import of a provider's method list. /// Adds an import of a provider's method list.
fn consumer(provider: &str, expected: Value) -> Value { fn consumer(provider: &str, expected: Value) -> Value {
@@ -404,7 +484,7 @@ async fn review_dropped_activation_completes() {
host.stop().await; host.stop().await;
} }
/// PROTOCOL 10.1: discovery fails after its fixed ten-second deadline and receives EOF. /// PROTOCOL 10.1: discovery fails after its fixed ten-second deadline and receives EOF.
/// PROTOCOL 3.7: old processes retain their declared grace while new calls use new code. /// PROTOCOL 3.7: old processes get five seconds regardless of declarations; new calls use new code.
/// PROTOCOL 3.7: retirement waits for all live admissions, including reverted versions. /// PROTOCOL 3.7: retirement waits for all live admissions, including reverted versions.
/// PROTOCOL 3.7: rejected discovery candidates never receive retirement hooks. /// PROTOCOL 3.7: rejected discovery candidates never receive retirement hooks.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -481,7 +561,19 @@ async fn activates_copy() {
world.plugin("notes", document("add"), document("add")); world.plugin("notes", document("add"), document("add"));
let host = world.start().await; let host = world.start().await;
let result = activate(&host, "notes").await.unwrap(); let result = activate(&host, "notes").await.unwrap();
assert!(result["notes"]["methods"].is_array()); assert_eq!(
result,
json!({"notes": {
"plugin":"notes", "wasm":"/plugins/notes/plugin.wasm",
"hash":fixture_hash(2), "previous":fixture_hash(1),
"method_count":1, "added":[], "removed":[], "changed":[],
"interface_check":"passed"
}})
);
assert_eq!(
call(&host, "notes.rpc.discover", json!({})).await,
document("add")
);
assert_eq!( assert_eq!(
call(&host, "notes.version", json!({})).await["version"], call(&host, "notes.version", json!({})).await["version"],
"2" "2"
@@ -512,7 +604,7 @@ async fn source_changes_do_not_change_candidate() {
); );
host.stop().await; host.stop().await;
} }
/// PROTOCOL 3.7: plugins without a declared grace period retain old calls before the default deadline. /// PROTOCOL 3.7: missing metadata also retains old calls until the uniform deadline.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn default_grace_retains_old_process() { async fn default_grace_retains_old_process() {
let world = TestWorld::new(); let world = TestWorld::new();
@@ -531,6 +623,104 @@ async fn default_grace_retains_old_process() {
assert_eq!(host.processes()[&pid].state, "running"); assert_eq!(host.processes()[&pid].state, "running");
host.stop().await; host.stop().await;
} }
/// PROTOCOL 3.7: metadata cannot shorten or extend the actual grace for active requests.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn uniform_grace_ignores_discovery_declarations() {
let world = TestWorld::new();
let declarations = [
("missing", None),
("default", Some(json!(60))),
("long", Some(json!(3600))),
("zero", Some(json!(0))),
("negative", Some(json!(-1))),
("invalid", Some(json!("long"))),
("huge", Some(json!(1e300))),
];
for (name, grace) in &declarations {
let mut old = document("work");
if let Some(grace) = grace {
old["x-sausage-grace"] = grace.clone();
}
world.plugin(name, old, document("work"));
}
let host = world.start().await;
let mut ready = host.subscribe();
let mut calls = Vec::new();
for (name, _) in &declarations {
let host = host.clone();
let method = format!("{name}.work");
calls.push(tokio::spawn(async move {
let result = host.call(&method, json!({})).await;
(result, std::time::Instant::now())
}));
}
// Guest notifications are events, not process/file polling.
for _ in &declarations {
let event = tokio::time::timeout(Duration::from_secs(30), ready.recv())
.await
.unwrap()
.unwrap();
assert_eq!(event["params"]["name"], "ready");
}
let before = std::time::Instant::now();
let plugins: Vec<_> = declarations
.iter()
.map(|(name, _)| json!({"plugin":name,"wasm":format!("/state/{name}/next.wasm")}))
.collect();
call(&host, "core.activate", json!({"plugins":plugins})).await;
// Use the existing discovery+stop test budget, not a new execution limit.
tokio::time::timeout(Duration::from_secs(15), async {
for task in calls {
let (result, ended) = task.await.unwrap();
assert_eq!(result.unwrap_err()["code"], -32002);
assert!(
ended.duration_since(before) >= Duration::from_secs(5),
"every declaration gets the full grace"
);
}
})
.await
.unwrap();
host.stop().await;
}
/// PROTOCOL 10.4/3.7: a watcher refresh uses the same policy as explicit activation.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn manual_copy_uses_uniform_grace() {
let world = TestWorld::new();
let mut old = document("work");
old["x-sausage-grace"] = json!(3600);
world.plugin("notes", old, document("work"));
let host = world.start().await;
// Wait for initial discovery without changing the installed version.
call(
&host,
"core.activate",
json!({"plugin":"notes","wasm":"/plugins/notes/plugin.wasm"}),
)
.await;
let mut ready = host.subscribe();
let work_host = host.clone();
let work = tokio::spawn(async move { work_host.call("notes.work", json!({})).await });
let event = tokio::time::timeout(Duration::from_secs(30), ready.recv())
.await
.unwrap()
.unwrap();
assert_eq!(event["params"]["name"], "ready");
let before = std::time::Instant::now();
std::fs::copy(fixture(2), world.0.join("plugins/notes/plugin.wasm")).unwrap();
assert_eq!(
tokio::time::timeout(Duration::from_secs(15), work)
.await
.unwrap()
.unwrap()
.unwrap_err()["code"],
-32002
);
assert!(before.elapsed() >= Duration::from_secs(5));
host.stop().await;
}
/// PROTOCOL 10.3: candidate imports must fit installed providers. /// PROTOCOL 10.3: candidate imports must fit installed providers.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn checks_candidate_imports() { async fn checks_candidate_imports() {
@@ -577,7 +767,12 @@ async fn activates_group() {
consumer("notes", document("other")), consumer("notes", document("other")),
); );
let host = world.start().await; let host = world.start().await;
call(&host,"core.activate",json!({"plugins":[{"plugin":"notes","wasm":"/state/notes/next.wasm"},{"plugin":"agent","wasm":"/state/agent/next.wasm"}]})).await; let result = call(&host,"core.activate",json!({"plugins":[{"plugin":"notes","wasm":"/state/notes/next.wasm"},{"plugin":"agent","wasm":"/state/agent/next.wasm"}]})).await;
assert_eq!(result.as_object().unwrap().len(), 2);
assert_eq!(result["notes"]["added"], json!(["other"]));
assert_eq!(result["notes"]["removed"], json!(["add"]));
assert_eq!(result["agent"]["method_count"], 1);
assert_eq!(result["agent"]["interface_check"], "passed");
assert_eq!( assert_eq!(
call(&host, "agent.version", json!({})).await["version"], call(&host, "agent.version", json!({})).await["version"],
"2" "2"
@@ -594,12 +789,13 @@ async fn force_skips_checks() {
consumer("absent", document("add")), consumer("absent", document("add")),
); );
let host = world.start().await; let host = world.start().await;
call( let result = call(
&host, &host,
"core.activate", "core.activate",
json!({"plugin":"notes","wasm":"/state/notes/next.wasm","force":true}), json!({"plugin":"notes","wasm":"/state/notes/next.wasm","force":true}),
) )
.await; .await;
assert_eq!(result["notes"]["interface_check"], "skipped");
host.stop().await; host.stop().await;
} }
/// PROTOCOL 10.3: revert activates the previous copy and preserves the replaced version. /// PROTOCOL 10.3: revert activates the previous copy and preserves the replaced version.
@@ -609,7 +805,11 @@ async fn reverts_previous_version() {
world.plugin("notes", document("add"), document("add")); world.plugin("notes", document("add"), document("add"));
let host = world.start().await; let host = world.start().await;
activate(&host, "notes").await.unwrap(); activate(&host, "notes").await.unwrap();
call(&host, "core.revert", json!({"plugin":"notes"})).await; let result = call(&host, "core.revert", json!({"plugin":"notes"})).await;
assert_eq!(result["notes"]["hash"], fixture_hash(1));
assert_eq!(result["notes"]["previous"], fixture_hash(2));
assert_eq!(result["notes"]["interface_check"], "passed");
assert!(result["notes"].get("methods").is_none());
assert_eq!( assert_eq!(
call(&host, "notes.version", json!({})).await["version"], call(&host, "notes.version", json!({})).await["version"],
"1" "1"
@@ -716,11 +916,11 @@ async fn graph_missing_import() {
assert!(dot.as_str().unwrap().contains("absent")); assert!(dot.as_str().unwrap().contains("absent"));
host.stop().await; host.stop().await;
} }
/// Checks the declared upgrade grace and dispatch to the new version. /// Checks that an explicit long declaration cannot extend grace; new calls use the new version.
async fn old_process_grace_and_new_calls() { async fn old_process_grace_and_new_calls() {
let world = TestWorld::new(); let world = TestWorld::new();
let mut old = document("add"); let mut old = document("add");
old["x-sausage-grace"] = json!(0.5); old["x-sausage-grace"] = json!(3600);
world.plugin("notes", old, document("add")); world.plugin("notes", old, document("add"));
let host = world.start().await; let host = world.start().await;
call(&host, "notes.hold", json!({})).await; call(&host, "notes.hold", json!({})).await;
@@ -737,6 +937,76 @@ async fn old_process_grace_and_new_calls() {
"2" "2"
); );
marker(world.0.join("state/notes/ended-1"), "ended").await; marker(world.0.join("state/notes/ended-1"), "ended").await;
let notice: Value = serde_json::from_slice(
&std::fs::read(world.0.join(format!("state/notes/superseded-{pid}"))).unwrap(),
)
.unwrap();
assert_eq!(notice["params"]["current"], fixture_hash(2));
host.stop().await;
}
/// PROTOCOL 3.7: every old process gets the new hash while still running, before grace ends.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn superseded_notifies_each_old_process() {
let world = TestWorld::new();
let mut old = document("hold");
old["x-sausage-grace"] = json!(60);
world.plugin("notes", old, document("hold"));
let host = world.start().await;
call(&host, "notes.hold", json!({})).await;
call(&host, "notes.hold", json!({})).await;
let pids: Vec<_> = host
.processes()
.into_iter()
.filter(|(_, process)| process.method == "hold")
.map(|(pid, _)| pid)
.collect();
assert_eq!(pids.len(), 2);
activate(&host, "notes").await.unwrap();
let expected =
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":fixture_hash(2)}})
.to_string();
for pid in pids {
// The notice arrives while the old process is still in its five-second window.
marker(
world.0.join(format!("state/notes/superseded-{pid}")),
&expected,
)
.await;
assert_eq!(host.processes()[&pid].state, "running");
}
assert_eq!(
call(&host, "notes.version", json!({})).await["version"],
"2"
);
host.stop().await;
}
/// PROTOCOL 3.7: a service that exits on the notice restarts before its grace deadline.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn superseded_service_exits_early() {
let world = TestWorld::new();
let mut old = document("serve");
old["x-sausage-grace"] = json!(60);
world.plugin("notes", old, document("serve"));
std::fs::write(
world.0.join("config/boot.yaml"),
json!([{"method":"notes.serve","params":{"exit_on_superseded":true}}]).to_string(),
)
.unwrap();
let host = world.start().await;
marker(world.0.join("state/notes/service"), "1").await;
let pid = host
.processes()
.into_iter()
.find(|(_, p)| p.method == "serve")
.unwrap()
.0;
activate(&host, "notes").await.unwrap();
marker(world.0.join("state/notes/service"), "2").await;
assert!(!host.processes().contains_key(&pid));
assert!(
!world.0.join("state/notes/ended-1").exists(),
"old service must exit on the notice, not EOF"
);
host.stop().await; host.stop().await;
} }
/// PROTOCOL 3.7: retirement receives the previous hash only after old processes end. /// PROTOCOL 3.7: retirement receives the previous hash only after old processes end.
@@ -821,7 +1091,7 @@ async fn upgrades_enter_the_journal() {
.as_array() .as_array()
.unwrap() .unwrap()
.iter() .iter()
.any(|e| e["line"] == "upgrade grace 60 seconds") .any(|e| e["line"] == "upgrade grace 5 seconds")
{ {
break entries; break entries;
} }
@@ -945,6 +1215,36 @@ async fn old_consumer_can_call_removed_provider_method() {
host.stop().await; host.stop().await;
} }
/// PROTOCOL 8: supervise actual WASI exits on either side of the readiness response.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn crashed_boot_service_restarts_before_and_after_ready() {
for when in ["before", "after"] {
let world = TestWorld::new();
world.plugin("notes", document("serve"), document("serve"));
std::fs::write(
world.0.join("config/boot.yaml"),
json!([{"method":"notes.serve","params":{"crash_once":when}}]).to_string(),
)
.unwrap();
let host = world.start().await;
marker(world.0.join("state/notes/service"), "1").await;
let entries = call(
&host,
"core.log",
json!({"grep":"service start notes.serve"}),
)
.await;
assert_eq!(entries.as_array().unwrap().len(), 2);
let processes = host.processes();
let services: Vec<_> = processes.values().filter(|p| p.method == "serve").collect();
assert_eq!(services.len(), 1);
assert_eq!(services[0].params, json!({"crash_once":when}));
let exits = call(&host, "core.log", json!({"plugin":"notes","grep":"exit 1"})).await;
assert_eq!(exits.as_array().unwrap().len(), 1);
host.stop().await;
}
}
/// PROTOCOL 10.2: activation and revert check interfaces but leave persistent state unchanged. /// PROTOCOL 10.2: activation and revert check interfaces but leave persistent state unchanged.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn state_is_outside_interface_checks() { async fn state_is_outside_interface_checks() {
+22 -1
View File
@@ -489,7 +489,28 @@ async fn oversized_current_turn_preserves_the_previous_cursor() {
.unwrap(); .unwrap();
let (body, reply) = server.next().await; let (body, reply) = server.next().await;
let serialized = input(&body, api).to_string(); let serialized = input(&body, api).to_string();
assert!(serialized.contains("[tool output truncated")); assert!(serialized.contains("Full output: /logs/agent/"));
let history = world.history();
let result: Value = history
.lines()
.map(|line| serde_json::from_str::<Value>(line).unwrap())
.find(|entry| {
entry["output"]
.as_str()
.is_some_and(|out| out.contains("Full output: "))
})
.unwrap();
let output = result["output"].as_str().unwrap();
let path = output
.split("Full output: ")
.nth(1)
.unwrap()
.split(". ")
.next()
.unwrap();
let archive = std::fs::read_to_string(world.0.join(path.trim_start_matches('/'))).unwrap();
assert!(archive.contains(&"x".repeat(200_000)));
assert!(output.len() <= 50 * 1024);
assert!(serialized.len() < 70_000); assert!(serialized.len() < 70_000);
reply reply
.send((413, json!({"type":"request_too_large"}))) .send((413, json!({"type":"request_too_large"})))
+413 -12
View File
@@ -12,6 +12,23 @@ impl TestWorld {
World::create(&path).unwrap(); World::create(&path).unwrap();
Self(path) Self(path)
} }
/// Short-deadline tests must not queue on another test's Cargo target lock.
async fn timeout_host(&self, name: &str) -> Host {
Host::start_with_cache(&self.0, &Path::new(CACHE).join("timeouts").join(name))
.await
.unwrap()
}
/// Creates a dependency-free plugin fixture.
fn simple(&self, name: &str, text: &str) {
let root = self.0.join(format!("plugins/{name}/source"));
std::fs::create_dir_all(root.join("src")).unwrap();
std::fs::write(
root.join("Cargo.toml"),
format!("[package]\nname='{name}'\nversion='0.1.0'\nedition='2024'\n"),
)
.unwrap();
std::fs::write(root.join("src/main.rs"), text).unwrap();
}
/// Copies a fixture into its plugin source folder. /// Copies a fixture into its plugin source folder.
fn source(&self, name: &str, source: &Path) { fn source(&self, name: &str, source: &Path) {
copy(source, &self.0.join(format!("plugins/{name}/source"))); copy(source, &self.0.join(format!("plugins/{name}/source")));
@@ -56,8 +73,28 @@ async fn build(world: &TestWorld, name: &str) -> Result<Value, Value> {
.register(&host); .register(&host);
let result = host.call("cargo.build", json!({"plugin":name})).await; let result = host.call("cargo.build", json!({"plugin":name})).await;
host.stop().await; host.stop().await;
let logs = match &result {
Ok(output) => &output["logs"],
Err(error) => &error["data"]["logs"],
};
check_logs(world, logs);
result result
} }
/// RPC paths point at full raw streams, outside pruned build folders.
fn check_logs(world: &TestWorld, logs: &Value) {
let mut total = 0;
for stream in ["stdout", "stderr"] {
let path = logs[stream]["path"].as_str().expect("log path");
assert!(path.starts_with("/logs/cargo/"));
assert!(path.ends_with(&format!(".{stream}.log")));
let bytes = std::fs::metadata(world.0.join(path.trim_start_matches('/')))
.unwrap()
.len();
assert_eq!(logs[stream]["bytes"], bytes);
total += bytes;
}
assert_eq!(logs["total_bytes"], total);
}
struct Valid { struct Valid {
world: TestWorld, world: TestWorld,
result: Value, result: Value,
@@ -189,6 +226,20 @@ async fn build_output_and_warnings() {
.unwrap() .unwrap()
.contains("fixture build warning") .contains("fixture build warning")
); );
let logs = &valid.result["logs"];
let raw = |stream: &str| {
std::fs::read_to_string(
valid.world.0.join(
logs[stream]["path"]
.as_str()
.unwrap()
.trim_start_matches('/'),
),
)
.unwrap()
};
assert!(raw("stdout").contains("\"reason\":\"compiler-artifact\""));
assert!(raw("stderr").contains("Finished "));
let output = valid.result["tests"]["output"].as_str().unwrap(); let output = valid.result["tests"]["output"].as_str().unwrap();
assert!(output.contains("INTEGRATION_CASE")); assert!(output.contains("INTEGRATION_CASE"));
assert!(!output.contains("Compiling ")); assert!(!output.contains("Compiling "));
@@ -227,14 +278,7 @@ async fn manifest_and_nested_sources() {
assert!(manifest.contains("optional = true")); assert!(manifest.contains("optional = true"));
let document: toml_edit::DocumentMut = manifest.parse().unwrap(); let document: toml_edit::DocumentMut = manifest.parse().unwrap();
let sdk = Path::new(document["target"]["cfg(target_arch = \"wasm32\")"]["dependencies"]["sausage_plugin"]["path"].as_str().unwrap()); let sdk = Path::new(document["target"]["cfg(target_arch = \"wasm32\")"]["dependencies"]["sausage_plugin"]["path"].as_str().unwrap());
assert_eq!(sdk.parent().unwrap(), Path::new(CACHE).join("sdk")); assert_eq!(sdk, valid.world.0.join("plugins/cargo/sdk"));
let hash = sdk.file_name().unwrap().to_str().unwrap();
assert_eq!(hash.len(), 64);
assert!(hash.bytes().all(|byte| byte.is_ascii_hexdigit()));
assert_eq!(
std::fs::read(sdk.join("src/lib.rs")).unwrap(),
std::fs::read(valid.world.0.join("plugins/cargo/sdk/src/lib.rs")).unwrap()
);
assert_eq!( assert_eq!(
std::fs::read_to_string(root.join("src/target/data")).unwrap(), std::fs::read_to_string(root.join("src/target/data")).unwrap(),
"nested target" "nested target"
@@ -260,6 +304,7 @@ async fn native_discovery_and_error_paths() {
.await .await
.unwrap_err(); .unwrap_err();
assert_eq!(error["code"], -32004); assert_eq!(error["code"], -32004);
check_logs(&world, &error["data"]["logs"]);
assert!( assert!(
error["data"]["output"] error["data"]["output"]
.as_str() .as_str()
@@ -431,7 +476,7 @@ async fn source_copy() {
assert!( assert!(
std::fs::read_to_string(valid.world.0.join("state/cargo/work/Cargo.toml")) std::fs::read_to_string(valid.world.0.join("state/cargo/work/Cargo.toml"))
.unwrap() .unwrap()
.contains(Path::new(CACHE).join("sdk").to_str().unwrap()) .contains(valid.world.0.join("plugins/cargo/sdk").to_str().unwrap())
); );
} }
/// PROTOCOL 11: compiler failures return compiler diagnostics. /// PROTOCOL 11: compiler failures return compiler diagnostics.
@@ -585,6 +630,96 @@ async fn unchanged_sdk() {
modified modified
); );
} }
/// PROTOCOL 11: an SDK edit is used by the next build on the same running host.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn world_sdk_edits_apply_without_restart() {
tokio::time::timeout(std::time::Duration::from_secs(180), async {
let world = TestWorld::new();
world.simple("sdk-probe", "fn main() {}\n#[test]\nfn world_sdk_probe() { assert_eq!(sausage_plugin::world_sdk_probe(), 1); }\n");
let manifest = world.0.join("plugins/sdk-probe/source/Cargo.toml");
let mut text = std::fs::read_to_string(&manifest).unwrap();
text.push_str("\n[dependencies]\nsausage-plugin = '0.1'\n");
std::fs::write(manifest, text).unwrap();
let host = Host::start_with_cache(&world.0, Path::new(CACHE)).await.unwrap();
let library = world.0.join("plugins/cargo/sdk/src/lib.rs");
let original = std::fs::read_to_string(&library).unwrap();
std::fs::write(&library, format!("{original}\npub fn world_sdk_probe() -> u8 {{ 1 }}\n")).unwrap();
let params = json!({"plugin":"sdk-probe", "timeout_secs":120, "test_timeout_secs":10});
let first = host.call("cargo.build", params.clone()).await.unwrap();
assert_eq!(first["tests"]["passed"], 1);
// Only the SDK changes: Cargo must not reuse its old compiled version.
std::fs::write(&library, format!("{original}\npub fn world_sdk_probe() -> u8 {{ 2 }}\n")).unwrap();
let second = host.call("cargo.build", params).await.unwrap_err();
assert_eq!(second["code"], -32004);
assert_eq!(second["data"]["tests"]["failed"], 1);
assert!(second["data"]["tests"]["output"].as_str().unwrap().contains("world_sdk_probe"));
host.stop().await;
}).await.expect("SDK edit regression timed out");
}
/// PROTOCOL 11: startup seeds a missing SDK, but never repairs or overwrites an existing one.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn world_sdk_survives_restart_and_missing_sdk_is_reseeded() {
tokio::time::timeout(std::time::Duration::from_secs(30), async {
let world = TestWorld::new();
let sdk = world.0.join("plugins/cargo/sdk");
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
let library = sdk.join("src/lib.rs");
let original = std::fs::read(&library).unwrap();
let manifest = sdk.join("Cargo.toml");
let edited = b"// world SDK edit must survive restart\n";
std::fs::write(&library, edited).unwrap();
std::fs::write(&manifest, "# edited manifest\n").unwrap();
let removed = sdk.join("macros/src/lib.rs");
std::fs::remove_file(&removed).unwrap();
host.stop().await;
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
assert_eq!(std::fs::read(&library).unwrap(), edited);
assert_eq!(
std::fs::read_to_string(&manifest).unwrap(),
"# edited manifest\n"
);
assert!(!removed.exists(), "existing SDK must not be repaired");
host.stop().await;
std::fs::remove_dir_all(&sdk).unwrap();
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
assert_eq!(std::fs::read(&library).unwrap(), original);
assert!(removed.is_file());
assert!(sdk.join("interface/src/lib.rs").is_file());
host.stop().await;
})
.await
.expect("SDK restart regression timed out");
}
/// PROTOCOL 11: removing the SDK while the host is running re-seeds it on the next build.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn world_sdk_missing_at_build_is_reseeded() {
tokio::time::timeout(std::time::Duration::from_secs(180), async {
let world = TestWorld::new();
world.simple("reseed-probe", "fn main() {}\n");
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
let sdk = world.0.join("plugins/cargo/sdk");
let original = std::fs::read(sdk.join("src/lib.rs")).unwrap();
std::fs::remove_dir_all(&sdk).unwrap();
host.call(
"cargo.build",
json!({"plugin":"reseed-probe", "timeout_secs":120, "test_timeout_secs":10}),
)
.await
.unwrap();
assert_eq!(std::fs::read(sdk.join("src/lib.rs")).unwrap(), original);
host.stop().await;
})
.await
.expect("SDK re-seed build regression timed out");
}
/// PROTOCOL 11: init can build a plugin without a running host. /// PROTOCOL 11: init can build a plugin without a running host.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn standalone_builder() { async fn standalone_builder() {
@@ -606,6 +741,259 @@ async fn standalone_builder() {
1 1
); );
} }
/// a named CPU-bound WASI test times out, while other instances still run.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn named_test_timeout_and_mutex_release() {
let world = TestWorld::new();
world.simple("test-timeout", "fn main() {} #[test] fn a_hung() { loop { std::hint::black_box(1); } } #[test] fn z_remaining() { eprintln!(\"REMAINING_TEST_RAN\"); }");
world.simple("next", "fn main() {} #[test] fn passes() {}");
let integration = world.0.join("plugins/test-timeout/source/tests");
std::fs::create_dir_all(&integration).unwrap();
std::fs::write(
integration.join("remaining.rs"),
"#[test] fn integration_remaining() { eprintln!(\"REMAINING_BINARY_RAN\"); }",
)
.unwrap();
let host = world.timeout_host("named").await;
let error = tokio::time::timeout(
std::time::Duration::from_secs(60),
host.call(
"cargo.build",
json!({"plugin":"test-timeout", "test_timeout_secs":1, "timeout_secs":45}),
),
)
.await
.unwrap()
.unwrap_err();
assert_eq!(error["code"], -32004);
assert_eq!(error["message"], "WASI tests failed");
assert_eq!(error["data"]["tests"]["failed"], 1);
assert_eq!(error["data"]["tests"]["passed"], 2);
let output = error["data"]["output"].as_str().unwrap();
assert!(output.contains("test a_hung ... FAILED"), "{output}");
assert!(output.contains("timed out after 1 s"), "{output}");
assert!(output.contains("REMAINING_TEST_RAN"), "{output}");
assert!(output.contains("REMAINING_BINARY_RAN"), "{output}");
let next = tokio::time::timeout(
std::time::Duration::from_secs(30),
host.call("cargo.build", json!({"plugin":"next"})),
)
.await
.unwrap()
.unwrap();
assert_eq!(next["tests"]["passed"], 1);
host.stop().await;
}
/// Wait for a build-script child without using timing sleeps.
async fn script_child(world: &TestWorld) -> u32 {
tokio::time::timeout(std::time::Duration::from_secs(60), async {
loop {
if let Ok(text) = std::fs::read_to_string(world.0.join("state/cargo/work/child.pid"))
&& let Ok(pid) = text.trim().parse()
{
break pid;
}
tokio::task::yield_now().await;
}
})
.await
.expect("build script never started")
}
/// A killed descendant may briefly remain as a zombie, but cannot keep executing.
async fn child_ended(child: u32) {
tokio::time::timeout(std::time::Duration::from_secs(10), async {
loop {
match std::fs::read_to_string(format!("/proc/{child}/stat")) {
Ok(status) if status.split_whitespace().nth(2) == Some("Z") => break,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => break,
_ => tokio::task::yield_now().await,
}
}
})
.await
.expect("cargo descendant survived cancellation/timeout");
}
/// an explicit small total budget kills build.rs and unblocks a queued build.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn build_timeout_and_queued_build() {
let world = TestWorld::new();
world.source("hang", &fixture("cargo-hang"));
world.simple("next", "fn main() {} #[test] fn passes() {}");
let host = world.timeout_host("queued").await;
let building = host.clone();
let task = tokio::spawn(async move {
building
.call("cargo.build", json!({"plugin":"hang", "timeout_secs":5}))
.await
});
let child = script_child(&world).await;
let queued_host = host.clone();
let queued = tokio::spawn(async move {
queued_host
.call("cargo.build", json!({"plugin":"next", "timeout_secs":3}))
.await
});
let error = task.await.unwrap().unwrap_err();
assert_eq!(error["code"], -32004);
assert_eq!(error["message"], "timeout");
assert_eq!(error["data"]["reason"], "timeout");
check_logs(&world, &error["data"]["logs"]);
assert_eq!(error["data"]["step"], "tests");
assert!(error["data"]["elapsed_secs"].as_f64().unwrap() >= 5.0);
assert!(!error["data"]["output"].as_str().unwrap().is_empty());
child_ended(child).await;
let next = tokio::time::timeout(std::time::Duration::from_secs(20), queued)
.await
.unwrap()
.unwrap()
.unwrap();
assert_eq!(next["tests"]["passed"], 1);
host.stop().await;
}
/// the same total deadline covers release compilation and retains test results.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn release_build_timeout_retains_tests() {
let world = TestWorld::new();
world.simple(
"release-timeout",
r#"fn main() {}
#[test] fn prepare_release() {
std::fs::write(concat!(env!("CARGO_MANIFEST_DIR"), "/release-marker"), "ready").unwrap();
}"#,
);
std::fs::write(
world.0.join("plugins/release-timeout/source/build.rs"),
r#"
fn main() {
println!("cargo:rerun-if-changed=release-marker");
if std::path::Path::new("release-marker").exists() {
std::process::Command::new("sh")
.args(["-c", "echo $$ > child.pid; exec sleep 1000"])
.status().unwrap();
}
}"#,
)
.unwrap();
let host = world.timeout_host("release").await;
let building = host.clone();
let task = tokio::spawn(async move {
building
.call(
"cargo.build",
json!({"plugin":"release-timeout", "timeout_secs":5}),
)
.await
});
let child = script_child(&world).await;
let error = tokio::time::timeout(std::time::Duration::from_secs(10), task)
.await
.unwrap()
.unwrap()
.unwrap_err();
assert_eq!(error["code"], -32004);
assert_eq!(error["data"]["reason"], "timeout");
check_logs(&world, &error["data"]["logs"]);
assert_eq!(error["data"]["step"], "release build");
assert_eq!(error["data"]["tests"]["passed"], 1);
assert!(error["data"]["elapsed_secs"].as_f64().unwrap() >= 5.0);
child_ended(child).await;
host.stop().await;
}
/// omission selects the real 900-second deadline, without a 15-minute test.
#[tokio::test]
async fn omitted_build_timeout_uses_default() {
let world = TestWorld::new();
world.source("hang", &fixture("cargo-hang"));
let host = world.timeout_host("default").await;
let building = host.clone();
let task =
tokio::spawn(async move { building.call("cargo.build", json!({"plugin":"hang"})).await });
let child = script_child(&world).await;
tokio::time::pause();
tokio::time::advance(std::time::Duration::from_secs(
sausage::cargo::DEFAULT_TIMEOUT_SECS,
))
.await;
// Let the awakened deadline observe the advanced clock before resuming it.
tokio::task::yield_now().await;
let error = task.await.unwrap().unwrap_err();
tokio::time::resume();
assert_eq!(error["data"]["reason"], "timeout");
assert!(error["data"]["elapsed_secs"].as_f64().unwrap() >= 900.0);
child_ended(child).await;
host.stop().await;
}
/// a caller ending stops its attached native build and releases the mutex.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn ended_caller_releases_build_mutex() {
let world = TestWorld::new();
world.source("hang", &fixture("cargo-hang"));
world.simple("next", "fn main() {} #[test] fn passes() {}");
let host = world.timeout_host("caller").await;
let (finish, finished) = tokio::sync::watch::channel(false);
host.register_native(
"caller",
std::sync::Arc::new(move |context, _, _| {
let mut finished = finished.clone();
Box::pin(async move {
tokio::spawn(
async move { context.call("cargo.build", json!({"plugin":"hang"})).await },
);
let _ = finished.wait_for(|done| *done).await;
Ok(json!({}))
})
}),
);
let calling = host.clone();
let caller = tokio::spawn(async move { calling.call("caller.run", json!({})).await });
let child = script_child(&world).await;
let queued_host = host.clone();
let queued = tokio::spawn(async move {
queued_host
.call("cargo.build", json!({"plugin":"next"}))
.await
});
finish.send_replace(true);
caller.await.unwrap().unwrap();
let next = tokio::time::timeout(std::time::Duration::from_secs(12), queued)
.await
.expect("ended caller retained mutex")
.unwrap()
.unwrap();
assert_eq!(next["tests"]["passed"], 1);
child_ended(child).await;
host.stop().await;
}
/// discovery exposes non-null defaults; invalid budgets cannot disable limits.
#[tokio::test]
async fn timeout_params_discovery_and_validation() {
let world = TestWorld::new();
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
let doc = host.call("cargo.rpc.discover", json!({})).await.unwrap();
let params = doc["methods"][0]["params"].as_array().unwrap();
for (name, default) in [("timeout_secs", 900), ("test_timeout_secs", 120)] {
let param = params.iter().find(|param| param["name"] == name).unwrap();
assert_eq!(param["schema"]["default"], default);
assert_eq!(param["schema"]["minimum"], 1);
assert!(!param["description"].as_str().unwrap().is_empty());
for value in [json!(0), json!(-1), json!(1.5), json!("1"), json!(null)] {
let mut args = json!({"plugin":"missing"});
args[name] = value;
let error = host.call("cargo.build", args).await.unwrap_err();
assert_eq!(error["code"], -32602, "{error}");
}
}
host.stop().await;
}
/// PROTOCOL 11: killing a native build also kills its build-script descendants. /// PROTOCOL 11: killing a native build also kills its build-script descendants.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn kill_process_group() { async fn kill_process_group() {
@@ -615,9 +1003,7 @@ async fn kill_process_group() {
); );
let world = TestWorld::new(); let world = TestWorld::new();
world.source("hang", &fixture("cargo-hang")); world.source("hang", &fixture("cargo-hang"));
let host = Host::start_with_cache(&world.0, Path::new(CACHE)) let host = world.timeout_host("kill").await;
.await
.unwrap();
let building = host.clone(); let building = host.clone();
let task = let task =
tokio::spawn(async move { building.call("cargo.build", json!({"plugin":"hang"})).await }); tokio::spawn(async move { building.call("cargo.build", json!({"plugin":"hang"})).await });
@@ -685,8 +1071,23 @@ async fn kill_process_group() {
.unwrap() .unwrap()
.is_empty() .is_empty()
); );
// Queue a build on the SAME builder before cancelling the lock holder.
world.simple("next", "fn main() {} #[test] fn passes() {}");
let queued_host = host.clone();
let queued = tokio::spawn(async move {
queued_host
.call("cargo.build", json!({"plugin":"next"}))
.await
});
host.call("core.kill", json!({"pid":pid})).await.unwrap(); host.call("core.kill", json!({"pid":pid})).await.unwrap();
assert!(task.await.unwrap().is_err()); assert!(task.await.unwrap().is_err());
child_ended(ancestor).await;
let next = tokio::time::timeout(std::time::Duration::from_secs(10), queued)
.await
.expect("cancelled build retained mutex")
.unwrap()
.unwrap();
assert_eq!(next["tests"]["passed"], 1);
tokio::time::timeout(std::time::Duration::from_secs(10), async { tokio::time::timeout(std::time::Duration::from_secs(10), async {
loop { loop {
let status = std::fs::read_to_string(format!("/proc/{child}/stat")); let status = std::fs::read_to_string(format!("/proc/{child}/stat"));
+53 -18
View File
@@ -19,6 +19,7 @@ fn binary() -> Command {
.arg("--cache-dir") .arg("--cache-dir")
.arg(CACHE) .arg(CACHE)
.env_remove("SAUSAGE_HEADLESS") .env_remove("SAUSAGE_HEADLESS")
.env_remove("SAUSAGE_LOOP")
.kill_on_drop(true); .kill_on_drop(true);
command command
} }
@@ -35,7 +36,7 @@ async fn status(child: &mut Child) -> std::process::ExitStatus {
} }
async fn journal_exists(world: &Path) { async fn journal_exists(world: &Path) {
tokio::time::timeout(Duration::from_secs(15), async { tokio::time::timeout(Duration::from_secs(15), async {
while !world.join("state/core/journal.sqlite").exists() { while !world.join("logs/core/journal.sqlite").exists() {
tokio::task::yield_now().await; tokio::task::yield_now().await;
} }
}) })
@@ -54,9 +55,15 @@ fn install_probe(world: &Path) {
std::fs::copy(probe(), world.join("plugins/probe/plugin.wasm")).unwrap(); std::fs::copy(probe(), world.join("plugins/probe/plugin.wasm")).unwrap();
} }
/// A failed agent build leaves no world. Bare sausage retries and publishes a complete world. /// Failed init leaves no world; successful init seeds the restart policy and a working agent.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn init_builds_agent() { async fn init_builds_agent() {
tokio::time::timeout(Duration::from_secs(300), init_builds_agent_inner())
.await
.expect("init and seeded-agent checks exceeded 300 seconds");
}
async fn init_builds_agent_inner() {
let temporary = tempfile::tempdir().unwrap(); let temporary = tempfile::tempdir().unwrap();
let world = temporary.path().join("world"); let world = temporary.path().join("world");
// Legacy configuration variables must not affect the templates. // Legacy configuration variables must not affect the templates.
@@ -81,7 +88,7 @@ async fn init_builds_agent() {
assert!(!failed.status.success()); assert!(!failed.status.success());
assert!(String::from_utf8_lossy(&failed.stderr).contains("nonexistent-sausage-rustc")); assert!(String::from_utf8_lossy(&failed.stderr).contains("nonexistent-sausage-rustc"));
assert!(!world.exists()); assert!(!world.exists());
let output = configured().output().await.unwrap(); let output = configured().arg("init").output().await.unwrap();
assert!( assert!(
output.status.success(), output.status.success(),
"stdout: {}\nstderr: {}", "stdout: {}\nstderr: {}",
@@ -91,6 +98,22 @@ async fn init_builds_agent() {
assert!( assert!(
String::from_utf8_lossy(&output.stdout).contains("first agent build takes a few minutes") String::from_utf8_lossy(&output.stdout).contains("first agent build takes a few minutes")
); );
// Bare sausage reopens the completed world instead of trying to recreate it.
assert!(configured().output().await.unwrap().status.success());
let prompt = std::fs::read_to_string(world.join("plugins/agent/source/src/prompt.md")).unwrap();
assert!(prompt.contains("HARD RULE: the mediator and all agents NEVER call `core.restart`"));
assert!(prompt.contains("without the person's explicit approval each time"));
assert!(prompt.contains("alias `core.exit`"));
for text in [
"`#loop`",
"SAUSAGE_LOOP=1",
"answers before",
"exit code 75",
] {
assert!(prompt.contains(text), "seeded prompt missing {text}");
}
let boot = std::fs::read_to_string(world.join("config/boot.yaml")).unwrap();
assert!(boot.contains("SAUSAGE_LOOP=1"));
let mut config: Value = let mut config: Value =
serde_yaml_ng::from_slice(&std::fs::read(world.join("config/agent.yaml")).unwrap()) serde_yaml_ng::from_slice(&std::fs::read(world.join("config/agent.yaml")).unwrap())
.unwrap(); .unwrap();
@@ -228,7 +251,7 @@ async fn signals_work_during_boot() {
tokio::time::timeout(Duration::from_secs(15), async { tokio::time::timeout(Duration::from_secs(15), async {
loop { loop {
let db = let db =
rusqlite::Connection::open(world.join("state/core/journal.sqlite")).unwrap(); rusqlite::Connection::open(world.join("logs/core/journal.sqlite")).unwrap();
if let Ok(2) = db.query_row( if let Ok(2) = db.query_row(
"SELECT count(*) FROM journal WHERE line='service start core.lock'", "SELECT count(*) FROM journal WHERE line='service start core.lock'",
[], [],
@@ -249,27 +272,39 @@ async fn signals_work_during_boot() {
/// Piped input supports commands, quit, and the restart exit code. /// Piped input supports commands, quit, and the restart exit code.
#[tokio::test] #[tokio::test]
async fn piped_console_and_exit_codes() { async fn piped_console_and_exit_codes() {
for (line, expected) in [("help\n", 0), ("quit\n", 0), ("core.exit\n", 75)] { for (line, supervised, expected) in [
("help\n", false, 0),
("quit\n", false, 0),
("core.restart\n", false, 0),
("core.exit\n", false, 0),
("core.restart\n", true, 75),
("core.exit\n", true, 75),
] {
let (_folder, world) = world(); let (_folder, world) = world();
let mut child = binary() let mut command = binary();
.arg("run") command.arg("run").arg(world);
.arg(world) if supervised {
command.env("SAUSAGE_LOOP", "1");
}
let mut child = command
.stdin(Stdio::piped()) .stdin(Stdio::piped())
.stdout(Stdio::piped()) .stdout(Stdio::piped())
.spawn() .spawn()
.unwrap(); .unwrap();
child // Supervised calls must finish via restart, not a racing console EOF.
.stdin let mut input = child.stdin.take().unwrap();
.take() input.write_all(line.as_bytes()).await.unwrap();
.unwrap() if !supervised {
.write_all(line.as_bytes()) drop(input);
.await }
.unwrap();
let output = tokio::time::timeout(Duration::from_secs(15), child.wait_with_output()) let output = tokio::time::timeout(Duration::from_secs(15), child.wait_with_output())
.await .await
.unwrap() .unwrap()
.unwrap(); .unwrap();
assert_eq!(output.status.code(), Some(expected)); assert_eq!(output.status.code(), Some(expected));
if !supervised && line.starts_with("core.") {
assert!(String::from_utf8_lossy(&output.stdout).contains("SAUSAGE_LOOP not set"));
}
if expected == 0 { if expected == 0 {
assert!(!output.stdout.is_empty()); assert!(!output.stdout.is_empty());
} }
@@ -301,7 +336,7 @@ async fn console_error_stops_host() {
.await .await
.unwrap(); .unwrap();
assert_eq!(status(&mut child).await.code(), Some(1)); assert_eq!(status(&mut child).await.code(), Some(1));
let db = rusqlite::Connection::open(world.join("state/core/journal.sqlite")).unwrap(); let db = rusqlite::Connection::open(world.join("logs/core/journal.sqlite")).unwrap();
let stopped: i64 = db let stopped: i64 = db
.query_row( .query_row(
"SELECT count(*) FROM journal WHERE plugin='probe' AND line='stop'", "SELECT count(*) FROM journal WHERE plugin='probe' AND line='stop'",
@@ -328,9 +363,9 @@ async fn interrupt_leaves_call_running() {
input.write_all(b"probe.busy\n").await.unwrap(); input.write_all(b"probe.busy\n").await.unwrap();
let pid = tokio::time::timeout(Duration::from_secs(20), async { let pid = tokio::time::timeout(Duration::from_secs(20), async {
loop { loop {
if world.join("state/core/journal.sqlite").exists() { if world.join("logs/core/journal.sqlite").exists() {
let db = let db =
rusqlite::Connection::open(world.join("state/core/journal.sqlite")).unwrap(); rusqlite::Connection::open(world.join("logs/core/journal.sqlite")).unwrap();
if let Ok(pid) = db.query_row( if let Ok(pid) = db.query_row(
"SELECT pid FROM journal WHERE plugin='probe' AND line='start busy {}'", "SELECT pid FROM journal WHERE plugin='probe' AND line='start busy {}'",
[], [],
+258 -41
View File
@@ -129,7 +129,7 @@ async fn stderr_flood_does_not_block_stdin() {
let payload = json!({"large":"x".repeat(1024*1024)}); let payload = json!({"large":"x".repeat(1024*1024)});
assert_eq!(call(&h, "probe.echo", payload.clone()).await, payload); assert_eq!(call(&h, "probe.echo", payload.clone()).await, payload);
h.stop().await; h.stop().await;
let db = rusqlite::Connection::open(w.0.join("state/core/journal.sqlite")).unwrap(); let db = rusqlite::Connection::open(w.0.join("logs/core/journal.sqlite")).unwrap();
let count: i64 = db let count: i64 = db
.query_row( .query_row(
"SELECT count(*) FROM journal WHERE line LIKE 'flood %'", "SELECT count(*) FROM journal WHERE line LIKE 'flood %'",
@@ -198,7 +198,7 @@ async fn null_error_is_success() {
async fn journal_recovers_after_sqlite_write_failure() { async fn journal_recovers_after_sqlite_write_failure() {
let w = TestWorld::new(); let w = TestWorld::new();
let h = w.start().await; let h = w.start().await;
let db = rusqlite::Connection::open(w.0.join("state/core/journal.sqlite")).unwrap(); let db = rusqlite::Connection::open(w.0.join("logs/core/journal.sqlite")).unwrap();
db.busy_timeout(std::time::Duration::from_secs(5)).unwrap(); db.busy_timeout(std::time::Duration::from_secs(5)).unwrap();
db.execute_batch("CREATE TRIGGER reject_writes BEFORE INSERT ON journal BEGIN SELECT RAISE(FAIL,'test failure'); END;").unwrap(); db.execute_batch("CREATE TRIGGER reject_writes BEFORE INSERT ON journal BEGIN SELECT RAISE(FAIL,'test failure'); END;").unwrap();
assert_eq!( assert_eq!(
@@ -206,6 +206,20 @@ async fn journal_recovers_after_sqlite_write_failure() {
json!({"failed_log":true}) json!({"failed_log":true})
); );
call(&h, "core.log", json!({})).await; call(&h, "core.log", json!({})).await;
let batches: Vec<Value> = std::fs::read_dir(w.0.join("logs/core/failed-batches"))
.unwrap()
.map(Result::unwrap)
// The writer atomically publishes .json files; its temporary files can
// still be incomplete while later process-exit entries are spooling.
.filter(|file| file.path().extension().is_some_and(|ext| ext == "json"))
.map(|file| serde_json::from_slice(&std::fs::read(file.path()).unwrap()).unwrap())
.collect();
assert!(
batches
.iter()
.flat_map(|batch| batch.as_array().unwrap())
.any(|entry| entry["line"].as_str().unwrap().contains("failed_log"))
);
db.execute_batch("DROP TRIGGER reject_writes;").unwrap(); db.execute_batch("DROP TRIGGER reject_writes;").unwrap();
assert_eq!( assert_eq!(
call(&h, "probe.echo", json!({"recovered":true})).await, call(&h, "probe.echo", json!({"recovered":true})).await,
@@ -490,14 +504,119 @@ async fn unimplemented_core_methods_return_method_not_found() {
async fn params_must_be_objects() { async fn params_must_be_objects() {
let w = TestWorld::new(); let w = TestWorld::new();
let h = w.start().await; let h = w.start().await;
for (params, kind) in [
(json!("{\"text\":\"hello\"}"), "a string"),
(json!([]), "an array"),
(json!(42), "a number"),
(json!(null), "null"),
(json!(true), "a boolean"),
] {
let expected = json!({
"code": -32602,
"message": format!("params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string.")
});
for method in ["probe.echo", "core.ps"] {
assert_eq!(h.call(method, params.clone()).await.unwrap_err(), expected);
}
assert_eq!(
h.call("core.call", json!({"method":"probe.echo", "params":params}))
.await
.unwrap_err(),
expected
);
}
let object = json!({"text":"hello"});
assert_eq!(call(&h, "probe.echo", object.clone()).await, object);
assert_eq!( assert_eq!(
h.call("probe.echo", json!([])).await.unwrap_err()["code"], call(
-32602 &h,
"core.call",
json!({"method":"probe.echo", "params":object})
)
.await,
object
);
assert_eq!(
call(&h, "core.call", json!({"method":"probe.echo"})).await,
json!({})
); );
h.stop().await; h.stop().await;
} }
/// PROTOCOL 3.1: stdout fast paths reject non-objects without changing defaults.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn stdout_params_must_be_objects() {
let w = TestWorld::new();
let h = w.start().await;
let mut notifications = h.subscribe();
for method in ["core.notify", "core.unlock", "probe.echo"] {
for (params, kind) in [
(json!("{}"), "a string"),
(json!([]), "an array"),
(json!(1.5), "a number"),
(json!(null), "null"),
(json!(false), "a boolean"),
] {
let response = call(
&h,
"probe.raw_call",
json!({
"jsonrpc":"2.0", "id":1, "method":method, "params":params
}),
)
.await;
assert_eq!(response["id"], 1);
assert_eq!(
response["error"],
json!({
"code": -32602,
"message": format!("params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string.")
})
);
}
let params = match method {
"core.notify" => json!({"name":"valid", "params":{"n":1}}),
"core.unlock" => json!({"path":"/state/not-held"}),
_ => json!({"n":1}),
};
let response = call(
&h,
"probe.raw_call",
json!({
"jsonrpc":"2.0", "id":1, "method":method, "params":params
}),
)
.await;
assert_eq!(
response["result"],
if method == "probe.echo" {
params
} else {
Value::Null
}
);
assert!(response.get("error").is_none());
let omitted = call(
&h,
"probe.raw_call",
json!({
"jsonrpc":"2.0", "id":1, "method":method
}),
)
.await;
match method {
"core.notify" => {
assert_eq!(omitted["error"]["message"], "name must be a string");
assert_eq!(notifications.try_recv().unwrap()["params"]["name"], "valid");
}
"core.unlock" => assert_eq!(omitted["error"]["message"], "path must be a string"),
_ => assert_eq!(omitted["result"], json!({})),
}
}
h.stop().await;
}
/// PROTOCOL 3.9: exit without answer reports status and last logs. /// PROTOCOL 3.9: exit without answer reports status and last logs.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn exit_without_answer_reports_status_and_last_logs() { async fn exit_without_answer_reports_status_and_last_logs() {
@@ -989,17 +1108,16 @@ async fn stop_closes_stdin_and_waits_for_exit() {
let w = TestWorld::new(); let w = TestWorld::new();
let h = w.start().await; let h = w.start().await;
let hc = h.clone(); let hc = h.clone();
let mut events = h.subscribe();
let pending = tokio::spawn(async move { hc.call("probe.wait_eof", json!({})).await }); let pending = tokio::spawn(async move { hc.call("probe.wait_eof", json!({})).await });
// Admission is not readiness: stop may otherwise close stdin before the
// request reaches the guest, so wait for the handler to enter its EOF loop.
let pid = tokio::time::timeout(std::time::Duration::from_secs(20), async { let pid = tokio::time::timeout(std::time::Duration::from_secs(20), async {
loop { loop {
if let Some((pid, _)) = h let event = events.recv().await.unwrap();
.processes() if event["params"]["name"] == "waiting-eof" {
.into_iter() break event["params"]["params"]["pid"].clone();
.find(|(_, p)| p.method == "wait_eof")
{
break pid;
} }
tokio::task::yield_now().await;
} }
}) })
.await .await
@@ -1014,19 +1132,6 @@ async fn stop_closes_stdin_and_waits_for_exit() {
h.stop().await; h.stop().await;
} }
/// PROTOCOL 9: core exit requests restart from its handler.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn core_exit_requests_restart_from_its_handler() {
let w = TestWorld::new();
let h = w.start().await;
assert_eq!(call(&h, "core.exit", json!({})).await, json!({}));
tokio::time::timeout(std::time::Duration::from_secs(1), h.exit_requested())
.await
.unwrap();
h.stop().await;
}
/// PROTOCOL 3.3: process ids are uuidv7 and versions are sha256. /// PROTOCOL 3.3: process ids are uuidv7 and versions are sha256.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn process_ids_are_uuidv7_and_versions_are_sha256() { async fn process_ids_are_uuidv7_and_versions_are_sha256() {
@@ -1081,7 +1186,7 @@ async fn journal_filters_and_entries_persist_in_sqlite() {
); );
h.stop().await; h.stop().await;
assert!( assert!(
std::fs::read(w.0.join("state/core/journal.sqlite")) std::fs::read(w.0.join("logs/core/journal.sqlite"))
.unwrap() .unwrap()
.starts_with(b"SQLite format 3") .starts_with(b"SQLite format 3")
); );
@@ -1267,8 +1372,9 @@ async fn journal_entries_have_uuid_time_identity_stream_and_line() {
async fn journal_has_wal_and_identity_indexes() { async fn journal_has_wal_and_identity_indexes() {
let w = TestWorld::new(); let w = TestWorld::new();
let h = w.start().await; let h = w.start().await;
assert!(!w.0.join("state/core/journal.sqlite").exists());
h.stop().await; h.stop().await;
let db = rusqlite::Connection::open(w.0.join("state/core/journal.sqlite")).unwrap(); let db = rusqlite::Connection::open(w.0.join("logs/core/journal.sqlite")).unwrap();
assert_eq!( assert_eq!(
db.query_row("PRAGMA journal_mode", [], |r| r.get::<_, String>(0)) db.query_row("PRAGMA journal_mode", [], |r| r.get::<_, String>(0))
.unwrap(), .unwrap(),
@@ -1288,19 +1394,6 @@ async fn journal_has_wal_and_identity_indexes() {
h.stop().await; h.stop().await;
} }
/// PROTOCOL 9: core exit works through core call.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn core_exit_works_through_core_call() {
let w = TestWorld::new();
let h = w.start().await;
assert_eq!(
call(&h, "core.call", json!({"method":"core.exit","params":{}})).await,
json!({})
);
assert!(h.restart_requested());
h.stop().await;
}
/// PROTOCOL 6: acquisitions from different processes keep their FIFO positions. /// PROTOCOL 6: acquisitions from different processes keep their FIFO positions.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn lock_acquisitions_from_different_processes_are_fifo() { async fn lock_acquisitions_from_different_processes_are_fifo() {
@@ -1931,9 +2024,67 @@ async fn detached_processes_need_explicit_termination() {
} }
} }
/// PROTOCOL 9: core.exit from boot ends the host executable with status 75. /// PROTOCOL 9: restart answers the detached caller, then uses normal stop and status 75.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn core_exit_sets_executable_status_75() { async fn core_restart_cleanly_stops_detached_caller() {
tokio::time::timeout(std::time::Duration::from_secs(30), async {
let w = TestWorld::new();
std::fs::write(
w.0.join("config/boot.yaml"),
br#"[{"method":"probe.restart","params":{}}]"#,
)
.unwrap();
let output = tokio::process::Command::new(env!("CARGO_BIN_EXE_sausage"))
.env("SAUSAGE_LOOP", "1")
.arg("--headless")
.arg("--cache-dir")
.arg(concat!(env!("CARGO_TARGET_TMPDIR"), "/cache"))
.arg("run")
.arg(&w.0)
.stdin(std::process::Stdio::null())
.kill_on_drop(true)
.output()
.await
.unwrap();
assert_eq!(
output.status.code(),
Some(75),
"{}",
String::from_utf8_lossy(&output.stderr)
);
assert_eq!(
std::fs::read_to_string(w.0.join("state/restart-answer")).unwrap(),
"received"
);
assert_eq!(
std::fs::read_to_string(w.0.join("state/restart-stop")).unwrap(),
"persisted after EOF"
);
let db = rusqlite::Connection::open(w.0.join("logs/core/journal.sqlite")).unwrap();
let count: i64 = db
.query_row(
"SELECT count(*) FROM journal WHERE line = 'restart stop hook finished'",
[],
|row| row.get(0),
)
.unwrap();
assert_eq!(count, 1);
let killed: i64 = db
.query_row(
"SELECT count(*) FROM journal WHERE line = 'kill'",
[],
|row| row.get(0),
)
.unwrap();
assert_eq!(killed, 0);
})
.await
.unwrap();
}
/// PROTOCOL 9: supervised core.exit from boot uses the restart shutdown path and code 75.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn core_exit_alias_sets_executable_status_75() {
let w = TestWorld::new(); let w = TestWorld::new();
std::fs::write( std::fs::write(
w.0.join("config/boot.yaml"), w.0.join("config/boot.yaml"),
@@ -1948,6 +2099,7 @@ async fn core_exit_sets_executable_status_75() {
.arg(concat!(env!("CARGO_TARGET_TMPDIR"), "/cache")) .arg(concat!(env!("CARGO_TARGET_TMPDIR"), "/cache"))
.arg("run") .arg("run")
.arg(&w.0) .arg(&w.0)
.env("SAUSAGE_LOOP", "1")
.stdin(std::process::Stdio::null()) .stdin(std::process::Stdio::null())
.kill_on_drop(true) .kill_on_drop(true)
.output(), .output(),
@@ -1962,3 +2114,68 @@ async fn core_exit_sets_executable_status_75() {
String::from_utf8_lossy(&output.stderr) String::from_utf8_lossy(&output.stderr)
); );
} }
/// PROTOCOL 7: logs move, state and the old journal do not; core.log merges both.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn journal_location_preserves_legacy_history() {
let w = TestWorld::new();
let old = w.0.join("state/core/journal.sqlite");
let db = rusqlite::Connection::open(&old).unwrap();
db.execute_batch("CREATE TABLE journal (id TEXT PRIMARY KEY, time TEXT, pid TEXT, plugin TEXT, stream TEXT, line TEXT);
INSERT INTO journal VALUES ('0000','old time','old-pid','old-plugin','core','legacy history');").unwrap();
drop(db);
let before = std::fs::read(&old).unwrap();
let history = w.0.join("state/core/console-history");
std::fs::write(&history, "keep console history").unwrap();
let h = w.start().await;
call(&h, "probe.echo", json!({})).await;
assert!(w.0.join("logs/core/journal.sqlite").exists());
let entries = call(&h, "core.log", json!({"limit":-1})).await;
assert_eq!(entries[0]["line"], "legacy history");
assert!(entries.as_array().unwrap().len() > 1);
assert_eq!(
call(&h, "core.log", json!({"pid":"old-pid"})).await[0]["line"],
"legacy history"
);
h.stop().await;
assert_eq!(std::fs::read(&old).unwrap(), before);
assert_eq!(
std::fs::read_to_string(history).unwrap(),
"keep console history"
);
}
/// PROTOCOL 7: the journal retains the full trap, not just its short status reason.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn full_trap_diagnostic_is_journaled() {
let w = TestWorld::new();
let h = w.start().await;
let error = tokio::time::timeout(
std::time::Duration::from_secs(20),
h.call("probe.trap", json!({})),
)
.await
.unwrap()
.unwrap_err();
assert_eq!(error["code"], -32002);
assert!(
error["data"]["status"]
.as_str()
.unwrap()
.starts_with("trap: wasm trap:")
);
let entries = call(
&h,
"core.log",
json!({"plugin":"probe","grep":"wasm backtrace"}),
)
.await;
assert!(
entries.as_array().unwrap().iter().any(|e| {
let text = e["line"].as_str().unwrap();
text.contains("Caused by:") && text.contains("wasm trap:")
}),
"{entries}"
);
h.stop().await;
}
+1
View File
@@ -7,3 +7,4 @@ mod common;
mod console; mod console;
mod core; mod core;
mod sdk; mod sdk;
mod watches;
+50
View File
@@ -178,6 +178,56 @@ async fn stop_runs_hook() {
assert_eq!(std::fs::read(world.0.join("stopped")).unwrap(), b"yes"); assert_eq!(std::fs::read(world.0.join("stopped")).unwrap(), b"yes");
host.stop().await; host.stop().await;
} }
/// PROTOCOL 3.7: an idle SDK listener opts into drain and exits before EOF/grace.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn superseded_idle_listener_drains_before_grace() {
let world = TestWorld::new();
let host = world.start().await;
let (notice, mut notices) = tokio::sync::mpsc::unbounded_channel();
host.register_native(
"other",
std::sync::Arc::new(move |_, method, _| {
let notice = notice.clone();
Box::pin(async move {
assert_eq!(method, "superseded");
notice.send(std::time::Instant::now()).unwrap();
Ok(json!({}))
})
}),
);
let pid = call(&host, "probe.drain_service", json!({})).await;
// Replace with a different real component; force only skips unrelated test consumers.
std::fs::copy(
fixtures().join("notes.wasm"),
world.0.join("state/next.wasm"),
)
.unwrap();
call(
&host,
"core.activate",
json!({"plugin":"probe","wasm":"/state/next.wasm","force":true}),
)
.await;
// Activation preparation/discovery is outside upgrade grace. Time from the
// hook's outgoing call, not from the start of candidate compilation.
let before = tokio::time::timeout(Duration::from_secs(60), notices.recv())
.await
.unwrap()
.unwrap();
// Waiting on its lifetime lock is an exit event, not process-liveness polling.
call(&host, "core.lock", json!({"path":"/drain-owner"})).await;
assert!(before.elapsed() < Duration::from_secs(5));
assert!(!host.processes().contains_key(pid.as_str().unwrap()));
assert!(
!world
.0
.join(format!("stopped-{}", pid.as_str().unwrap()))
.exists(),
"drain must not run on_stop (discovery processes may run their own hooks)"
);
host.stop().await;
}
/// SDK framing carries payloads above 64 KiB in both directions. /// SDK framing carries payloads above 64 KiB in both directions.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn large_messages() { async fn large_messages() {
+174
View File
@@ -0,0 +1,174 @@
//! Real SDK watch callbacks and raw WASI filesystem mutations share the host namespace.
use super::common::{CACHE, probe, sdk_probe};
use sausage::{Host, world::World};
use serde_json::{Value, json};
use std::time::Duration;
// Same call bound as core.rs; no sleeps or polling.
async fn call(h: &Host, method: &str, p: Value) -> Value {
tokio::time::timeout(Duration::from_secs(20), h.call(method, p))
.await
.unwrap()
.unwrap()
}
async fn event(h: &Host, rx: &mut tokio::sync::broadcast::Receiver<Value>, name: &str) -> Value {
let result = tokio::time::timeout(Duration::from_secs(20), async {
loop {
let e = rx.recv().await.unwrap();
if e["params"]["name"] == name {
return e["params"]["params"].clone();
}
}
})
.await;
match result {
Ok(v) => v,
Err(e) => {
eprintln!(
"JOURNAL {}",
h.call("core.log", json!({"limit":100})).await.unwrap()
);
panic!("{e}");
}
}
}
async fn start() -> (tempfile::TempDir, Host) {
let dir = tempfile::tempdir().unwrap();
let world = World::create(&dir.path().join("world")).unwrap();
for (name, wasm) in [("probe", probe()), ("sdk", sdk_probe())] {
std::fs::create_dir_all(world.plugin_dir(name)).unwrap();
std::fs::copy(wasm, world.plugin_dir(name).join("plugin.wasm")).unwrap();
}
let host = Host::start_with_cache(&world.root, std::path::Path::new(CACHE))
.await
.unwrap();
(dir, host)
}
fn watch(h: &Host, path: &str) -> tokio::task::JoinHandle<sausage::RpcResult> {
let (h, path) = (h.clone(), path.to_owned());
tokio::spawn(async move { h.call("sdk.watch", json!({"path":path})).await })
}
async fn mutate(h: &Host, op: &str, path: &str) {
call(h, "probe.mutate", json!({"op":op,"path":path})).await;
}
/// PROTOCOL 4: pre-arm state is in the first scan; each post-scan mutation wakes
/// the SDK without completing the watch. Covers descriptor/stream/truncate paths.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn armed_scan_and_later_changes() {
let (_dir, h) = start().await;
mutate(&h, "mkdir", "/state/mailbox").await;
call(
&h,
"probe.write",
json!({"path":"/state/mailbox/before", "text":"before"}),
)
.await;
let mut rx = h.subscribe();
let task = watch(&h, "/state/mailbox/../mailbox/");
let armed = event(&h, &mut rx, "armed").await;
assert_eq!(armed["entries"], json!(["before"]));
assert!(!task.is_finished());
for (op, expected) in [
("descriptor", "descriptor"),
("stream", "streamptor"),
("append", "streamptorappend"),
("size", ""),
("descriptor", "descriptor"),
("truncate", ""),
] {
mutate(&h, op, "/state/mailbox/before").await;
let mut scan = event(&h, &mut rx, "changed").await;
while scan["text"] != expected {
scan = event(&h, &mut rx, "changed").await;
}
}
mutate(&h, "burst", "/state/mailbox").await;
let mut scanned = event(&h, &mut rx, "changed").await;
while !scanned["entries"].as_array().unwrap().contains(&json!("4")) {
scanned = event(&h, &mut rx, "changed").await;
}
while rx.try_recv().is_ok() {}
mutate(&h, "delete", "/state/mailbox/4").await;
let mut scanned = event(&h, &mut rx, "changed").await;
while scanned["entries"].as_array().unwrap().contains(&json!("4")) {
scanned = event(&h, &mut rx, "changed").await;
}
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
assert!(task.await.unwrap().is_err());
h.stop().await;
}
/// PROTOCOL 4: atomic publication, rename source/destination, mounted paths,
/// absent paths, and caller-death cleanup. No external host writes are required.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn rename_delete_and_mounts() {
let (_dir, h) = start().await;
for base in ["/state/inbox", "/run/inbox"] {
let mut rx = h.subscribe();
let task = watch(&h, base);
let armed = event(&h, &mut rx, "armed").await;
assert_eq!(armed["entries"], json!([]));
let staging = format!("{base}-staging");
mutate(&h, "mkdir", &staging).await;
call(
&h,
"probe.mutate",
json!({"op":"rename","path":staging,"to":base}),
)
.await;
event(&h, &mut rx, "changed").await;
mutate(&h, "atomic", &format!("{base}/published")).await;
let mut scan = event(&h, &mut rx, "changed").await;
while !scan["entries"]
.as_array()
.unwrap()
.contains(&json!("published"))
{
scan = event(&h, &mut rx, "changed").await;
}
assert_eq!(
call(
&h,
"probe.read",
json!({"path":format!("{base}/published")})
)
.await,
"complete"
);
while rx.try_recv().is_ok() {}
call(
&h,
"probe.mutate",
json!({"op":"rename","path":base,"to":format!("{base}-moved")}),
)
.await;
let mut scan = event(&h, &mut rx, "changed").await;
while scan["entries"] != json!([]) {
scan = event(&h, &mut rx, "changed").await;
}
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
assert!(task.await.unwrap().is_err());
}
// /self watches bind to this caller, and mutations are visible via /proc/<pid>.
let mut rx = h.subscribe();
let task = watch(&h, "/self");
let armed = event(&h, &mut rx, "armed").await;
assert!(
armed["entries"]
.as_array()
.unwrap()
.contains(&json!("owned"))
);
event(&h, &mut rx, "changed").await;
assert_eq!(
call(
&h,
"probe.read",
json!({"path":format!("/proc/{}/owned", armed["pid"].as_str().unwrap())})
)
.await,
"visible via proc"
);
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
assert!(task.await.unwrap().is_err());
h.stop().await;
}
+41 -1
View File
@@ -84,12 +84,52 @@ fn main() {
.unwrap(); .unwrap();
json!({"retired":version}) json!({"retired":version})
} }
"work" => {
// Keep an actual request unanswered through grace; notices do not drain it.
send(
json!({"jsonrpc":"2.0","method":"core.notify","params":{"name":"ready","params":{}}}),
);
for _ in lines {}
return;
}
"serve" | "hold" => { "serve" | "hold" => {
// One genuine process failure, followed by a healthy service, without
// timers or liveness polling in either the fixture or supervisor.
let crash = request["params"]["crash_once"].as_str();
let first = crash.is_some()
&& std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(format!("/state/{plugin}/crashed-once"))
.is_ok();
if first && crash == Some("before") {
std::process::exit(1);
}
send(json!({"jsonrpc":"2.0","id":request["id"],"result":{"version":version}})); send(json!({"jsonrpc":"2.0","id":request["id"],"result":{"version":version}}));
if first && crash == Some("after") {
std::process::exit(1);
}
if request["method"] == "serve" { if request["method"] == "serve" {
std::fs::write(format!("/state/{plugin}/service"), version).unwrap(); std::fs::write(format!("/state/{plugin}/service"), version).unwrap();
} }
for _ in lines {} for line in lines {
let Ok(message) = serde_json::from_str::<Value>(&line.unwrap()) else {
continue;
};
if message["method"] == "rpc.superseded" {
std::fs::write(
format!(
"/state/{plugin}/superseded-{}",
std::env::var("SAUSAGE_PID").unwrap()
),
message.to_string(),
)
.unwrap();
if request["params"]["exit_on_superseded"] == true {
return;
}
}
}
std::fs::write(format!("/state/{plugin}/ended-{version}"), "ended").unwrap(); std::fs::write(format!("/state/{plugin}/ended-{version}"), "ended").unwrap();
return; return;
} }
+89
View File
@@ -83,6 +83,9 @@ fn unknown_id() {
} }
fn wait_eof(lines: &mut Lines) { fn wait_eof(lines: &mut Lines) {
std::fs::write("/state/waiting", "yes").unwrap(); std::fs::write("/state/waiting", "yes").unwrap();
send(json!({"jsonrpc":"2.0","method":"core.notify","params":{
"name":"waiting-eof", "params":{"pid":std::env::var("SAUSAGE_PID").unwrap()}
}}));
for _ in lines {} for _ in lines {}
std::fs::write("/state/stopped", "yes").unwrap(); std::fs::write("/state/stopped", "yes").unwrap();
} }
@@ -376,6 +379,7 @@ fn dispatch(request: &Value, lines: &mut Lines) -> Option<Value> {
let params = &request["params"]; let params = &request["params"];
let result = match request["method"].as_str().unwrap() { let result = match request["method"].as_str().unwrap() {
"batch" => batch(lines), "batch" => batch(lines),
"trap" => std::arch::wasm32::unreachable(),
"nested_environment" => nested_environment(lines), "nested_environment" => nested_environment(lines),
"forget" => forget(lines), "forget" => forget(lines),
"forgotten" => { "forgotten" => {
@@ -447,13 +451,33 @@ fn dispatch(request: &Value, lines: &mut Lines) -> Option<Value> {
burst(request); burst(request);
return None; return None;
} }
"restart" => {
assert_eq!(
lock(&json!({"path":"/state/restart-stop"}), lines),
json!({"held":true})
);
assert_eq!(
outgoing_call(&json!({"method":"core.restart", "params":{}}), lines),
json!({})
);
std::fs::write("/state/restart-answer", "received").unwrap();
serve(request, lines);
std::fs::write("/state/restart-stop", "persisted after EOF").unwrap();
eprintln!("restart stop hook finished");
return None;
}
"serve" => { "serve" => {
serve(request, lines); serve(request, lines);
return None; return None;
} }
"network" => network(params), "network" => network(params),
"mutate" => mutate(params),
"read" => read_file(params), "read" => read_file(params),
"write" => write_file(params), "write" => write_file(params),
"raw_call" => {
send(params.clone());
receive(lines)
}
"call" => outgoing_call(params, lines), "call" => outgoing_call(params, lines),
"lock" => lock(params, lines), "lock" => lock(params, lines),
_ => { _ => {
@@ -498,3 +522,68 @@ fn main() {
})); }));
} }
} }
/// Separate mutation calls let the host await a scan before each subsequent change.
fn mutate(p: &Value) -> Value {
use wasip2::filesystem::{
preopens,
types::{DescriptorFlags as D, OpenFlags as O, PathFlags as P},
};
let path = p["path"].as_str().unwrap();
match p["op"].as_str().unwrap() {
"mkdir" => std::fs::create_dir_all(path).unwrap(),
"rename" => std::fs::rename(path, p["to"].as_str().unwrap()).unwrap(),
"delete" => std::fs::remove_file(path).unwrap(),
"rmdir" => std::fs::remove_dir(path).unwrap(),
"atomic" => {
let temp = format!("{path}.tmp");
std::fs::write(&temp, "complete").unwrap();
std::fs::rename(temp, path).unwrap();
}
"burst" => {
for i in 0..5 {
std::fs::write(format!("{path}/{i}"), "complete").unwrap();
}
}
op => {
let root = preopens::get_directories().remove(0).0;
let file = root
.open_at(
P::empty(),
path.trim_start_matches('/'),
O::empty(),
D::READ | D::WRITE,
)
.unwrap();
match op {
"descriptor" => {
file.write(b"descriptor", 0).unwrap();
}
"stream" => {
file.write_via_stream(0)
.unwrap()
.blocking_write_and_flush(b"stream")
.unwrap();
}
"append" => {
file.append_via_stream()
.unwrap()
.blocking_write_and_flush(b"append")
.unwrap();
}
"size" => file.set_size(0).unwrap(),
"truncate" => {
root.open_at(
P::empty(),
path.trim_start_matches('/'),
O::TRUNCATE,
D::WRITE,
)
.unwrap();
}
_ => panic!("unknown mutation"),
}
}
}
json!({})
}
+2 -2
View File
@@ -80,7 +80,7 @@
# `nix run .#loop -- <world>` runs the newest `sausage run <world>` # `nix run .#loop -- <world>` runs the newest `sausage run <world>`
# (without a folder: bare `sausage`, i.e. ./world, created if needed) # (without a folder: bare `sausage`, i.e. ./world, created if needed)
# from the public repo, again and again: after a crash or a call of # from the public repo, again and again: after a crash or a call of
# `core.exit` it refreshes the flake and starts the host again, so # `core.restart` (exit code 75) it refreshes the flake and starts the host again, so
# the world updates itself. `quit` (exit code 0) ends the loop. # the world updates itself. `quit` (exit code 0) ends the loop.
# SAUSAGE_FLAKE overrides where it fetches from (any flake URL, # SAUSAGE_FLAKE overrides where it fetches from (any flake URL,
# e.g. a fork or a local checkout). # e.g. a fork or a local checkout).
@@ -90,7 +90,7 @@
# No folder: bare `sausage` (creates ./world first if needed). # No folder: bare `sausage` (creates ./world first if needed).
if [ $# -gt 0 ]; then set -- run "$@"; fi if [ $# -gt 0 ]; then set -- run "$@"; fi
while true; do while true; do
nix run --refresh "$url#default" -- "$@" SAUSAGE_LOOP=1 nix run --refresh "$url#default" -- "$@"
code=$? code=$?
if [ "$code" -eq 0 ]; then exit 0; fi if [ "$code" -eq 0 ]; then exit 0; fi
echo "sausage-loop: sausage exited with $code, starting the newest version in 2 s …" >&2 echo "sausage-loop: sausage exited with $code, starting the newest version in 2 s …" >&2
+27 -7
View File
@@ -153,6 +153,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
let mut stop = quote!(); let mut stop = quote!();
let mut has_stop = false; let mut has_stop = false;
let mut has_retired = false; let mut has_retired = false;
let mut has_superseded = false;
for member in &mut item.items { for member in &mut item.items {
let ImplItem::Fn(f) = member else { continue }; let ImplItem::Fn(f) = member else { continue };
let retired = f.attrs.iter().any(|a| { let retired = f.attrs.iter().any(|a| {
@@ -161,6 +162,12 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
.last() .last()
.is_some_and(|s| s.ident == "on_retired") .is_some_and(|s| s.ident == "on_retired")
}); });
let superseded = f.attrs.iter().any(|a| {
a.path()
.segments
.last()
.is_some_and(|s| s.ident == "on_superseded")
});
let stopping = f.attrs.iter().any(|a| { let stopping = f.attrs.iter().any(|a| {
a.path() a.path()
.segments .segments
@@ -179,11 +186,16 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
"only one on_retired method is allowed", "only one on_retired method is allowed",
)); ));
} }
if superseded && std::mem::replace(&mut has_superseded, true) {
return Err(syn::Error::new(
f.span(),
"only one on_superseded method is allowed",
));
}
f.attrs.retain(|a| { f.attrs.retain(|a| {
!a.path() !a.path().segments.last().is_some_and(|s| {
.segments s.ident == "on_retired" || s.ident == "on_stop" || s.ident == "on_superseded"
.last() })
.is_some_and(|s| s.ident == "on_retired" || s.ident == "on_stop")
}); });
documented(&f.attrs, f.span())?; documented(&f.attrs, f.span())?;
if f.sig.inputs.is_empty() if f.sig.inputs.is_empty()
@@ -224,7 +236,9 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
stop = quote!(fn stop(&self)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=()> + '_>> {Box::pin(async move {let _=self.#name() #await_;})}); stop = quote!(fn stop(&self)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=()> + '_>> {Box::pin(async move {let _=self.#name() #await_;})});
continue; continue;
} }
let wire = if retired { let wire = if superseded {
"rpc.superseded".to_string()
} else if retired {
"rpc.retired".to_string() "rpc.retired".to_string()
} else { } else {
name.unraw().to_string() name.unraw().to_string()
@@ -253,7 +267,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
}; };
let doc = docs(&f.attrs); let doc = docs(&f.attrs);
arms.push(quote!(#wire => { fn params_type<T: ::sausage_plugin::Params>() {} params_type::<#p>(); let result=self.#name(#(#args),*) #await_ ?; Ok(::sausage_plugin::serde_json::to_value(result)?) })); arms.push(quote!(#wire => { fn params_type<T: ::sausage_plugin::Params>() {} params_type::<#p>(); let result=self.#name(#(#args),*) #await_ ?; Ok(::sausage_plugin::serde_json::to_value(result)?) }));
if !retired { if !retired && !superseded {
entries.push(quote!(::sausage_plugin::method::<#p,#output>(&mut doc,#wire,#doc);)); entries.push(quote!(::sausage_plugin::method::<#p,#output>(&mut doc,#wire,#doc);));
} }
} }
@@ -270,7 +284,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
fn methods_document()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace ::sausage_plugin::finish_document(doc)} fn methods_document()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace ::sausage_plugin::finish_document(doc)}
fn describe()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace #(doc.endpoint::<#types>(#names);)* ::sausage_plugin::finish_document(doc)} fn describe()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace #(doc.endpoint::<#types>(#names);)* ::sausage_plugin::finish_document(doc)}
fn discover(&self)->::sausage_plugin::serde_json::Value {<Self as ::sausage_plugin::Interface>::describe()} fn discover(&self)->::sausage_plugin::serde_json::Value {<Self as ::sausage_plugin::Interface>::describe()}
fn dispatch<'a>(&'a self, method:&'a str, params: ::sausage_plugin::serde_json::Value,ctx:&'a ::sausage_plugin::Ctx)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=::sausage_plugin::Result<::sausage_plugin::serde_json::Value>>+'a>> {Box::pin(async move {if !params.is_object() {return Err(::sausage_plugin::Error::new(-32602,"params must be an object"));} match method {#(#arms,)* _=>Err(::sausage_plugin::Error::new(-32601,"method not found"))}})} fn dispatch<'a>(&'a self, method:&'a str, params: ::sausage_plugin::serde_json::Value,ctx:&'a ::sausage_plugin::Ctx)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=::sausage_plugin::Result<::sausage_plugin::serde_json::Value>>+'a>> {Box::pin(async move {if !params.is_object() {return Err(::sausage_plugin::Error::params_not_object(&params));} match method {#(#arms,)* _=>Err(::sausage_plugin::Error::new(-32601,"method not found"))}})}
#stop #stop
}}) }})
})(); })();
@@ -328,6 +342,12 @@ pub fn import(input: TokenStream) -> TokenStream {
pub fn on_retired(_: TokenStream, input: TokenStream) -> TokenStream { pub fn on_retired(_: TokenStream, input: TokenStream) -> TokenStream {
input input
} }
/// Mark the method that handles `rpc.superseded` at the start of upgrade grace.
/// Calls still work here. Use `sausage_plugin::drain` to finish active work and exit.
#[proc_macro_attribute]
pub fn on_superseded(_: TokenStream, input: TokenStream) -> TokenStream {
input
}
/// Mark the hook that runs at stdin EOF without a live call transport. /// Mark the hook that runs at stdin EOF without a live call transport.
#[proc_macro_attribute] #[proc_macro_attribute]
pub fn on_stop(_: TokenStream, input: TokenStream) -> TokenStream { pub fn on_stop(_: TokenStream, input: TokenStream) -> TokenStream {
+184 -2
View File
@@ -1,5 +1,19 @@
//! Outgoing core calls and notification callbacks. //! Outgoing core calls and notification callbacks.
use super::*; use super::*;
/// Removes a dropped call's local routing state, without cancelling host work.
struct PendingCall(u64);
impl Drop for PendingCall {
fn drop(&mut self) {
let pending = STATE.with(|s| s.borrow_mut().pending.remove(&self.0));
if let Some(mut pending) = pending {
// A lock may still be granted: release it when its late response arrives.
if let Some(lease) = pending.unlock.take() {
STATE.with(|s| s.borrow_mut().abandoned_locks.insert(self.0, lease));
}
// Drop user-owned callback captures outside the STATE borrow.
}
}
}
/// An outgoing request builder. /// An outgoing request builder.
#[must_use = "await the call to send it"] #[must_use = "await the call to send it"]
pub struct Call { pub struct Call {
@@ -26,6 +40,21 @@ impl Call {
self self
} }
/// Receive progress notifications before the result arrives. /// Receive progress notifications before the result arrives.
/// Dropping the call future removes its local pending entry and callback;
/// later notifications and responses are ignored. It does not cancel host work.
///
/// For `core.watch`, poll the call future alongside work in the handler, then
/// drop it when the handler returns. Do not put it in `background()` just to
/// keep the watch alive: background tasks keep the process alive, while the
/// watch only ends when the process ends. Dropping the future does not cancel
/// the host watch, but it lets an otherwise idle process exit normally. A
/// long-lived service should keep one watch per purpose for its lifetime,
/// rather than repeatedly opening and dropping watches on the host.
///
/// Scan only after `armed`, and rescan on `changed`. Events may coalesce, but
/// are not rate-limited: watch the narrowest useful path to avoid rescanning
/// for unrelated writes (for example, watch a mailbox, not all session data).
/// Do not use endpoint connect retries to wait for file publication.
pub fn on_notify(mut self, f: impl FnMut(&str, Value) + 'static) -> Self { pub fn on_notify(mut self, f: impl FnMut(&str, Value) + 'static) -> Self {
self.notify = Some(Box::new(f)); self.notify = Some(Box::new(f));
self self
@@ -37,7 +66,7 @@ impl IntoFuture for Call {
fn into_future(self) -> Self::IntoFuture { fn into_future(self) -> Self::IntoFuture {
Box::pin(async move { Box::pin(async move {
if !self.params.is_object() { if !self.params.is_object() {
return Err(Error::new(-32602, "params must be an object")); return Err(Error::params_not_object(&self.params));
} }
let (tx, rx) = oneshot::channel(); let (tx, rx) = oneshot::channel();
let id = STATE.with(|s| { let id = STATE.with(|s| {
@@ -60,6 +89,7 @@ impl IntoFuture for Call {
); );
Ok(id) Ok(id)
})?; })?;
let _pending = PendingCall(id);
let (method, params) = if self.detach { let (method, params) = if self.detach {
( (
"core.call".to_string(), "core.call".to_string(),
@@ -69,7 +99,8 @@ impl IntoFuture for Call {
(self.method, self.params) (self.method, self.params)
}; };
if let Err(e) = send(json!({"jsonrpc":"2.0","id":id,"method":method,"params":params})) { if let Err(e) = send(json!({"jsonrpc":"2.0","id":id,"method":method,"params":params})) {
STATE.with(|s| s.borrow_mut().pending.remove(&id)); let pending = STATE.with(|s| s.borrow_mut().pending.remove(&id));
drop(pending);
return Err(e); return Err(e);
} }
rx.await rx.await
@@ -77,3 +108,154 @@ impl IntoFuture for Call {
}) })
} }
} }
#[cfg(test)]
mod tests {
use super::{STATE, State, block_on, call, clear_pending, lock, receive};
use crate::Result;
use futures::{FutureExt, future::LocalBoxFuture};
use serde_json::{Value, json};
use std::{cell::RefCell, future::IntoFuture, rc::Rc, task::Context};
fn activate() {
STATE.with(|s| {
*s.borrow_mut() = State {
active: true,
..State::default()
};
});
}
fn start(
mut call: LocalBoxFuture<'static, Result<Value>>,
) -> (u64, LocalBoxFuture<'static, Result<Value>>) {
assert!(
call.as_mut()
.poll(&mut Context::from_waker(std::task::Waker::noop()))
.is_pending()
);
(STATE.with(|s| s.borrow().next), call)
}
fn notification(id: u64, name: &str) {
receive(json!({"method":"core.notify","params":{"call":id,"name":name,"params":{}}}));
}
fn response(id: u64, result: Value) {
receive(json!({"id":id,"result":result}));
}
#[test]
fn dropped_call_removes_pending_and_callback() {
activate();
let events = Rc::new(RefCell::new(Vec::new()));
let captured = events.clone();
let (id, future) = start(
call("core.watch", json!({"path":"/mailbox"}))
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
.into_future(),
);
notification(id, "armed");
assert_eq!(&*events.borrow(), &["armed"]);
assert_eq!(Rc::strong_count(&events), 2);
drop(future);
STATE.with(|s| {
assert!(s.borrow().pending.is_empty());
assert!(s.borrow().abandoned_locks.is_empty());
});
assert_eq!(Rc::strong_count(&events), 1);
notification(id, "changed");
response(id, Value::Null);
receive(json!({"id":id,"error":{"code":-32000,"message":"late error"}}));
assert_eq!(&*events.borrow(), &["armed"]);
}
#[test]
fn kept_watch_delivers_until_result() {
activate();
let events = Rc::new(RefCell::new(Vec::new()));
let captured = events.clone();
let (id, future) = start(
call("core.watch", json!({"path":"/mailbox"}))
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
.into_future(),
);
notification(id, "armed");
notification(id, "changed");
notification(id, "changed");
response(id, json!({"done":true}));
assert_eq!(block_on(future).unwrap(), json!({"done":true}));
notification(id, "late");
assert_eq!(&*events.borrow(), &["armed", "changed", "changed"]);
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
assert_eq!(Rc::strong_count(&events), 1);
}
#[test]
fn callback_can_drop_its_own_call_without_being_restored() {
activate();
let holder = Rc::new(RefCell::new(None));
let captured = holder.clone();
let (id, future) = start(
call("core.watch", json!({"path":"/mailbox"}))
.on_notify(move |_, _| {
// Panics on a second invocation; the first must remove the entry.
drop(captured.borrow_mut().take().expect("live call"));
})
.into_future(),
);
*holder.borrow_mut() = Some(future);
notification(id, "armed");
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
notification(id, "changed");
response(id, Value::Null);
assert_eq!(Rc::strong_count(&holder), 1);
}
#[test]
fn eof_cleanup_can_drop_callback_owned_futures() {
activate();
let holder = Rc::new(RefCell::new(None));
let weak = Rc::downgrade(&holder);
let captured = holder.clone();
let (_, future) = start(
call("core.watch", json!({"path":"/mailbox"}))
.on_notify(move |_, _| {
let _ = &captured;
})
.into_future(),
);
*holder.borrow_mut() = Some(future);
drop(holder);
STATE.with(|s| s.borrow_mut().stopping = true);
clear_pending();
assert!(weak.upgrade().is_none());
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
}
#[test]
fn unpolled_call_never_registers() {
activate();
drop(call("core.watch", json!({"path":"/mailbox"})).into_future());
STATE.with(|s| {
assert!(s.borrow().pending.is_empty());
assert_eq!(s.borrow().next, 0);
});
}
#[test]
fn dropped_lock_keeps_only_late_grant_cleanup() {
activate();
let (id, future) = start(
lock("/late")
.into_future()
.map(|r| r.map(|_| Value::Null))
.boxed_local(),
);
drop(future);
STATE.with(|s| {
assert!(s.borrow().pending.is_empty());
assert_eq!(s.borrow().abandoned_locks.len(), 1);
});
notification(id, "ignored");
response(id, json!({"held":true}));
STATE.with(|s| assert!(s.borrow().abandoned_locks.is_empty()));
}
}
+7 -3
View File
@@ -65,7 +65,7 @@ impl<T: serde::de::DeserializeOwned + 'static> IntoFuture for EndpointCall<T> {
} }
} }
/// Publish a declared interface and keep its listener alive until stdin EOF. /// Publish a declared interface and keep its listener alive until stdin EOF or [`drain`].
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Result<()> { pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Result<()> {
use wstd::iter::AsyncIterator; use wstd::iter::AsyncIterator;
@@ -80,7 +80,7 @@ pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Resul
while let Some(Ok(stream)) = incoming.next().await { while let Some(Ok(stream)) = incoming.next().await {
let value = value.clone(); let value = value.clone();
let name = name.clone(); let name = name.clone();
crate::transport::endpoint_task(async move { background(async move {
let _ = serve_connection(stream, value, name).await; let _ = serve_connection(stream, value, name).await;
}); });
} }
@@ -211,7 +211,11 @@ async fn serve_connection<T: Interface>(
use futures::{FutureExt, StreamExt}; use futures::{FutureExt, StreamExt};
let (mut reader, mut writer) = stream.split(); let (mut reader, mut writer) = stream.split();
let mut lines = crate::lines::Lines::default(); let mut lines = crate::lines::Lines::default();
while let Some(request) = lines.rpc(&mut reader).await? { while let Some(request) = crate::transport::until_drained(lines.rpc(&mut reader))
.await
.transpose()?
.flatten()
{
let method = request["method"].as_str().unwrap_or_default(); let method = request["method"].as_str().unwrap_or_default();
let (tx, mut rx) = futures::channel::mpsc::unbounded(); let (tx, mut rx) = futures::channel::mpsc::unbounded();
let ctx = Ctx { let ctx = Ctx {
+19
View File
@@ -15,6 +15,25 @@ pub struct Error {
pub data: Option<Value>, pub data: Option<Value>,
} }
impl Error { impl Error {
/// Build the object-params diagnostic used by calls and generated dispatch.
#[doc(hidden)]
pub fn params_not_object(params: &Value) -> Self {
let kind = match params {
Value::String(_) => "a string",
Value::Array(_) => "an array",
Value::Number(_) => "a number",
Value::Null => "null",
Value::Bool(_) => "a boolean",
Value::Object(_) => "an object",
};
Self::new(
-32602,
format!(
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
),
)
}
/// Construct an error without structured details. /// Construct an error without structured details.
pub fn new(code: i64, message: impl Into<String>) -> Self { pub fn new(code: i64, message: impl Into<String>) -> Self {
Self { Self {
+24 -3
View File
@@ -356,13 +356,20 @@
//! //!
//! The interface check covers calls, not state. Old and new processes share files in `/state`. //! The interface check covers calls, not state. Old and new processes share files in `/state`.
//! `core.revert` restores code, not state. Preserve compatibility or migrate the state. //! `core.revert` restores code, not state. Preserve compatibility or migrate the state.
//! `#[rpc(grace_seconds = N)]` declares the grace period for old processes. The default is 60 seconds. //! Core gives every old process 5 seconds of upgrade grace, then closes stdin and allows
//! a separate 5-second stop budget before killing it. Discovery-only processes are excluded.
//! `#[rpc(grace_seconds = N)]` still emits legacy `x-sausage-grace` metadata, but core ignores it.
//! New calls use the new version immediately. Old processes can still use old imports during their grace period. //! New calls use the new version immediately. Old processes can still use old imports during their grace period.
//! [`macro@on_superseded`] handles the notice at grace start, with normal calls still available.
//! It is opt-in: without a hook the process continues as before until core stops it.
//! Call [`drain`] in this hook to close SDK endpoint listeners and idle connections,
//! let active handlers and background tasks finish, and exit without waiting for grace.
//! Custom listeners and long-lived background tasks must be ended by the plugin itself.
//! [`macro@on_retired`] handles `rpc.retired` after the last process of an old version ends. //! [`macro@on_retired`] handles `rpc.retired` after the last process of an old version ends.
//! This hook is best effort. If the host stops first, the hook does not run. //! This hook is best effort. If the host stops first, the hook does not run.
//! New processes must handle unmigrated state. Check a migration marker and migrate if it is absent. //! New processes must handle unmigrated state. Check a migration marker and migrate if it is absent.
//! See section 3.7 of `/proc/core/PROTOCOL.md`. //! See section 3.7 of `/proc/core/PROTOCOL.md`.
//! Declare at most one stop hook and one retirement hook per interface. //! Declare at most one hook of each kind per interface.
//! //!
//! ```no_run //! ```no_run
//! use sausage_plugin::{rpc, schema, state_dir, write_atomic, Result}; //! use sausage_plugin::{rpc, schema, state_dir, write_atomic, Result};
@@ -374,8 +381,21 @@
//! /// The SHA-256 of the previous WASM file. //! /// The SHA-256 of the previous WASM file.
//! previous: String, //! previous: String,
//! } //! }
//! /// The version now receiving new calls.
//! #[schema]
//! struct SupersededParams {
//! /// The SHA-256 of the newly active WASM file.
//! current: String,
//! }
//! #[rpc] //! #[rpc]
//! impl Migrator { //! impl Migrator {
//! /// Finish current work and exit early during upgrade grace.
//! #[sausage_plugin::on_superseded]
//! fn superseded(&self, p: SupersededParams) -> Result<()> {
//! eprintln!("replaced by {}", p.current);
//! sausage_plugin::drain();
//! Ok(())
//! }
//! /// Record a retired version. Normal methods must also handle missing markers. //! /// Record a retired version. Normal methods must also handle missing markers.
//! #[sausage_plugin::on_retired] //! #[sausage_plugin::on_retired]
//! fn retired(&self, p: RetiredParams) -> Result<()> { //! fn retired(&self, p: RetiredParams) -> Result<()> {
@@ -392,6 +412,7 @@ mod error;
mod helpers; mod helpers;
mod imports; mod imports;
mod lines; mod lines;
mod sqlite;
mod transport; mod transport;
pub use discovery::*; pub use discovery::*;
pub use endpoints::*; pub use endpoints::*;
@@ -400,6 +421,6 @@ pub use helpers::*;
pub use imports::*; pub use imports::*;
#[doc(hidden)] #[doc(hidden)]
pub use inventory; pub use inventory;
pub use sausage_plugin_macros::{import, on_retired, on_stop, rpc, schema, test}; pub use sausage_plugin_macros::{import, on_retired, on_stop, on_superseded, rpc, schema, test};
pub use transport::*; pub use transport::*;
pub use {schemars, serde, serde_json}; pub use {schemars, serde, serde_json};
+43
View File
@@ -0,0 +1,43 @@
//! Shared SQLite open policy, expanded against the plugin's own rusqlite version.
/// Open SQLite under an already-held Sausage host lock. The caller must depend
/// on `rusqlite` (bundled) and hold the SAME host lock for EVERY connection's
/// entire lifetime, including reads and old/new plugin versions. This macro does
/// not acquire the lock. Do not use it for the host journal or databases accessed
/// outside Sausage. Native builds retain OS locks; WASI selects `unix-none`, so
/// crash-leftover dotfile lock directories cannot block recovery.
///
/// Temporary tables/sorts stay in memory before any schema access. Writable opens
/// use DELETE journals; SQLite rolls back hot journals itself. Never delete a
/// leftover journal. Read-only opens cannot recover a hot journal: recover with
/// a writable open first.
///
/// The flags form preserves the caller's open flags. A macro avoids coupling the
/// SDK to a particular rusqlite/libsqlite3-sys version (the host and plugins use
/// different versions, which cannot coexist as dependencies due to `links`).
///
/// ```ignore
/// let _guard = sausage_plugin::lock("/state/example/db.lock").await?;
/// let c = sausage_plugin::sqlite_open!("/state/example/db.sqlite")?;
/// // Query here; c must drop before _guard.
/// ```
#[macro_export]
macro_rules! sqlite_open {
($path:expr) => {
$crate::sqlite_open!($path, ::rusqlite::OpenFlags::default())
};
($path:expr, $flags:expr) => {{
(|| -> ::rusqlite::Result<::rusqlite::Connection> {
let flags = $flags;
#[cfg(target_os = "wasi")]
let c = ::rusqlite::Connection::open_with_flags_and_vfs($path, flags, "unix-none")?;
#[cfg(not(target_os = "wasi"))]
let c = ::rusqlite::Connection::open_with_flags($path, flags)?;
c.pragma_update(None, "temp_store", "MEMORY")?;
if !flags.contains(::rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY) {
c.pragma_update(None, "journal_mode", "DELETE")?;
}
Ok(c)
})()
}};
}
+65 -10
View File
@@ -96,8 +96,11 @@ struct State {
active: bool, active: bool,
endpoints: HashMap<std::any::TypeId, String>, endpoints: HashMap<std::any::TypeId, String>,
stopping: bool, stopping: bool,
draining: bool,
next: u64, next: u64,
pending: HashMap<u64, Pending>, pending: HashMap<u64, Pending>,
// Only leases, never callbacks or response senders, survive dropped lock acquisitions.
abandoned_locks: HashMap<u64, std::rc::Rc<Lease>>,
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
tasks: usize, tasks: usize,
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
@@ -108,6 +111,18 @@ struct State {
listeners: HashMap<u64, futures::future::AbortHandle>, listeners: HashMap<u64, futures::future::AbortHandle>,
} }
thread_local! { static STATE: RefCell<State> = RefCell::new(State::default()); } thread_local! { static STATE: RefCell<State> = RefCell::new(State::default()); }
#[cfg(any(target_os = "wasi", test))]
fn clear_pending() {
let pending = STATE.with(|s| {
let mut s = s.borrow_mut();
(
std::mem::take(&mut s.pending),
std::mem::take(&mut s.abandoned_locks),
)
});
// Callback captures may own other call futures whose drop guards use STATE.
drop(pending);
}
pub(crate) fn endpoint_name<T: 'static>() -> Result<String> { pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
STATE STATE
.with(|s| { .with(|s| {
@@ -125,17 +140,48 @@ pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
} }
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
pub(crate) fn endpoint_task(future: impl Future<Output = ()> + 'static) { pub(crate) fn endpoint_task(future: impl Future<Output = ()> + 'static) {
background(async move {
let _ = until_drained(future).await;
});
}
/// Cancel an accept/read wait when draining, but never an active handler.
#[cfg(target_os = "wasi")]
pub(crate) async fn until_drained<F: Future>(future: F) -> Option<F::Output> {
let (handle, registration) = futures::future::AbortHandle::new_pair(); let (handle, registration) = futures::future::AbortHandle::new_pair();
let id = STATE.with(|s| { let id = STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
if s.draining {
return None;
}
s.next += 1; s.next += 1;
let id = s.next; let id = s.next;
s.listeners.insert(id, handle); s.listeners.insert(id, handle);
id Some(id)
}); })?;
background(async move { let result = futures::future::Abortable::new(future, registration)
let _ = futures::future::Abortable::new(future, registration).await; .await
STATE.with(|s| s.borrow_mut().listeners.remove(&id)); .ok();
STATE.with(|s| s.borrow_mut().listeners.remove(&id));
result
}
/// Stop accepting SDK endpoint work, then exit when handlers and background tasks finish.
///
/// Calls and notifications still work. This does not run the EOF stop hook or cancel
/// background tasks: services with their own listeners/tasks must arrange for those
/// to finish too. Repeated calls are harmless. Core's upgrade grace remains the bound.
pub fn drain() {
STATE.with(|s| {
let mut s = s.borrow_mut();
s.draining = true;
#[cfg(target_os = "wasi")]
{
for (_, listener) in s.listeners.drain() {
listener.abort();
}
if let Some(w) = s.wake.take() {
w.wake();
}
}
}); });
} }
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
@@ -192,16 +238,16 @@ pub fn test_run<F: Future>(f: F) -> F::Output {
STATE.with(|s| { STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
s.stopping = true; s.stopping = true;
s.pending.clear();
}); });
clear_pending();
}); });
let result = f.await; let result = f.await;
drop(reader); drop(reader);
STATE.with(|s| { STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
s.active = false; s.active = false;
s.pending.clear();
}); });
clear_pending();
result result
}) })
} }
@@ -274,7 +320,16 @@ pub fn receive(value: Value) {
} }
let _ = p.tx.send(result); let _ = p.tx.send(result);
} else { } else {
eprintln!("note: response with unknown ID {}", value["id"]); let lease = value["id"]
.as_u64()
.and_then(|id| STATE.with(|s| s.borrow_mut().abandoned_locks.remove(&id)));
if let Some(lease) = lease {
lease
.held
.set(value.get("error").is_none() && value["result"]["held"] == true);
} else {
eprintln!("note: response with unknown ID {}", value["id"]);
}
} }
} }
@@ -324,11 +379,11 @@ async fn serve<T: Interface>(value: T) {
STATE.with(|s| { STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
s.stopping = true; s.stopping = true;
s.pending.clear();
for (_, listener) in s.listeners.drain() { for (_, listener) in s.listeners.drain() {
listener.abort(); listener.abort();
} }
}); });
clear_pending();
value.stop().await; value.stop().await;
break; break;
} }
@@ -379,8 +434,8 @@ async fn serve<T: Interface>(value: T) {
STATE.with(|s| { STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
s.active = false; s.active = false;
s.pending.clear();
}); });
clear_pending();
let _ = std::io::stdout().flush(); let _ = std::io::stdout().flush();
std::process::exit(0); std::process::exit(0);
} }
+124
View File
@@ -2,6 +2,12 @@
use sausage_plugin::serde_json::{Value, json}; use sausage_plugin::serde_json::{Value, json};
use sausage_plugin::{Ctx, Result, call, lock, rpc, schema}; use sausage_plugin::{Ctx, Result, call, lock, rpc, schema};
struct Probe; struct Probe;
/// A virtual path to observe.
#[schema]
struct WatchPath {
/// Directory or file, not necessarily present yet.
path: String,
}
/// The values to add. /// The values to add.
#[schema] #[schema]
struct Add { struct Add {
@@ -16,6 +22,12 @@ struct Retired {
/// The previous hash. /// The previous hash.
previous: String, previous: String,
} }
/// The replacement version.
#[schema]
struct Superseded {
/// The current hash.
current: String,
}
/// A large payload. /// A large payload.
#[schema] #[schema]
struct Echo { struct Echo {
@@ -24,6 +36,98 @@ struct Echo {
} }
#[rpc(grace_seconds = 12,endpoints(control=Control))] #[rpc(grace_seconds = 12,endpoints(control=Control))]
impl Probe { impl Probe {
/// Scan only after the watch is armed; keep the call open until process death.
async fn watch(&self, p: WatchPath) -> Result<()> {
let path = p.path.clone();
call("core.watch", json!({"path":p.path}))
.on_notify(move |name, _| {
if name == "armed" && path == "/self" {
std::fs::write("/self/owned", "visible via proc").unwrap();
}
let mut entries: Vec<_> = std::fs::read_dir(&path)
.into_iter()
.flatten()
.filter_map(std::result::Result::ok)
.map(|e| e.file_name().into_string().unwrap())
.collect();
entries.sort();
let text = std::fs::read_to_string(format!("{path}/before")).ok();
Ctx::default()
.notify(
name,
json!({"pid":sausage_plugin::pid(), "entries":entries, "text":text}),
)
.unwrap();
})
.await?;
Ok(())
}
/// Poll the watch in the handler, then return idle without a background task.
async fn watch_once(&self, p: WatchPath) -> Result<()> {
use std::{
cell::RefCell,
rc::Rc,
task::{Poll, Waker},
};
let state = Rc::new(RefCell::new((false, None::<Waker>)));
let received = state.clone();
let mut watch = std::pin::pin!(std::future::IntoFuture::into_future(
call("core.watch", json!({"path":p.path})).on_notify(move |name, _| {
if name == "armed" {
let mut state = received.borrow_mut();
state.0 = true;
if let Some(waker) = state.1.take() {
waker.wake();
}
}
})
));
std::future::poll_fn(|cx| {
if let Poll::Ready(result) = watch.as_mut().poll(cx) {
return Poll::Ready(result.map(|_| ()));
}
let mut state = state.borrow_mut();
state.1 = Some(cx.waker().clone());
if state.0 {
Poll::Ready(Ok(()))
} else {
Poll::Pending
}
})
.await
}
/// Drop one watch while another still delivers, in a process that remains alive.
async fn drop_watch(&self) -> Result<Vec<String>> {
use std::{cell::RefCell, future::IntoFuture, rc::Rc, task::Poll};
let events = Rc::new(RefCell::new(Vec::new()));
let captured = events.clone();
let mut dropped = call("core.watch", json!({"path":"/dropped"}))
.on_notify(move |_, _| captured.borrow_mut().push("dropped".to_owned()))
.into_future();
std::future::poll_fn(|cx| {
assert!(dropped.as_mut().poll(cx).is_pending());
Poll::Ready(())
})
.await;
drop(dropped);
let captured = events.clone();
let mut kept = call("core.watch", json!({"path":"/kept"}))
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
.into_future();
std::future::poll_fn(|cx| {
assert!(kept.as_mut().poll(cx).is_pending());
Poll::Ready(())
})
.await;
// The harness sends late traffic and kept-watch events before answering.
call("other.barrier", json!({})).await?;
drop(kept);
let result = events.borrow().clone();
Ok(result)
}
/// Publish a control endpoint until EOF. /// Publish a control endpoint until EOF.
async fn serve(&self) -> Result<()> { async fn serve(&self) -> Result<()> {
sausage_plugin::listen("/endpoint", std::rc::Rc::new(Control)).await sausage_plugin::listen("/endpoint", std::rc::Rc::new(Control)).await
@@ -54,6 +158,11 @@ impl Probe {
sausage_plugin::listen("/state/control", std::rc::Rc::new(Control)).await?; sausage_plugin::listen("/state/control", std::rc::Rc::new(Control)).await?;
Ok(sausage_plugin::pid()) Ok(sausage_plugin::pid())
} }
/// Keep an idle listener and ownership lock until opt-in drain ends this process.
async fn drain_service(&self) -> Result<String> {
lock("/drain-owner").await?.keep();
self.service().await
}
/// Add two numbers. /// Add two numbers.
/// ///
/// The result is their sum. /// The result is their sum.
@@ -111,6 +220,15 @@ impl Probe {
fn retired(&self, p: Retired) -> Result<String> { fn retired(&self, p: Retired) -> Result<String> {
Ok(p.previous) Ok(p.previous)
} }
/// Drain on replacement while calls are still available.
#[sausage_plugin::on_superseded]
async fn superseded(&self, p: Superseded) -> Result<()> {
call("other.superseded", json!({"current":p.current})).await?;
sausage_plugin::drain();
sausage_plugin::drain();
Ctx::default().notify("draining", json!({}))?;
Ok(())
}
/// Wait until EOF, then run the stop hook. /// Wait until EOF, then run the stop hook.
async fn wait(&self) -> Result<String> { async fn wait(&self) -> Result<String> {
call("other.wait", json!({})).await?; call("other.wait", json!({})).await?;
@@ -155,6 +273,7 @@ impl Probe {
#[sausage_plugin::on_stop] #[sausage_plugin::on_stop]
async fn stop(&self) { async fn stop(&self) {
std::fs::write("/stopped", b"yes").unwrap(); std::fs::write("/stopped", b"yes").unwrap();
std::fs::write(format!("/stopped-{}", sausage_plugin::pid()), b"yes").unwrap();
Ctx::default().notify("stopped", json!({})).unwrap(); Ctx::default().notify("stopped", json!({})).unwrap();
assert!(call("other.no", json!({})).await.is_err()); assert!(call("other.no", json!({})).await.is_err());
} }
@@ -193,6 +312,11 @@ impl Control {
ctx.notify("endpoint-progress", json!({"value":7}))?; ctx.notify("endpoint-progress", json!({"value":7}))?;
Ok(7) Ok(7)
} }
/// Finish an in-flight call after draining begins.
async fn wait(&self) -> Result<String> {
call("other.inflight", json!({})).await?;
Ok("finished".into())
}
/// End the endpoint process after the response. /// End the endpoint process after the response.
fn stop(&self, ctx: &Ctx) -> Result<()> { fn stop(&self, ctx: &Ctx) -> Result<()> {
ctx.exit_after_response(); ctx.exit_after_response();
+47
View File
@@ -116,6 +116,53 @@ fn discovery_and_dispatch() {
.into(); .into();
assert_eq!(internal.code, -32603); assert_eq!(internal.code, -32603);
} }
#[sausage_plugin::test]
async fn params_must_be_objects() {
for (params, kind) in [
(json!("{\"a\":2,\"b\":3}"), "a string"),
(json!([]), "an array"),
(json!(42), "a number"),
(json!(null), "null"),
(json!(true), "a boolean"),
] {
let message = format!(
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
);
let outgoing = sausage_plugin::call("plugin.add", params.clone())
.await
.unwrap_err();
assert_eq!(outgoing.code, -32602);
assert_eq!(outgoing.message, message);
for method in ["add", "choice", "unknown"] {
let incoming = Plugin
.dispatch(method, params.clone(), &Ctx::default())
.await
.unwrap_err();
assert_eq!(incoming.code, -32602);
assert_eq!(incoming.message, message);
}
}
assert_eq!(
Plugin
.dispatch("add", json!({"a":2,"b":3}), &Ctx::default())
.await
.unwrap(),
5
);
assert_eq!(
Plugin
.dispatch("defaults", json!({}), &Ctx::default())
.await
.unwrap(),
json!({"value":""})
);
// An object reaches transport validation rather than the params rejection.
let outgoing = sausage_plugin::call("plugin.add", json!({"a":2,"b":3}))
.await
.unwrap_err();
assert!(outgoing.message.contains("no core transport"));
}
#[sausage_plugin::test] #[sausage_plugin::test]
async fn missing_core_fails_without_wait() { async fn missing_core_fails_without_wait() {
assert!( assert!(
+11
View File
@@ -0,0 +1,11 @@
struct Supersedes;
#[sausage_plugin::rpc]
impl Supersedes {
/// First superseded hook.
#[sausage_plugin::on_superseded]
fn a(&self) -> sausage_plugin::Result<()> { Ok(()) }
/// Second superseded hook.
#[sausage_plugin::on_superseded]
fn b(&self) -> sausage_plugin::Result<()> { Ok(()) }
}
fn main() {}
+5
View File
@@ -0,0 +1,5 @@
error: only one on_superseded method is allowed
--> tests/ui/superseded.rs:7:5
|
7 | /// Second superseded hook.
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
+127 -4
View File
@@ -34,10 +34,8 @@ impl Process {
let mut s = String::new(); let mut s = String::new();
if self.output.read_line(&mut s).unwrap() == 0 { if self.output.read_line(&mut s).unwrap() == 0 {
let mut stderr = String::new(); let mut stderr = String::new();
let _ = std::io::Read::read_to_string( let _ =
self.child.stderr.as_mut().unwrap(), std::io::Read::read_to_string(self.child.stderr.as_mut().unwrap(), &mut stderr);
&mut stderr,
);
panic!("unexpected EOF from wasmtime. Its stderr:\n{stderr}"); panic!("unexpected EOF from wasmtime. Its stderr:\n{stderr}");
} }
if let Ok(v) = sausage_plugin::serde_json::from_str::<Value>(&s) if let Ok(v) = sausage_plugin::serde_json::from_str::<Value>(&s)
@@ -110,6 +108,34 @@ fn protocol() {
} }
p.finishes(); p.finishes();
} }
// A handler-owned watch does not keep an idle process alive after return,
// even though the host never answers the watch and stdin remains open.
let mut p = start(&wasm, &dir);
p.request("watch_once", json!({"path":"/mailbox"}));
let watch = p.read();
assert_eq!(watch["method"], "core.watch");
p.send(json!({"jsonrpc":"2.0","method":"core.notify","params":{"call":watch["id"],"name":"armed","params":{}}}));
assert_eq!(p.read()["result"], Value::Null);
p.finishes();
// Keep the process alive after dropping a call: late traffic is ignored,
// while the kept watch continues to deliver every notification.
let mut p = start(&wasm, &dir);
p.request("drop_watch", json!({}));
let dropped = p.read();
let kept = p.read();
let barrier = p.read();
assert_eq!(dropped["params"]["path"], "/dropped");
assert_eq!(kept["params"]["path"], "/kept");
assert_eq!(barrier["method"], "other.barrier");
for watch in [&dropped, &kept] {
for name in ["armed", "changed", "changed"] {
p.send(json!({"jsonrpc":"2.0","method":"core.notify","params":{"call":watch["id"],"name":name,"params":{}}}));
}
}
p.answer(&dropped, Value::Null);
p.answer(&barrier, json!({}));
assert_eq!(p.read()["result"], json!(["armed", "changed", "changed"]));
p.finishes();
let mut p = start(&wasm, &dir); let mut p = start(&wasm, &dir);
p.request("rpc.discover", json!({})); p.request("rpc.discover", json!({}));
let d = p.read(); let d = p.read();
@@ -203,6 +229,103 @@ fn protocol() {
assert_eq!(outgoing["method"], "other.background"); assert_eq!(outgoing["method"], "other.background");
p.answer(&outgoing, json!({})); p.answer(&outgoing, json!({}));
assert!(p.child.wait().unwrap().success()); assert!(p.child.wait().unwrap().success());
// Replacement is a notification, not EOF. Drain keeps both endpoint handlers
// and background calls alive, while closing idle connections and the listener.
assert!(
!d["result"]["methods"]
.as_array()
.unwrap()
.iter()
.any(|m| m["name"] == "rpc.superseded" || m["name"] == "superseded")
);
std::fs::remove_file(dir.join("stopped")).unwrap();
let mut provider = start(&wasm, &dir);
provider.request("serve", json!({}));
assert_eq!(provider.read()["result"], Value::Null);
provider.request("start", json!({}));
let first = provider.read();
let second = provider.read();
let background = if first["method"] == "other.background" {
first
} else {
second
};
assert_eq!(background["method"], "other.background");
let address = std::fs::read_to_string(dir.join("endpoint"))
.unwrap()
.split_whitespace()
.next()
.unwrap()
.trim_start_matches("tcp:")
.to_owned();
let mut idle = std::net::TcpStream::connect(&address).unwrap();
idle.set_read_timeout(Some(std::time::Duration::from_secs(4)))
.unwrap();
writeln!(
idle,
"{}",
json!({"jsonrpc":"2.0","id":0,"method":"rpc.endpoint","params":{}})
)
.unwrap();
let mut idle = BufReader::new(idle);
let mut line = String::new();
idle.read_line(&mut line).unwrap();
assert_eq!(
sausage_plugin::serde_json::from_str::<Value>(&line).unwrap()["id"],
0
);
let mut active = std::net::TcpStream::connect(&address).unwrap();
active
.set_read_timeout(Some(std::time::Duration::from_secs(4)))
.unwrap();
writeln!(
active,
"{}",
json!({"jsonrpc":"2.0","id":1,"method":"wait","params":{}})
)
.unwrap();
let inflight = provider.read();
assert_eq!(inflight["method"], "other.inflight");
provider
.send(json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":"new-hash"}}));
let hook = provider.read();
assert_eq!(hook["method"], "other.superseded");
assert_eq!(hook["params"]["current"], "new-hash");
provider.answer(&hook, json!({}));
assert_eq!(provider.read()["params"]["name"], "draining");
line.clear();
assert_eq!(idle.read_line(&mut line).unwrap(), 0);
// drain() wakes independent accept/read cancellations. Idle EOF does not
// order listener destruction: TCP may still complete a handshake, but that
// connection must close without serving work. Do not retry the connect.
if let Ok(mut late) = std::net::TcpStream::connect(&address) {
eprintln!("TCP handshake raced listener cancellation; waiting for close");
// Reuse the four-second socket bound used by idle/active above.
late.set_read_timeout(Some(std::time::Duration::from_secs(4)))
.unwrap();
let closed = std::io::Read::read(&mut late, &mut [0]);
assert!(
matches!(closed, Ok(0))
|| matches!(&closed, Err(e) if e.kind() == std::io::ErrorKind::ConnectionReset),
"late connection survived drain: {closed:?}"
);
}
assert!(provider.child.try_wait().unwrap().is_none());
provider.answer(&inflight, json!({}));
line.clear();
BufReader::new(active).read_line(&mut line).unwrap();
assert_eq!(
sausage_plugin::serde_json::from_str::<Value>(&line).unwrap()["result"],
"finished"
);
assert!(provider.child.try_wait().unwrap().is_none());
provider.answer(&background, json!({}));
provider.finishes();
assert!(std::net::TcpStream::connect(&address).is_err());
assert!(
!dir.join("stopped").exists(),
"drain is not the EOF stop hook"
);
let provider_doc = &d["result"]; let provider_doc = &d["result"];
let imports = target.join("imports"); let imports = target.join("imports");
std::fs::create_dir_all(&imports).unwrap(); std::fs::create_dir_all(&imports).unwrap();
+1
View File
@@ -5,6 +5,7 @@ pub mod generated;
pub mod host; pub mod host;
pub mod memory; pub mod memory;
pub mod mounts; pub mod mounts;
pub mod observe;
pub mod process; pub mod process;
mod readonly; mod readonly;
mod streams; mod streams;
+615
View File
@@ -0,0 +1,615 @@
//! Observes successful namespace and open-file mutations, even if their caller is cancelled.
use crate::*;
use std::{
collections::BTreeMap,
sync::{Mutex, Weak},
};
use tokio::sync::{OwnedRwLockReadGuard, OwnedRwLockWriteGuard, RwLock};
type Namespace = Mutex<BTreeMap<String, Weak<RwLock<()>>>>;
enum Held {
Read { _guard: OwnedRwLockReadGuard<()> },
Write { _guard: OwnedRwLockWriteGuard<()> },
}
type TrackedPath = Mutex<Option<String>>;
type Path = Arc<TrackedPath>;
type Callback = dyn Fn(&str) + Send + Sync;
/// Shared observation for mount namespaces. Paths supplied to the callback are virtual paths.
/// Callbacks run after filesystem locks have been released; they must not block.
#[derive(Clone)]
pub struct Observer {
changed: Arc<Callback>,
paths: Arc<Mutex<Vec<Weak<TrackedPath>>>>,
// Only open-registration versus rename/remove needs ordering. Lock the affected
// subtrees, not all IO: a blocked open must not stall unrelated namespaces.
namespace: Arc<Namespace>,
}
impl Observer {
pub fn new(changed: impl Fn(&str) + Send + Sync + 'static) -> Self {
Self {
changed: Arc::new(changed),
paths: Arc::default(),
namespace: Arc::default(),
}
}
/// Wrap a layer at its canonical namespace prefix (e.g. `/run`).
pub fn layer(&self, layer: Arc<dyn Layer>, prefix: &str, process: Process) -> Arc<dyn Layer> {
Arc::new(Observed {
layer,
prefix: prefix.trim_end_matches('/').into(),
observer: self.clone(),
process,
})
}
/// Opens share their path and ancestors; rename/remove exclusively hold their
/// endpoints, sharing ancestors. Sorted acquisition avoids cross-rename deadlocks.
/// No descriptor IO or notification holds these guards.
async fn order(&self, endpoints: &[(&str, bool)]) -> Vec<Held> {
let mut modes = BTreeMap::new();
for &(path, exclusive) in endpoints {
modes
.entry(path.to_owned())
.and_modify(|mode| *mode |= exclusive)
.or_insert(exclusive);
let mut ancestor = path;
while let Some((parent, _)) = ancestor.rsplit_once('/') {
let parent = if parent.is_empty() { "/" } else { parent };
modes.entry(parent.to_owned()).or_insert(false);
if parent == "/" {
break;
}
ancestor = parent;
}
}
let locks: Vec<_> = {
let mut namespace = self.namespace.lock().unwrap();
namespace.retain(|_, lock| lock.strong_count() != 0);
modes
.into_iter()
.map(|(path, exclusive)| {
let lock = namespace.entry(path).or_default();
let strong = lock.upgrade().unwrap_or_else(|| {
let strong = Arc::new(RwLock::new(()));
*lock = Arc::downgrade(&strong);
strong
});
(strong, exclusive)
})
.collect()
};
let mut held = Vec::new();
for (lock, exclusive) in locks {
held.push(if exclusive {
Held::Write {
_guard: lock.write_owned().await,
}
} else {
Held::Read {
_guard: lock.read_owned().await,
}
});
}
held
}
fn path(&self, path: String) -> Path {
let path = Arc::new(Mutex::new(Some(path)));
let mut paths = self.paths.lock().unwrap();
paths.retain(|p| p.strong_count() != 0);
paths.push(Arc::downgrade(&path));
path
}
fn moved(&self, from: &str, to: Option<&str>) {
let mut paths = self.paths.lock().unwrap();
paths.retain(|p| {
let Some(p) = p.upgrade() else { return false };
let mut path = p.lock().unwrap();
if let Some(old) = path.as_deref() {
if under(old, from) {
*path = to.map(|to| format!("{to}{}", &old[from.len()..]));
} else if to.is_some_and(|to| under(old, to)) {
// A replaced destination's old descriptor no longer names a visible file.
*path = None;
}
}
true
});
}
}
/// Component-aware descendant matching, including the path itself.
pub fn under(path: &str, base: &str) -> bool {
base == "/" || path == base || path.strip_prefix(base).is_some_and(|s| s.starts_with('/'))
}
struct Observed {
layer: Arc<dyn Layer>,
prefix: String,
observer: Observer,
process: Process,
}
impl Observed {
fn path(&self, path: &str) -> String {
let path = format!("{}{}", self.prefix, path.trim_end_matches('/'));
if path.is_empty() { "/".into() } else { path }
}
}
struct Handle {
file: Arc<dyn File>,
path: Path,
observer: Observer,
process: Process,
}
impl Handle {
async fn mutate<T: Send + 'static>(
&self,
f: impl Future<Output = Result<T>> + Send + 'static,
) -> Result<T> {
let (path, observer) = (self.path.clone(), self.observer.clone());
self.process
.operation(async move {
let result = f.await?;
let path = path.lock().unwrap().clone();
if let Some(path) = path {
(observer.changed)(&path);
}
Ok(result)
})
.await
}
}
#[async_trait]
impl Layer for Observed {
async fn stat(&self, path: &str, follow: bool) -> Result<Stat> {
self.layer.stat(path, follow).await
}
async fn readlink(&self, path: &str) -> Result<String> {
self.layer.readlink(path).await
}
async fn list(&self, path: &str) -> Result<Vec<Entry>> {
self.layer.list(path).await
}
async fn open(&self, path: &str, options: Open) -> Result<Arc<dyn File>> {
let (layer, local, path, observer, process) = (
self.layer.clone(),
path.to_owned(),
self.path(path),
self.observer.clone(),
self.process.clone(),
);
self.process
.operation(async move {
let order = observer.order(&[(&path, false)]).await;
let file = layer.open(&local, options).await?;
let handle = Arc::new(Handle {
file,
path: observer.path(path.clone()),
observer: observer.clone(),
process,
});
drop(order);
if options.create || options.truncate {
(observer.changed)(&path);
}
Ok(handle as Arc<dyn File>)
})
.await
}
async fn mkdir(&self, path: &str) -> Result<()> {
let (layer, local, path, observer) = (
self.layer.clone(),
path.to_owned(),
self.path(path),
self.observer.clone(),
);
self.process
.operation(async move {
layer.mkdir(&local).await?;
(observer.changed)(&path);
Ok(())
})
.await
}
async fn remove_file(&self, path: &str) -> Result<()> {
self.remove(path, false).await
}
async fn remove_dir(&self, path: &str) -> Result<()> {
self.remove(path, true).await
}
async fn rename(&self, from: &str, to: &str) -> Result<()> {
let (layer, a, b, from, to, observer) = (
self.layer.clone(),
from.to_owned(),
to.to_owned(),
self.path(from),
self.path(to),
self.observer.clone(),
);
self.process
.operation(async move {
let order = observer.order(&[(&from, true), (&to, true)]).await;
layer.rename(&a, &b).await?;
if from != to {
observer.moved(&from, Some(&to));
}
drop(order);
(observer.changed)(&from);
(observer.changed)(&to);
Ok(())
})
.await
}
}
impl Observed {
async fn remove(&self, path: &str, directory: bool) -> Result<()> {
let (layer, local, path, observer) = (
self.layer.clone(),
path.to_owned(),
self.path(path),
self.observer.clone(),
);
self.process
.operation(async move {
let order = observer.order(&[(&path, true)]).await;
if directory {
layer.remove_dir(&local).await?;
} else {
layer.remove_file(&local).await?;
}
observer.moved(&path, None);
drop(order);
(observer.changed)(&path);
Ok(())
})
.await
}
}
#[async_trait]
impl File for Handle {
async fn stat(&self) -> Result<Stat> {
self.file.stat().await
}
async fn read(&self, offset: u64, length: usize) -> Result<Vec<u8>> {
self.file.read(offset, length).await
}
async fn write(&self, offset: u64, bytes: Vec<u8>) -> Result<usize> {
let f = self.file.clone();
self.mutate(async move { f.write(offset, bytes).await })
.await
}
async fn append(&self, bytes: Vec<u8>) -> Result<usize> {
let f = self.file.clone();
self.mutate(async move { f.append(bytes).await }).await
}
async fn set_size(&self, size: u64) -> Result<()> {
let f = self.file.clone();
self.mutate(async move { f.set_size(size).await }).await
}
async fn sync(&self) -> Result<()> {
self.file.sync().await
}
async fn set_times(
&self,
accessed: Option<SystemTime>,
modified: Option<SystemTime>,
) -> Result<()> {
let f = self.file.clone();
self.mutate(async move { f.set_times(accessed, modified).await })
.await
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::sync::mpsc;
#[test]
fn component_matching() {
assert!(under("/mailbox/a", "/mailbox"));
assert!(under("/mailbox", "/mailbox"));
assert!(under("/anything", "/"));
assert!(!under("/mailboxes/a", "/mailbox"));
}
/// Every visible mutation is delivered after the layer call, including rebased
/// open handles in another process's namespace. Removed handles stay usable.
#[tokio::test]
async fn handles_follow_rename_and_unlink() {
for host in [false, true] {
let temp = tempfile::tempdir().unwrap();
let process = Process::default();
let layer: Arc<dyn Layer> = if host {
Arc::new(Host::new(temp.path(), process.clone()).unwrap())
} else {
Arc::new(Memory::default())
};
let (tx, mut rx) = mpsc::unbounded_channel();
let observer = Observer::new(move |p| {
tx.send(p.to_owned()).unwrap();
});
let a = observer.layer(layer.clone(), "/run", process.clone());
let b = observer.layer(layer, "/run", process);
a.mkdir("/dir").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/dir");
let f = a
.open(
"/dir/a",
Open {
create: true,
write: true,
..Open::default()
},
)
.await
.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/dir/a");
b.rename("/dir", "/moved").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/dir");
assert_eq!(rx.recv().await.unwrap(), "/run/moved");
f.write(0, b"new".to_vec()).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/moved/a");
assert_eq!(
b.open("/moved/a", Open::default())
.await
.unwrap()
.read(0, 3)
.await
.unwrap(),
b"new"
);
let old = b
.open(
"/old",
Open {
create: true,
write: true,
..Open::default()
},
)
.await
.unwrap();
rx.recv().await.unwrap();
b.rename("/moved/a", "/old").await.unwrap();
rx.recv().await.unwrap();
rx.recv().await.unwrap();
old.write(0, b"invisible".to_vec()).await.unwrap();
assert!(rx.try_recv().is_err());
f.append(b"!".to_vec()).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/old");
b.remove_file("/old").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/old");
f.set_size(0).await.unwrap();
assert!(rx.try_recv().is_err());
assert!(b.remove_file("/missing").await.is_err());
assert!(rx.try_recv().is_err());
}
}
/// A pending exclusive namespace change must not block a sibling's opens,
/// even though Tokio's RwLock prefers queued writers over new readers.
#[tokio::test]
async fn namespace_order_is_scoped_and_shared() {
async fn ready<T>(future: impl Future<Output = T>) -> T {
let mut future = std::pin::pin!(future);
std::future::poll_fn(|cx| match future.as_mut().poll(cx) {
std::task::Poll::Ready(value) => std::task::Poll::Ready(value),
std::task::Poll::Pending => panic!("unrelated namespace operation blocked"),
})
.await
}
let observer = Observer::new(|_| {});
let open = observer.order(&[("/run/slow/file", false)]).await;
let concurrent_open = ready(observer.order(&[("/run/slow/file", false)])).await;
let mut rename =
std::pin::pin!(observer.order(&[("/run/slow", true), ("/run/moved", true)]));
std::future::poll_fn(|cx| {
assert!(rename.as_mut().poll(cx).is_pending());
std::task::Poll::Ready(())
})
.await;
// Same mounted layer, different subtree; also a different mounted layer.
let sibling_open = ready(observer.order(&[("/run/sibling/file", false)])).await;
let host_open = ready(observer.order(&[("/state/file", false)])).await;
drop(sibling_open);
let sibling_rename =
ready(observer.order(&[("/run/sibling", true), ("/run/other", true)])).await;
drop((sibling_rename, host_open, concurrent_open, open));
let renamed = ready(rename).await;
// A descendant open waits for its ancestor's rename until rebasing is done.
let mut descendant = std::pin::pin!(observer.order(&[("/run/moved/file", false)]));
std::future::poll_fn(|cx| {
assert!(descendant.as_mut().poll(cx).is_pending());
std::task::Poll::Ready(())
})
.await;
drop(renamed);
drop(ready(descendant).await);
drop(
observer
.order(&[("/run/moved/file", true), ("/run/moved", true)])
.await,
);
drop(
observer
.order(&[("/run/moved", true), ("/run/moved", true)])
.await,
);
// Dead weak entries are pruned, rather than retaining every path ever opened.
observer.order(&[("/", false)]).await;
assert_eq!(observer.namespace.lock().unwrap().len(), 1);
}
/// Pause an open after obtaining its real identity, before registering its
/// observed handle, like a slow host syscall or /self/log query.
struct SlowOpen {
memory: Arc<Memory>,
gate: Mutex<
Option<(
tokio::sync::oneshot::Sender<()>,
tokio::sync::oneshot::Receiver<()>,
)>,
>,
}
#[async_trait]
impl Layer for SlowOpen {
async fn stat(&self, p: &str, follow: bool) -> Result<Stat> {
self.memory.stat(p, follow).await
}
async fn list(&self, p: &str) -> Result<Vec<Entry>> {
self.memory.list(p).await
}
async fn open(&self, p: &str, options: Open) -> Result<Arc<dyn File>> {
let f = self.memory.open(p, options).await?;
let gate = self.gate.lock().unwrap().take();
if let Some((started, release)) = gate {
started.send(()).unwrap();
release.await.unwrap();
}
Ok(f)
}
async fn mkdir(&self, p: &str) -> Result<()> {
self.memory.mkdir(p).await
}
async fn remove_file(&self, p: &str) -> Result<()> {
self.memory.remove_file(p).await
}
async fn remove_dir(&self, p: &str) -> Result<()> {
self.memory.remove_dir(p).await
}
async fn rename(&self, a: &str, b: &str) -> Result<()> {
self.memory.rename(a, b).await
}
}
#[tokio::test]
async fn blocked_io_does_not_stall_other_processes() {
let memory = Arc::new(Memory::default());
memory.mkdir("/slow").await.unwrap();
let options = Open {
create: true,
write: true,
..Open::default()
};
let raw = memory.open("/slow/file", options).await.unwrap();
let (started, opening) = tokio::sync::oneshot::channel();
let (release, released) = tokio::sync::oneshot::channel();
let (tx, mut rx) = mpsc::unbounded_channel();
let observer = Observer::new(move |p| {
tx.send(p.to_owned()).unwrap();
});
let slow = observer.layer(
Arc::new(SlowOpen {
memory: memory.clone(),
gate: Mutex::new(Some((started, released))),
}),
"/run",
Process::default(),
);
let other = observer.layer(memory, "/run", Process::default());
let opening_task = tokio::spawn(async move {
slow.open(
"/slow/file",
Open {
write: true,
..Open::default()
},
)
.await
});
opening.await.unwrap();
let renamer = other.clone();
let renamed = tokio::spawn(async move { renamer.rename("/slow", "/moved").await });
// These must complete while the first process's open is still blocked,
// and a conflicting rename is waiting to rebase its registered handle.
other
.open(
"/",
Open {
directory: true,
..Open::default()
},
)
.await
.unwrap();
let f = other.open("/other", options).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/other");
f.write(0, b"visible".to_vec()).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/other");
other.mkdir("/sibling").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/sibling");
other.rename("/other", "/sibling/file").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/other");
assert_eq!(rx.recv().await.unwrap(), "/run/sibling/file");
other.remove_file("/sibling/file").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/sibling/file");
assert!(!renamed.is_finished());
release.send(()).unwrap();
let opened = opening_task.await.unwrap().unwrap();
renamed.await.unwrap().unwrap();
rx.recv().await.unwrap();
rx.recv().await.unwrap();
opened.set_size(0).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/moved/file");
// No namespace guard surrounds descriptor IO: a rename completes while
// this write is held in flight. Its notification uses the rebased path.
let handle = Handle {
file: raw.clone(),
path: observer.path("/run/moved/file".into()),
observer,
process: Process::default(),
};
let (started, writing) = tokio::sync::oneshot::channel();
let (release, released) = tokio::sync::oneshot::channel();
let write = tokio::spawn(async move {
handle
.mutate(async move {
started.send(()).unwrap();
released.await.unwrap();
raw.write(0, b"after rename".to_vec()).await
})
.await
});
writing.await.unwrap();
other.rename("/moved", "/again").await.unwrap();
rx.recv().await.unwrap();
rx.recv().await.unwrap();
other.open("/again/file", Open::default()).await.unwrap();
release.send(()).unwrap();
write.await.unwrap().unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/again/file");
}
/// Killing a waiting writer does not discard observation of a completed mutation;
/// the process's idle barrier includes the callback, not only the system call.
#[tokio::test]
async fn cancellation_keeps_mutation_and_observation() {
let (tx, mut rx) = mpsc::unbounded_channel();
let observer = Observer::new(move |p| {
tx.send(p.to_owned()).unwrap();
});
let process = Process::default();
let layer = observer.layer(Arc::new(Memory::default()), "/", process.clone());
let order = observer.order(&[("/published", true)]).await;
let (started, ready) = tokio::sync::oneshot::channel();
let p = process.clone();
let task = tokio::spawn(async move {
// Signal from inside the tracked operation, not before its guard exists.
p.operation(async move {
started.send(()).unwrap();
layer
.open(
"/published",
Open {
create: true,
write: true,
..Open::default()
},
)
.await
})
.await
});
ready.await.unwrap();
task.abort();
let _ = task.await;
drop(order);
process.idle().await;
assert_eq!(rx.try_recv().unwrap(), "/published");
}
}
+16 -1
View File
@@ -36,7 +36,22 @@ impl Process {
.await .await
.map_err(std::io::Error::other)? .map_err(std::io::Error::other)?
} }
/// Wait for all outstanding system calls. /// Complete an asynchronous mutation and its observation even if the guest is cancelled.
/// Its guard participates in idle(), so no mutation outlives process cleanup.
pub async fn operation<T: Send + 'static>(
&self,
operation: impl std::future::Future<Output = std::io::Result<T>> + Send + 'static,
) -> std::io::Result<T> {
self.0.count.fetch_add(1, Ordering::SeqCst);
let guard = Guard(self.0.clone());
tokio::spawn(async move {
let _guard = guard;
operation.await
})
.await
.map_err(std::io::Error::other)?
}
/// Wait for all outstanding system calls and observed mutations.
/// First abort the guest task. Then await its JoinHandle, or drop its store. /// First abort the guest task. Then await its JoinHandle, or drop its store.
/// Only then await idle before releasing locks. Abort alone does not stop a running poll. /// Only then await idle before releasing locks. Abort alone does not stop a running poll.
/// This ordering also prevents later memory writes, which need no blocking guard. /// This ordering also prevents later memory writes, which need no blocking guard.