Compare commits
19
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c7d478754d | ||
|
|
facaf1d9b1 | ||
|
|
ee88adf6c6 | ||
|
|
6dc21cd8dc | ||
|
|
5be5b3b397 | ||
|
|
a95b5287cf | ||
|
|
959ddf46ff | ||
|
|
4da8a28392 | ||
|
|
c54edea596 | ||
|
|
9ef73e6bc7 | ||
|
|
dc81f78c20 | ||
|
|
9a8c143856 | ||
|
|
a14f051f97 | ||
|
|
e26260d36e | ||
|
|
037377fd27 | ||
|
|
4be9ae41de | ||
|
|
168795b408 | ||
|
|
8c46d9b9d0 | ||
|
|
c15e50d752 |
Generated
+2
@@ -2092,7 +2092,9 @@ dependencies = [
|
|||||||
"rusqlite",
|
"rusqlite",
|
||||||
"rustyline",
|
"rustyline",
|
||||||
"sausage-interface",
|
"sausage-interface",
|
||||||
|
"sausage-plugin",
|
||||||
"sausage-vfs",
|
"sausage-vfs",
|
||||||
|
"schemars 1.2.3",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
"serde_yaml_ng",
|
"serde_yaml_ng",
|
||||||
|
|||||||
@@ -61,6 +61,20 @@ a plugin, or a change that the agent makes to itself.
|
|||||||
|
|
||||||
## Small choices
|
## Small choices
|
||||||
|
|
||||||
|
- **Restart is a clean stop supervised by `#loop`.** `core.restart` requires
|
||||||
|
the host startup marker `SAUSAGE_LOOP=1`, which the loop exports for its
|
||||||
|
invocation. Without a supervisor, restarting would only stop the world.
|
||||||
|
A single pending request answers first, allows 250 ms for delivery, then
|
||||||
|
signals the existing shutdown path; it does not use `process::exit` or
|
||||||
|
bypass detached-process cleanup, lock release, or journal draining. The
|
||||||
|
executable returns dedicated restart code **75** after cleanup. `#loop`
|
||||||
|
refreshes and starts again on 75 (and other nonzero exits); 0 stops it.
|
||||||
|
`core.exit` is a compatibility alias with the same supervisor gate,
|
||||||
|
response grace, and coalescing; new callers use `core.restart`.
|
||||||
|
**HARD RULE: the mediator and all agents NEVER call `core.restart` without
|
||||||
|
the person's explicit approval each time.** This
|
||||||
|
approval rule also covers legacy `core.exit`; it must not bypass the gate.
|
||||||
|
|
||||||
- **YAML for configuration, JSON for data.** A person edits configuration, so
|
- **YAML for configuration, JSON for data.** A person edits configuration, so
|
||||||
it needs comments. `init` writes commented templates instead of asking
|
it needs comments. `init` writes commented templates instead of asking
|
||||||
questions, so there is one way to configure a world.
|
questions, so there is one way to configure a world.
|
||||||
@@ -70,6 +84,16 @@ a plugin, or a change that the agent makes to itself.
|
|||||||
that occur only in WASI, for example a missing WASI function or a panic at
|
that occur only in WASI, for example a missing WASI function or a panic at
|
||||||
start. The test runner uses the same Wasmtime and the same mounts as the
|
start. The test runner uses the same Wasmtime and the same mounts as the
|
||||||
core.
|
core.
|
||||||
|
- **Tests never run without a timeout.** `cargo.build` serializes its work
|
||||||
|
folder, so one hung test or build script must not block all later builds.
|
||||||
|
`timeout_secs` defaults to 900 seconds after mutex acquisition, excluding
|
||||||
|
queue time; `test_timeout_secs` defaults to 120 seconds per WASI instance.
|
||||||
|
Both are optional positive integers, never null/unlimited. Test timeouts name
|
||||||
|
the failing test and let other instances finish. Build deadlines and
|
||||||
|
cancellation kill cargo's process group before dropping the mutex; timeout
|
||||||
|
errors retain the phase, elapsed time, and recent output. Source preparation
|
||||||
|
and publication retain serialization while their blocking filesystem work
|
||||||
|
finishes, even if cancelled, to avoid overlapping writes to the work folder.
|
||||||
- **The SDK is not published.** `cargo` points `sausage-plugin = "0.1"` at
|
- **The SDK is not published.** `cargo` points `sausage-plugin = "0.1"` at
|
||||||
its own copy, which it also shows in `/plugins/cargo/sdk/`.
|
its own copy, which it also shows in `/plugins/cargo/sdk/`.
|
||||||
- **Compiled code is a cache, not state.** It lives outside the world, in
|
- **Compiled code is a cache, not state.** It lives outside the world, in
|
||||||
|
|||||||
+209
-18
@@ -19,7 +19,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
|
|||||||
| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
|
| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
|
||||||
| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
|
| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
|
||||||
| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
|
| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
|
||||||
| service | A call in `/config/boot.yaml`. The core starts it at boot and again after an upgrade of its plugin. |
|
| service | A call in `/config/boot.yaml`. The core starts it at boot, restarts it after a crash, and starts it again after an upgrade of its plugin. |
|
||||||
| import | An interface of another plugin that a plugin uses. |
|
| import | An interface of another plugin that a plugin uses. |
|
||||||
| provider, consumer | The plugin that has an interface, and a plugin that imports it. |
|
| provider, consumer | The plugin that has an interface, and a plugin that imports it. |
|
||||||
| journal | The central log of the world. |
|
| journal | The central log of the world. |
|
||||||
@@ -34,7 +34,8 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
|
|||||||
source/ the source. `cargo.build` builds it from here.
|
source/ the source. `cargo.build` builds it from here.
|
||||||
imports/<plugin>.json descriptions of the imported plugins
|
imports/<plugin>.json descriptions of the imported plugins
|
||||||
state/<plugin>/ persistent state of a plugin, by convention
|
state/<plugin>/ persistent state of a plugin, by convention
|
||||||
state/core/journal.sqlite the journal
|
logs/core/journal.sqlite the journal
|
||||||
|
state/core/ console history (state, not logs)
|
||||||
config/ global configuration
|
config/ global configuration
|
||||||
boot.yaml the services
|
boot.yaml the services
|
||||||
agent.yaml the API, base URL, model, and key of the agent
|
agent.yaml the API, base URL, model, and key of the agent
|
||||||
@@ -69,7 +70,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
|
|||||||
`notes.rpc.discover` sends it `rpc.discover`.
|
`notes.rpc.discover` sends it `rpc.discover`.
|
||||||
- `core.<method>` is a method of the core (section 9).
|
- `core.<method>` is a method of the core (section 9).
|
||||||
- Method names that start with `rpc.` are reserved: `rpc.discover`,
|
- Method names that start with `rpc.` are reserved: `rpc.discover`,
|
||||||
`rpc.retired`, and `rpc.endpoint`.
|
`rpc.superseded`, `rpc.retired`, and `rpc.endpoint`.
|
||||||
|
|
||||||
### 3.3 What a call does
|
### 3.3 What a call does
|
||||||
|
|
||||||
@@ -152,9 +153,17 @@ A process is an instance of one plugin version. When a new version becomes
|
|||||||
active (section 10.3):
|
active (section 10.3):
|
||||||
|
|
||||||
1. New calls get the new version at once.
|
1. New calls get the new version at once.
|
||||||
2. Each process of an older version gets a grace period to finish its work.
|
2. At grace start, the core sends each process of an older version the
|
||||||
The default is 60 seconds. A plugin can declare its own value in
|
JSON-RPC notification `rpc.superseded {"current": "<sha256 of the new file>"}`
|
||||||
`rpc.discover` (`x-sausage-grace`). Then the core stops the process.
|
on stdin, without an ID. The process is still running: calls and responses
|
||||||
|
continue to work. It can stop accepting work, finish its in-flight calls,
|
||||||
|
and exit early. Ignoring the notice changes nothing. The grace period is
|
||||||
|
the upper bound before the core stops the process (section 3.6); its
|
||||||
|
duration is 5 seconds for every plugin. Discovery metadata, including an
|
||||||
|
explicit `x-sausage-grace`, does not change it. After grace, stdin closes
|
||||||
|
and the separate 5-second stop budget applies (section 3.6): at most two
|
||||||
|
5-second windows. Discovery-only processes do not receive this notice or
|
||||||
|
enter upgrade grace; they follow the discovery lifetime (section 10.1).
|
||||||
3. When the old process of a service has ended, the core starts the service
|
3. When the old process of a service has ended, the core starts the service
|
||||||
again with the new version.
|
again with the new version.
|
||||||
4. When the last process of an older version has ended, the core starts a
|
4. When the last process of an older version has ended, the core starts a
|
||||||
@@ -213,6 +222,7 @@ these areas fails.
|
|||||||
/proc/
|
/proc/
|
||||||
core/
|
core/
|
||||||
PROTOCOL.md this file
|
PROTOCOL.md this file
|
||||||
|
discover.json generated core.rpc.discover document
|
||||||
graph.json plugins, imports, and the result of each check
|
graph.json plugins, imports, and the result of each check
|
||||||
graph.dot the same graph for Graphviz
|
graph.dot the same graph for Graphviz
|
||||||
locks.json held locks and waiting processes
|
locks.json held locks and waiting processes
|
||||||
@@ -240,6 +250,40 @@ Rules:
|
|||||||
|
|
||||||
Principle: **inspect through files, act through calls.**
|
Principle: **inspect through files, act through calls.**
|
||||||
|
|
||||||
|
### File watching
|
||||||
|
|
||||||
|
`core.watch {"path": "/state/session/mailbox"}` stays open for the caller
|
||||||
|
process's lifetime. It needs a request ID and a running plugin caller.
|
||||||
|
Its first `core.notify` is `"armed"`, with empty params. Only then scan the
|
||||||
|
files. Writes before arming are visible in that scan; writes after arming
|
||||||
|
produce `"changed"` notifications (also with empty params). Rescan after
|
||||||
|
each notification; files, not event counts, are the truth.
|
||||||
|
|
||||||
|
Notifications follow successful plugin-originated file mutations: creation,
|
||||||
|
truncation, descriptor/stream writes, rename (both source and destination),
|
||||||
|
and deletion, including mounted `/run` and `/self` files. Paths are normalized
|
||||||
|
in the virtual namespace; `/self` means the caller's `/proc/<pid>` folder.
|
||||||
|
A watch matches itself and descendants, not sibling prefixes. Moving or
|
||||||
|
removing an ancestor also invalidates it. Paths need not exist when armed.
|
||||||
|
Open file writes follow plugin renames; unlinked/replaced handles no longer
|
||||||
|
name visible files. External host writes are outside this contract.
|
||||||
|
|
||||||
|
Changes may coalesce, including a change already visible in the initial
|
||||||
|
scan, but a change after the last notification always produces another.
|
||||||
|
Delivery holds no filesystem locks. Observation does not serialize descriptor
|
||||||
|
IO; only overlapping opens and rename/removal wait for one another.
|
||||||
|
Watch the narrowest useful path: delivery is not rate-limited, so watching a
|
||||||
|
busy parent can cause a rescan for every write.
|
||||||
|
|
||||||
|
The host-side watch ends only when its caller process exits, not on a change
|
||||||
|
notification; there is no cancellation API or polling/retry loop. SDK callers
|
||||||
|
should poll the watch future alongside their handler's work and drop it on
|
||||||
|
return, not keep it in a background task that prevents an idle process from
|
||||||
|
exiting. Dropping the future removes the SDK's pending entry and callback, so
|
||||||
|
late notifications and responses are ignored; it does not cancel the host
|
||||||
|
watch. A long-lived service keeps one watch per purpose for its lifetime,
|
||||||
|
rather than repeatedly opening and dropping watches.
|
||||||
|
|
||||||
## 5. Endpoints
|
## 5. Endpoints
|
||||||
|
|
||||||
An endpoint is a JSON-RPC interface that a running process serves over TCP,
|
An endpoint is a JSON-RPC interface that a running process serves over TCP,
|
||||||
@@ -279,9 +323,23 @@ for example the control interface of a web server.
|
|||||||
|
|
||||||
## 7. The journal
|
## 7. The journal
|
||||||
|
|
||||||
- An SQLite file at `/state/core/journal.sqlite`. Only the core writes it.
|
- New entries go to `/logs/core/journal.sqlite`. Only the core writes it.
|
||||||
|
An existing `/state/core/journal.sqlite` stays in place and is opened
|
||||||
|
read-only; queries merge both databases by entry ID before applying the
|
||||||
|
limit. The host never renames a live SQLite/WAL trio. Console history stays
|
||||||
|
in `/state/core`. No journal or log files are automatically pruned.
|
||||||
|
- If a journal transaction fails, the complete batch (including original
|
||||||
|
IDs and timestamps) is saved in an exclusive, synced JSON file under
|
||||||
|
`/logs/core/failed-batches/`; the host reports its path on stderr. These
|
||||||
|
files are recovery artifacts, not query results: recovery/import is
|
||||||
|
deliberate, using IDs to avoid duplicates, and never deletes the source.
|
||||||
|
If spooling also fails, entries stay in memory and retry on the next
|
||||||
|
command (no polling); at writer shutdown their full text is reported to
|
||||||
|
host stderr. Storage failure plus host loss can still lose unsaved data.
|
||||||
- Each entry has a UUIDv7 (it sorts by time), the process ID, the plugin, the
|
- Each entry has a UUIDv7 (it sorts by time), the process ID, the plugin, the
|
||||||
stream (`stderr` or `core`), and the line.
|
stream (`stderr` or `core`), and the line.
|
||||||
|
- Traps retain their full diagnostic text in the journal; short exit-status
|
||||||
|
displays are UTF-8-boundary-safe.
|
||||||
- The core writes its own events too: start (with method and params), exit
|
- The core writes its own events too: start (with method and params), exit
|
||||||
status, stop, kill, upgrades, and the warning about discarded stdout.
|
status, stop, kill, upgrades, and the warning about discarded stdout.
|
||||||
- `core.log {plugin?, pid?, since?, until?, grep?, limit?}` queries the
|
- `core.log {plugin?, pid?, since?, until?, grep?, limit?}` queries the
|
||||||
@@ -302,10 +360,29 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
|
|||||||
- A service is a detached call with the parent `core`. The core waits for
|
- A service is a detached call with the parent `core`. The core waits for
|
||||||
its result before it starts the next one. The result shows that the
|
its result before it starts the next one. The result shows that the
|
||||||
service is ready.
|
service is ready.
|
||||||
- If a service fails, the core writes the error to the journal and continues.
|
- If a service fails before answering, the core writes the error to the
|
||||||
- After an upgrade of the plugin of a service, the core starts the service
|
journal and continues booting. A failure to admit a process (for example,
|
||||||
again with the new version (section 3.7).
|
a missing plugin or invalid parameters) is logged, not retried in a loop.
|
||||||
- The core does not restart other detached processes.
|
- The core supervises the process lifetime, not its readiness response. A
|
||||||
|
nonzero exit, trap or failed host task restarts the same service with its
|
||||||
|
original method and parameters, including a crash before readiness. A
|
||||||
|
readiness response alone never triggers a restart; a clean exit does not
|
||||||
|
count as a crash or trigger an immediate restart.
|
||||||
|
- More than 5 crashes within 10 seconds gives up on that service and writes
|
||||||
|
`service gave up` to the journal. Each service has an independent sliding
|
||||||
|
window; crashes exactly 10 seconds old expire. There is no retry delay or
|
||||||
|
periodic liveness check. Giving up is latched for this host lifetime:
|
||||||
|
neither elapsed time nor an upgrade re-arms it.
|
||||||
|
- After an upgrade of the plugin of a service, the same supervisor starts
|
||||||
|
it again with the current version after its old process ends (section
|
||||||
|
3.7). Superseded exits are retirement, not crashes. Admission and cleanup
|
||||||
|
reserve a single owner/attempt, so an upgrade racing a crash cannot start
|
||||||
|
two owners. A matching boot-service start from `rpc.retired` is ignored
|
||||||
|
(returns `null`), even while restart is pending or after giving up; older
|
||||||
|
plugins' fallback starts do not bypass supervision.
|
||||||
|
- An explicit `core.stop` or `core.kill` disables automatic attempts for
|
||||||
|
that service, including during an upgrade. Host shutdown starts no new
|
||||||
|
processes. The core does not restart other detached processes.
|
||||||
- A singleton is a service with a fixed endpoint in `/run`. Only the core
|
- A singleton is a service with a fixed endpoint in `/run`. Only the core
|
||||||
starts it, so two instances cannot start at the same time.
|
starts it, so two instances cannot start at the same time.
|
||||||
|
|
||||||
@@ -313,16 +390,59 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
|
|||||||
|
|
||||||
| Method | Params | Result |
|
| Method | Params | Result |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
|
| `core.rpc.discover` | none | generated OpenRPC 1.2 document for every native core method |
|
||||||
| `core.call` | `method`, `params`, `detach`? | the result of the method |
|
| `core.call` | `method`, `params`, `detach`? | the result of the method |
|
||||||
|
| `core.watch` | `path` | stays open; `armed`, then `changed` notifications (section 4) |
|
||||||
| `core.notify` | `name`, `params` | none (a notification) |
|
| `core.notify` | `name`, `params` | none (a notification) |
|
||||||
| `core.stop` | `pid` | `{}` when the process has ended |
|
| `core.stop` | `pid` | `{}` when the process has ended |
|
||||||
| `core.kill` | `pid` | `{}` |
|
| `core.kill` | `pid` | `{}` |
|
||||||
| `core.lock` | `path`, `wait`? | `{"held": bool}` |
|
| `core.lock` | `path`, `wait`? | `{"held": bool}` |
|
||||||
| `core.unlock` | `path` | none (a notification) |
|
| `core.unlock` | `path` | none (a notification) |
|
||||||
| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents |
|
| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | short activation summaries, keyed by plugin |
|
||||||
| `core.revert` | `plugin`, `force`? | like `core.activate` |
|
| `core.revert` | `plugin`, `force`? | like `core.activate` |
|
||||||
| `core.log` | filters (section 7) | journal entries |
|
| `core.log` | filters (section 7) | journal entries |
|
||||||
| `core.exit` | none | `{}`. Then the host stops with exit code 75. |
|
| `core.restart` | none | `{}`. With `SAUSAGE_LOOP=1`, schedule clean shutdown and exit code 75. |
|
||||||
|
| `core.exit` | none | Alias of `core.restart`: same supervisor gate, response, coalescing, and exit code 75. |
|
||||||
|
|
||||||
|
`core.rpc.discover` and `/proc/core/discover.json` expose the same document,
|
||||||
|
with unprefixed method names, just like a plugin's discovery document. The
|
||||||
|
core generates it from the parameter and result types used by its dispatch;
|
||||||
|
there is no separate JSON schema file to maintain. Copy the proc file to
|
||||||
|
`imports/core.json` and use `import!("core")` to get typed calls, including
|
||||||
|
`core::lock` and `core::log`. Dotted wire names become underscores in Rust
|
||||||
|
(`rpc.discover` becomes `rpc_discover`). Import checks recognize the native
|
||||||
|
core as a provider without requiring a `plugins/core` directory. Forwarded
|
||||||
|
results and discovery documents contain arbitrary JSON; other core results
|
||||||
|
are typed, including activation summaries and journal entries.
|
||||||
|
|
||||||
|
### Host restart
|
||||||
|
|
||||||
|
**HARD RULE: the mediator and all agents NEVER call `core.restart` without
|
||||||
|
the person's explicit approval each time.** The same approval rule applies
|
||||||
|
to the legacy alias `core.exit`; it uses the same gate and pending request.
|
||||||
|
|
||||||
|
`core.restart {}` is enabled only when the host starts with `SAUSAGE_LOOP=1`.
|
||||||
|
The host captures the marker at startup. An absent marker (or any value other
|
||||||
|
than `1`) returns error -32600: `restart needs the loop supervisor
|
||||||
|
(SAUSAGE_LOOP not set); start with SAUSAGE_LOOP=1`, without scheduling an exit.
|
||||||
|
This is a supervisor marker, not a permission system.
|
||||||
|
|
||||||
|
The method returns `{}` immediately and schedules the shutdown request after
|
||||||
|
250 milliseconds to let the response reach the caller before stdin closes.
|
||||||
|
Repeated requests while a restart is pending return `{}` without scheduling
|
||||||
|
another restart or extending that delay. The executable then uses the same
|
||||||
|
`Host::stop` path as a normal stop: reject new processes, stop all processes
|
||||||
|
(including detached ones) in parallel with the normal five-second grace,
|
||||||
|
wait for outstanding file operations, release locks, and drain journal writes.
|
||||||
|
Only after that does the executable return the dedicated restart exit code
|
||||||
|
**75**. The library signals this path; it never kills its embedding process.
|
||||||
|
|
||||||
|
`nix run .#loop -- <world>` sets `SAUSAGE_LOOP=1` on the host invocation and
|
||||||
|
starts the newest host again after exit code 75 (and, as before, other nonzero
|
||||||
|
exits). Exit code 0 ends the loop. If using an older loop script, the person
|
||||||
|
must set `SAUSAGE_LOOP=1` in that supervisor's environment; do not set it for
|
||||||
|
an unsupervised host. The NixEnv service invokes this flake's `#loop`, so a
|
||||||
|
new loop script supplies the marker without changes to NixEnv.
|
||||||
|
|
||||||
Everything else is a file.
|
Everything else is a file.
|
||||||
|
|
||||||
@@ -341,7 +461,8 @@ Params are by name. The document can have these extension fields:
|
|||||||
`methods`.
|
`methods`.
|
||||||
- `x-sausage-imports`: for each provider, the part of its document that the
|
- `x-sausage-imports`: for each provider, the part of its document that the
|
||||||
plugin uses.
|
plugin uses.
|
||||||
- `x-sausage-grace`: the grace period in seconds.
|
- `x-sausage-grace`: legacy grace declaration in seconds. The core ignores
|
||||||
|
this field; upgrade grace is always 5 seconds (section 3.7).
|
||||||
|
|
||||||
The schema of params describes what the provider can read. The schema of a
|
The schema of params describes what the provider can read. The schema of a
|
||||||
result describes what the provider writes. Descriptions are part of the
|
result describes what the provider writes. Descriptions are part of the
|
||||||
@@ -364,7 +485,8 @@ imported method:
|
|||||||
- There are no version numbers. This check replaces them.
|
- There are no version numbers. This check replaces them.
|
||||||
- The check covers a fixed subset of JSON Schema: objects with `properties`
|
- The check covers a fixed subset of JSON Schema: objects with `properties`
|
||||||
and `required`, the primitive types, arrays, optional and nullable values,
|
and `required`, the primitive types, arrays, optional and nullable values,
|
||||||
string enums, integer ranges, and `$ref`. A construct outside the subset
|
string enums, integer ranges, typed dictionaries (`additionalProperties`
|
||||||
|
with a value schema), and `$ref`. A construct outside the subset
|
||||||
fails the check.
|
fails the check.
|
||||||
- The check covers calls, not state. Old and new processes share the files in
|
- The check covers calls, not state. Old and new processes share the files in
|
||||||
`/state`, and `core.revert` restores only code. A plugin must keep its
|
`/state`, and `core.revert` restores only code. A plugin must keep its
|
||||||
@@ -389,6 +511,26 @@ imported method:
|
|||||||
If a check fails, nothing changes, and the error -32003 lists each problem,
|
If a check fails, nothing changes, and the error -32003 lists each problem,
|
||||||
with a counterexample where possible.
|
with a counterexample where possible.
|
||||||
|
|
||||||
|
The result is an object keyed by plugin name. Each value contains:
|
||||||
|
|
||||||
|
- `plugin`: the plugin name.
|
||||||
|
- `wasm`: the installed world path (`/plugins/<name>/plugin.wasm`).
|
||||||
|
- `hash`: the SHA-256 of the installed copy.
|
||||||
|
- `previous`: the replaced version's SHA-256, or `null` on first installation.
|
||||||
|
For an identical-hash no-op, this is the current hash.
|
||||||
|
- `method_count`: the number of methods in the new main interface.
|
||||||
|
- `added`, `removed`, `changed`: sorted method-name arrays compared with the
|
||||||
|
previous discovery document. A change to a method description or any
|
||||||
|
transitively referenced local schema counts as changed. Unrelated schemas
|
||||||
|
and endpoint interfaces do not affect these arrays. With no previous
|
||||||
|
document, all methods are added.
|
||||||
|
- `interface_check`: `"passed"` after successful checks, or `"skipped"` with
|
||||||
|
`force: true` (not a claim of compatibility).
|
||||||
|
|
||||||
|
The result does not include schemas. The full document remains available
|
||||||
|
through `<plugin>.rpc.discover` and `/proc/core/plugins/<name>/discover.json`.
|
||||||
|
An identical-hash no-op and `core.revert` return the same summary shape.
|
||||||
|
|
||||||
- The core runs one activation at a time.
|
- The core runs one activation at a time.
|
||||||
- `plugins: [...]` activates several plugins together. The checks then use
|
- `plugins: [...]` activates several plugins together. The checks then use
|
||||||
the new set.
|
the new set.
|
||||||
@@ -418,13 +560,46 @@ builds `/plugins/notes/source/`:
|
|||||||
3. It builds the release `.wasm` for `wasm32-wasip2`. The tests and the build
|
3. It builds the release `.wasm` for `wasm32-wasip2`. The tests and the build
|
||||||
share the compiled dependencies.
|
share the compiled dependencies.
|
||||||
4. It returns `{"wasm": "/state/cargo/builds/<n>/notes.wasm", "tests":
|
4. It returns `{"wasm": "/state/cargo/builds/<n>/notes.wasm", "tests":
|
||||||
{"passed", "failed", "output"}, "warnings"}`. If a step fails, it returns
|
{"passed", "failed", "output"}, "warnings", "logs"}`. If a step fails, it returns
|
||||||
an error with the compiler or test output.
|
an error with the compiler or test output.
|
||||||
|
|
||||||
|
- Before running commands, it creates `/logs/cargo/<YYYY-MM-DD>/<build-id>.stdout.log`
|
||||||
|
and `.stderr.log` (UTC date, unique build ID). Both test and release phases
|
||||||
|
append their raw bytes directly to these files, before any parsing, progress
|
||||||
|
filtering or UTF-8 conversion. This includes successful JSON build output.
|
||||||
|
`/logs` is a top-level world folder, created if missing, not under `/state`.
|
||||||
|
- Success results and error -32004 `data` include `logs`:
|
||||||
|
`{"stdout":{"path":"/logs/cargo/...stdout.log","bytes":123},
|
||||||
|
"stderr":{"path":"/logs/cargo/...stderr.log","bytes":45},"total_bytes":168}`.
|
||||||
|
Counts are raw bytes, not characters. `logs` is null only if log creation
|
||||||
|
itself fails. Preparation failures otherwise include empty log files.
|
||||||
|
Inline diagnostics may be filtered or shortened; the files are the full truth.
|
||||||
|
- Logs remain after failure, total/test timeout, and cancellation. Cancellation
|
||||||
|
need not return a result; its files remain discoverable in `/logs/cargo/`.
|
||||||
|
Pruning numeric `/state/cargo/builds/` artifact folders does not prune logs.
|
||||||
|
- Optional `timeout_secs` is an integer of at least 1, default **900** (not
|
||||||
|
null or unlimited). The wall-clock budget starts when the build acquires its
|
||||||
|
mutex, excluding queue time, and covers tests and release compilation together.
|
||||||
|
On expiry, cargo and its descendant process group are killed and the mutex is
|
||||||
|
released. Error -32004 has reason/message `timeout`, with `data.reason`,
|
||||||
|
`data.elapsed_secs`, `data.step` (`tests` or `release build`), `data.tests`,
|
||||||
|
and `data.output` containing the last 50 output lines.
|
||||||
|
- Optional `test_timeout_secs` is an integer of at least 1, default **120**.
|
||||||
|
Each WASI test instance has this wall-clock budget. On expiry, the instance
|
||||||
|
is killed, its named test fails with `timed out after N s`, and remaining
|
||||||
|
tests continue. The overall build budget still applies.
|
||||||
|
- Cancellation kills cargo's process group, including build scripts and test
|
||||||
|
runners, before releasing the build mutex. An attached build also stops when
|
||||||
|
its caller ends (under the normal process stopping deadline).
|
||||||
- A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal
|
- A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal
|
||||||
entries, like other calls.
|
entries, like other calls.
|
||||||
- `/plugins/cargo/sdk/` shows the source of the SDK. The dependency
|
- `/plugins/cargo/sdk/` is the SDK used by builds. The dependency
|
||||||
`sausage-plugin = "0.1"` points to the same files in the host's cache.
|
`sausage-plugin = "0.1"` points directly to this folder, including its macros
|
||||||
|
and interface crate. Edits apply to the next build without a host restart.
|
||||||
|
- The host seeds a missing SDK folder from its embedded copy at startup or
|
||||||
|
before a build. An existing folder is never overwritten or repaired, even
|
||||||
|
after a host upgrade. To reset to the host's bundled SDK, remove the whole
|
||||||
|
folder; the next startup or build re-seeds it. Back up local edits first.
|
||||||
- Compiled crates stay in the host's cache, outside the world, and all worlds
|
- Compiled crates stay in the host's cache, outside the world, and all worlds
|
||||||
share them. So a second build compiles only what changed.
|
share them. So a second build compiles only what changed.
|
||||||
- Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2`
|
- Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2`
|
||||||
@@ -443,3 +618,19 @@ Tests:
|
|||||||
files through `env!("CARGO_MANIFEST_DIR")`.
|
files through `env!("CARGO_MANIFEST_DIR")`.
|
||||||
- The test runner answers `core.lock`, `core.unlock`, `core.notify`, and
|
- The test runner answers `core.lock`, `core.unlock`, `core.notify`, and
|
||||||
`core.log`. A call to a plugin fails: a test has no other plugins.
|
`core.log`. A call to a plugin fails: a test has no other plugins.
|
||||||
|
|
||||||
|
## 12. Embedded agent tool output
|
||||||
|
|
||||||
|
The agent shipped by `sausage init` uses a context-size guard of 50 KiB or
|
||||||
|
2000 lines, not a data-retention limit. Oversized tool results, including
|
||||||
|
errors, are saved in full under `/logs/agent/<date>/default/` before history
|
||||||
|
and provider delivery receive an excerpt. Overflow JSON string values are
|
||||||
|
rendered as physical text lines so the reference's one-based inclusive line
|
||||||
|
ranges can be followed with `read_file {path, line_offset, line_limit}`
|
||||||
|
(zero-based offset; default 200 lines). Huge single lines give exact byte
|
||||||
|
ranges and continuation offsets. Command/log/build/test results show the
|
||||||
|
tail; other tools show the head. Reading an existing `/logs` file excerpts
|
||||||
|
that same file without making another archive. Archives are exclusive and
|
||||||
|
never automatically deleted. If archival fails, the agent reports the error
|
||||||
|
to stderr and keeps the complete result instead of discarding it. Existing
|
||||||
|
history formats and paths are unchanged.
|
||||||
|
|||||||
@@ -45,9 +45,18 @@ nix run git+https://gitea.surma.technology/surma/sausage#loop # ./
|
|||||||
nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world
|
nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world
|
||||||
```
|
```
|
||||||
|
|
||||||
When the host crashes or a plugin calls `core.exit`, the loop fetches the
|
When the host crashes or a plugin calls `core.restart`, the loop fetches the
|
||||||
newest commit and starts it again. `quit` ends the loop. Set `SAUSAGE_FLAKE`
|
newest commit and starts it again. The loop sets `SAUSAGE_LOOP=1` to enable
|
||||||
to run another flake URL instead, for example your fork.
|
that RPC; without it, restart returns an error and leaves the host running.
|
||||||
|
Restart answers first, then cleanly stops all processes and exits with code
|
||||||
|
75. `quit` (code 0) ends the loop. Set `SAUSAGE_FLAKE` to run another flake
|
||||||
|
URL instead, for example your fork.
|
||||||
|
|
||||||
|
**The mediator and all agents NEVER call `core.restart` without the person's
|
||||||
|
explicit approval each time.** This also applies to legacy `core.exit`, an
|
||||||
|
alias with the same supervisor gate, response-before-exit behavior, and
|
||||||
|
request coalescing.
|
||||||
|
See [the protocol](PROTOCOL.md#host-restart) for the shutdown contract.
|
||||||
|
|
||||||
From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the
|
From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the
|
||||||
same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
|
same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
|
||||||
@@ -57,7 +66,7 @@ same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
|
|||||||
|
|
||||||
`--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host
|
`--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host
|
||||||
runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls
|
runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls
|
||||||
`core.exit` (exit code 75, so the loop starts the newest version):
|
`core.restart` (requires `SAUSAGE_LOOP=1`; exit code 75 restarts the loop):
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world
|
nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world
|
||||||
@@ -84,10 +93,22 @@ plugins/<name>/plugin.wasm the current version of a plugin
|
|||||||
plugins/<name>/source/ its source
|
plugins/<name>/source/ its source
|
||||||
state/<plugin>/ the state of a plugin, by convention
|
state/<plugin>/ the state of a plugin, by convention
|
||||||
state/core/journal.sqlite the journal: stderr of all processes, and core events
|
state/core/journal.sqlite the journal: stderr of all processes, and core events
|
||||||
config/boot.yaml the services that start with the world
|
config/boot.yaml ordered boot services, supervised for crashes (PROTOCOL §8)
|
||||||
config/agent.yaml the configuration of the agent
|
config/agent.yaml the configuration of the agent
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Boot services restart on process crashes, not on readiness responses. More
|
||||||
|
than 5 crashes within 10 seconds stops automatic attempts for that service
|
||||||
|
and journals `service gave up`; an upgrade does not reset that decision.
|
||||||
|
Explicitly stopping or killing a service also leaves it stopped for this
|
||||||
|
host lifetime. See [Boot and services](PROTOCOL.md#8-boot-and-services).
|
||||||
|
|
||||||
|
Rust builds use the SDK in `plugins/cargo/sdk/` directly. The host seeds this
|
||||||
|
folder only when it is missing; restarts and host upgrades preserve existing
|
||||||
|
files. SDK edits take effect on the next `cargo.build`, without a restart.
|
||||||
|
To reset to the host's bundled SDK, back up any edits and remove the whole
|
||||||
|
folder; the next startup or build re-seeds it.
|
||||||
|
|
||||||
Each process also sees `/proc` (the processes and the state of the core),
|
Each process also sees `/proc` (the processes and the state of the core),
|
||||||
`/self` (its own folder in `/proc`), and `/run` (shared files in memory).
|
`/self` (its own folder in `/proc`), and `/run` (shared files in memory).
|
||||||
|
|
||||||
|
|||||||
+28
-9
@@ -94,19 +94,38 @@ impl Agent {
|
|||||||
return Ok(final_text);
|
return Ok(final_text);
|
||||||
}
|
}
|
||||||
for (id, name, arguments) in calls {
|
for (id, name, arguments) in calls {
|
||||||
let output = match serde_json::from_str(&arguments) {
|
let args = serde_json::from_str::<serde_json::Value>(&arguments);
|
||||||
Ok(args) => tools::run(&name, &args)
|
let output = match &args {
|
||||||
|
Ok(args) => tools::run(&name, args)
|
||||||
.await
|
.await
|
||||||
.unwrap_or_else(|error| json!({"error":error})),
|
.unwrap_or_else(|error| json!({"error":error})),
|
||||||
Err(error) => json!({"error":format!("arguments are not valid JSON: {error}")}),
|
Err(error) => json!({"error":format!("arguments are not valid JSON: {error}")}),
|
||||||
};
|
};
|
||||||
append(
|
let output = if name == "read_file"
|
||||||
&mut entries,
|
&& args
|
||||||
Entry::ToolResult {
|
.as_ref()
|
||||||
id,
|
.ok()
|
||||||
output: tools::limit_output(output.to_string()),
|
.and_then(|a| a["path"].as_str())
|
||||||
},
|
.is_some_and(|p| p.starts_with("/logs/"))
|
||||||
)?;
|
{
|
||||||
|
output["content"]
|
||||||
|
.as_str()
|
||||||
|
.map(str::to_owned)
|
||||||
|
.unwrap_or_else(|| output.to_string())
|
||||||
|
} else {
|
||||||
|
output.to_string()
|
||||||
|
};
|
||||||
|
let tail = args
|
||||||
|
.as_ref()
|
||||||
|
.ok()
|
||||||
|
.and_then(|a| a["method"].as_str())
|
||||||
|
.is_some_and(|m| {
|
||||||
|
[".exec", ".log", ".build", ".test"]
|
||||||
|
.iter()
|
||||||
|
.any(|suffix| m.ends_with(suffix))
|
||||||
|
});
|
||||||
|
let output = tools::overflow::guard(output, "default", &id, tail);
|
||||||
|
append(&mut entries, Entry::ToolResult { id, output })?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,258 @@
|
|||||||
|
//! Context-size guard; full output is in the referenced file. Never a data limit.
|
||||||
|
use std::io::{self, Write};
|
||||||
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
|
|
||||||
|
// One policy point: Pi's 50 KiB / 2000 lines, including room for the reference.
|
||||||
|
const CONTEXT_LIMIT: (usize, usize) = (50 * 1024, 2000);
|
||||||
|
static NEXT: AtomicU64 = AtomicU64::new(0);
|
||||||
|
|
||||||
|
fn component(s: &str) -> String {
|
||||||
|
s.bytes()
|
||||||
|
.take(80)
|
||||||
|
.map(|b| {
|
||||||
|
if b.is_ascii_alphanumeric() || b == b'-' || b == b'_' {
|
||||||
|
b as char
|
||||||
|
} else {
|
||||||
|
'_'
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn exceeds_limit(output: &str) -> bool {
|
||||||
|
output.len() > CONTEXT_LIMIT.0 || output.lines().count() > CONTEXT_LIMIT.1
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Save before excerpting. On storage failure return all data, not a false reference.
|
||||||
|
pub fn guard(output: String, session: &str, id: &str, tail: bool) -> String {
|
||||||
|
if !exceeds_limit(&output) {
|
||||||
|
return output;
|
||||||
|
}
|
||||||
|
// Decode only results that crossed the context guard. Archive physical
|
||||||
|
// string lines, not JSON escapes, even if decoding makes the text smaller.
|
||||||
|
let output = serde_json::from_str::<serde_json::Value>(&output)
|
||||||
|
.map(|value| super::render(&value))
|
||||||
|
.unwrap_or(output);
|
||||||
|
let pid = std::env::var("SAUSAGE_PID").unwrap_or_else(|_| "test".into());
|
||||||
|
let dir = format!("/logs/agent/{}/{}", date(), component(session));
|
||||||
|
let saved = (|| -> io::Result<String> {
|
||||||
|
std::fs::create_dir_all(&dir)?;
|
||||||
|
// create_new prevents overwriting even if an ID is reused or a process restarts.
|
||||||
|
loop {
|
||||||
|
let path = format!(
|
||||||
|
"{dir}/{}-{}-{}.txt",
|
||||||
|
component(id),
|
||||||
|
component(&pid),
|
||||||
|
NEXT.fetch_add(1, Ordering::Relaxed)
|
||||||
|
);
|
||||||
|
match std::fs::OpenOptions::new()
|
||||||
|
.write(true)
|
||||||
|
.create_new(true)
|
||||||
|
.open(&path)
|
||||||
|
{
|
||||||
|
Ok(mut f) => {
|
||||||
|
f.write_all(output.as_bytes())?;
|
||||||
|
return Ok(path);
|
||||||
|
}
|
||||||
|
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
|
||||||
|
Err(e) => return Err(e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
match saved {
|
||||||
|
Ok(path) => excerpt(&output, &path, tail, 0, output.len()),
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("overflow archive failed: {e}; retaining complete tool result");
|
||||||
|
output
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn date() -> String {
|
||||||
|
let days = std::time::SystemTime::now()
|
||||||
|
.duration_since(std::time::UNIX_EPOCH)
|
||||||
|
.unwrap_or_default()
|
||||||
|
.as_secs()
|
||||||
|
/ 86400;
|
||||||
|
let z = days as i64 + 719468;
|
||||||
|
let era = z / 146097;
|
||||||
|
let doe = z - era * 146097;
|
||||||
|
let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365;
|
||||||
|
let y = yoe + era * 400;
|
||||||
|
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
|
||||||
|
let mp = (5 * doy + 2) / 153;
|
||||||
|
let d = doy - (153 * mp + 2) / 5 + 1;
|
||||||
|
let m = mp + if mp < 10 { 3 } else { -9 };
|
||||||
|
format!("{:04}-{m:02}-{d:02}", y + i64::from(m <= 2))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Excerpt an existing archive in place. Byte/line ranges refer to that file,
|
||||||
|
/// including when a caller requests an oversized line or the entire archive.
|
||||||
|
pub fn existing(bytes: &[u8], path: &str, args: &serde_json::Value) -> String {
|
||||||
|
let text = String::from_utf8_lossy(bytes);
|
||||||
|
let (mut from, mut to) = if let Some(line) = args["line_offset"].as_u64() {
|
||||||
|
let from = text
|
||||||
|
.split_inclusive('\n')
|
||||||
|
.take(line as usize)
|
||||||
|
.map(str::len)
|
||||||
|
.sum::<usize>();
|
||||||
|
let len = text[from..]
|
||||||
|
.split_inclusive('\n')
|
||||||
|
.take(args["line_limit"].as_u64().unwrap_or(200) as usize)
|
||||||
|
.map(str::len)
|
||||||
|
.sum::<usize>();
|
||||||
|
(from, from + len)
|
||||||
|
} else {
|
||||||
|
let from = (args["offset"].as_u64().unwrap_or(0) as usize).min(text.len());
|
||||||
|
(
|
||||||
|
from,
|
||||||
|
args["max_bytes"].as_u64().map_or(text.len(), |n| {
|
||||||
|
from.saturating_add(n as usize).min(text.len())
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
};
|
||||||
|
while !text.is_char_boundary(from) {
|
||||||
|
from += 1;
|
||||||
|
}
|
||||||
|
while to > from && !text.is_char_boundary(to) {
|
||||||
|
to -= 1;
|
||||||
|
}
|
||||||
|
to = to.max(from);
|
||||||
|
excerpt(&text, path, false, from, to)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn excerpt(output: &str, path: &str, tail: bool, from: usize, to: usize) -> String {
|
||||||
|
let selected = &output[from..to];
|
||||||
|
let lines: Vec<&str> = selected.split_inclusive('\n').collect();
|
||||||
|
let budget = CONTEXT_LIMIT.0.saturating_sub(1024);
|
||||||
|
let mut start = 0;
|
||||||
|
let mut end = lines.len();
|
||||||
|
let mut bytes = 0;
|
||||||
|
if tail {
|
||||||
|
start = end;
|
||||||
|
while start > 0
|
||||||
|
&& end - start < CONTEXT_LIMIT.1.saturating_sub(2)
|
||||||
|
&& bytes + lines[start - 1].len() <= budget
|
||||||
|
{
|
||||||
|
start -= 1;
|
||||||
|
bytes += lines[start].len();
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
end = 0;
|
||||||
|
while end < lines.len()
|
||||||
|
&& end < CONTEXT_LIMIT.1.saturating_sub(2)
|
||||||
|
&& bytes + lines[end].len() <= budget
|
||||||
|
{
|
||||||
|
bytes += lines[end].len();
|
||||||
|
end += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut shown = lines[start..end].concat();
|
||||||
|
// A single huge line: preserve valid UTF-8 and explicitly label a partial line.
|
||||||
|
let partial = start == end;
|
||||||
|
if partial && !lines.is_empty() {
|
||||||
|
if tail {
|
||||||
|
start = lines.len() - 1;
|
||||||
|
let line = lines[start];
|
||||||
|
let mut from = line.len().saturating_sub(budget);
|
||||||
|
while !line.is_char_boundary(from) {
|
||||||
|
from += 1;
|
||||||
|
}
|
||||||
|
shown.push_str(&line[from..]);
|
||||||
|
} else {
|
||||||
|
let mut to = budget.min(lines[0].len());
|
||||||
|
while !lines[0].is_char_boundary(to) {
|
||||||
|
to -= 1;
|
||||||
|
}
|
||||||
|
shown.push_str(&lines[0][..to]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut byte_start = from + lines[..start].iter().map(|s| s.len()).sum::<usize>();
|
||||||
|
if partial && tail {
|
||||||
|
byte_start = to - shown.len();
|
||||||
|
}
|
||||||
|
let byte_end = byte_start + shown.len();
|
||||||
|
let line_start = output[..byte_start].bytes().filter(|b| *b == b'\n').count();
|
||||||
|
let line_end = line_start + shown.split_inclusive('\n').count();
|
||||||
|
let total_lines = output.split_inclusive('\n').count();
|
||||||
|
let partial = partial
|
||||||
|
|| (from > 0 && !output[..from].ends_with('\n'))
|
||||||
|
|| (to < output.len() && !output[..to].ends_with('\n'));
|
||||||
|
let read = if partial {
|
||||||
|
format!(
|
||||||
|
"Shown byte range [{byte_start},{byte_end}); continue with read_file path={path}, offset={byte_end}, max_bytes={budget} (or choose an earlier byte offset)"
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
format!(
|
||||||
|
"Read with read_file path={path}, line_offset={line_start}, line_limit={}; next line_offset={line_end}",
|
||||||
|
line_end - line_start
|
||||||
|
)
|
||||||
|
};
|
||||||
|
format!(
|
||||||
|
"{shown}\n[Showing lines {}-{} of {}{}; total {} bytes. Full output: {path}. {read}]",
|
||||||
|
line_start + 1,
|
||||||
|
line_end,
|
||||||
|
total_lines,
|
||||||
|
if partial { " (partial line)" } else { "" },
|
||||||
|
output.len()
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Zero-based line offsets; byte-based reads remain unchanged when absent.
|
||||||
|
pub fn read_lines(bytes: &[u8], offset: usize, limit: usize) -> String {
|
||||||
|
String::from_utf8_lossy(bytes)
|
||||||
|
.split_inclusive('\n')
|
||||||
|
.skip(offset)
|
||||||
|
.take(limit)
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
#[test]
|
||||||
|
fn full_archive_and_context_guard() {
|
||||||
|
let original = "😀 line\n".repeat(5400);
|
||||||
|
for tail in [false, true] {
|
||||||
|
let shown = guard(original.clone(), "guard-test", "call", tail);
|
||||||
|
assert!(shown.len() <= CONTEXT_LIMIT.0);
|
||||||
|
assert!(shown.lines().count() <= CONTEXT_LIMIT.1);
|
||||||
|
assert!(shown.contains(if tail {
|
||||||
|
"Showing lines 3403-5400"
|
||||||
|
} else {
|
||||||
|
"Showing lines 1-1998"
|
||||||
|
}));
|
||||||
|
let path = shown
|
||||||
|
.split("Full output: ")
|
||||||
|
.nth(1)
|
||||||
|
.unwrap()
|
||||||
|
.split(". Read with")
|
||||||
|
.next()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(std::fs::read_to_string(path).unwrap(), original);
|
||||||
|
}
|
||||||
|
assert_eq!(guard("small".into(), "test", "id", false), "small");
|
||||||
|
}
|
||||||
|
#[test]
|
||||||
|
fn archive_failure_does_not_discard_data() {
|
||||||
|
std::fs::create_dir_all(format!("/logs/agent/{}", date())).unwrap();
|
||||||
|
std::fs::write(format!("/logs/agent/{}/not-a-directory", date()), "block").unwrap();
|
||||||
|
let output = "no loss\n".repeat(6000);
|
||||||
|
assert_eq!(
|
||||||
|
guard(output.clone(), "not-a-directory", "test", false),
|
||||||
|
output
|
||||||
|
);
|
||||||
|
}
|
||||||
|
#[test]
|
||||||
|
fn huge_line_and_exact_boundaries() {
|
||||||
|
for tail in [false, true] {
|
||||||
|
let result = guard("😀".repeat(40000), "test", "utf8", tail);
|
||||||
|
assert!(result.len() <= CONTEXT_LIMIT.0);
|
||||||
|
assert!(result.contains("partial line"));
|
||||||
|
}
|
||||||
|
let exact = "a\n".repeat(CONTEXT_LIMIT.1);
|
||||||
|
assert_eq!(guard(exact.clone(), "test", "exact", false), exact);
|
||||||
|
assert_eq!(read_lines(b"one\ntwo\nthree", 1, 1), "two\n");
|
||||||
|
assert_eq!(read_lines(b"one", 3, 1), "");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,6 +3,8 @@ You are the Sausage agent. Answer the person. Use tools to inspect or change the
|
|||||||
Read the contract at `/proc/core/PROTOCOL.md`. The Rust plugin SDK and its documentation are at `/plugins/cargo/sdk/`.
|
Read the contract at `/proc/core/PROTOCOL.md`. The Rust plugin SDK and its documentation are at `/plugins/cargo/sdk/`.
|
||||||
Build a plugin with `cargo.build`. Activate its WASM file with `core.activate`.
|
Build a plugin with `cargo.build`. Activate its WASM file with `core.activate`.
|
||||||
Your own source is at `/plugins/agent/source/`. You can change and rebuild it like any other plugin.
|
Your own source is at `/plugins/agent/source/`. You can change and rebuild it like any other plugin.
|
||||||
|
HARD RULE: the mediator and all agents NEVER call `core.restart` (or its alias `core.exit`) without the person's explicit approval each time.
|
||||||
|
Restart requires the host to run under `#loop` (`SAUSAGE_LOOP=1`); it answers before clean shutdown with exit code 75.
|
||||||
Never call `agent.chat`. It would wait on the history lock that your current turn holds.
|
Never call `agent.chat`. It would wait on the history lock that your current turn holds.
|
||||||
Write logs to stderr. Use call results and notifications for communication.
|
Write logs to stderr. Use call results and notifications for communication.
|
||||||
The interface check does not cover persistent state. Preserve state compatibility when you change a plugin.
|
The interface check does not cover persistent state. Preserve state compatibility when you change a plugin.
|
||||||
|
|||||||
+85
-23
@@ -4,21 +4,47 @@ use crate::llm::Tool;
|
|||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
use std::io;
|
use std::io;
|
||||||
|
|
||||||
/// Maximum UTF-8 bytes in a serialized tool result, including its truncation marker.
|
#[path = "overflow.rs"]
|
||||||
pub const MAX_OUTPUT_BYTES: usize = 65_536;
|
pub mod overflow;
|
||||||
|
|
||||||
/// Bound every tool result before it enters history or a provider request.
|
/// Human-readable, lossless text representation of tool JSON. String values
|
||||||
pub fn limit_output(mut output: String) -> String {
|
/// retain real newlines, with JSON-quoted field/index labels for structure.
|
||||||
const MARKER: &str = "\n[tool output truncated at 65536 bytes]";
|
fn render(value: &Value) -> String {
|
||||||
if output.len() > MAX_OUTPUT_BYTES {
|
fn visit(value: &Value, label: &str, out: &mut String) {
|
||||||
let mut end = MAX_OUTPUT_BYTES - MARKER.len();
|
match value {
|
||||||
while !output.is_char_boundary(end) {
|
Value::Object(fields) => {
|
||||||
end -= 1;
|
if fields.is_empty() {
|
||||||
|
out.push_str(&format!("{label}: {{}}\n"));
|
||||||
}
|
}
|
||||||
output.truncate(end);
|
for (key, value) in fields {
|
||||||
output.push_str(MARKER);
|
visit(
|
||||||
|
value,
|
||||||
|
&format!("{label}[{}]", serde_json::to_string(key).unwrap()),
|
||||||
|
out,
|
||||||
|
);
|
||||||
}
|
}
|
||||||
output
|
}
|
||||||
|
Value::Array(items) => {
|
||||||
|
if items.is_empty() {
|
||||||
|
out.push_str(&format!("{label}: []\n"));
|
||||||
|
}
|
||||||
|
for (i, value) in items.iter().enumerate() {
|
||||||
|
visit(value, &format!("{label}[{i}]"), out);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Value::String(text) => {
|
||||||
|
out.push_str(&format!("{label} (string, {} bytes):\n", text.len()));
|
||||||
|
out.push_str(text);
|
||||||
|
out.push('\n');
|
||||||
|
}
|
||||||
|
_ => {
|
||||||
|
out.push_str(&format!("{label}: {value}\n"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut out = String::new();
|
||||||
|
visit(value, "$", &mut out);
|
||||||
|
out
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Describe the five tools sent to either inference API.
|
/// Describe the five tools sent to either inference API.
|
||||||
@@ -26,7 +52,7 @@ pub fn definitions() -> Vec<Tool> {
|
|||||||
vec![
|
vec![
|
||||||
Tool {
|
Tool {
|
||||||
name: "call",
|
name: "call",
|
||||||
description: "Call a plugin method through the core. Await its result. Set detach to let its process survive this caller. Output is capped at 65536 bytes with a truncation marker.",
|
description: "Call a plugin method through the core. Await its result. Set detach to let its process survive this caller. Large results show an excerpt with a full-output /logs reference.",
|
||||||
parameters: schema(
|
parameters: schema(
|
||||||
json!({
|
json!({
|
||||||
"method": {"type": "string", "description": "Qualified plugin method"},
|
"method": {"type": "string", "description": "Qualified plugin method"},
|
||||||
@@ -38,19 +64,21 @@ pub fn definitions() -> Vec<Tool> {
|
|||||||
},
|
},
|
||||||
Tool {
|
Tool {
|
||||||
name: "read_file",
|
name: "read_file",
|
||||||
description: "Read a text file in the world, including `/proc`. `offset` and `max_bytes` are optional. Output is capped at 65536 bytes with a truncation marker.",
|
description: "Read a text file in the world, including `/proc`. `offset`/`max_bytes` select bytes; `line_offset`/`line_limit` select zero-based lines (default 200). Large results show an excerpt with a full-output /logs reference.",
|
||||||
parameters: schema(
|
parameters: schema(
|
||||||
json!({
|
json!({
|
||||||
"path": {"type": "string"},
|
"path": {"type": "string"},
|
||||||
"offset": {"type": "integer", "description": "Byte offset (default 0)"},
|
"offset": {"type": "integer", "description": "Byte offset (default 0)"},
|
||||||
"max_bytes": {"type": "integer", "description": "Optional maximum number of bytes to read"}
|
"max_bytes": {"type": "integer", "description": "Optional maximum number of bytes to read"},
|
||||||
|
"line_offset": {"type": "integer", "description": "Zero-based starting line; overrides bytes"},
|
||||||
|
"line_limit": {"type": "integer", "description": "Number of lines; default 200"}
|
||||||
}),
|
}),
|
||||||
&["path"],
|
&["path"],
|
||||||
),
|
),
|
||||||
},
|
},
|
||||||
Tool {
|
Tool {
|
||||||
name: "write_file",
|
name: "write_file",
|
||||||
description: "Write a text file in the world. Parent folders are created. Output is capped at 65536 bytes with a truncation marker.",
|
description: "Write a text file in the world. Parent folders are created. Large results show an excerpt with a full-output /logs reference.",
|
||||||
parameters: schema(
|
parameters: schema(
|
||||||
json!({"path": {"type": "string"}, "content": {"type": "string"}}),
|
json!({"path": {"type": "string"}, "content": {"type": "string"}}),
|
||||||
&["path", "content"],
|
&["path", "content"],
|
||||||
@@ -58,12 +86,12 @@ pub fn definitions() -> Vec<Tool> {
|
|||||||
},
|
},
|
||||||
Tool {
|
Tool {
|
||||||
name: "list_dir",
|
name: "list_dir",
|
||||||
description: "List entries in a world directory. Returns each name, type, and size. Output is capped at 65536 bytes with a truncation marker.",
|
description: "List entries in a world directory. Returns each name, type, and size. Large results show an excerpt with a full-output /logs reference.",
|
||||||
parameters: schema(json!({"path": {"type": "string"}}), &["path"]),
|
parameters: schema(json!({"path": {"type": "string"}}), &["path"]),
|
||||||
},
|
},
|
||||||
Tool {
|
Tool {
|
||||||
name: "copy_dir",
|
name: "copy_dir",
|
||||||
description: "Copy a directory tree, for example a plugin source to a draft folder. Skips `target/` folders. Output is capped at 65536 bytes with a truncation marker.",
|
description: "Copy a directory tree, for example a plugin source to a draft folder. Skips `target/` folders. Large results show an excerpt with a full-output /logs reference.",
|
||||||
parameters: schema(
|
parameters: schema(
|
||||||
json!({"from": {"type": "string"}, "to": {"type": "string"}}),
|
json!({"from": {"type": "string"}, "to": {"type": "string"}}),
|
||||||
&["from", "to"],
|
&["from", "to"],
|
||||||
@@ -103,6 +131,16 @@ pub async fn run(name: &str, args: &Value) -> sausage_plugin::Result<Value> {
|
|||||||
fn read_file(args: &Value) -> io::Result<Value> {
|
fn read_file(args: &Value) -> io::Result<Value> {
|
||||||
let path = args["path"].as_str().unwrap_or_default();
|
let path = args["path"].as_str().unwrap_or_default();
|
||||||
let bytes = std::fs::read(path)?;
|
let bytes = std::fs::read(path)?;
|
||||||
|
if path.starts_with("/logs/") {
|
||||||
|
return Ok(
|
||||||
|
json!({"path":path,"size":bytes.len(),"content":overflow::existing(&bytes, path, args)}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if let Some(line) = args["line_offset"].as_u64() {
|
||||||
|
return Ok(
|
||||||
|
json!({"path":path,"size":bytes.len(),"line_offset":line,"content":overflow::read_lines(&bytes, line as usize, args["line_limit"].as_u64().unwrap_or(200) as usize)}),
|
||||||
|
);
|
||||||
|
}
|
||||||
let offset = (args["offset"].as_u64().unwrap_or(0) as usize).min(bytes.len());
|
let offset = (args["offset"].as_u64().unwrap_or(0) as usize).min(bytes.len());
|
||||||
let end = args["max_bytes"].as_u64().map_or(bytes.len(), |max| {
|
let end = args["max_bytes"].as_u64().map_or(bytes.len(), |max| {
|
||||||
offset.saturating_add(max as usize).min(bytes.len())
|
offset.saturating_add(max as usize).min(bytes.len())
|
||||||
@@ -185,11 +223,35 @@ fn copy_tree(from: &str, to: &str) -> io::Result<u64> {
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
#[test]
|
#[test]
|
||||||
fn tool_output_limit_preserves_utf8_and_marks_truncation() {
|
fn overflow_json_can_be_followed_by_line_without_rearchiving() {
|
||||||
let limited = super::limit_output("😀".repeat(40_000));
|
let value = serde_json::json!({"stdout":"hello 😀\n".repeat(4000),"exit":1,"error":{"text":"failure\n".repeat(4000)}});
|
||||||
assert!(limited.len() <= super::MAX_OUTPUT_BYTES);
|
let expected = super::render(&value);
|
||||||
assert!(limited.ends_with("[tool output truncated at 65536 bytes]"));
|
let shown = super::overflow::guard(value.to_string(), "json-test", "call", true);
|
||||||
assert_eq!(super::limit_output("small".into()), "small");
|
assert!(shown.len() <= 50 * 1024 && shown.lines().count() <= 2000);
|
||||||
|
let path = shown
|
||||||
|
.split("Full output: ")
|
||||||
|
.nth(1)
|
||||||
|
.unwrap()
|
||||||
|
.split(". Read with")
|
||||||
|
.next()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(std::fs::read_to_string(path).unwrap(), expected);
|
||||||
|
let parent = std::path::Path::new(path).parent().unwrap();
|
||||||
|
let count = std::fs::read_dir(parent).unwrap().count();
|
||||||
|
let args = serde_json::json!({"path":path,"line_offset":30,"line_limit":5});
|
||||||
|
let read = super::read_file(&args).unwrap();
|
||||||
|
let read = read["content"].as_str().unwrap();
|
||||||
|
assert!(read.contains("Showing lines 31-35"));
|
||||||
|
assert!(read.contains(path));
|
||||||
|
assert_eq!(std::fs::read_dir(parent).unwrap().count(), count);
|
||||||
|
assert!(
|
||||||
|
super::overflow::existing(
|
||||||
|
expected.as_bytes(),
|
||||||
|
path,
|
||||||
|
&serde_json::json!({"line_offset":u64::MAX})
|
||||||
|
)
|
||||||
|
.contains("Full output:")
|
||||||
|
);
|
||||||
}
|
}
|
||||||
#[test]
|
#[test]
|
||||||
fn copy_rejects_a_destination_inside_the_source() {
|
fn copy_rejects_a_destination_inside_the_source() {
|
||||||
|
|||||||
@@ -17,6 +17,8 @@ http-body-util = "0.1"
|
|||||||
include_dir = "0.7"
|
include_dir = "0.7"
|
||||||
serde = { version = "1", features = ["derive"] }
|
serde = { version = "1", features = ["derive"] }
|
||||||
serde_json = "1"
|
serde_json = "1"
|
||||||
|
schemars = "1"
|
||||||
|
sausage-plugin = { path = "../sdk" }
|
||||||
serde_yaml_ng = "0.10"
|
serde_yaml_ng = "0.10"
|
||||||
tokio = { version = "1", features = ["full"] }
|
tokio = { version = "1", features = ["full"] }
|
||||||
toml_edit = "0.25"
|
toml_edit = "0.25"
|
||||||
@@ -36,6 +38,7 @@ chrono = { version = "0.4.42", default-features = false, features = ["clock"] }
|
|||||||
tempfile = "3.27.0"
|
tempfile = "3.27.0"
|
||||||
|
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
|
tokio = { version = "1", features = ["test-util"] }
|
||||||
hyper = { version = "1.12.0", features = ["server", "http1"] }
|
hyper = { version = "1.12.0", features = ["server", "http1"] }
|
||||||
hyper-util = { version = "0.1.21", features = ["tokio"] }
|
hyper-util = { version = "0.1.21", features = ["tokio"] }
|
||||||
sausage-interface = { path = "../interface" }
|
sausage-interface = { path = "../interface" }
|
||||||
|
|||||||
+108
-12
@@ -1,11 +1,51 @@
|
|||||||
//! Serialized candidate-copy activation and revert under PROTOCOL.md section 10.3.
|
//! Serialized candidate-copy activation and revert under PROTOCOL.md section 10.3.
|
||||||
use crate::{
|
use crate::{
|
||||||
Host, RpcResult, error,
|
Host, RpcResult,
|
||||||
|
core::{ActivationSummary, InterfaceCheck},
|
||||||
|
error,
|
||||||
graph::{Interface, io_error, problems, read_hashed},
|
graph::{Interface, io_error, problems, read_hashed},
|
||||||
};
|
};
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
use std::{collections::BTreeMap, path::PathBuf};
|
use std::{collections::BTreeMap, path::PathBuf};
|
||||||
|
|
||||||
|
/// Method descriptions plus their reachable local references, without expanding cycles.
|
||||||
|
fn methods(document: &Value) -> BTreeMap<String, Value> {
|
||||||
|
fn references(value: &Value, document: &Value, found: &mut BTreeMap<String, Value>) {
|
||||||
|
match value {
|
||||||
|
Value::Object(object) => {
|
||||||
|
if let Some(reference) = object.get("$ref").and_then(Value::as_str)
|
||||||
|
&& let Some(pointer) = reference.strip_prefix('#')
|
||||||
|
&& !found.contains_key(reference)
|
||||||
|
&& let Some(target) = document.pointer(pointer)
|
||||||
|
{
|
||||||
|
found.insert(reference.into(), target.clone());
|
||||||
|
references(target, document, found);
|
||||||
|
}
|
||||||
|
for child in object.values() {
|
||||||
|
references(child, document, found);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Value::Array(array) => {
|
||||||
|
for child in array {
|
||||||
|
references(child, document, found);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
document["methods"]
|
||||||
|
.as_array()
|
||||||
|
.into_iter()
|
||||||
|
.flatten()
|
||||||
|
.filter_map(|method| {
|
||||||
|
let name = method["name"].as_str()?;
|
||||||
|
let mut found = BTreeMap::new();
|
||||||
|
references(method, document, &mut found);
|
||||||
|
Some((name.into(), json!({"method":method,"references":found})))
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
struct Candidate {
|
struct Candidate {
|
||||||
plugin: String,
|
plugin: String,
|
||||||
copy: PathBuf,
|
copy: PathBuf,
|
||||||
@@ -113,7 +153,31 @@ impl Host {
|
|||||||
.map(|candidate| candidate.plugin.clone())
|
.map(|candidate| candidate.plugin.clone())
|
||||||
.collect();
|
.collect();
|
||||||
self.refresh_graph_except(&names).await?;
|
self.refresh_graph_except(&names).await?;
|
||||||
let previous = self.0.graph.lock().unwrap().plugins.clone();
|
let mut previous = self.0.graph.lock().unwrap().plugins.clone();
|
||||||
|
// Startup discovery may not yet have visited a replaced plugin. Discover
|
||||||
|
// its immutable old copy for the diff, but never prevent repairing it.
|
||||||
|
for candidate in candidates.iter() {
|
||||||
|
if let Some(old) = previous.get_mut(&candidate.plugin)
|
||||||
|
&& old.document.is_null()
|
||||||
|
&& old.error.is_none()
|
||||||
|
{
|
||||||
|
if old.hash == candidate.interface.hash {
|
||||||
|
old.document = candidate.interface.document.clone();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let directory = self.0.world.plugin_dir(&candidate.plugin);
|
||||||
|
if let Ok((bytes, hash)) = read_hashed(directory.join("plugin.wasm")).await {
|
||||||
|
let copy = directory.join(format!("discover-{}.wasm", uuid::Uuid::now_v7()));
|
||||||
|
if tokio::fs::write(©, bytes).await.is_ok()
|
||||||
|
&& let Ok(interface) =
|
||||||
|
self.discover_copy(&candidate.plugin, ©, hash).await
|
||||||
|
{
|
||||||
|
*old = interface;
|
||||||
|
}
|
||||||
|
let _ = tokio::fs::remove_file(copy).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
let mut proposed = previous.clone();
|
let mut proposed = previous.clone();
|
||||||
for candidate in candidates.iter() {
|
for candidate in candidates.iter() {
|
||||||
proposed.insert(candidate.plugin.clone(), candidate.interface.clone());
|
proposed.insert(candidate.plugin.clone(), candidate.interface.clone());
|
||||||
@@ -148,7 +212,7 @@ impl Host {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
self.install_candidates(candidates, previous, proposed)
|
self.install_candidates(candidates, previous, proposed, params["force"] != true)
|
||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
/// Installs staged files and snapshots all old admissions under the shared write gate.
|
/// Installs staged files and snapshots all old admissions under the shared write gate.
|
||||||
@@ -157,6 +221,7 @@ impl Host {
|
|||||||
candidates: &[Candidate],
|
candidates: &[Candidate],
|
||||||
previous: BTreeMap<String, Interface>,
|
previous: BTreeMap<String, Interface>,
|
||||||
proposed: BTreeMap<String, Interface>,
|
proposed: BTreeMap<String, Interface>,
|
||||||
|
checked: bool,
|
||||||
) -> RpcResult {
|
) -> RpcResult {
|
||||||
let admission = self.0.admission.write().await;
|
let admission = self.0.admission.write().await;
|
||||||
// Every candidate already has a temporary name before any installed file changes.
|
// Every candidate already has a temporary name before any installed file changes.
|
||||||
@@ -183,25 +248,56 @@ impl Host {
|
|||||||
.map_err(io_error)?;
|
.map_err(io_error)?;
|
||||||
}
|
}
|
||||||
let mut upgrades = Vec::new();
|
let mut upgrades = Vec::new();
|
||||||
let mut documents = serde_json::Map::new();
|
let mut summaries = BTreeMap::new();
|
||||||
for candidate in candidates {
|
for candidate in candidates {
|
||||||
documents.insert(
|
let old = previous.get(&candidate.plugin);
|
||||||
|
let before = methods(old.map(|old| &old.document).unwrap_or(&Value::Null));
|
||||||
|
let after = methods(&candidate.interface.document);
|
||||||
|
let added: Vec<_> = after
|
||||||
|
.keys()
|
||||||
|
.filter(|name| !before.contains_key(*name))
|
||||||
|
.cloned()
|
||||||
|
.collect();
|
||||||
|
let removed: Vec<_> = before
|
||||||
|
.keys()
|
||||||
|
.filter(|name| !after.contains_key(*name))
|
||||||
|
.cloned()
|
||||||
|
.collect();
|
||||||
|
let changed: Vec<_> = after
|
||||||
|
.iter()
|
||||||
|
.filter(|(name, method)| before.get(*name).is_some_and(|old| old != *method))
|
||||||
|
.map(|(name, _)| name.clone())
|
||||||
|
.collect();
|
||||||
|
summaries.insert(
|
||||||
candidate.plugin.clone(),
|
candidate.plugin.clone(),
|
||||||
candidate.interface.document.clone(),
|
ActivationSummary {
|
||||||
|
plugin: candidate.plugin.clone(),
|
||||||
|
wasm: format!("/plugins/{}/plugin.wasm", candidate.plugin),
|
||||||
|
hash: candidate.interface.hash.clone(),
|
||||||
|
previous: old
|
||||||
|
.map(|old| old.hash.clone())
|
||||||
|
.filter(|hash| !hash.is_empty()),
|
||||||
|
method_count: candidate.interface.document["methods"]
|
||||||
|
.as_array()
|
||||||
|
.map_or(0, Vec::len),
|
||||||
|
added,
|
||||||
|
removed,
|
||||||
|
changed,
|
||||||
|
interface_check: if checked {
|
||||||
|
InterfaceCheck::Passed
|
||||||
|
} else {
|
||||||
|
InterfaceCheck::Skipped
|
||||||
|
},
|
||||||
|
},
|
||||||
);
|
);
|
||||||
if let Some(old) = previous
|
if let Some(old) = previous
|
||||||
.get(&candidate.plugin)
|
.get(&candidate.plugin)
|
||||||
.filter(|old| !old.hash.is_empty() && old.hash != candidate.interface.hash)
|
.filter(|old| !old.hash.is_empty() && old.hash != candidate.interface.hash)
|
||||||
{
|
{
|
||||||
let grace = old.document["x-sausage-grace"]
|
|
||||||
.as_f64()
|
|
||||||
.filter(|value| value.is_finite() && *value >= 0.0)
|
|
||||||
.unwrap_or(60.0);
|
|
||||||
upgrades.push(self.snapshot_upgrade(
|
upgrades.push(self.snapshot_upgrade(
|
||||||
&candidate.plugin,
|
&candidate.plugin,
|
||||||
old.hash.clone(),
|
old.hash.clone(),
|
||||||
&candidate.interface.hash,
|
&candidate.interface.hash,
|
||||||
grace,
|
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
self.0.journal.write(
|
self.0.journal.write(
|
||||||
@@ -216,6 +312,6 @@ impl Host {
|
|||||||
for upgrade in upgrades {
|
for upgrade in upgrades {
|
||||||
self.begin_upgrade(upgrade);
|
self.begin_upgrade(upgrade);
|
||||||
}
|
}
|
||||||
Ok(Value::Object(documents))
|
Ok(json!(summaries))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,6 +2,9 @@
|
|||||||
# Each method is a plugin.method name. params contains its named arguments.
|
# Each method is a plugin.method name. params contains its named arguments.
|
||||||
# The core waits for each result before it starts the next service.
|
# The core waits for each result before it starts the next service.
|
||||||
# After a plugin upgrade, the core restarts that plugin's services.
|
# After a plugin upgrade, the core restarts that plugin's services.
|
||||||
|
# Host restart is separate: #loop sets SAUSAGE_LOOP=1 to enable core.restart
|
||||||
|
# (and its alias core.exit). It answers before clean shutdown with exit code 75.
|
||||||
|
# Agents need the person's explicit approval each time; never add it as a service.
|
||||||
# Example (replace [] with this list):
|
# Example (replace [] with this list):
|
||||||
# - method: www.start
|
# - method: www.start
|
||||||
# params:
|
# params:
|
||||||
|
|||||||
+435
-58
@@ -3,12 +3,44 @@ use crate::{Host, world::World};
|
|||||||
use anyhow::{Context, Result, bail};
|
use anyhow::{Context, Result, bail};
|
||||||
use include_dir::{Dir, include_dir};
|
use include_dir::{Dir, include_dir};
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
use sha2::{Digest, Sha256};
|
use std::time::Duration;
|
||||||
use std::{
|
use std::{
|
||||||
path::{Path, PathBuf},
|
path::{Path, PathBuf},
|
||||||
sync::Arc,
|
sync::Arc,
|
||||||
};
|
};
|
||||||
use tokio::process::Command;
|
use tokio::{process::Command, time::Instant};
|
||||||
|
|
||||||
|
/// Default wall-clock budget after acquiring the build mutex (not queue time).
|
||||||
|
pub const DEFAULT_TIMEOUT_SECS: u64 = 900;
|
||||||
|
/// Default wall-clock budget for each WASI test instance.
|
||||||
|
pub const DEFAULT_TEST_TIMEOUT_SECS: u64 = 120;
|
||||||
|
|
||||||
|
/// Wall-clock budgets for a build and its individual WASI tests.
|
||||||
|
#[derive(Clone, Copy, Debug)]
|
||||||
|
pub struct BuildOptions {
|
||||||
|
pub timeout_secs: u64,
|
||||||
|
pub test_timeout_secs: u64,
|
||||||
|
}
|
||||||
|
impl Default for BuildOptions {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
timeout_secs: DEFAULT_TIMEOUT_SECS,
|
||||||
|
test_timeout_secs: DEFAULT_TEST_TIMEOUT_SECS,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
/// Optional RPC seconds are positive integers; null does not disable a deadline.
|
||||||
|
fn seconds(params: &Value, name: &str, default: u64) -> Result<u64, Value> {
|
||||||
|
match params.get(name) {
|
||||||
|
None => Ok(default),
|
||||||
|
Some(value) => value
|
||||||
|
.as_u64()
|
||||||
|
.filter(|seconds| *seconds >= 1)
|
||||||
|
.ok_or_else(|| {
|
||||||
|
crate::error(-32602, &format!("{name} must be an integer of at least 1"))
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The native build plugin.
|
/// The native build plugin.
|
||||||
pub const ID: &str = "cargo";
|
pub const ID: &str = "cargo";
|
||||||
@@ -25,9 +57,78 @@ pub struct TestResults {
|
|||||||
pub failed: u64,
|
pub failed: u64,
|
||||||
pub output: String,
|
pub output: String,
|
||||||
}
|
}
|
||||||
|
/// Raw, unfiltered output shared by the test and release phases of one build.
|
||||||
|
#[derive(Clone, Debug, serde::Serialize)]
|
||||||
|
pub struct BuildLogs {
|
||||||
|
pub stdout: StreamLog,
|
||||||
|
pub stderr: StreamLog,
|
||||||
|
pub total_bytes: u64,
|
||||||
|
}
|
||||||
|
#[derive(Clone, Debug, serde::Serialize)]
|
||||||
|
pub struct StreamLog {
|
||||||
|
pub path: String,
|
||||||
|
pub bytes: u64,
|
||||||
|
}
|
||||||
|
/// Persistent descriptors are inherited by cargo, so cancellation cannot lose
|
||||||
|
/// bytes already written by cargo or its descendants. No buffered writer is used.
|
||||||
|
struct LogFiles {
|
||||||
|
stdout: std::fs::File,
|
||||||
|
stderr: std::fs::File,
|
||||||
|
stdout_path: PathBuf,
|
||||||
|
stderr_path: PathBuf,
|
||||||
|
}
|
||||||
|
impl LogFiles {
|
||||||
|
fn create(world: &Path) -> Result<Self> {
|
||||||
|
let directory = world
|
||||||
|
.join("logs/cargo")
|
||||||
|
.join(chrono::Utc::now().format("%Y-%m-%d").to_string());
|
||||||
|
std::fs::create_dir_all(&directory).context("create cargo log directory")?;
|
||||||
|
let id = uuid::Uuid::now_v7();
|
||||||
|
let stdout_path = directory.join(format!("{id}.stdout.log"));
|
||||||
|
let stderr_path = directory.join(format!("{id}.stderr.log"));
|
||||||
|
let create = |path: &Path| {
|
||||||
|
std::fs::OpenOptions::new()
|
||||||
|
.create_new(true)
|
||||||
|
.append(true)
|
||||||
|
.open(path)
|
||||||
|
.with_context(|| format!("create {}", path.display()))
|
||||||
|
};
|
||||||
|
Ok(Self {
|
||||||
|
stdout: create(&stdout_path)?,
|
||||||
|
stderr: create(&stderr_path)?,
|
||||||
|
stdout_path,
|
||||||
|
stderr_path,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
fn info(&self, world: &Path) -> Result<BuildLogs> {
|
||||||
|
let stream = |file: &std::fs::File, path: &Path| -> Result<StreamLog> {
|
||||||
|
Ok(StreamLog {
|
||||||
|
path: format!("/{}", path.strip_prefix(world)?.display()),
|
||||||
|
bytes: file.metadata()?.len(),
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let stdout = stream(&self.stdout, &self.stdout_path)?;
|
||||||
|
let stderr = stream(&self.stderr, &self.stderr_path)?;
|
||||||
|
Ok(BuildLogs {
|
||||||
|
total_bytes: stdout.bytes + stderr.bytes,
|
||||||
|
stdout,
|
||||||
|
stderr,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
fn read(&self, path: &Path, offset: u64) -> Result<Vec<u8>> {
|
||||||
|
use std::io::{Read, Seek, SeekFrom};
|
||||||
|
let mut file =
|
||||||
|
std::fs::File::open(path).with_context(|| format!("read {}", path.display()))?;
|
||||||
|
file.seek(SeekFrom::Start(offset))?;
|
||||||
|
let mut bytes = Vec::new();
|
||||||
|
file.read_to_end(&mut bytes)?;
|
||||||
|
Ok(bytes)
|
||||||
|
}
|
||||||
|
}
|
||||||
/// A successful build, usable without a running host.
|
/// A successful build, usable without a running host.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub struct BuildOutput {
|
pub struct BuildOutput {
|
||||||
|
pub logs: Option<BuildLogs>,
|
||||||
pub wasm: PathBuf,
|
pub wasm: PathBuf,
|
||||||
pub warnings: String,
|
pub warnings: String,
|
||||||
pub tests: TestResults,
|
pub tests: TestResults,
|
||||||
@@ -35,9 +136,12 @@ pub struct BuildOutput {
|
|||||||
/// A failed compiler or test phase, with its complete diagnostics.
|
/// A failed compiler or test phase, with its complete diagnostics.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub struct BuildFailure {
|
pub struct BuildFailure {
|
||||||
|
pub logs: Option<BuildLogs>,
|
||||||
pub reason: String,
|
pub reason: String,
|
||||||
pub tests: TestResults,
|
pub tests: TestResults,
|
||||||
pub output: String,
|
pub output: String,
|
||||||
|
pub elapsed_secs: Option<f64>,
|
||||||
|
pub step: Option<&'static str>,
|
||||||
}
|
}
|
||||||
impl std::fmt::Display for BuildFailure {
|
impl std::fmt::Display for BuildFailure {
|
||||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
@@ -48,9 +152,12 @@ impl std::error::Error for BuildFailure {}
|
|||||||
impl From<anyhow::Error> for BuildFailure {
|
impl From<anyhow::Error> for BuildFailure {
|
||||||
fn from(error: anyhow::Error) -> Self {
|
fn from(error: anyhow::Error) -> Self {
|
||||||
Self {
|
Self {
|
||||||
|
logs: None,
|
||||||
reason: "build preparation failed".into(),
|
reason: "build preparation failed".into(),
|
||||||
output: format!("{error:#}"),
|
output: format!("{error:#}"),
|
||||||
tests: TestResults::default(),
|
tests: TestResults::default(),
|
||||||
|
elapsed_secs: None,
|
||||||
|
step: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -61,10 +168,9 @@ pub struct Cargo {
|
|||||||
runner: PathBuf,
|
runner: PathBuf,
|
||||||
target: PathBuf,
|
target: PathBuf,
|
||||||
sdk: PathBuf,
|
sdk: PathBuf,
|
||||||
sdk_files: Arc<std::collections::BTreeMap<PathBuf, Vec<u8>>>,
|
|
||||||
}
|
}
|
||||||
impl Cargo {
|
impl Cargo {
|
||||||
/// Installs changed SDK files and selects the current sausage executable as runner.
|
/// Seeds a missing world SDK and selects the current sausage executable as runner.
|
||||||
pub fn new(world: World, cache: &Path) -> Result<Self> {
|
pub fn new(world: World, cache: &Path) -> Result<Self> {
|
||||||
let mut runner = std::env::current_exe()?;
|
let mut runner = std::env::current_exe()?;
|
||||||
if runner
|
if runner
|
||||||
@@ -75,9 +181,23 @@ impl Cargo {
|
|||||||
}
|
}
|
||||||
Self::with_runner(world, runner, cache)
|
Self::with_runner(world, runner, cache)
|
||||||
}
|
}
|
||||||
/// Installs the SDK and accepts an explicit sausage binary, including for init integrations.
|
/// Seeds a missing world SDK and accepts an explicit sausage binary, including for init integrations.
|
||||||
pub fn with_runner(world: World, runner: PathBuf, cache: &Path) -> Result<Self> {
|
pub fn with_runner(world: World, runner: PathBuf, cache: &Path) -> Result<Self> {
|
||||||
let sdk = world.plugin_dir(ID).join("sdk");
|
let sdk = world.plugin_dir(ID).join("sdk");
|
||||||
|
Self::seed_sdk(&sdk)?;
|
||||||
|
Ok(Self {
|
||||||
|
sdk,
|
||||||
|
target: std::path::absolute(cache)?.join("cargo-target"),
|
||||||
|
world,
|
||||||
|
runner,
|
||||||
|
builds: Arc::new(tokio::sync::Mutex::new(())),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
/// Seeds only an absent SDK; an existing world owns all of its SDK files.
|
||||||
|
fn seed_sdk(sdk: &Path) -> Result<()> {
|
||||||
|
if sdk.try_exists()? {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
let mut files = std::collections::BTreeMap::new();
|
let mut files = std::collections::BTreeMap::new();
|
||||||
collect_files(&SDK, Path::new("src"), &mut files);
|
collect_files(&SDK, Path::new("src"), &mut files);
|
||||||
collect_files(&MACROS, Path::new("macros/src"), &mut files);
|
collect_files(&MACROS, Path::new("macros/src"), &mut files);
|
||||||
@@ -115,26 +235,23 @@ impl Cargo {
|
|||||||
}
|
}
|
||||||
files.insert(destination, document.to_string().into_bytes());
|
files.insert(destination, document.to_string().into_bytes());
|
||||||
}
|
}
|
||||||
let mut hash = Sha256::new();
|
let parent = sdk.parent().context("SDK has no parent")?;
|
||||||
|
std::fs::create_dir_all(parent).with_context(|| format!("create {}", parent.display()))?;
|
||||||
|
let staging = tempfile::Builder::new()
|
||||||
|
.prefix("seed-sdk-")
|
||||||
|
.tempdir_in(parent)
|
||||||
|
.with_context(|| format!("stage SDK in {}", parent.display()))?;
|
||||||
for (path, bytes) in &files {
|
for (path, bytes) in &files {
|
||||||
let name = path.to_str().unwrap().as_bytes();
|
write_changed(&staging.path().join(path), bytes)?;
|
||||||
hash.update((name.len() as u64).to_le_bytes());
|
|
||||||
hash.update(name);
|
|
||||||
hash.update((bytes.len() as u64).to_le_bytes());
|
|
||||||
hash.update(bytes);
|
|
||||||
write_changed(&sdk.join(path), bytes)?;
|
|
||||||
}
|
}
|
||||||
let cached = std::path::absolute(cache)?
|
// Another builder may have seeded the world while we prepared the files.
|
||||||
.join("sdk")
|
if !sdk.try_exists()?
|
||||||
.join(format!("{:x}", hash.finalize()));
|
&& let Err(error) = std::fs::rename(staging.path(), sdk)
|
||||||
Ok(Self {
|
&& !sdk.try_exists()?
|
||||||
sdk: cached,
|
{
|
||||||
sdk_files: Arc::new(files),
|
return Err(error).with_context(|| format!("seed {}", sdk.display()));
|
||||||
target: std::path::absolute(cache)?.join("cargo-target"),
|
}
|
||||||
world,
|
Ok(())
|
||||||
runner,
|
|
||||||
builds: Arc::new(tokio::sync::Mutex::new(())),
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
/// Registers cargo.build as a cancellable native process.
|
/// Registers cargo.build as a cancellable native process.
|
||||||
pub fn register(self, host: &Host) {
|
pub fn register(self, host: &Host) {
|
||||||
@@ -142,19 +259,59 @@ impl Cargo {
|
|||||||
host.register_native(ID,Arc::new(move |_context,method,params| {
|
host.register_native(ID,Arc::new(move |_context,method,params| {
|
||||||
let builder=builder.clone();
|
let builder=builder.clone();
|
||||||
Box::pin(async move {
|
Box::pin(async move {
|
||||||
if method=="rpc.discover" { return Ok(json!({"openrpc":"1.2.6","info":{"title":"cargo","version":"native"},"methods":[{"name":"build","paramStructure":"by-name","params":[{"name":"plugin","required":true,"schema":{"type":"string"}}],"result":{"name":"build","schema":{"type":"object","required":["wasm","tests","warnings"],"properties":{"wasm":{"type":"string"},"warnings":{"type":"string"},"tests":{"type":"object","required":["passed","failed","output"],"properties":{"passed":{"type":"integer","minimum":0},"failed":{"type":"integer","minimum":0},"output":{"type":"string"}}}}}},"errors":[{"code":-32004,"message":"Build failed"}]}]})); }
|
if method=="rpc.discover" { return Ok(json!({"openrpc":"1.2.6","info":{"title":"cargo","version":"native"},"methods":[{"name":"build","paramStructure":"by-name","description":"Copies source, runs bounded WASI tests, then builds a release component. Deadlines kill cargo and its descendants; cancellation releases the build mutex.","params":[{"name":"plugin","required":true,"schema":{"type":"string"}},{"name":"timeout_secs","required":false,"description":"Total wall-clock limit in seconds for tests and release build after acquiring the build mutex (queue time excluded). Default 900; minimum 1. Cannot be disabled.","schema":{"type":"integer","minimum":1,"default":DEFAULT_TIMEOUT_SECS}},{"name":"test_timeout_secs","required":false,"description":"Wall-clock limit in seconds for each WASI test instance. Default 120; minimum 1. A timed-out test is named and fails; remaining tests continue.","schema":{"type":"integer","minimum":1,"default":DEFAULT_TEST_TIMEOUT_SECS}}],"result":{"name":"build","schema":{"type":"object","required":["wasm","tests","warnings","logs"],"properties":{"logs":{"type":"object","required":["stdout","stderr","total_bytes"],"properties":{"stdout":{"type":"object","required":["path","bytes"],"properties":{"path":{"type":"string"},"bytes":{"type":"integer","minimum":0}}},"stderr":{"type":"object","required":["path","bytes"],"properties":{"path":{"type":"string"},"bytes":{"type":"integer","minimum":0}}},"total_bytes":{"type":"integer","minimum":0}}},"wasm":{"type":"string"},"warnings":{"type":"string"},"tests":{"type":"object","required":["passed","failed","output"],"properties":{"passed":{"type":"integer","minimum":0},"failed":{"type":"integer","minimum":0},"output":{"type":"string"}}}}}},"errors":[{"code":-32004,"message":"Build failed"}]}]})); }
|
||||||
if method!="build" { return Err(crate::error(-32601,"cargo method not found")); }
|
if method!="build" { return Err(crate::error(-32601,"cargo method not found")); }
|
||||||
let plugin=params["plugin"].as_str().ok_or_else(||crate::error(-32602,"plugin must be a string"))?;
|
let plugin=params["plugin"].as_str().ok_or_else(||crate::error(-32602,"plugin must be a string"))?;
|
||||||
match builder.build(&builder.world.plugin_dir(plugin).join("source")).await {
|
let options = BuildOptions {
|
||||||
Ok(output)=>Ok(json!({"wasm":format!("/{}",output.wasm.strip_prefix(&builder.world.root).unwrap().display()),"tests":output.tests,"warnings":output.warnings})),
|
timeout_secs: seconds(¶ms, "timeout_secs", DEFAULT_TIMEOUT_SECS)?,
|
||||||
Err(failure)=>Err(json!({"code":-32004,"message":failure.reason,"data":{"tests":failure.tests,"output":failure.output}})),
|
test_timeout_secs: seconds(¶ms, "test_timeout_secs", DEFAULT_TEST_TIMEOUT_SECS)?,
|
||||||
|
};
|
||||||
|
match builder.build_with_options(&builder.world.plugin_dir(plugin).join("source"), options).await {
|
||||||
|
Ok(output)=>Ok(json!({"wasm":format!("/{}",output.wasm.strip_prefix(&builder.world.root).unwrap().display()),"tests":output.tests,"warnings":output.warnings,"logs":output.logs})),
|
||||||
|
Err(failure)=>Err(json!({"code":-32004,"message":failure.reason,"data":{"reason":failure.reason,"tests":failure.tests,"output":failure.output,"elapsed_secs":failure.elapsed_secs,"step":failure.step,"logs":failure.logs}})),
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
/// Copies source once, runs WASI tests, then publishes a release component under state/cargo/builds.
|
/// Copies source once, runs WASI tests, then publishes a release component under state/cargo/builds.
|
||||||
pub async fn build(&self, source: &Path) -> std::result::Result<BuildOutput, BuildFailure> {
|
pub async fn build(&self, source: &Path) -> std::result::Result<BuildOutput, BuildFailure> {
|
||||||
|
self.build_with_options(source, BuildOptions::default())
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
/// Like build, with explicit positive wall-clock budgets. Waiting for the mutex is unbounded.
|
||||||
|
pub async fn build_with_options(
|
||||||
|
&self,
|
||||||
|
source: &Path,
|
||||||
|
options: BuildOptions,
|
||||||
|
) -> std::result::Result<BuildOutput, BuildFailure> {
|
||||||
|
let logs = LogFiles::create(&self.world.root)?;
|
||||||
|
let result = self.build_logged(source, options, &logs).await;
|
||||||
|
let info = logs.info(&self.world.root)?;
|
||||||
|
match result {
|
||||||
|
Ok(mut output) => {
|
||||||
|
output.logs = Some(info);
|
||||||
|
Ok(output)
|
||||||
|
}
|
||||||
|
Err(mut failure) => {
|
||||||
|
failure.logs = Some(info);
|
||||||
|
Err(failure)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
async fn build_logged(
|
||||||
|
&self,
|
||||||
|
source: &Path,
|
||||||
|
options: BuildOptions,
|
||||||
|
logs: &LogFiles,
|
||||||
|
) -> std::result::Result<BuildOutput, BuildFailure> {
|
||||||
|
if options.timeout_secs == 0 || options.test_timeout_secs == 0 {
|
||||||
|
return Err(anyhow::anyhow!("timeouts must be at least 1 second").into());
|
||||||
|
}
|
||||||
let build_guard = self.builds.clone().lock_owned().await;
|
let build_guard = self.builds.clone().lock_owned().await;
|
||||||
|
let started = Instant::now();
|
||||||
|
let deadline = started
|
||||||
|
.checked_add(Duration::from_secs(options.timeout_secs))
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("timeout_secs is too large"))?;
|
||||||
let work = self.world.state_dir(ID).join("work");
|
let work = self.world.state_dir(ID).join("work");
|
||||||
let target = self.target.clone();
|
let target = self.target.clone();
|
||||||
let temporary = self.world.state_dir(ID).join("tmp");
|
let temporary = self.world.state_dir(ID).join("tmp");
|
||||||
@@ -162,27 +319,8 @@ impl Cargo {
|
|||||||
let source = source.to_path_buf();
|
let source = source.to_path_buf();
|
||||||
let copy = work.clone();
|
let copy = work.clone();
|
||||||
let sdk = self.sdk.clone();
|
let sdk = self.sdk.clone();
|
||||||
let sdk_files = self.sdk_files.clone();
|
|
||||||
let build_guard = tokio::task::spawn_blocking(move || -> Result<_> {
|
let build_guard = tokio::task::spawn_blocking(move || -> Result<_> {
|
||||||
// Host startup installs only the readable world copy. Builds populate the cache.
|
Self::seed_sdk(&sdk)?;
|
||||||
if !sdk.is_dir() {
|
|
||||||
let parent = sdk.parent().unwrap();
|
|
||||||
std::fs::create_dir_all(parent)
|
|
||||||
.with_context(|| format!("create {}", parent.display()))?;
|
|
||||||
let staging = tempfile::Builder::new()
|
|
||||||
.prefix("install-")
|
|
||||||
.tempdir_in(parent)
|
|
||||||
.with_context(|| format!("stage SDK in {}", parent.display()))?;
|
|
||||||
for (path, bytes) in sdk_files.iter() {
|
|
||||||
write_changed(&staging.path().join(path), bytes)?;
|
|
||||||
}
|
|
||||||
// Another builder can publish the same complete SDK first.
|
|
||||||
if let Err(error) = std::fs::rename(staging.path(), &sdk)
|
|
||||||
&& !sdk.is_dir()
|
|
||||||
{
|
|
||||||
return Err(error).with_context(|| format!("install {}", sdk.display()));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if cleanup.exists() {
|
if cleanup.exists() {
|
||||||
std::fs::remove_dir_all(&cleanup)
|
std::fs::remove_dir_all(&cleanup)
|
||||||
.with_context(|| format!("remove {}", cleanup.display()))?;
|
.with_context(|| format!("remove {}", cleanup.display()))?;
|
||||||
@@ -208,16 +346,35 @@ impl Cargo {
|
|||||||
.await
|
.await
|
||||||
.map_err(anyhow::Error::from)??;
|
.map_err(anyhow::Error::from)??;
|
||||||
let mut test = Command::new("cargo");
|
let mut test = Command::new("cargo");
|
||||||
test.args(["test", "--release", "--target", TARGET, "--target-dir"])
|
test.args([
|
||||||
|
"test",
|
||||||
|
"--no-fail-fast",
|
||||||
|
"--release",
|
||||||
|
"--target",
|
||||||
|
TARGET,
|
||||||
|
"--target-dir",
|
||||||
|
])
|
||||||
.arg(&target)
|
.arg(&target)
|
||||||
.current_dir(&work)
|
.current_dir(&work)
|
||||||
.env("TMPDIR", &temporary)
|
.env("TMPDIR", &temporary)
|
||||||
|
.env(
|
||||||
|
"SAUSAGE_TEST_TIMEOUT_SECS",
|
||||||
|
options.test_timeout_secs.to_string(),
|
||||||
|
)
|
||||||
.arg("--config")
|
.arg("--config")
|
||||||
.arg(format!(
|
.arg(format!(
|
||||||
"target.wasm32-wasip2.runner={}",
|
"target.wasm32-wasip2.runner={}",
|
||||||
json!([self.runner, "test-runner"])
|
json!([self.runner, "test-runner"])
|
||||||
));
|
));
|
||||||
let output = run_command(test).await?;
|
let output = run_command(
|
||||||
|
test,
|
||||||
|
started,
|
||||||
|
deadline,
|
||||||
|
"tests",
|
||||||
|
TestResults::default(),
|
||||||
|
logs,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
let stdout = String::from_utf8_lossy(&output.stdout);
|
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||||
let tests = TestResults {
|
let tests = TestResults {
|
||||||
@@ -231,12 +388,14 @@ impl Cargo {
|
|||||||
};
|
};
|
||||||
if !output.status.success() {
|
if !output.status.success() {
|
||||||
return Err(BuildFailure {
|
return Err(BuildFailure {
|
||||||
|
logs: None,
|
||||||
reason: "WASI tests failed".into(),
|
reason: "WASI tests failed".into(),
|
||||||
output: tests.output.clone(),
|
output: tests.output.clone(),
|
||||||
tests,
|
tests,
|
||||||
|
elapsed_secs: None,
|
||||||
|
step: None,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
let result = async {
|
|
||||||
let mut build = Command::new("cargo");
|
let mut build = Command::new("cargo");
|
||||||
build
|
build
|
||||||
.args([
|
.args([
|
||||||
@@ -249,7 +408,16 @@ impl Cargo {
|
|||||||
])
|
])
|
||||||
.arg(&target)
|
.arg(&target)
|
||||||
.current_dir(&work);
|
.current_dir(&work);
|
||||||
let output = run_command(build).await?;
|
let output = run_command(
|
||||||
|
build,
|
||||||
|
started,
|
||||||
|
deadline,
|
||||||
|
"release build",
|
||||||
|
tests.clone(),
|
||||||
|
logs,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
let result = async {
|
||||||
let stdout = String::from_utf8_lossy(&output.stdout);
|
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||||
let diagnostics = stderr.to_string();
|
let diagnostics = stderr.to_string();
|
||||||
@@ -282,6 +450,7 @@ impl Cargo {
|
|||||||
})
|
})
|
||||||
.await??;
|
.await??;
|
||||||
Ok(BuildOutput {
|
Ok(BuildOutput {
|
||||||
|
logs: None,
|
||||||
wasm: destination,
|
wasm: destination,
|
||||||
warnings: strip_progress(&diagnostics),
|
warnings: strip_progress(&diagnostics),
|
||||||
tests: tests.clone(),
|
tests: tests.clone(),
|
||||||
@@ -289,9 +458,12 @@ impl Cargo {
|
|||||||
}
|
}
|
||||||
.await;
|
.await;
|
||||||
result.map_err(|error: anyhow::Error| BuildFailure {
|
result.map_err(|error: anyhow::Error| BuildFailure {
|
||||||
|
logs: None,
|
||||||
reason: "release build failed".into(),
|
reason: "release build failed".into(),
|
||||||
output: format!("{error:#}"),
|
output: format!("{error:#}"),
|
||||||
tests,
|
tests,
|
||||||
|
elapsed_secs: None,
|
||||||
|
step: None,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -306,16 +478,81 @@ impl Drop for ProcessGroup {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
/// Starts one process group and retains its guard across all output collection.
|
/// Starts one process group and retains its guard across all output collection.
|
||||||
async fn run_command(mut command: Command) -> Result<std::process::Output> {
|
async fn run_command(
|
||||||
|
mut command: Command,
|
||||||
|
started: Instant,
|
||||||
|
deadline: Instant,
|
||||||
|
step: &'static str,
|
||||||
|
mut tests: TestResults,
|
||||||
|
logs: &LogFiles,
|
||||||
|
) -> std::result::Result<std::process::Output, BuildFailure> {
|
||||||
command
|
command
|
||||||
.process_group(0)
|
.process_group(0)
|
||||||
.kill_on_drop(true)
|
.kill_on_drop(true)
|
||||||
.stdin(std::process::Stdio::null())
|
.stdin(std::process::Stdio::null())
|
||||||
.stdout(std::process::Stdio::piped())
|
// Child writes go straight to persistent files: no pipe buffers or reader
|
||||||
.stderr(std::process::Stdio::piped());
|
// tasks can be dropped on cancellation. Parsing happens only afterwards.
|
||||||
let child = command.spawn().context("start cargo")?;
|
.stdout(logs.stdout.try_clone().context("clone stdout log")?)
|
||||||
let _group = ProcessGroup(child.id().unwrap());
|
.stderr(logs.stderr.try_clone().context("clone stderr log")?);
|
||||||
Ok(child.wait_with_output().await?)
|
let out_start = logs.stdout.metadata().context("stat stdout log")?.len();
|
||||||
|
let err_start = logs.stderr.metadata().context("stat stderr log")?.len();
|
||||||
|
let mut child = command.spawn().context("start cargo")?;
|
||||||
|
let group = ProcessGroup(child.id().unwrap());
|
||||||
|
|
||||||
|
let (status, timed_out) = {
|
||||||
|
let collect = child.wait();
|
||||||
|
tokio::pin!(collect);
|
||||||
|
tokio::select! {
|
||||||
|
biased;
|
||||||
|
_ = tokio::time::sleep_until(deadline) => {
|
||||||
|
// Kill the entire tree before waiting for EOF/reaping cargo. The guard also
|
||||||
|
// covers cancellation while collecting, and runs before the mutex drops.
|
||||||
|
let _ = nix::sys::signal::killpg(nix::unistd::Pid::from_raw(group.0 as i32), nix::sys::signal::Signal::SIGKILL);
|
||||||
|
(collect.await, true)
|
||||||
|
}
|
||||||
|
status = &mut collect => (status, false),
|
||||||
|
}
|
||||||
|
};
|
||||||
|
let status = status.map_err(anyhow::Error::from)?;
|
||||||
|
// Also stop any descendants that kept the streams open after cargo exited.
|
||||||
|
drop(group);
|
||||||
|
let out = logs.read(&logs.stdout_path, out_start)?;
|
||||||
|
let err = logs.read(&logs.stderr_path, err_start)?;
|
||||||
|
if timed_out {
|
||||||
|
let output = format!(
|
||||||
|
"{}\n{}",
|
||||||
|
String::from_utf8_lossy(&out),
|
||||||
|
String::from_utf8_lossy(&err)
|
||||||
|
);
|
||||||
|
let output = output
|
||||||
|
.trim_end()
|
||||||
|
.lines()
|
||||||
|
.rev()
|
||||||
|
.take(50)
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.into_iter()
|
||||||
|
.rev()
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join("\n");
|
||||||
|
if step == "tests" {
|
||||||
|
tests.passed = test_count(&String::from_utf8_lossy(&out), "passed");
|
||||||
|
tests.failed = test_count(&String::from_utf8_lossy(&out), "failed");
|
||||||
|
tests.output = output.clone();
|
||||||
|
}
|
||||||
|
return Err(BuildFailure {
|
||||||
|
logs: None,
|
||||||
|
reason: "timeout".into(),
|
||||||
|
tests,
|
||||||
|
output,
|
||||||
|
elapsed_secs: Some(started.elapsed().as_secs_f64()),
|
||||||
|
step: Some(step),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(std::process::Output {
|
||||||
|
status,
|
||||||
|
stdout: out,
|
||||||
|
stderr: err,
|
||||||
|
})
|
||||||
}
|
}
|
||||||
/// Copies only source, excluding build outputs and git internals at every level.
|
/// Copies only source, excluding build outputs and git internals at every level.
|
||||||
fn copy_source(source: &Path, destination: &Path, root: bool) -> Result<()> {
|
fn copy_source(source: &Path, destination: &Path, root: bool) -> Result<()> {
|
||||||
@@ -412,7 +649,7 @@ fn write_changed(path: &Path, bytes: &[u8]) -> Result<()> {
|
|||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
/// Collects embedded source files for deterministic hashing and installation.
|
/// Collects embedded source files for seeding a missing world SDK.
|
||||||
fn collect_files(
|
fn collect_files(
|
||||||
directory: &Dir<'_>,
|
directory: &Dir<'_>,
|
||||||
destination: &Path,
|
destination: &Path,
|
||||||
@@ -457,3 +694,143 @@ fn strip_progress(output: &str) -> String {
|
|||||||
.collect::<Vec<_>>()
|
.collect::<Vec<_>>()
|
||||||
.join("\n")
|
.join("\n")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod timeout_tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
/// Streams retain exact bytes, including non-UTF8, progress and JSON; phases append.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn raw_streams_append_before_parsing() {
|
||||||
|
let directory = tempfile::tempdir().unwrap();
|
||||||
|
let logs = LogFiles::create(directory.path()).unwrap();
|
||||||
|
for _ in 0..2 {
|
||||||
|
let mut command = Command::new("sh");
|
||||||
|
command.args(["-c", "printf '\\377raw\\r'; printf ' Compiling progress\\n{\"reason\":\"compiler-artifact\"}\\n' >&2; exit 3"]);
|
||||||
|
let now = Instant::now();
|
||||||
|
let output = tokio::time::timeout(
|
||||||
|
Duration::from_secs(5),
|
||||||
|
run_command(
|
||||||
|
command,
|
||||||
|
now,
|
||||||
|
now + Duration::from_secs(3),
|
||||||
|
"tests",
|
||||||
|
TestResults::default(),
|
||||||
|
&logs,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert!(!output.status.success());
|
||||||
|
assert_eq!(output.stdout, b"\xffraw\r");
|
||||||
|
assert!(String::from_utf8_lossy(&output.stderr).contains("Compiling progress"));
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read(&logs.stdout_path).unwrap(),
|
||||||
|
b"\xffraw\r\xffraw\r"
|
||||||
|
);
|
||||||
|
let info = logs.info(directory.path()).unwrap();
|
||||||
|
assert_eq!(info.stdout.bytes, 10);
|
||||||
|
assert_eq!(info.total_bytes, info.stdout.bytes + info.stderr.bytes);
|
||||||
|
assert!(info.stdout.path.starts_with("/logs/cargo/"));
|
||||||
|
assert!(!directory.path().join("state/cargo").exists());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cancellation leaves kernel-written data intact, even without a newline.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn cancelled_command_keeps_logs() {
|
||||||
|
use notify::Watcher;
|
||||||
|
let directory = tempfile::tempdir().unwrap();
|
||||||
|
let logs = LogFiles::create(directory.path()).unwrap();
|
||||||
|
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel();
|
||||||
|
let mut watcher =
|
||||||
|
notify::recommended_watcher(move |event: notify::Result<notify::Event>| {
|
||||||
|
if let Ok(event) = event {
|
||||||
|
if event
|
||||||
|
.paths
|
||||||
|
.iter()
|
||||||
|
.any(|path| path.file_name().is_some_and(|name| name == "ready"))
|
||||||
|
{
|
||||||
|
let _ = tx.send(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
watcher
|
||||||
|
.watch(directory.path(), notify::RecursiveMode::NonRecursive)
|
||||||
|
.unwrap();
|
||||||
|
let mut command = Command::new("sh");
|
||||||
|
command.args(["-c", "printf 'partial\\377'; printf 'stderr\\r' >&2; touch \"$1/ready\"; exec sleep 1000", "sh"]).arg(directory.path());
|
||||||
|
let now = Instant::now();
|
||||||
|
let task = tokio::spawn(async move {
|
||||||
|
run_command(
|
||||||
|
command,
|
||||||
|
now,
|
||||||
|
now + Duration::from_secs(60),
|
||||||
|
"tests",
|
||||||
|
TestResults::default(),
|
||||||
|
&logs,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
tokio::time::timeout(Duration::from_secs(5), rx.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
task.abort();
|
||||||
|
assert!(task.await.unwrap_err().is_cancelled());
|
||||||
|
let paths: Vec<_> = std::fs::read_dir(directory.path().join("logs/cargo"))
|
||||||
|
.unwrap()
|
||||||
|
.flat_map(|day| std::fs::read_dir(day.unwrap().path()).unwrap())
|
||||||
|
.map(|file| file.unwrap().path())
|
||||||
|
.collect();
|
||||||
|
assert_eq!(paths.len(), 2);
|
||||||
|
for path in paths {
|
||||||
|
let expected: &[u8] = if path.to_string_lossy().ends_with(".stdout.log") {
|
||||||
|
b"partial\xff"
|
||||||
|
} else {
|
||||||
|
b"stderr\r"
|
||||||
|
};
|
||||||
|
assert_eq!(std::fs::read(path).unwrap(), expected);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Deadline errors retain the shared start time, phase, test totals, and output tail.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn timeout_diagnostics() {
|
||||||
|
let mut command = Command::new("sh");
|
||||||
|
command.args([
|
||||||
|
"-c",
|
||||||
|
"i=0; while [ $i -lt 100 ]; do echo diagnostic-$i; i=$((i+1)); done; exec sleep 1000",
|
||||||
|
]);
|
||||||
|
let directory = tempfile::tempdir().unwrap();
|
||||||
|
let logs = LogFiles::create(directory.path()).unwrap();
|
||||||
|
let now = Instant::now();
|
||||||
|
let failure = run_command(
|
||||||
|
command,
|
||||||
|
now - Duration::from_secs(10),
|
||||||
|
now + Duration::from_secs(1),
|
||||||
|
"release build",
|
||||||
|
TestResults {
|
||||||
|
passed: 7,
|
||||||
|
failed: 0,
|
||||||
|
output: "successful tests".into(),
|
||||||
|
},
|
||||||
|
&logs,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert_eq!(failure.reason, "timeout");
|
||||||
|
assert_eq!(failure.step, Some("release build"));
|
||||||
|
assert!(failure.elapsed_secs.unwrap() >= 11.0);
|
||||||
|
assert_eq!(failure.tests.passed, 7);
|
||||||
|
assert_eq!(failure.tests.output, "successful tests");
|
||||||
|
assert_eq!(failure.output.lines().count(), 50);
|
||||||
|
assert!(failure.output.contains("diagnostic-99"));
|
||||||
|
assert!(!failure.output.contains("diagnostic-0\n"));
|
||||||
|
let raw = std::fs::read_to_string(&logs.stdout_path).unwrap();
|
||||||
|
assert_eq!(raw.lines().count(), 100);
|
||||||
|
assert!(raw.starts_with("diagnostic-0\n"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+148
-66
@@ -1,77 +1,159 @@
|
|||||||
//! The native core methods and their JSON parameters.
|
//! Native core dispatch and OpenRPC, generated from one typed method registry.
|
||||||
|
mod types;
|
||||||
|
use types::*;
|
||||||
|
pub(crate) use types::{ActivationSummary, Held, InterfaceCheck};
|
||||||
|
|
||||||
use crate::{Host, RpcResult, error};
|
use crate::{Host, RpcResult, error};
|
||||||
|
use sausage_plugin::{Empty, document, finish_document, method};
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
|
use std::sync::OnceLock;
|
||||||
|
|
||||||
|
// Every dispatched method must supply docs, a parameter type, and a result
|
||||||
|
// type. The same entry generates its descriptor; neither list can drift.
|
||||||
|
macro_rules! core_methods {
|
||||||
|
($host:ident, $caller:ident, $id:ident, $wire:ident, $p:ident;
|
||||||
|
$($name:literal: $params:ty => $result:ty, $docs:literal $body:block)*) => {
|
||||||
|
#[cfg(test)]
|
||||||
|
const METHOD_NAMES: &[&str] = &[$($name),*];
|
||||||
|
|
||||||
|
pub(crate) fn discover() -> &'static Value {
|
||||||
|
static DOCUMENT: OnceLock<Value> = OnceLock::new();
|
||||||
|
DOCUMENT.get_or_init(|| {
|
||||||
|
let mut doc = document("core", env!("CARGO_PKG_VERSION"));
|
||||||
|
$(method::<$params, $result>(&mut doc, $name, $docs);)*
|
||||||
|
finish_document(doc)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
impl Host {
|
impl Host {
|
||||||
/// Execute a core method without starting a guest process.
|
/// Execute a native method without starting a guest process.
|
||||||
pub(crate) async fn core(
|
pub(crate) async fn core(
|
||||||
&self,
|
&$host,
|
||||||
caller: &str,
|
$caller: &str,
|
||||||
id: Option<Value>,
|
$id: Option<Value>,
|
||||||
method: &str,
|
$wire: &str,
|
||||||
p: Value,
|
$p: Value,
|
||||||
) -> RpcResult {
|
) -> RpcResult {
|
||||||
match method {
|
match $wire {
|
||||||
"call" => {
|
$($name => {
|
||||||
self.route(
|
let $p: $params = params($p)?;
|
||||||
caller.into(),
|
let result: $result = $body;
|
||||||
id,
|
Ok(json!(result))
|
||||||
required_str(&p, "method")?.into(),
|
},)*
|
||||||
p.get("params").cloned().unwrap_or(json!({})),
|
|
||||||
p["detach"].as_bool().unwrap_or(false),
|
|
||||||
)
|
|
||||||
.await
|
|
||||||
}
|
|
||||||
"notify" => {
|
|
||||||
self.notify(
|
|
||||||
caller,
|
|
||||||
required_str(&p, "name")?,
|
|
||||||
p.get("params").cloned().unwrap_or(json!({})),
|
|
||||||
);
|
|
||||||
Ok(Value::Null)
|
|
||||||
}
|
|
||||||
"kill" => {
|
|
||||||
let pid = required_str(&p, "pid")?;
|
|
||||||
let ended = self
|
|
||||||
.0
|
|
||||||
.state
|
|
||||||
.lock()
|
|
||||||
.unwrap()
|
|
||||||
.processes
|
|
||||||
.get(pid)
|
|
||||||
.map(|p| p.ended.subscribe());
|
|
||||||
self.kill(pid);
|
|
||||||
if let Some(mut ended) = ended {
|
|
||||||
let _ = ended.wait_for(|ended| *ended).await;
|
|
||||||
}
|
|
||||||
Ok(json!({}))
|
|
||||||
}
|
|
||||||
"stop" => {
|
|
||||||
if let Some(mut n) = self.begin_stop(required_str(&p, "pid")?) {
|
|
||||||
let _ = n.wait_for(|ended| *ended).await;
|
|
||||||
}
|
|
||||||
Ok(json!({}))
|
|
||||||
}
|
|
||||||
"lock" => {
|
|
||||||
self.lock(caller, required_str(&p, "path")?, p["wait"] != false)
|
|
||||||
.await
|
|
||||||
}
|
|
||||||
"unlock" => {
|
|
||||||
self.unlock(caller, required_str(&p, "path")?);
|
|
||||||
Ok(Value::Null)
|
|
||||||
}
|
|
||||||
"log" => Ok(self.0.journal.query(p).await),
|
|
||||||
"activate" | "revert" => self.activation(caller, method, p).await,
|
|
||||||
"exit" => {
|
|
||||||
self.0.exit.send_replace(true);
|
|
||||||
Ok(json!({}))
|
|
||||||
}
|
|
||||||
_ => Err(error(-32601, "method not found")),
|
_ => Err(error(-32601, "method not found")),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
fn required_str<'a>(params: &'a Value, name: &str) -> Result<&'a str, Value> {
|
};
|
||||||
params[name]
|
}
|
||||||
.as_str()
|
|
||||||
.ok_or_else(|| error(-32602, &format!("{name} must be a string")))
|
core_methods! { self, caller, id, wire, p;
|
||||||
|
"rpc.discover": Empty => Json,
|
||||||
|
"Describe every native core method as OpenRPC 1.2, generated from its wire types." {
|
||||||
|
let _ = p;
|
||||||
|
Json(discover().clone())
|
||||||
|
}
|
||||||
|
"watch": Path => (),
|
||||||
|
"Watch a path for the running caller's lifetime. Needs a request ID; sends armed, then changed notifications with empty params. Scan after armed and rescan after changed." {
|
||||||
|
self.watch(caller, id, &p.path).await?;
|
||||||
|
}
|
||||||
|
"call": Call => Json,
|
||||||
|
"Call a qualified method, optionally detached. Returns that method's result unchanged." {
|
||||||
|
Json(self.route(caller.into(), id, p.method, p.params.0, p.detach.unwrap_or(false)).await?)
|
||||||
|
}
|
||||||
|
"notify": Notify => (),
|
||||||
|
"Notify the caller about progress. Normally sent as a notification, without an ID." {
|
||||||
|
self.notify(caller, &p.name, p.params.0);
|
||||||
|
}
|
||||||
|
"kill": Pid => Empty,
|
||||||
|
"Kill a process immediately; return after it has ended. An unknown process is a no-op." {
|
||||||
|
let ended = self.0.state.lock().unwrap().processes.get(&p.pid).map(|p| p.ended.subscribe());
|
||||||
|
self.kill(&p.pid);
|
||||||
|
if let Some(mut ended) = ended {
|
||||||
|
let _ = ended.wait_for(|ended| *ended).await;
|
||||||
|
}
|
||||||
|
Empty {}
|
||||||
|
}
|
||||||
|
"stop": Pid => Empty,
|
||||||
|
"Close a process's stdin and wait until it ends, killing it after the five-second stop budget. An unknown process is a no-op." {
|
||||||
|
if let Some(mut ended) = self.begin_stop(&p.pid) {
|
||||||
|
let _ = ended.wait_for(|ended| *ended).await;
|
||||||
|
}
|
||||||
|
Empty {}
|
||||||
|
}
|
||||||
|
"lock": Lock => Held,
|
||||||
|
"Acquire an advisory FIFO lock. Wait unless wait is false; locks are released when their owner ends." {
|
||||||
|
result(self.lock(caller, &p.path, p.wait.unwrap_or(true)).await?)?
|
||||||
|
}
|
||||||
|
"unlock": Path => (),
|
||||||
|
"Release an advisory lock held by this caller. Normally sent as a notification, without an ID." {
|
||||||
|
self.unlock(caller, &p.path);
|
||||||
|
}
|
||||||
|
"log": Log => Vec<crate::journal::Entry>,
|
||||||
|
"Query the journal. Return the newest matching entries in ascending entry-ID order." {
|
||||||
|
result(self.0.journal.query(json!(p)).await)?
|
||||||
|
}
|
||||||
|
"activate": Activate => ActivationResult,
|
||||||
|
"Copy, discover, check and install one or more plugin WASMs. Return short summaries keyed by plugin name; force skips interface checks." {
|
||||||
|
result(self.activation(caller, wire, json!(p)).await?)?
|
||||||
|
}
|
||||||
|
"revert": Revert => ActivationResult,
|
||||||
|
"Restore one or more plugins from plugin.prev.wasm, using the same checks and summaries as activate." {
|
||||||
|
result(self.activation(caller, wire, json!(p)).await?)?
|
||||||
|
}
|
||||||
|
"restart": Empty => Empty,
|
||||||
|
"Request a clean host restart. Requires the person's explicit approval each time and the SAUSAGE_LOOP=1 supervisor marker; otherwise fails with -32600." {
|
||||||
|
let _ = p;
|
||||||
|
self.request_restart()?;
|
||||||
|
Empty {}
|
||||||
|
}
|
||||||
|
"exit": Empty => Empty,
|
||||||
|
"Legacy alias of restart: same explicit-approval rule, supervisor gate and coalesced shutdown request." {
|
||||||
|
let _ = p;
|
||||||
|
self.request_restart()?;
|
||||||
|
Empty {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::{runtime::Runtime, world::World};
|
||||||
|
use std::collections::BTreeSet;
|
||||||
|
|
||||||
|
/// Every descriptor names a dispatched method, and vice versa.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn discovery_covers_dispatch_exactly() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let world = dir.path().join("world");
|
||||||
|
World::create(&world).unwrap();
|
||||||
|
let host = Host::test_world(&world, Runtime::new(&dir.path().join("cache")).unwrap())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let doc = host.call("core.rpc.discover", json!({})).await.unwrap();
|
||||||
|
let names: BTreeSet<_> = doc["methods"]
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.iter()
|
||||||
|
.map(|m| {
|
||||||
|
assert!(!m["description"].as_str().unwrap().is_empty());
|
||||||
|
m["name"].as_str().unwrap()
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
assert_eq!(names, METHOD_NAMES.iter().copied().collect());
|
||||||
|
assert_eq!(names.len(), doc["methods"].as_array().unwrap().len());
|
||||||
|
for name in names {
|
||||||
|
let response = host.call(&format!("core.{name}"), json!({})).await;
|
||||||
|
if let Err(error) = response {
|
||||||
|
assert_ne!(error["code"], -32601, "descriptor without dispatch: {name}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
host.call("core.no_such_method", json!({}))
|
||||||
|
.await
|
||||||
|
.unwrap_err()["code"],
|
||||||
|
-32601
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,200 @@
|
|||||||
|
//! Native wire types shared by dispatch, discovery, and result producers.
|
||||||
|
use sausage_plugin::{Params, SchemaType, schema};
|
||||||
|
use serde::{Deserialize, Deserializer, Serialize};
|
||||||
|
use serde_json::Value;
|
||||||
|
use std::collections::BTreeMap;
|
||||||
|
|
||||||
|
// Core forwarding/discovery carry arbitrary JSON, and activation returns a map
|
||||||
|
// keyed by plugin name. These native contracts are wider than #[schema]'s
|
||||||
|
// plugin subset; schemars still derives them from their actual wire types.
|
||||||
|
#[derive(Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
#[serde(transparent)]
|
||||||
|
pub(super) struct Json(pub Value);
|
||||||
|
impl SchemaType for Json {}
|
||||||
|
|
||||||
|
/// Forward a call, optionally detached from its caller.
|
||||||
|
#[derive(Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
pub(super) struct Call {
|
||||||
|
/// Qualified target method.
|
||||||
|
pub method: String,
|
||||||
|
/// Target parameters; omitted means an empty object.
|
||||||
|
#[serde(default = "empty_params")]
|
||||||
|
pub params: Json,
|
||||||
|
/// Detach the new process; default false.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub detach: Option<bool>,
|
||||||
|
}
|
||||||
|
fn empty_params() -> Json {
|
||||||
|
Json(serde_json::json!({}))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Notify the caller of progress.
|
||||||
|
#[schema]
|
||||||
|
pub(super) struct Notify {
|
||||||
|
/// Notification name.
|
||||||
|
pub name: String,
|
||||||
|
/// Notification payload; omitted means an empty object.
|
||||||
|
#[serde(default = "empty_params")]
|
||||||
|
pub params: Json,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Select a process.
|
||||||
|
#[schema]
|
||||||
|
pub(super) struct Pid {
|
||||||
|
/// Process ID.
|
||||||
|
pub pid: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Select a virtual path or advisory lock name.
|
||||||
|
#[schema]
|
||||||
|
pub(super) struct Path {
|
||||||
|
/// Path to observe or unlock.
|
||||||
|
pub path: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Acquire an advisory FIFO lock.
|
||||||
|
#[derive(Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
pub(super) struct Lock {
|
||||||
|
/// Advisory lock name (usually a state file path).
|
||||||
|
pub path: String,
|
||||||
|
/// Wait for ownership unless false; omitted or null means true.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub wait: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Lock ownership after acquisition.
|
||||||
|
#[schema]
|
||||||
|
pub(crate) struct Held {
|
||||||
|
/// Whether the caller holds the lock.
|
||||||
|
pub held: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Query journal entries, returned in ascending ID order.
|
||||||
|
#[derive(Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
pub(super) struct Log {
|
||||||
|
/// Exact plugin name.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub plugin: Option<String>,
|
||||||
|
/// Exact process ID.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub pid: Option<String>,
|
||||||
|
/// Inclusive lower entry ID.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub since: Option<String>,
|
||||||
|
/// Inclusive upper entry ID.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub until: Option<String>,
|
||||||
|
/// Literal substring in the line.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub grep: Option<String>,
|
||||||
|
/// Maximum newest entries; default 100, negative means no limit.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub limit: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Keep the existing core's forgiving optional filters/flags: values of the
|
||||||
|
// wrong type act like omitted values, rather than adding new validation.
|
||||||
|
fn optional<'de, T: serde::de::DeserializeOwned, D: Deserializer<'de>>(
|
||||||
|
deserializer: D,
|
||||||
|
) -> Result<Option<T>, D::Error> {
|
||||||
|
Ok(serde_json::from_value(Value::deserialize(deserializer)?).ok())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One candidate for activation.
|
||||||
|
#[schema]
|
||||||
|
pub(super) struct ActivatePlugin {
|
||||||
|
/// Plugin name.
|
||||||
|
pub plugin: String,
|
||||||
|
/// Candidate WASM world path.
|
||||||
|
pub wasm: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Activate one candidate, or a group checked and installed together.
|
||||||
|
#[derive(Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
pub(super) struct Activate {
|
||||||
|
/// Single plugin name, used when plugins is omitted.
|
||||||
|
pub plugin: Option<String>,
|
||||||
|
/// Single candidate WASM world path, used when plugins is omitted.
|
||||||
|
pub wasm: Option<String>,
|
||||||
|
/// Group of candidates; takes precedence over plugin and wasm. Must not be empty.
|
||||||
|
pub plugins: Option<Vec<ActivatePlugin>>,
|
||||||
|
/// Skip interface checks only when true; default false.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub force: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One plugin to restore from plugin.prev.wasm.
|
||||||
|
#[schema]
|
||||||
|
pub(super) struct RevertPlugin {
|
||||||
|
/// Plugin name.
|
||||||
|
pub plugin: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Revert one plugin or a group to their previous WASM files.
|
||||||
|
#[derive(Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
pub(super) struct Revert {
|
||||||
|
/// Single plugin name, used when plugins is omitted.
|
||||||
|
pub plugin: Option<String>,
|
||||||
|
/// Group of plugins; takes precedence over plugin. Must not be empty.
|
||||||
|
pub plugins: Option<Vec<RevertPlugin>>,
|
||||||
|
/// Skip interface checks only when true; default false.
|
||||||
|
#[serde(default, deserialize_with = "optional")]
|
||||||
|
pub force: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A short summary of one installed plugin.
|
||||||
|
#[schema]
|
||||||
|
pub(crate) struct ActivationSummary {
|
||||||
|
/// Plugin name.
|
||||||
|
pub plugin: String,
|
||||||
|
/// Installed world path.
|
||||||
|
pub wasm: String,
|
||||||
|
/// SHA-256 of the installed copy.
|
||||||
|
pub hash: String,
|
||||||
|
/// Replaced hash, or null on first installation.
|
||||||
|
pub previous: Option<String>,
|
||||||
|
/// Number of methods in the main interface.
|
||||||
|
pub method_count: usize,
|
||||||
|
/// Sorted names added to the interface.
|
||||||
|
pub added: Vec<String>,
|
||||||
|
/// Sorted names removed from the interface.
|
||||||
|
pub removed: Vec<String>,
|
||||||
|
/// Sorted names whose description or reachable schema changed.
|
||||||
|
pub changed: Vec<String>,
|
||||||
|
/// Whether interface checks passed or were skipped.
|
||||||
|
pub interface_check: InterfaceCheck,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Outcome of the activation's interface check.
|
||||||
|
#[derive(Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
#[schemars(inline)]
|
||||||
|
#[serde(rename_all = "lowercase")]
|
||||||
|
pub(crate) enum InterfaceCheck {
|
||||||
|
/// The interface checks passed.
|
||||||
|
Passed,
|
||||||
|
/// The caller explicitly skipped the checks.
|
||||||
|
Skipped,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SchemaType for InterfaceCheck {}
|
||||||
|
|
||||||
|
#[derive(Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
#[serde(transparent)]
|
||||||
|
pub(super) struct ActivationResult(pub BTreeMap<String, ActivationSummary>);
|
||||||
|
impl SchemaType for ActivationResult {}
|
||||||
|
|
||||||
|
// Native optional fields retain the core's lenient deserializers, which are
|
||||||
|
// intentionally outside the SDK attribute macro's plugin schema subset.
|
||||||
|
macro_rules! native_params {
|
||||||
|
($($name:ty),*) => { $(impl SchemaType for $name {} impl Params for $name {})* };
|
||||||
|
}
|
||||||
|
native_params!(Call, Lock, Log, Activate, Revert);
|
||||||
|
|
||||||
|
// Parameter decoding stays at the core boundary, not in the schema builder.
|
||||||
|
pub(super) fn params<P: Params + serde::de::DeserializeOwned>(p: Value) -> Result<P, Value> {
|
||||||
|
serde_json::from_value(p).map_err(|error| crate::error(-32602, &error.to_string()))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn result<R: serde::de::DeserializeOwned>(value: Value) -> Result<R, Value> {
|
||||||
|
serde_json::from_value(value).map_err(|error| crate::error(-32603, &error.to_string()))
|
||||||
|
}
|
||||||
+23
-3
@@ -47,29 +47,49 @@ impl Host {
|
|||||||
tracker: Process,
|
tracker: Process,
|
||||||
) -> anyhow::Result<Mounts> {
|
) -> anyhow::Result<Mounts> {
|
||||||
let root = self.0.world.root.clone();
|
let root = self.0.world.root.clone();
|
||||||
|
let operations = tracker.clone();
|
||||||
let host = tracker
|
let host = tracker
|
||||||
.clone()
|
.clone()
|
||||||
.blocking(move || sausage_vfs::Host::new(root, tracker))
|
.blocking(move || sausage_vfs::Host::new(root, tracker))
|
||||||
.await?;
|
.await?;
|
||||||
let mut mounts = Mounts::default();
|
let mut mounts = Mounts::default();
|
||||||
mounts.mount("/", Arc::new(host), false);
|
mounts.mount(
|
||||||
|
"/",
|
||||||
|
self.0
|
||||||
|
.observer
|
||||||
|
.layer(Arc::new(host), "/", operations.clone()),
|
||||||
|
false,
|
||||||
|
);
|
||||||
mounts.mount("/proc", Arc::new(Proc(self.clone())), true);
|
mounts.mount("/proc", Arc::new(Proc(self.clone())), true);
|
||||||
mounts.mount(
|
mounts.mount(
|
||||||
"/self",
|
"/self",
|
||||||
|
self.0.observer.layer(
|
||||||
Arc::new(SelfFolder {
|
Arc::new(SelfFolder {
|
||||||
host: self.clone(),
|
host: self.clone(),
|
||||||
pid,
|
pid: pid.clone(),
|
||||||
memory,
|
memory,
|
||||||
}),
|
}),
|
||||||
|
&format!("/proc/{}", pid.as_deref().unwrap_or("console")),
|
||||||
|
operations.clone(),
|
||||||
|
),
|
||||||
|
false,
|
||||||
|
);
|
||||||
|
mounts.mount(
|
||||||
|
"/run",
|
||||||
|
self.0
|
||||||
|
.observer
|
||||||
|
.layer(Arc::new(self.0.run.clone()), "/run", operations),
|
||||||
false,
|
false,
|
||||||
);
|
);
|
||||||
mounts.mount("/run", Arc::new(self.0.run.clone()), false);
|
|
||||||
Ok(mounts)
|
Ok(mounts)
|
||||||
}
|
}
|
||||||
fn core_files(&self) -> Generated {
|
fn core_files(&self) -> Generated {
|
||||||
let mut files = Generated::default();
|
let mut files = Generated::default();
|
||||||
let protocol = include_bytes!("../../PROTOCOL.md");
|
let protocol = include_bytes!("../../PROTOCOL.md");
|
||||||
files.insert("/PROTOCOL.md", move || Ok(protocol.to_vec()));
|
files.insert("/PROTOCOL.md", move || Ok(protocol.to_vec()));
|
||||||
|
files.insert("/discover.json", || {
|
||||||
|
Ok(serde_json::to_vec(crate::core::discover()).unwrap())
|
||||||
|
});
|
||||||
let graph = self.graph_json();
|
let graph = self.graph_json();
|
||||||
files.insert("/graph.json", move || Ok(graph.clone()));
|
files.insert("/graph.json", move || Ok(graph.clone()));
|
||||||
let dot = self.graph_dot();
|
let dot = self.graph_dot();
|
||||||
|
|||||||
+10
-10
@@ -61,6 +61,15 @@ pub(crate) fn problems(
|
|||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
// Core is a native provider, not an installed plugin or cached WASM.
|
||||||
|
if provider == "core" {
|
||||||
|
for problem in
|
||||||
|
sausage_interface::compat::check(expected, crate::core::discover())
|
||||||
|
{
|
||||||
|
failures.push(json!({"consumer":consumer,"provider":provider,"status":"broken","method":problem.method,"message":problem.message,"counterexample":problem.counterexample,"display":problem.to_string()}));
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
let Some(actual) = plugins.get(provider) else {
|
let Some(actual) = plugins.get(provider) else {
|
||||||
failures.push(json!({"consumer":consumer,"provider":provider,"status":"missing","message":"the provider does not exist"}));
|
failures.push(json!({"consumer":consumer,"provider":provider,"status":"missing","message":"the provider does not exist"}));
|
||||||
continue;
|
continue;
|
||||||
@@ -223,16 +232,7 @@ impl Host {
|
|||||||
if let Some(old) = old_plugins.get(plugin).filter(|old| {
|
if let Some(old) = old_plugins.get(plugin).filter(|old| {
|
||||||
!old.hash.is_empty() && old.hash != current.hash && current.error.is_none()
|
!old.hash.is_empty() && old.hash != current.hash && current.error.is_none()
|
||||||
}) {
|
}) {
|
||||||
let grace = old.document["x-sausage-grace"]
|
upgrades.push(self.snapshot_upgrade(plugin, old.hash.clone(), ¤t.hash));
|
||||||
.as_f64()
|
|
||||||
.filter(|value| value.is_finite() && *value >= 0.0)
|
|
||||||
.unwrap_or(60.0);
|
|
||||||
upgrades.push(self.snapshot_upgrade(
|
|
||||||
plugin,
|
|
||||||
old.hash.clone(),
|
|
||||||
¤t.hash,
|
|
||||||
grace,
|
|
||||||
));
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
self.publish_graph(plugins);
|
self.publish_graph(plugins);
|
||||||
|
|||||||
+19
-4
@@ -26,6 +26,8 @@ pub(crate) struct Inner {
|
|||||||
pub(crate) journal: Arc<journal::Journal>,
|
pub(crate) journal: Arc<journal::Journal>,
|
||||||
pub(crate) notifications: tokio::sync::broadcast::Sender<Value>,
|
pub(crate) notifications: tokio::sync::broadcast::Sender<Value>,
|
||||||
pub(crate) exit: watch::Sender<bool>,
|
pub(crate) exit: watch::Sender<bool>,
|
||||||
|
pub(crate) loop_supervised: bool,
|
||||||
|
pub(crate) restart_pending: std::sync::atomic::AtomicBool,
|
||||||
pub(crate) components: components::Components,
|
pub(crate) components: components::Components,
|
||||||
/// Calls hold it for reading from the choice of a version until their
|
/// Calls hold it for reading from the choice of a version until their
|
||||||
/// process is in the table. Activation holds it for writing while it
|
/// process is in the table. Activation holds it for writing while it
|
||||||
@@ -35,6 +37,8 @@ pub(crate) struct Inner {
|
|||||||
/// processes, and upgrade tasks do nothing.
|
/// processes, and upgrade tasks do nothing.
|
||||||
pub(crate) shutdown: std::sync::atomic::AtomicBool,
|
pub(crate) shutdown: std::sync::atomic::AtomicBool,
|
||||||
pub(crate) run: sausage_vfs::Memory,
|
pub(crate) run: sausage_vfs::Memory,
|
||||||
|
pub(crate) watches: crate::watches::Watches,
|
||||||
|
pub(crate) observer: sausage_vfs::observe::Observer,
|
||||||
pub(crate) graph: Mutex<crate::graph::Graph>,
|
pub(crate) graph: Mutex<crate::graph::Graph>,
|
||||||
pub(crate) activations: tokio::sync::Mutex<()>,
|
pub(crate) activations: tokio::sync::Mutex<()>,
|
||||||
pub(crate) watcher: Mutex<Option<notify::RecommendedWatcher>>,
|
pub(crate) watcher: Mutex<Option<notify::RecommendedWatcher>>,
|
||||||
@@ -64,8 +68,11 @@ impl Host {
|
|||||||
Ok((World::open(&path)?, Runtime::new(&cache)?))
|
Ok((World::open(&path)?, Runtime::new(&cache)?))
|
||||||
})
|
})
|
||||||
.await??;
|
.await??;
|
||||||
let journal =
|
let journal = journal::Journal::open_with_legacy(
|
||||||
journal::Journal::open(world.state_dir("core").join("journal.sqlite")).await?;
|
world.root.join("logs/core/journal.sqlite"),
|
||||||
|
Some(world.state_dir("core").join("journal.sqlite")),
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
let host = Self::with_parts(world, runtime, journal);
|
let host = Self::with_parts(world, runtime, journal);
|
||||||
let cargo_world = host.0.world.clone();
|
let cargo_world = host.0.world.clone();
|
||||||
let cargo = tokio::task::spawn_blocking(move || {
|
let cargo = tokio::task::spawn_blocking(move || {
|
||||||
@@ -86,6 +93,10 @@ impl Host {
|
|||||||
))
|
))
|
||||||
}
|
}
|
||||||
fn with_parts(world: World, runtime: Arc<Runtime>, journal: journal::Journal) -> Self {
|
fn with_parts(world: World, runtime: Arc<Runtime>, journal: journal::Journal) -> Self {
|
||||||
|
let watches = crate::watches::Watches::default();
|
||||||
|
let events = watches.clone();
|
||||||
|
let observer =
|
||||||
|
sausage_vfs::observe::Observer::new(move |path| crate::watches::changed(&events, path));
|
||||||
Self(Arc::new(Inner {
|
Self(Arc::new(Inner {
|
||||||
world,
|
world,
|
||||||
runtime,
|
runtime,
|
||||||
@@ -98,10 +109,14 @@ impl Host {
|
|||||||
}),
|
}),
|
||||||
notifications: tokio::sync::broadcast::channel(1024).0,
|
notifications: tokio::sync::broadcast::channel(1024).0,
|
||||||
exit: watch::channel(false).0,
|
exit: watch::channel(false).0,
|
||||||
|
loop_supervised: crate::restart::enabled(std::env::var_os("SAUSAGE_LOOP").as_deref()),
|
||||||
|
restart_pending: std::sync::atomic::AtomicBool::new(false),
|
||||||
components: components::Components::default(),
|
components: components::Components::default(),
|
||||||
admission: tokio::sync::RwLock::new(()),
|
admission: tokio::sync::RwLock::new(()),
|
||||||
shutdown: std::sync::atomic::AtomicBool::new(false),
|
shutdown: std::sync::atomic::AtomicBool::new(false),
|
||||||
run: sausage_vfs::Memory::default(),
|
run: sausage_vfs::Memory::default(),
|
||||||
|
watches,
|
||||||
|
observer,
|
||||||
graph: Mutex::new(crate::graph::Graph::default()),
|
graph: Mutex::new(crate::graph::Graph::default()),
|
||||||
activations: tokio::sync::Mutex::new(()),
|
activations: tokio::sync::Mutex::new(()),
|
||||||
watcher: Mutex::new(None),
|
watcher: Mutex::new(None),
|
||||||
@@ -143,12 +158,12 @@ impl Host {
|
|||||||
)
|
)
|
||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
/// Wait until a core.exit handler requests a restart.
|
/// Wait until core.restart (or its alias core.exit) requests shutdown after its response grace.
|
||||||
pub async fn exit_requested(&self) {
|
pub async fn exit_requested(&self) {
|
||||||
let mut exit = self.0.exit.subscribe();
|
let mut exit = self.0.exit.subscribe();
|
||||||
let _ = exit.wait_for(|requested| *requested).await;
|
let _ = exit.wait_for(|requested| *requested).await;
|
||||||
}
|
}
|
||||||
/// Return whether a core.exit handler requested a restart.
|
/// Return whether a restart has reached the normal shutdown path.
|
||||||
pub fn restart_requested(&self) -> bool {
|
pub fn restart_requested(&self) -> bool {
|
||||||
*self.0.exit.borrow()
|
*self.0.exit.borrow()
|
||||||
}
|
}
|
||||||
|
|||||||
+287
-19
@@ -1,19 +1,31 @@
|
|||||||
//! A dedicated SQLite writer batches entries and serves ordered queries without propagating disk failures into the host.
|
//! A dedicated SQLite writer batches entries and serves ordered queries without propagating disk failures into the host.
|
||||||
use rusqlite::{Connection, params};
|
use rusqlite::{Connection, OpenFlags, params};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
|
use std::io::Write;
|
||||||
use std::{
|
use std::{
|
||||||
collections::HashSet,
|
collections::HashSet,
|
||||||
path::PathBuf,
|
path::PathBuf,
|
||||||
sync::{Mutex, mpsc},
|
sync::{Mutex, mpsc},
|
||||||
};
|
};
|
||||||
use tokio::sync::oneshot;
|
use tokio::sync::oneshot;
|
||||||
struct Entry {
|
/// One journal entry, with its original ID and timestamp.
|
||||||
|
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
|
||||||
|
pub(crate) struct Entry {
|
||||||
|
/// UTC timestamp.
|
||||||
|
time: String,
|
||||||
|
/// UUIDv7 entry ID, ordered by time.
|
||||||
id: String,
|
id: String,
|
||||||
|
/// Process ID.
|
||||||
pid: String,
|
pid: String,
|
||||||
|
/// Plugin name.
|
||||||
plugin: String,
|
plugin: String,
|
||||||
|
/// stderr or core.
|
||||||
stream: String,
|
stream: String,
|
||||||
|
/// Complete log line.
|
||||||
line: String,
|
line: String,
|
||||||
}
|
}
|
||||||
|
impl sausage_plugin::SchemaType for Entry {}
|
||||||
enum Command {
|
enum Command {
|
||||||
Write(Entry),
|
Write(Entry),
|
||||||
Query(Value, oneshot::Sender<Value>),
|
Query(Value, oneshot::Sender<Value>),
|
||||||
@@ -21,44 +33,83 @@ enum Command {
|
|||||||
pub(crate) struct Journal {
|
pub(crate) struct Journal {
|
||||||
sender: mpsc::Sender<Command>,
|
sender: mpsc::Sender<Command>,
|
||||||
warned: Mutex<HashSet<String>>,
|
warned: Mutex<HashSet<String>>,
|
||||||
|
spool: PathBuf,
|
||||||
}
|
}
|
||||||
impl Journal {
|
impl Journal {
|
||||||
pub(crate) async fn memory() -> anyhow::Result<Self> {
|
pub(crate) async fn memory() -> anyhow::Result<Self> {
|
||||||
Self::open(PathBuf::from(":memory:")).await
|
Self::open(PathBuf::from(":memory:")).await
|
||||||
}
|
}
|
||||||
pub(crate) async fn open(path: PathBuf) -> anyhow::Result<Self> {
|
pub(crate) async fn open(path: PathBuf) -> anyhow::Result<Self> {
|
||||||
|
Self::open_with_legacy(path, None).await
|
||||||
|
}
|
||||||
|
pub(crate) async fn open_with_legacy(
|
||||||
|
path: PathBuf,
|
||||||
|
legacy: Option<PathBuf>,
|
||||||
|
) -> anyhow::Result<Self> {
|
||||||
|
let spool = path
|
||||||
|
.parent()
|
||||||
|
.unwrap_or(std::path::Path::new("."))
|
||||||
|
.join("failed-batches");
|
||||||
|
let worker_spool = spool.clone();
|
||||||
let (sender, receiver) = mpsc::channel();
|
let (sender, receiver) = mpsc::channel();
|
||||||
let (ready, initialized) = oneshot::channel();
|
let (ready, initialized) = oneshot::channel();
|
||||||
std::thread::spawn(move || match open_database(path) {
|
std::thread::spawn(move || {
|
||||||
Ok(mut db) => {
|
match (|| -> anyhow::Result<_> {
|
||||||
|
let db = open_database(path)?;
|
||||||
|
// Never rename, checkpoint, or write the old SQLite/WAL files.
|
||||||
|
let old = legacy
|
||||||
|
.filter(|p| p.exists())
|
||||||
|
.map(|p| Connection::open_with_flags(p, OpenFlags::SQLITE_OPEN_READ_ONLY))
|
||||||
|
.transpose()?;
|
||||||
|
Ok((db, old))
|
||||||
|
})() {
|
||||||
|
Ok((mut db, old)) => {
|
||||||
let _ = ready.send(Ok(()));
|
let _ = ready.send(Ok(()));
|
||||||
|
let mut pending = Vec::new();
|
||||||
while let Ok(first) = receiver.recv() {
|
while let Ok(first) = receiver.recv() {
|
||||||
let mut batch = vec![first];
|
let mut batch = vec![first];
|
||||||
batch.extend(receiver.try_iter().take(255));
|
batch.extend(receiver.try_iter().take(255));
|
||||||
write_batch(&mut db, batch);
|
pending = write_batch(&mut db, old.as_ref(), &worker_spool, pending, batch);
|
||||||
|
}
|
||||||
|
if !pending.is_empty() {
|
||||||
|
// Both storage locations failed. Last resort: retain full text in host stderr.
|
||||||
|
eprintln!(
|
||||||
|
"journal shutdown with unpersisted entries: {}",
|
||||||
|
serde_json::to_string(&pending).unwrap()
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Err(error) => {
|
Err(error) => {
|
||||||
eprintln!("journal open failed: {error}");
|
eprintln!("journal open failed: {error}");
|
||||||
let _ = ready.send(Err(error));
|
let _ = ready.send(Err(error));
|
||||||
}
|
}
|
||||||
|
}
|
||||||
});
|
});
|
||||||
initialized.await??;
|
initialized.await??;
|
||||||
Ok(Self {
|
Ok(Self {
|
||||||
sender,
|
sender,
|
||||||
warned: Mutex::new(HashSet::new()),
|
warned: Mutex::new(HashSet::new()),
|
||||||
|
spool,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
pub(crate) fn write(&self, pid: &str, plugin: &str, stream: &str, line: &str) {
|
pub(crate) fn write(&self, pid: &str, plugin: &str, stream: &str, line: &str) {
|
||||||
let entry = Entry {
|
let entry = Entry {
|
||||||
|
time: chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Millis, true),
|
||||||
id: uuid::Uuid::now_v7().to_string(),
|
id: uuid::Uuid::now_v7().to_string(),
|
||||||
pid: pid.into(),
|
pid: pid.into(),
|
||||||
plugin: plugin.into(),
|
plugin: plugin.into(),
|
||||||
stream: stream.into(),
|
stream: stream.into(),
|
||||||
line: line.into(),
|
line: line.into(),
|
||||||
};
|
};
|
||||||
if self.sender.send(Command::Write(entry)).is_err() {
|
if let Err(mpsc::SendError(Command::Write(entry))) = self.sender.send(Command::Write(entry))
|
||||||
eprintln!("journal writer stopped");
|
{
|
||||||
|
eprintln!("journal writer stopped; saving entry outside SQLite");
|
||||||
|
if let Err(error) = spool_batch(&self.spool, std::slice::from_ref(&entry)) {
|
||||||
|
eprintln!(
|
||||||
|
"journal spool failed: {error}; unpersisted entry: {}",
|
||||||
|
serde_json::to_string(&entry).unwrap()
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
pub(crate) fn warning(&self, pid: &str, plugin: &str) {
|
pub(crate) fn warning(&self, pid: &str, plugin: &str) {
|
||||||
@@ -94,28 +145,94 @@ fn open_database(path: PathBuf) -> anyhow::Result<Connection> {
|
|||||||
CREATE INDEX IF NOT EXISTS journal_plugin ON journal(plugin);")?;
|
CREATE INDEX IF NOT EXISTS journal_plugin ON journal(plugin);")?;
|
||||||
Ok(db)
|
Ok(db)
|
||||||
}
|
}
|
||||||
fn write_batch(db: &mut Connection, batch: Vec<Command>) {
|
/// Exclusive, synced JSON files preserve failed transactions without a busy retry loop.
|
||||||
|
/// Recovery is deliberate: files are never automatically removed or replayed.
|
||||||
|
fn spool_batch(dir: &std::path::Path, entries: &[Entry]) -> anyhow::Result<PathBuf> {
|
||||||
|
std::fs::create_dir_all(dir)?;
|
||||||
|
let path = dir.join(format!("{}.json", uuid::Uuid::now_v7()));
|
||||||
|
// A failed/partial write must not look like a complete recovery batch.
|
||||||
|
let mut file = tempfile::NamedTempFile::new_in(dir)?;
|
||||||
|
serde_json::to_writer(&mut file, entries)?;
|
||||||
|
file.write_all(b"\n")?;
|
||||||
|
file.as_file().sync_all()?;
|
||||||
|
file.persist_noclobber(&path)?;
|
||||||
|
std::fs::File::open(dir)?.sync_all()?;
|
||||||
|
eprintln!(
|
||||||
|
"journal failed batch preserved: {} ({} entries)",
|
||||||
|
path.display(),
|
||||||
|
entries.len()
|
||||||
|
);
|
||||||
|
Ok(path)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn write_batch(
|
||||||
|
db: &mut Connection,
|
||||||
|
old: Option<&Connection>,
|
||||||
|
spool: &std::path::Path,
|
||||||
|
mut pending: Vec<Entry>,
|
||||||
|
batch: Vec<Command>,
|
||||||
|
) -> Vec<Entry> {
|
||||||
|
let mut queries = Vec::new();
|
||||||
|
for command in batch {
|
||||||
|
match command {
|
||||||
|
Command::Write(entry) => pending.push(entry),
|
||||||
|
Command::Query(filters, response) => queries.push((filters, response)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !pending.is_empty() {
|
||||||
let write = (|| -> rusqlite::Result<()> {
|
let write = (|| -> rusqlite::Result<()> {
|
||||||
let transaction = db.transaction()?;
|
let transaction = db.transaction()?;
|
||||||
for command in &batch {
|
for e in &pending {
|
||||||
if let Command::Write(e) = command {
|
transaction.execute("INSERT OR IGNORE INTO journal (id,time,pid,plugin,stream,line) VALUES (?1,?2,?3,?4,?5,?6)",params![e.id,e.time,e.pid,e.plugin,e.stream,e.line])?;
|
||||||
transaction.execute("INSERT INTO journal (id,time,pid,plugin,stream,line) VALUES (?1,strftime('%Y-%m-%dT%H:%M:%fZ','now'),?2,?3,?4,?5)",params![e.id,e.pid,e.plugin,e.stream,e.line])?;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
transaction.commit()
|
transaction.commit()
|
||||||
})();
|
})();
|
||||||
if let Err(error) = write {
|
match write {
|
||||||
|
Ok(()) => pending.clear(),
|
||||||
|
Err(error) => {
|
||||||
eprintln!("journal write failed: {error}");
|
eprintln!("journal write failed: {error}");
|
||||||
|
match spool_batch(spool, &pending) {
|
||||||
|
Ok(_) => pending.clear(),
|
||||||
|
Err(error) => eprintln!(
|
||||||
|
"journal spool failed: {error}; retaining {} entries in memory until next command",
|
||||||
|
pending.len()
|
||||||
|
),
|
||||||
}
|
}
|
||||||
for command in batch {
|
}
|
||||||
if let Command::Query(filters, response) = command {
|
}
|
||||||
let value = query_database(db, &filters).unwrap_or_else(|error| {
|
}
|
||||||
|
for (filters, response) in queries {
|
||||||
|
let value = query_all(db, old, &filters).unwrap_or_else(|error| {
|
||||||
eprintln!("journal query failed: {error}");
|
eprintln!("journal query failed: {error}");
|
||||||
json!([])
|
json!([])
|
||||||
});
|
});
|
||||||
let _ = response.send(value);
|
let _ = response.send(value);
|
||||||
}
|
}
|
||||||
|
pending
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn query_all(
|
||||||
|
db: &Connection,
|
||||||
|
old: Option<&Connection>,
|
||||||
|
filters: &Value,
|
||||||
|
) -> rusqlite::Result<Value> {
|
||||||
|
let mut entries = query_database(db, filters)?.as_array().unwrap().clone();
|
||||||
|
if let Some(old) = old {
|
||||||
|
entries.extend(
|
||||||
|
query_database(old, filters)?
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.iter()
|
||||||
|
.cloned(),
|
||||||
|
);
|
||||||
|
entries.sort_by(|a, b| a["id"].as_str().cmp(&b["id"].as_str()));
|
||||||
|
entries.dedup_by(|a, b| a["id"] == b["id"]);
|
||||||
|
let limit = filters["limit"].as_i64().unwrap_or(100);
|
||||||
|
if limit >= 0 {
|
||||||
|
entries.drain(..entries.len().saturating_sub(limit as usize));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(json!(entries))
|
||||||
}
|
}
|
||||||
fn query_database(db: &Connection, filters: &Value) -> rusqlite::Result<Value> {
|
fn query_database(db: &Connection, filters: &Value) -> rusqlite::Result<Value> {
|
||||||
let mut statement = db.prepare(
|
let mut statement = db.prepare(
|
||||||
@@ -124,9 +241,26 @@ fn query_database(db: &Connection, filters: &Value) -> rusqlite::Result<Value> {
|
|||||||
AND (?3 IS NULL OR id>=?3) AND (?4 IS NULL OR id<=?4)
|
AND (?3 IS NULL OR id>=?3) AND (?4 IS NULL OR id<=?4)
|
||||||
AND (?5 IS NULL OR instr(line,?5)>0) ORDER BY id DESC LIMIT ?6",
|
AND (?5 IS NULL OR instr(line,?5)>0) ORDER BY id DESC LIMIT ?6",
|
||||||
)?;
|
)?;
|
||||||
let rows=statement.query_map(params![filters["plugin"].as_str(),filters["pid"].as_str(),filters["since"].as_str(),filters["until"].as_str(),filters["grep"].as_str(),filters["limit"].as_i64().unwrap_or(100)],|row| Ok(json!({
|
let rows = statement.query_map(
|
||||||
"id":row.get::<_,String>(0)?,"time":row.get::<_,String>(1)?,"pid":row.get::<_,String>(2)?,"plugin":row.get::<_,String>(3)?,"stream":row.get::<_,String>(4)?,"line":row.get::<_,String>(5)?,
|
params![
|
||||||
})))?;
|
filters["plugin"].as_str(),
|
||||||
|
filters["pid"].as_str(),
|
||||||
|
filters["since"].as_str(),
|
||||||
|
filters["until"].as_str(),
|
||||||
|
filters["grep"].as_str(),
|
||||||
|
filters["limit"].as_i64().unwrap_or(100)
|
||||||
|
],
|
||||||
|
|row| {
|
||||||
|
Ok(json!(Entry {
|
||||||
|
id: row.get(0)?,
|
||||||
|
time: row.get(1)?,
|
||||||
|
pid: row.get(2)?,
|
||||||
|
plugin: row.get(3)?,
|
||||||
|
stream: row.get(4)?,
|
||||||
|
line: row.get(5)?,
|
||||||
|
}))
|
||||||
|
},
|
||||||
|
)?;
|
||||||
let mut entries: Vec<_> = rows.collect::<rusqlite::Result<_>>()?;
|
let mut entries: Vec<_> = rows.collect::<rusqlite::Result<_>>()?;
|
||||||
entries.reverse();
|
entries.reverse();
|
||||||
Ok(json!(entries))
|
Ok(json!(entries))
|
||||||
@@ -135,6 +269,140 @@ fn query_database(db: &Connection, filters: &Value) -> rusqlite::Result<Value> {
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
fn entry(line: &str) -> Entry {
|
||||||
|
Entry {
|
||||||
|
id: uuid::Uuid::now_v7().to_string(),
|
||||||
|
time: "2026-10-26T12:00:00.000Z".into(),
|
||||||
|
pid: "pid".into(),
|
||||||
|
plugin: "test".into(),
|
||||||
|
stream: "core".into(),
|
||||||
|
line: line.into(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn failed_transaction_is_spooled_in_full_and_queries_finish() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let mut db = open_database(PathBuf::from(":memory:")).unwrap();
|
||||||
|
db.execute_batch("PRAGMA query_only=ON").unwrap();
|
||||||
|
let original = "😀 full failure\n".repeat(6000);
|
||||||
|
let (tx, mut rx) = oneshot::channel();
|
||||||
|
let pending = write_batch(
|
||||||
|
&mut db,
|
||||||
|
None,
|
||||||
|
dir.path(),
|
||||||
|
vec![],
|
||||||
|
vec![
|
||||||
|
Command::Write(entry(&original)),
|
||||||
|
Command::Query(json!({}), tx),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
assert!(pending.is_empty());
|
||||||
|
assert_eq!(rx.try_recv().unwrap(), json!([]));
|
||||||
|
let path = std::fs::read_dir(dir.path())
|
||||||
|
.unwrap()
|
||||||
|
.next()
|
||||||
|
.unwrap()
|
||||||
|
.unwrap()
|
||||||
|
.path();
|
||||||
|
let saved: Vec<Entry> = serde_json::from_slice(&std::fs::read(path).unwrap()).unwrap();
|
||||||
|
assert_eq!(saved.len(), 1);
|
||||||
|
assert_eq!(saved[0].line, original);
|
||||||
|
assert_eq!(saved[0].time, "2026-10-26T12:00:00.000Z");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn failed_spool_retries_on_next_command_without_duplicates() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let blocked = dir.path().join("blocked");
|
||||||
|
std::fs::write(&blocked, "not a directory").unwrap();
|
||||||
|
let mut db = open_database(PathBuf::from(":memory:")).unwrap();
|
||||||
|
db.execute_batch("PRAGMA query_only=ON").unwrap();
|
||||||
|
let pending = write_batch(
|
||||||
|
&mut db,
|
||||||
|
None,
|
||||||
|
&blocked,
|
||||||
|
vec![],
|
||||||
|
vec![Command::Write(entry("retained"))],
|
||||||
|
);
|
||||||
|
assert_eq!(pending.len(), 1);
|
||||||
|
db.execute_batch("PRAGMA query_only=OFF").unwrap();
|
||||||
|
let (tx, mut rx) = oneshot::channel();
|
||||||
|
let pending = write_batch(
|
||||||
|
&mut db,
|
||||||
|
None,
|
||||||
|
&blocked,
|
||||||
|
pending,
|
||||||
|
vec![Command::Query(json!({}), tx)],
|
||||||
|
);
|
||||||
|
assert!(pending.is_empty());
|
||||||
|
assert_eq!(rx.try_recv().unwrap()[0]["line"], "retained");
|
||||||
|
assert_eq!(
|
||||||
|
query_all(&db, None, &json!({}))
|
||||||
|
.unwrap()
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn legacy_wal_and_new_journal_share_filters_order_and_limit() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let legacy = dir.path().join("state/core/journal.sqlite");
|
||||||
|
let mut old = open_database(legacy.clone()).unwrap();
|
||||||
|
write_batch(
|
||||||
|
&mut old,
|
||||||
|
None,
|
||||||
|
dir.path(),
|
||||||
|
vec![],
|
||||||
|
vec![Command::Write(entry("old match"))],
|
||||||
|
);
|
||||||
|
// Leave the WAL connection open: opening the new journal must not move its files.
|
||||||
|
let wal = legacy.with_extension("sqlite-wal");
|
||||||
|
let before = std::fs::read(&wal).unwrap();
|
||||||
|
let new = dir.path().join("logs/core/journal.sqlite");
|
||||||
|
let journal = Journal::open_with_legacy(new.clone(), Some(legacy.clone()))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
journal.write("pid", "test", "core", "new match");
|
||||||
|
let entries = tokio::time::timeout(
|
||||||
|
std::time::Duration::from_secs(5),
|
||||||
|
journal.query(json!({"pid":"pid","grep":"match","limit":-1})),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(entries.as_array().unwrap().len(), 2);
|
||||||
|
assert_eq!(entries[0]["line"], "old match");
|
||||||
|
assert_eq!(entries[1]["line"], "new match");
|
||||||
|
assert_eq!(
|
||||||
|
journal.query(json!({"limit":1})).await[0]["line"],
|
||||||
|
"new match"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
journal.query(json!({"until":entries[0]["id"]})).await[0]["line"],
|
||||||
|
"old match"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
journal
|
||||||
|
.query(json!({"since":entries[1]["id"],"plugin":"test"}))
|
||||||
|
.await[0]["line"],
|
||||||
|
"new match"
|
||||||
|
);
|
||||||
|
assert_eq!(journal.query(json!({"grep":"absent"})).await, json!([]));
|
||||||
|
assert_eq!(std::fs::read(wal).unwrap(), before);
|
||||||
|
assert!(legacy.exists() && new.exists());
|
||||||
|
assert_eq!(
|
||||||
|
query_database(&old, &json!({}))
|
||||||
|
.unwrap()
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// PROTOCOL 7: completed processes do not leave warning identities in memory.
|
/// PROTOCOL 7: completed processes do not leave warning identities in memory.
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn completion_forgets_warning_identity() {
|
async fn completion_forgets_warning_identity() {
|
||||||
|
|||||||
@@ -12,11 +12,13 @@ mod journal;
|
|||||||
mod locks;
|
mod locks;
|
||||||
mod native;
|
mod native;
|
||||||
mod process;
|
mod process;
|
||||||
|
mod restart;
|
||||||
mod routing;
|
mod routing;
|
||||||
pub mod runtime;
|
pub mod runtime;
|
||||||
mod services;
|
mod services;
|
||||||
mod upgrades;
|
mod upgrades;
|
||||||
mod watcher;
|
mod watcher;
|
||||||
|
mod watches;
|
||||||
pub mod world;
|
pub mod world;
|
||||||
pub use host::Host;
|
pub use host::Host;
|
||||||
pub use native::{NativeContext, NativeFuture, NativePlugin};
|
pub use native::{NativeContext, NativeFuture, NativePlugin};
|
||||||
|
|||||||
+4
-4
@@ -1,6 +1,6 @@
|
|||||||
//! Advisory FIFO locks and cleanup of ended processes.
|
//! Advisory FIFO locks and cleanup of ended processes.
|
||||||
use crate::error;
|
|
||||||
use crate::{Host, RpcResult, host::State};
|
use crate::{Host, RpcResult, host::State};
|
||||||
|
use crate::{core::Held, error};
|
||||||
use serde_json::json;
|
use serde_json::json;
|
||||||
use std::collections::VecDeque;
|
use std::collections::VecDeque;
|
||||||
use tokio::sync::oneshot;
|
use tokio::sync::oneshot;
|
||||||
@@ -26,7 +26,7 @@ impl Host {
|
|||||||
}
|
}
|
||||||
if let Some(lock) = state.locks.get_mut(path) {
|
if let Some(lock) = state.locks.get_mut(path) {
|
||||||
if !wait {
|
if !wait {
|
||||||
return Ok(json!({"held":false}));
|
return Ok(json!(Held { held: false }));
|
||||||
}
|
}
|
||||||
let (tx, rx) = oneshot::channel();
|
let (tx, rx) = oneshot::channel();
|
||||||
lock.queue.push_back((caller.into(), tx));
|
lock.queue.push_back((caller.into(), tx));
|
||||||
@@ -46,7 +46,7 @@ impl Host {
|
|||||||
Some(rx) => rx
|
Some(rx) => rx
|
||||||
.await
|
.await
|
||||||
.unwrap_or_else(|_| Err(error(-32603, "lock canceled"))),
|
.unwrap_or_else(|_| Err(error(-32603, "lock canceled"))),
|
||||||
None => Ok(json!({"held":true})),
|
None => Ok(json!(Held { held: true })),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/// Release a held lock only when this process owns it.
|
/// Release a held lock only when this process owns it.
|
||||||
@@ -80,7 +80,7 @@ impl Host {
|
|||||||
pub(crate) fn grant(state: &mut State, path: &str) {
|
pub(crate) fn grant(state: &mut State, path: &str) {
|
||||||
if let Some(mut lock) = state.locks.remove(path) {
|
if let Some(mut lock) = state.locks.remove(path) {
|
||||||
while let Some((owner, tx)) = lock.queue.pop_front() {
|
while let Some((owner, tx)) = lock.queue.pop_front() {
|
||||||
if tx.send(Ok(json!({"held":true}))).is_ok() {
|
if tx.send(Ok(json!(Held { held: true }))).is_ok() {
|
||||||
lock.owner = owner;
|
lock.owner = owner;
|
||||||
state.locks.insert(path.into(), lock);
|
state.locks.insert(path.into(), lock);
|
||||||
break;
|
break;
|
||||||
|
|||||||
+22
-1
@@ -33,6 +33,8 @@ pub(crate) struct Process {
|
|||||||
pub(crate) id: Option<Value>,
|
pub(crate) id: Option<Value>,
|
||||||
pub(crate) answer: Option<oneshot::Sender<RpcResult>>,
|
pub(crate) answer: Option<oneshot::Sender<RpcResult>>,
|
||||||
pub(crate) ended: watch::Sender<bool>,
|
pub(crate) ended: watch::Sender<bool>,
|
||||||
|
pub(crate) stopped: bool,
|
||||||
|
pub(crate) superseded: bool,
|
||||||
}
|
}
|
||||||
impl Host {
|
impl Host {
|
||||||
/// Called only after the execution task ends. Wait for the store and file operations before releasing locks.
|
/// Called only after the execution task ends. Wait for the store and file operations before releasing locks.
|
||||||
@@ -57,6 +59,7 @@ impl Host {
|
|||||||
.map(|_| self.0.journal.query_later(json!({"pid":pid,"limit":20})));
|
.map(|_| self.0.journal.query_later(json!({"pid":pid,"limit":20})));
|
||||||
self.0.journal.forget(pid);
|
self.0.journal.forget(pid);
|
||||||
Self::release_locks(&mut state, pid);
|
Self::release_locks(&mut state, pid);
|
||||||
|
self.service_ended(&mut state, pid, status, p.stopped, p.superseded);
|
||||||
let status = status.to_string();
|
let status = status.to_string();
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
if let Some(answer) = p.answer {
|
if let Some(answer) = p.answer {
|
||||||
@@ -84,13 +87,24 @@ impl Host {
|
|||||||
}
|
}
|
||||||
/// Close stdin and start the same five-second deadline for each attached child.
|
/// Close stdin and start the same five-second deadline for each attached child.
|
||||||
pub(crate) fn begin_stop(&self, pid: &str) -> Option<watch::Receiver<bool>> {
|
pub(crate) fn begin_stop(&self, pid: &str) -> Option<watch::Receiver<bool>> {
|
||||||
|
self.stop_process(pid, false)
|
||||||
|
}
|
||||||
|
/// Grace expiry is retirement, not an operator stopping the service.
|
||||||
|
pub(crate) fn begin_retire(&self, pid: &str) -> Option<watch::Receiver<bool>> {
|
||||||
|
self.stop_process(pid, true)
|
||||||
|
}
|
||||||
|
fn stop_process(&self, pid: &str, retirement: bool) -> Option<watch::Receiver<bool>> {
|
||||||
let mut state = self.0.state.lock().unwrap();
|
let mut state = self.0.state.lock().unwrap();
|
||||||
let p = state.processes.get_mut(pid)?;
|
let p = state.processes.get_mut(pid)?;
|
||||||
|
if !retirement {
|
||||||
|
p.superseded = false;
|
||||||
|
}
|
||||||
let ended = p.ended.subscribe();
|
let ended = p.ended.subscribe();
|
||||||
if p.info.state == "stopping" {
|
if p.info.state == "stopping" {
|
||||||
return Some(ended);
|
return Some(ended);
|
||||||
}
|
}
|
||||||
p.info.state = "stopping".into();
|
p.info.state = "stopping".into();
|
||||||
|
p.stopped = true;
|
||||||
p.input.take();
|
p.input.take();
|
||||||
self.0.journal.write(pid, &p.info.plugin, "core", "stop");
|
self.0.journal.write(pid, &p.info.plugin, "core", "stop");
|
||||||
for lock in state.locks.values_mut() {
|
for lock in state.locks.values_mut() {
|
||||||
@@ -110,17 +124,24 @@ impl Host {
|
|||||||
let pid = pid.to_string();
|
let pid = pid.to_string();
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
tokio::time::sleep(std::time::Duration::from_secs(5)).await;
|
tokio::time::sleep(std::time::Duration::from_secs(5)).await;
|
||||||
h.kill(&pid);
|
h.kill_process(&pid, retirement);
|
||||||
});
|
});
|
||||||
Some(ended)
|
Some(ended)
|
||||||
}
|
}
|
||||||
/// Abort a process task and perform its ended-state cleanup.
|
/// Abort a process task and perform its ended-state cleanup.
|
||||||
pub(crate) fn kill(&self, pid: &str) {
|
pub(crate) fn kill(&self, pid: &str) {
|
||||||
|
self.kill_process(pid, false);
|
||||||
|
}
|
||||||
|
fn kill_process(&self, pid: &str, retirement: bool) {
|
||||||
let mut state = self.0.state.lock().unwrap();
|
let mut state = self.0.state.lock().unwrap();
|
||||||
for lock in state.locks.values_mut() {
|
for lock in state.locks.values_mut() {
|
||||||
lock.queue.retain(|(queued, _)| queued != pid);
|
lock.queue.retain(|(queued, _)| queued != pid);
|
||||||
}
|
}
|
||||||
if let Some(p) = state.processes.get_mut(pid) {
|
if let Some(p) = state.processes.get_mut(pid) {
|
||||||
|
if !retirement {
|
||||||
|
p.superseded = false;
|
||||||
|
}
|
||||||
|
p.stopped = true;
|
||||||
p.info.state = "stopping".into();
|
p.info.state = "stopping".into();
|
||||||
p.input.take();
|
p.input.take();
|
||||||
self.0.journal.write(pid, &p.info.plugin, "core", "kill");
|
self.0.journal.write(pid, &p.info.plugin, "core", "kill");
|
||||||
|
|||||||
@@ -0,0 +1,129 @@
|
|||||||
|
//! Supervisor-gated restart requests; the executable still owns normal shutdown.
|
||||||
|
use crate::{Host, RpcResult, error};
|
||||||
|
use serde_json::json;
|
||||||
|
use std::{ffi::OsStr, sync::atomic::Ordering, time::Duration};
|
||||||
|
|
||||||
|
pub(crate) fn enabled(value: Option<&OsStr>) -> bool {
|
||||||
|
value == Some(OsStr::new("1"))
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Host {
|
||||||
|
pub(crate) fn request_restart(&self) -> RpcResult {
|
||||||
|
if !self.0.loop_supervised {
|
||||||
|
return Err(error(
|
||||||
|
-32600,
|
||||||
|
"restart needs the loop supervisor (SAUSAGE_LOOP not set); start with SAUSAGE_LOOP=1",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if !self.0.restart_pending.swap(true, Ordering::SeqCst) {
|
||||||
|
let host = self.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
// Let the RPC response reach the caller before closing any stdin,
|
||||||
|
// including that of a detached gateway carrying this response.
|
||||||
|
tokio::time::sleep(Duration::from_millis(250)).await;
|
||||||
|
host.0.exit.send_replace(true);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(json!({}))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::{runtime::Runtime, world::World};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::time::timeout;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn only_explicit_loop_marker_enables_restart() {
|
||||||
|
assert!(!enabled(None));
|
||||||
|
for value in ["", "0", "true", "yes", "2"] {
|
||||||
|
assert!(!enabled(Some(OsStr::new(value))));
|
||||||
|
}
|
||||||
|
assert!(enabled(Some(OsStr::new("1"))));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn host(supervised: bool) -> (tempfile::TempDir, Host) {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
World::create(&dir.path().join("world")).unwrap();
|
||||||
|
let runtime = Runtime::new(&dir.path().join("cache")).unwrap();
|
||||||
|
let mut host = Host::test_world(&dir.path().join("world"), runtime)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
// No global environment mutation: parallel tests have independent gates.
|
||||||
|
Arc::get_mut(&mut host.0).unwrap().loop_supervised = supervised;
|
||||||
|
(dir, host)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn restart_without_supervisor_never_triggers_exit() {
|
||||||
|
timeout(Duration::from_secs(10), async {
|
||||||
|
let (_dir, host) = host(false).await;
|
||||||
|
for method in ["core.restart", "core.exit"] {
|
||||||
|
let error = host.call(method, json!({})).await.unwrap_err();
|
||||||
|
assert_eq!(error["code"], -32600);
|
||||||
|
assert!(
|
||||||
|
error["message"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.contains("SAUSAGE_LOOP not set")
|
||||||
|
);
|
||||||
|
let indirect = host
|
||||||
|
.call("core.call", json!({"method":method}))
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert_eq!(indirect, error);
|
||||||
|
}
|
||||||
|
assert!(!host.0.restart_pending.load(Ordering::SeqCst));
|
||||||
|
assert!(
|
||||||
|
timeout(Duration::from_millis(300), host.exit_requested())
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn restart_and_exit_share_response_grace_and_coalescing() {
|
||||||
|
timeout(Duration::from_secs(10), async {
|
||||||
|
for first in ["core.restart", "core.exit"] {
|
||||||
|
let (_dir, host) = host(true).await;
|
||||||
|
// Same dispatch as a plugin's core.call, without killing this runner.
|
||||||
|
assert_eq!(
|
||||||
|
host.call("core.call", json!({"method":first}))
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
json!({})
|
||||||
|
);
|
||||||
|
assert!(!host.restart_requested());
|
||||||
|
assert!(host.0.restart_pending.load(Ordering::SeqCst));
|
||||||
|
assert_eq!(
|
||||||
|
host.call("core.restart", json!({})).await.unwrap(),
|
||||||
|
json!({})
|
||||||
|
);
|
||||||
|
timeout(Duration::from_secs(1), host.exit_requested())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
// A duplicate after the trigger must not schedule a second task.
|
||||||
|
host.0.exit.send_replace(false);
|
||||||
|
assert_eq!(
|
||||||
|
host.call("core.restart", json!({})).await.unwrap(),
|
||||||
|
json!({})
|
||||||
|
);
|
||||||
|
assert_eq!(host.call("core.exit", json!({})).await.unwrap(), json!({}));
|
||||||
|
assert!(
|
||||||
|
timeout(Duration::from_millis(300), host.exit_requested())
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
+60
-9
@@ -7,6 +7,23 @@ use serde_json::{Value, json};
|
|||||||
use tokio::sync::{mpsc, oneshot, watch};
|
use tokio::sync::{mpsc, oneshot, watch};
|
||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
fn params_not_object(params: &Value) -> Value {
|
||||||
|
let kind = match params {
|
||||||
|
Value::String(_) => "a string",
|
||||||
|
Value::Array(_) => "an array",
|
||||||
|
Value::Number(_) => "a number",
|
||||||
|
Value::Null => "null",
|
||||||
|
Value::Bool(_) => "a boolean",
|
||||||
|
Value::Object(_) => "an object",
|
||||||
|
};
|
||||||
|
error(
|
||||||
|
-32602,
|
||||||
|
&format!(
|
||||||
|
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
impl Host {
|
impl Host {
|
||||||
/// Queue an incoming response or notification for its destination.
|
/// Queue an incoming response or notification for its destination.
|
||||||
pub(crate) fn send(&self, pid: &str, value: Value) {
|
pub(crate) fn send(&self, pid: &str, value: Value) {
|
||||||
@@ -34,15 +51,20 @@ impl Host {
|
|||||||
.processes
|
.processes
|
||||||
.get(pid)
|
.get(pid)
|
||||||
.and_then(|p| p.answer.as_ref().map(|_| (p.caller.clone(), p.id.clone())));
|
.and_then(|p| p.answer.as_ref().map(|_| (p.caller.clone(), p.id.clone())));
|
||||||
if let Some((caller, id)) = target {
|
if let Some((caller, Some(id))) = target {
|
||||||
let notification = json!({
|
self.call_notify(&caller, &id, name, params);
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"method": "core.notify",
|
|
||||||
"params": { "call": id, "name": name, "params": params }
|
|
||||||
});
|
|
||||||
self.send(&caller, notification);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/// Deliver a notification on an open core call using the same envelope as plugin progress.
|
||||||
|
pub(crate) fn call_notify(&self, caller: &str, id: &Value, name: &str, params: Value) {
|
||||||
|
self.send(
|
||||||
|
caller,
|
||||||
|
json!({
|
||||||
|
"jsonrpc":"2.0", "method":"core.notify",
|
||||||
|
"params": {"call":id, "name":name, "params":params}
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
/// Validate and route a request, then start its native or WASI process.
|
/// Validate and route a request, then start its native or WASI process.
|
||||||
pub(crate) fn route(
|
pub(crate) fn route(
|
||||||
&self,
|
&self,
|
||||||
@@ -69,7 +91,7 @@ impl Host {
|
|||||||
let host = self.clone();
|
let host = self.clone();
|
||||||
Box::pin(async move {
|
Box::pin(async move {
|
||||||
if !params.is_object() {
|
if !params.is_object() {
|
||||||
return Err(error(-32602, "params must be an object"));
|
return Err(params_not_object(¶ms));
|
||||||
}
|
}
|
||||||
if caller != "console" && caller != "core" {
|
if caller != "console" && caller != "core" {
|
||||||
let state = host.0.state.lock().unwrap();
|
let state = host.0.state.lock().unwrap();
|
||||||
@@ -149,6 +171,33 @@ impl Host {
|
|||||||
{
|
{
|
||||||
return Err(error(-32600, "caller stopped or ended"));
|
return Err(error(-32600, "caller stopped or ended"));
|
||||||
}
|
}
|
||||||
|
// Older plugins may still try to start their boot service from
|
||||||
|
// rpc.retired. The supervisor owns it even while a restart is
|
||||||
|
// pending, or after it has given up. Do not create a second owner.
|
||||||
|
if state
|
||||||
|
.processes
|
||||||
|
.get(&caller)
|
||||||
|
.is_some_and(|p| p.info.method == "rpc.retired")
|
||||||
|
&& state
|
||||||
|
.services
|
||||||
|
.iter()
|
||||||
|
.any(|s| s.method == format!("{plugin}.{method}") && s.params == params)
|
||||||
|
{
|
||||||
|
return Ok(Value::Null);
|
||||||
|
}
|
||||||
|
if caller == "core"
|
||||||
|
&& let Some(service) = state
|
||||||
|
.services
|
||||||
|
.iter_mut()
|
||||||
|
.find(|s| Some(&s.request) == id.as_ref())
|
||||||
|
{
|
||||||
|
if !service.enabled || service.owner.is_some() {
|
||||||
|
return Err(error(-32600, "service attempt is no longer current"));
|
||||||
|
}
|
||||||
|
service.owner = Some(pid.clone());
|
||||||
|
service.pending = false;
|
||||||
|
service.version = Some(info.version.clone());
|
||||||
|
}
|
||||||
host.0
|
host.0
|
||||||
.journal
|
.journal
|
||||||
.write(&pid, plugin, "core", &format!("start {method} {params}"));
|
.write(&pid, plugin, "core", &format!("start {method} {params}"));
|
||||||
@@ -165,6 +214,8 @@ impl Host {
|
|||||||
id: id.clone(),
|
id: id.clone(),
|
||||||
answer: if id.is_some() { Some(tx) } else { None },
|
answer: if id.is_some() { Some(tx) } else { None },
|
||||||
ended: watch::channel(false).0,
|
ended: watch::channel(false).0,
|
||||||
|
stopped: false,
|
||||||
|
superseded: false,
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -238,7 +289,7 @@ impl Host {
|
|||||||
// Deliver progress before a following response removes its destination.
|
// Deliver progress before a following response removes its destination.
|
||||||
let params = v.get("params").cloned().unwrap_or(json!({}));
|
let params = v.get("params").cloned().unwrap_or(json!({}));
|
||||||
let result = if !params.is_object() {
|
let result = if !params.is_object() {
|
||||||
Err(error(-32602, "params must be an object"))
|
Err(params_not_object(¶ms))
|
||||||
} else if method == "core.unlock" {
|
} else if method == "core.unlock" {
|
||||||
if let Some(path) = params["path"].as_str() {
|
if let Some(path) = params["path"].as_str() {
|
||||||
self.unlock(pid, path);
|
self.unlock(pid, path);
|
||||||
|
|||||||
+23
-1
@@ -212,7 +212,11 @@ fn short_trap(full: &str) -> String {
|
|||||||
.find_map(|l| l.find("wasm trap:").map(|i| l[i..].to_string()))
|
.find_map(|l| l.find("wasm trap:").map(|i| l[i..].to_string()))
|
||||||
.unwrap_or_else(|| full.lines().next().unwrap_or("trap").to_string());
|
.unwrap_or_else(|| full.lines().next().unwrap_or("trap").to_string());
|
||||||
let mut line = line.trim().to_string();
|
let mut line = line.trim().to_string();
|
||||||
line.truncate(300);
|
let mut end = line.len().min(300);
|
||||||
|
while !line.is_char_boundary(end) {
|
||||||
|
end -= 1;
|
||||||
|
}
|
||||||
|
line.truncate(end);
|
||||||
line
|
line
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -361,6 +365,7 @@ impl Runtime {
|
|||||||
});
|
});
|
||||||
|
|
||||||
let (err_tx, err_rx) = mpsc::channel(1024);
|
let (err_tx, err_rx) = mpsc::channel(1024);
|
||||||
|
let trap_log = log.clone();
|
||||||
let stderr_task = tokio::spawn(async move {
|
let stderr_task = tokio::spawn(async move {
|
||||||
let mut reader = BufReader::new(host_err);
|
let mut reader = BufReader::new(host_err);
|
||||||
let mut bytes = Vec::new();
|
let mut bytes = Vec::new();
|
||||||
@@ -400,6 +405,9 @@ impl Runtime {
|
|||||||
Some(I32Exit(c)) => Exit::Code(*c),
|
Some(I32Exit(c)) => Exit::Code(*c),
|
||||||
None => {
|
None => {
|
||||||
let full = format!("{e:?}");
|
let full = format!("{e:?}");
|
||||||
|
if let Some((journal, pid, plugin)) = &trap_log {
|
||||||
|
journal.write(pid, plugin, "core", &full);
|
||||||
|
}
|
||||||
Exit::Trap(short_trap(&full))
|
Exit::Trap(short_trap(&full))
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -416,3 +424,17 @@ impl Runtime {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod trap_text_tests {
|
||||||
|
use super::*;
|
||||||
|
#[test]
|
||||||
|
fn short_trap_display_is_utf8_safe() {
|
||||||
|
let full = format!("backtrace\nwasm trap: {}\nmore detail", "😀".repeat(1000));
|
||||||
|
let status = short_trap(&full);
|
||||||
|
assert!(status.len() <= 300);
|
||||||
|
assert!(status.starts_with("wasm trap:"));
|
||||||
|
assert_eq!(short_trap(""), "trap");
|
||||||
|
assert_eq!(short_trap("first line\nsecond"), "first line");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+129
-16
@@ -1,12 +1,36 @@
|
|||||||
//! Ordered boot services and their identities for upgrade restarts (PROTOCOL.md section 8).
|
//! Ordered boot calls and the single restart owner for each service (PROTOCOL.md section 8).
|
||||||
use crate::Host;
|
use crate::{Host, host::State};
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
#[derive(Clone)]
|
use std::{collections::VecDeque, sync::atomic::Ordering, time::Duration};
|
||||||
|
use tokio::time::Instant;
|
||||||
|
|
||||||
|
// Systemd-derived values:
|
||||||
|
// https://www.man7.org/linux/man-pages/man5/systemd-system.conf.5.html.
|
||||||
|
const CRASH_LIMIT: usize = 5;
|
||||||
|
const CRASH_WINDOW: Duration = Duration::from_secs(10);
|
||||||
|
|
||||||
pub(crate) struct Service {
|
pub(crate) struct Service {
|
||||||
pub(crate) method: String,
|
pub(crate) method: String,
|
||||||
pub(crate) params: Value,
|
pub(crate) params: Value,
|
||||||
pub(crate) request: Value,
|
pub(crate) request: Value,
|
||||||
pub(crate) starting: std::sync::Arc<std::sync::atomic::AtomicBool>,
|
pub(crate) owner: Option<String>,
|
||||||
|
pub(crate) pending: bool,
|
||||||
|
pub(crate) version: Option<String>,
|
||||||
|
pub(crate) enabled: bool,
|
||||||
|
crashes: VecDeque<Instant>,
|
||||||
|
}
|
||||||
|
impl Service {
|
||||||
|
fn crashed(&mut self, now: Instant) -> bool {
|
||||||
|
while self
|
||||||
|
.crashes
|
||||||
|
.front()
|
||||||
|
.is_some_and(|t| now.duration_since(*t) >= CRASH_WINDOW)
|
||||||
|
{
|
||||||
|
self.crashes.pop_front();
|
||||||
|
}
|
||||||
|
self.crashes.push_back(now);
|
||||||
|
self.crashes.len() <= CRASH_LIMIT
|
||||||
|
}
|
||||||
}
|
}
|
||||||
impl Host {
|
impl Host {
|
||||||
/// Starts discovery in the background, then waits for each boot service in order.
|
/// Starts discovery in the background, then waits for each boot service in order.
|
||||||
@@ -24,30 +48,119 @@ impl Host {
|
|||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
/// Records a unique request identity before the service can answer or exit.
|
/// Register once, before admission or readiness. Restarts reuse this registration.
|
||||||
pub(crate) async fn start_service(&self, method: String, params: Value) {
|
pub(crate) async fn start_service(&self, method: String, params: Value) {
|
||||||
if self.0.shutdown.load(std::sync::atomic::Ordering::SeqCst) {
|
let request = json!(uuid::Uuid::now_v7().to_string());
|
||||||
|
{
|
||||||
|
let mut state = self.0.state.lock().unwrap();
|
||||||
|
if self.0.shutdown.load(Ordering::SeqCst) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
let request = json!(uuid::Uuid::now_v7().to_string());
|
state.services.push(Service {
|
||||||
let starting = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(true));
|
method,
|
||||||
self.0.state.lock().unwrap().services.push(Service {
|
params,
|
||||||
method: method.clone(),
|
|
||||||
params: params.clone(),
|
|
||||||
request: request.clone(),
|
request: request.clone(),
|
||||||
starting: starting.clone(),
|
owner: None,
|
||||||
|
enabled: true,
|
||||||
|
pending: true,
|
||||||
|
version: None,
|
||||||
|
crashes: VecDeque::new(),
|
||||||
});
|
});
|
||||||
|
}
|
||||||
|
self.run_service(request).await;
|
||||||
|
}
|
||||||
|
/// Await readiness only; process cleanup, not this response, decides whether to restart.
|
||||||
|
async fn run_service(&self, request: Value) {
|
||||||
|
let (method, params) = {
|
||||||
|
let state = self.0.state.lock().unwrap();
|
||||||
|
if self.0.shutdown.load(Ordering::SeqCst) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let Some(service) = state
|
||||||
|
.services
|
||||||
|
.iter()
|
||||||
|
.find(|s| s.request == request && s.enabled)
|
||||||
|
else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
(service.method.clone(), service.params.clone())
|
||||||
|
};
|
||||||
self.0
|
self.0
|
||||||
.journal
|
.journal
|
||||||
.write("core", "core", "core", &format!("service start {method}"));
|
.write("core", "core", "core", &format!("service start {method}"));
|
||||||
if let Err(error) = self
|
let result = self
|
||||||
.route("core".into(), Some(request), method, params, true)
|
.route("core".into(), Some(request.clone()), method, params, true)
|
||||||
.await
|
.await;
|
||||||
{
|
{
|
||||||
|
let mut state = self.0.state.lock().unwrap();
|
||||||
|
if let Some(service) = state.services.iter_mut().find(|s| s.request == request) {
|
||||||
|
service.pending = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Err(error) = result {
|
||||||
|
// Admission failures have no process lifetime to supervise. A process crash
|
||||||
|
// before readiness is handled by finish just like one after readiness.
|
||||||
self.0
|
self.0
|
||||||
.journal
|
.journal
|
||||||
.write("core", "core", "core", &format!("service failed {error}"));
|
.write("core", "core", "core", &format!("service failed {error}"));
|
||||||
}
|
}
|
||||||
starting.store(false, std::sync::atomic::Ordering::SeqCst);
|
}
|
||||||
|
/// Called under the state lock after the old owner and its files/locks are gone.
|
||||||
|
pub(crate) fn service_ended(
|
||||||
|
&self,
|
||||||
|
state: &mut State,
|
||||||
|
pid: &str,
|
||||||
|
status: &str,
|
||||||
|
stopped: bool,
|
||||||
|
superseded: bool,
|
||||||
|
) {
|
||||||
|
let Some(service) = state
|
||||||
|
.services
|
||||||
|
.iter_mut()
|
||||||
|
.find(|s| s.owner.as_deref() == Some(pid))
|
||||||
|
else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
service.owner = None;
|
||||||
|
if self.0.shutdown.load(Ordering::SeqCst) || !service.enabled {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if !superseded {
|
||||||
|
if stopped {
|
||||||
|
service.enabled = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if status == "exit 0" {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if !service.crashed(Instant::now()) {
|
||||||
|
service.enabled = false;
|
||||||
|
self.0.journal.write(
|
||||||
|
"core",
|
||||||
|
"core",
|
||||||
|
"core",
|
||||||
|
&format!(
|
||||||
|
"service gave up {}: more than {CRASH_LIMIT} crashes within {} seconds",
|
||||||
|
service.method,
|
||||||
|
CRASH_WINDOW.as_secs(),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
self.queue_service(service);
|
||||||
|
}
|
||||||
|
/// Reserve the next attempt before spawning it; upgrades and crashes share this owner.
|
||||||
|
pub(crate) fn queue_service(&self, service: &mut Service) {
|
||||||
|
service.request = json!(uuid::Uuid::now_v7().to_string());
|
||||||
|
service.pending = true;
|
||||||
|
let request = service.request.clone();
|
||||||
|
let host = self.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
host.run_service(request).await;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests;
|
||||||
|
|||||||
@@ -0,0 +1,356 @@
|
|||||||
|
//! Deterministic service supervision tests: native tasks stand in for guest lifetimes.
|
||||||
|
use super::*;
|
||||||
|
use crate::{NativePlugin, runtime::Runtime};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::sync::{mpsc, oneshot, watch};
|
||||||
|
|
||||||
|
struct Rig {
|
||||||
|
host: Host,
|
||||||
|
starts: mpsc::UnboundedReceiver<(String, oneshot::Sender<()>)>,
|
||||||
|
_world: tempfile::TempDir,
|
||||||
|
}
|
||||||
|
impl Rig {
|
||||||
|
async fn new() -> Self {
|
||||||
|
let world = tempfile::tempdir().unwrap();
|
||||||
|
let runtime = Runtime::new(world.path()).unwrap();
|
||||||
|
let host = Host::test_world(world.path(), runtime).await.unwrap();
|
||||||
|
let (tx, starts) = mpsc::unbounded_channel();
|
||||||
|
let plugin: NativePlugin = Arc::new(move |ctx, method, _| {
|
||||||
|
let tx = tx.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
if method != "serve" {
|
||||||
|
return Ok(Value::Null);
|
||||||
|
}
|
||||||
|
let (end, ended) = oneshot::channel();
|
||||||
|
tx.send((ctx.pid, end)).unwrap();
|
||||||
|
let _ = ended.await;
|
||||||
|
Ok(Value::Null)
|
||||||
|
})
|
||||||
|
});
|
||||||
|
host.register_native("demo", plugin.clone());
|
||||||
|
host.register_native("other", plugin);
|
||||||
|
Self {
|
||||||
|
host,
|
||||||
|
starts,
|
||||||
|
_world: world,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
async fn start(&mut self, method: &str) -> (String, oneshot::Sender<()>) {
|
||||||
|
let host = self.host.clone();
|
||||||
|
let method = method.to_string();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
host.start_service(method, json!({"kept":true})).await;
|
||||||
|
});
|
||||||
|
self.next().await
|
||||||
|
}
|
||||||
|
async fn next(&mut self) -> (String, oneshot::Sender<()>) {
|
||||||
|
// Reuse the host integration call bound (activation.rs::call).
|
||||||
|
tokio::time::timeout(Duration::from_secs(60), self.starts.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
fn ended(&self, pid: &str) -> watch::Receiver<bool> {
|
||||||
|
self.host.0.state.lock().unwrap().processes[pid]
|
||||||
|
.ended
|
||||||
|
.subscribe()
|
||||||
|
}
|
||||||
|
async fn crash(&self, pid: &str) {
|
||||||
|
let mut ended = self.ended(pid);
|
||||||
|
// Abort execution without asking the supervisor for an intentional stop.
|
||||||
|
self.host.0.state.lock().unwrap().processes[pid]
|
||||||
|
.abort
|
||||||
|
.as_ref()
|
||||||
|
.unwrap()
|
||||||
|
.abort();
|
||||||
|
ended.wait_for(|done| *done).await.unwrap();
|
||||||
|
}
|
||||||
|
async fn clean(&self, pid: &str, end: oneshot::Sender<()>) {
|
||||||
|
let mut ended = self.ended(pid);
|
||||||
|
end.send(()).unwrap();
|
||||||
|
ended.wait_for(|done| *done).await.unwrap();
|
||||||
|
}
|
||||||
|
fn assert_single(&self, pid: &str, registrations: usize) {
|
||||||
|
let state = self.host.0.state.lock().unwrap();
|
||||||
|
assert_eq!(state.services.len(), registrations);
|
||||||
|
let p = &state.processes[pid];
|
||||||
|
assert_eq!(p.info.params, json!({"kept":true}));
|
||||||
|
assert!(p.info.detached);
|
||||||
|
assert_eq!(p.caller, "core");
|
||||||
|
assert_eq!(
|
||||||
|
state
|
||||||
|
.services
|
||||||
|
.iter()
|
||||||
|
.filter(|s| s.owner.as_deref() == Some(pid))
|
||||||
|
.count(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
}
|
||||||
|
fn no_start(&mut self) {
|
||||||
|
assert!(self.starts.try_recv().is_err());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn crash_before_and_after_readiness_observes_process_not_response() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (pid, _end) = rig.start("demo.serve").await;
|
||||||
|
rig.assert_single(&pid, 1);
|
||||||
|
rig.crash(&pid).await; // No ready RPC response was sent.
|
||||||
|
let (pid, end) = rig.next().await;
|
||||||
|
rig.assert_single(&pid, 1);
|
||||||
|
rig.host.answer(&pid, Ok(json!({"ready":true})));
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
rig.no_start(); // Ready does not end the service lifetime.
|
||||||
|
rig.crash(&pid).await;
|
||||||
|
drop(end);
|
||||||
|
let (pid, end) = rig.next().await;
|
||||||
|
rig.assert_single(&pid, 1);
|
||||||
|
rig.clean(&pid, end).await;
|
||||||
|
rig.no_start(); // A clean, intentional return is not a crash.
|
||||||
|
rig.host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn fifth_restarts_sixth_gives_up_and_journals_once() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (mut pid, mut end) = rig.start("demo.serve").await;
|
||||||
|
for crash in 1..=CRASH_LIMIT {
|
||||||
|
rig.crash(&pid).await;
|
||||||
|
drop(end);
|
||||||
|
(pid, end) = rig.next().await;
|
||||||
|
rig.assert_single(&pid, 1);
|
||||||
|
assert_eq!(
|
||||||
|
rig.host.0.state.lock().unwrap().services[0].crashes.len(),
|
||||||
|
crash
|
||||||
|
);
|
||||||
|
}
|
||||||
|
rig.crash(&pid).await;
|
||||||
|
drop(end);
|
||||||
|
rig.no_start();
|
||||||
|
let logs = rig
|
||||||
|
.host
|
||||||
|
.call("core.log", json!({"grep":"service gave up"}))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(logs.as_array().unwrap().len(), 1);
|
||||||
|
assert!(
|
||||||
|
logs[0]["line"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.contains("more than 5 crashes within 10 seconds")
|
||||||
|
);
|
||||||
|
assert!(!rig.host.0.state.lock().unwrap().services[0].enabled);
|
||||||
|
// A retired hook cannot bypass give-up while its supervisor has no owner.
|
||||||
|
rig.host.register_native(
|
||||||
|
"fallback",
|
||||||
|
Arc::new(move |ctx, _, _| {
|
||||||
|
Box::pin(async move {
|
||||||
|
ctx.call(
|
||||||
|
"core.call",
|
||||||
|
json!({"method":"demo.serve","params":{"kept":true},"detach":true}),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
rig.host
|
||||||
|
.call("fallback.rpc.retired", json!({}))
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
Value::Null
|
||||||
|
);
|
||||||
|
rig.no_start();
|
||||||
|
// Neither time passing nor an upgrade re-arms a given-up service.
|
||||||
|
tokio::time::advance(CRASH_WINDOW).await;
|
||||||
|
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
|
||||||
|
rig.host.begin_upgrade(upgrade);
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
rig.no_start();
|
||||||
|
rig.host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sliding_window_boundary_and_progress() {
|
||||||
|
let now = Instant::now();
|
||||||
|
let service = || Service {
|
||||||
|
method: "demo.serve".into(),
|
||||||
|
params: json!({}),
|
||||||
|
request: json!("request"),
|
||||||
|
owner: None,
|
||||||
|
enabled: true,
|
||||||
|
pending: false,
|
||||||
|
version: None,
|
||||||
|
crashes: VecDeque::new(),
|
||||||
|
};
|
||||||
|
let mut before = service();
|
||||||
|
let mut exact = service();
|
||||||
|
let mut after = service();
|
||||||
|
for _ in 0..CRASH_LIMIT {
|
||||||
|
assert!(before.crashed(now));
|
||||||
|
assert!(exact.crashed(now));
|
||||||
|
assert!(after.crashed(now));
|
||||||
|
}
|
||||||
|
assert!(!before.crashed(now + CRASH_WINDOW - Duration::from_nanos(1)));
|
||||||
|
assert!(exact.crashed(now + CRASH_WINDOW));
|
||||||
|
assert_eq!(exact.crashes.len(), 1);
|
||||||
|
assert!(after.crashed(now + CRASH_WINDOW + Duration::from_nanos(1)));
|
||||||
|
// Repeated progress beyond the window never accumulates old failures.
|
||||||
|
for step in 2..=CRASH_LIMIT as u32 + 1 {
|
||||||
|
assert!(exact.crashed(now + CRASH_WINDOW * step));
|
||||||
|
assert_eq!(exact.crashes.len(), 1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn services_have_independent_crash_budgets() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (mut a, mut a_end) = rig.start("demo.serve").await;
|
||||||
|
let (b, b_end) = rig.start("other.serve").await;
|
||||||
|
for _ in 0..CRASH_LIMIT {
|
||||||
|
rig.crash(&a).await;
|
||||||
|
drop(a_end);
|
||||||
|
(a, a_end) = rig.next().await;
|
||||||
|
}
|
||||||
|
rig.crash(&a).await;
|
||||||
|
drop(a_end);
|
||||||
|
rig.crash(&b).await;
|
||||||
|
drop(b_end);
|
||||||
|
let (b, b_end) = rig.next().await;
|
||||||
|
rig.assert_single(&b, 2);
|
||||||
|
rig.clean(&b, b_end).await;
|
||||||
|
rig.no_start();
|
||||||
|
rig.host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn upgrade_crash_race_has_one_restart_and_retired_fallback_cannot_duplicate_it() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (pid, end) = rig.start("demo.serve").await;
|
||||||
|
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
|
||||||
|
rig.crash(&pid).await; // Exits between snapshot and begin_upgrade.
|
||||||
|
drop(end);
|
||||||
|
rig.host.begin_upgrade(upgrade);
|
||||||
|
let (pid, end) = rig.next().await;
|
||||||
|
rig.assert_single(&pid, 1);
|
||||||
|
assert!(
|
||||||
|
rig.host.0.state.lock().unwrap().services[0]
|
||||||
|
.crashes
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
// A native retired hook exercises the same routing path as a guest fallback.
|
||||||
|
rig.host.register_native(
|
||||||
|
"fallback",
|
||||||
|
Arc::new(move |ctx, _, _| {
|
||||||
|
Box::pin(async move {
|
||||||
|
ctx.call(
|
||||||
|
"core.call",
|
||||||
|
json!({"method":"demo.serve","params":{"kept":true},"detach":true}),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
rig.host
|
||||||
|
.call("fallback.rpc.retired", json!({}))
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
Value::Null
|
||||||
|
);
|
||||||
|
rig.no_start();
|
||||||
|
rig.clean(&pid, end).await;
|
||||||
|
rig.host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn crash_before_upgrade_snapshot_also_has_one_restart() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (pid, end) = rig.start("demo.serve").await;
|
||||||
|
rig.crash(&pid).await;
|
||||||
|
drop(end);
|
||||||
|
// Restart may be pending admission or already admitted: snapshot handles both.
|
||||||
|
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "native");
|
||||||
|
rig.host.begin_upgrade(upgrade);
|
||||||
|
let (pid, end) = rig.next().await;
|
||||||
|
rig.assert_single(&pid, 1);
|
||||||
|
rig.no_start();
|
||||||
|
rig.clean(&pid, end).await;
|
||||||
|
rig.host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn pending_admission_is_single_owned_across_upgrade() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (pid, end) = rig.start("demo.serve").await;
|
||||||
|
let host = rig.host.clone();
|
||||||
|
let admission = host.0.admission.write().await;
|
||||||
|
rig.crash(&pid).await;
|
||||||
|
drop(end);
|
||||||
|
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "native");
|
||||||
|
rig.host.begin_upgrade(upgrade);
|
||||||
|
assert!(rig.host.0.state.lock().unwrap().services[0].pending);
|
||||||
|
drop(admission);
|
||||||
|
let (pid, end) = rig.next().await;
|
||||||
|
rig.assert_single(&pid, 1);
|
||||||
|
rig.no_start();
|
||||||
|
rig.clean(&pid, end).await;
|
||||||
|
rig.host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn shutdown_rejects_restart_already_waiting_for_admission() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (pid, end) = rig.start("demo.serve").await;
|
||||||
|
let host = rig.host.clone();
|
||||||
|
let admission = host.0.admission.write().await;
|
||||||
|
rig.crash(&pid).await;
|
||||||
|
drop(end);
|
||||||
|
rig.host.stop().await;
|
||||||
|
drop(admission);
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
rig.no_start();
|
||||||
|
assert!(rig.host.processes().is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn clean_service_stays_dormant_until_upgrade_and_explicit_stop_stays_disabled() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (pid, end) = rig.start("demo.serve").await;
|
||||||
|
rig.clean(&pid, end).await;
|
||||||
|
rig.no_start();
|
||||||
|
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
|
||||||
|
rig.host.begin_upgrade(upgrade);
|
||||||
|
let (pid, end) = rig.next().await;
|
||||||
|
rig.assert_single(&pid, 1);
|
||||||
|
// An explicit stop stays disabled even if the task subsequently exits cleanly.
|
||||||
|
let mut ended = rig.host.begin_stop(&pid).unwrap();
|
||||||
|
end.send(()).unwrap();
|
||||||
|
ended.wait_for(|done| *done).await.unwrap();
|
||||||
|
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "newer");
|
||||||
|
rig.host.begin_upgrade(upgrade);
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
rig.no_start();
|
||||||
|
rig.host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn intentional_stop_and_shutdown_never_restart_even_during_upgrade() {
|
||||||
|
let mut rig = Rig::new().await;
|
||||||
|
let (pid, end) = rig.start("demo.serve").await;
|
||||||
|
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
|
||||||
|
// An explicit kill after supersession overrides the automatic upgrade restart.
|
||||||
|
rig.host
|
||||||
|
.call("core.kill", json!({"pid":pid}))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
drop(end);
|
||||||
|
rig.host.begin_upgrade(upgrade);
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
rig.no_start();
|
||||||
|
let (_pid, end) = rig.start("other.serve").await;
|
||||||
|
rig.host.stop().await;
|
||||||
|
drop(end);
|
||||||
|
rig.no_start();
|
||||||
|
}
|
||||||
+19
-2
@@ -64,12 +64,21 @@ async fn instance(
|
|||||||
("SAUSAGE_PLUGIN".into(), "test".into()),
|
("SAUSAGE_PLUGIN".into(), "test".into()),
|
||||||
("SAUSAGE_PARENT".into(), "core".into()),
|
("SAUSAGE_PARENT".into(), "core".into()),
|
||||||
]);
|
]);
|
||||||
|
let timeout_secs = std::env::var("SAUSAGE_TEST_TIMEOUT_SECS")
|
||||||
|
.ok()
|
||||||
|
.and_then(|value| value.parse::<u64>().ok())
|
||||||
|
.filter(|value| *value >= 1)
|
||||||
|
.unwrap_or(sausage::cargo::DEFAULT_TEST_TIMEOUT_SECS);
|
||||||
|
let deadline = tokio::time::Instant::now()
|
||||||
|
.checked_add(std::time::Duration::from_secs(timeout_secs))
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("test_timeout_secs is too large"))?;
|
||||||
|
let (ended, mut ended_rx) = tokio::sync::watch::channel(true);
|
||||||
let mut guest = runtime
|
let mut guest = runtime
|
||||||
.start_with_args(
|
.start_with_args(
|
||||||
&component,
|
&component,
|
||||||
InstanceConfig {
|
InstanceConfig {
|
||||||
filesystem,
|
filesystem,
|
||||||
store_ended: tokio::sync::watch::channel(true).0,
|
store_ended: ended,
|
||||||
network: true,
|
network: true,
|
||||||
env,
|
env,
|
||||||
},
|
},
|
||||||
@@ -88,6 +97,7 @@ async fn instance(
|
|||||||
let mut stdout_open = true;
|
let mut stdout_open = true;
|
||||||
let mut stderr_open = true;
|
let mut stderr_open = true;
|
||||||
let mut handlers = tokio::task::JoinSet::new();
|
let mut handlers = tokio::task::JoinSet::new();
|
||||||
|
let result = tokio::time::timeout_at(deadline, async {
|
||||||
while stdout_open || stderr_open || !pending.is_empty() {
|
while stdout_open || stderr_open || !pending.is_empty() {
|
||||||
tokio::select! {
|
tokio::select! {
|
||||||
line=async { if let Some(line) = pending.pop_front() { Some(line) } else { guest.stdout.recv().await } }, if stdout_open || !pending.is_empty()=>match line {
|
line=async { if let Some(line) = pending.pop_front() { Some(line) } else { guest.stdout.recv().await } }, if stdout_open || !pending.is_empty()=>match line {
|
||||||
@@ -129,11 +139,18 @@ async fn instance(
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
Ok::<_, anyhow::Error>(guest.wait().await)
|
||||||
|
}).await;
|
||||||
handlers.abort_all();
|
handlers.abort_all();
|
||||||
let exit = guest.wait().await;
|
// Dropping the instance aborts even a CPU-bound WASI loop. Wait for the
|
||||||
|
// store and outstanding filesystem work before removing its test world.
|
||||||
|
drop(guest);
|
||||||
|
let _ = ended_rx.wait_for(|ended| *ended).await;
|
||||||
let _ = stderr_task.await;
|
let _ = stderr_task.await;
|
||||||
tracker.idle().await;
|
tracker.idle().await;
|
||||||
host.stop().await;
|
host.stop().await;
|
||||||
|
let exit =
|
||||||
|
result.map_err(|_| anyhow::anyhow!("timed out after {timeout_secs} s\n{output}"))??;
|
||||||
Ok(Run { exit, output })
|
Ok(Run { exit, output })
|
||||||
}
|
}
|
||||||
/// Extracts libtest names from its stable list format.
|
/// Extracts libtest names from its stable list format.
|
||||||
|
|||||||
+35
-40
@@ -1,9 +1,15 @@
|
|||||||
//! Grace periods, service restarts, and retirement of replaced versions (PROTOCOL.md section 3.7).
|
//! Grace periods, service restarts, and retirement of replaced versions (PROTOCOL.md section 3.7).
|
||||||
use crate::{Host, services::Service};
|
use crate::Host;
|
||||||
use serde_json::json;
|
use serde_json::json;
|
||||||
use std::{collections::BTreeMap, sync::atomic::Ordering, time::Duration};
|
use std::{collections::BTreeMap, sync::atomic::Ordering, time::Duration};
|
||||||
use tokio::sync::watch;
|
use tokio::sync::watch;
|
||||||
|
|
||||||
|
/// Uniform host policy (PROTOCOL 3.7), independent of discovery metadata.
|
||||||
|
const UPGRADE_GRACE: Duration = Duration::from_secs(5);
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests;
|
||||||
|
|
||||||
pub(crate) struct OldProcess {
|
pub(crate) struct OldProcess {
|
||||||
pid: String,
|
pid: String,
|
||||||
version: String,
|
version: String,
|
||||||
@@ -12,9 +18,8 @@ pub(crate) struct OldProcess {
|
|||||||
pub(crate) struct Upgrade {
|
pub(crate) struct Upgrade {
|
||||||
plugin: String,
|
plugin: String,
|
||||||
previous: String,
|
previous: String,
|
||||||
grace: Duration,
|
current: String,
|
||||||
processes: Vec<OldProcess>,
|
processes: Vec<OldProcess>,
|
||||||
services: Vec<(Service, Option<watch::Receiver<bool>>)>,
|
|
||||||
}
|
}
|
||||||
impl Host {
|
impl Host {
|
||||||
/// Snapshots old admissions while activation holds the admission write guard.
|
/// Snapshots old admissions while activation holds the admission write guard.
|
||||||
@@ -23,57 +28,59 @@ impl Host {
|
|||||||
plugin: &str,
|
plugin: &str,
|
||||||
previous: String,
|
previous: String,
|
||||||
current: &str,
|
current: &str,
|
||||||
grace: f64,
|
|
||||||
) -> Upgrade {
|
) -> Upgrade {
|
||||||
let mut state = self.0.state.lock().unwrap();
|
let mut state = self.0.state.lock().unwrap();
|
||||||
let processes = state
|
let processes = state
|
||||||
.processes
|
.processes
|
||||||
.iter()
|
.iter_mut()
|
||||||
.filter(|(_, process)| {
|
.filter(|(_, process)| {
|
||||||
process.info.plugin == plugin
|
process.info.plugin == plugin
|
||||||
&& process.info.version != current
|
&& process.info.version != current
|
||||||
&& process.info.method != "rpc.discover"
|
&& process.info.method != "rpc.discover"
|
||||||
})
|
})
|
||||||
.map(|(pid, process)| OldProcess {
|
.map(|(pid, process)| {
|
||||||
|
// The service supervisor owns the restart, including an exit on
|
||||||
|
// the superseded notice before the grace timer is installed.
|
||||||
|
process.superseded = !process.stopped;
|
||||||
|
OldProcess {
|
||||||
pid: pid.clone(),
|
pid: pid.clone(),
|
||||||
version: process.info.version.clone(),
|
version: process.info.version.clone(),
|
||||||
ended: process.ended.subscribe(),
|
ended: process.ended.subscribe(),
|
||||||
|
}
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
let mut services = Vec::new();
|
// A cleanly returned service is dormant until an upgrade. A pending
|
||||||
let mut retained = Vec::new();
|
// admission will choose the current version itself. Disabled services
|
||||||
for service in std::mem::take(&mut state.services) {
|
// (explicit stop or crash-limit give-up) must never be re-armed here.
|
||||||
if service
|
for service in &mut state.services {
|
||||||
|
if service.enabled
|
||||||
|
&& !service.pending
|
||||||
|
&& service.owner.is_none()
|
||||||
|
&& service.version.as_deref() != Some(current)
|
||||||
|
&& service
|
||||||
.method
|
.method
|
||||||
.split_once('.')
|
.split_once('.')
|
||||||
.is_some_and(|(name, _)| name == plugin)
|
.is_some_and(|(name, _)| name == plugin)
|
||||||
{
|
{
|
||||||
let process = state
|
self.queue_service(service);
|
||||||
.processes
|
|
||||||
.values()
|
|
||||||
.find(|process| process.id.as_ref() == Some(&service.request));
|
|
||||||
if process.is_some_and(|process| process.info.version == current)
|
|
||||||
|| (process.is_none() && service.starting.load(Ordering::SeqCst))
|
|
||||||
{
|
|
||||||
retained.push(service);
|
|
||||||
} else {
|
|
||||||
services.push((service, process.map(|process| process.ended.subscribe())));
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
retained.push(service);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
state.services = retained;
|
|
||||||
Upgrade {
|
Upgrade {
|
||||||
plugin: plugin.into(),
|
plugin: plugin.into(),
|
||||||
previous,
|
previous,
|
||||||
grace: Duration::from_secs_f64(grace),
|
current: current.into(),
|
||||||
processes,
|
processes,
|
||||||
services,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/// Lets old calls finish, stops them after grace, and starts each lifecycle hook once.
|
/// Lets old calls finish, stops them after grace, and starts each lifecycle hook once.
|
||||||
pub(crate) fn begin_upgrade(&self, upgrade: Upgrade) {
|
pub(crate) fn begin_upgrade(&self, upgrade: Upgrade) {
|
||||||
|
// Queue the notice before returning from activation, without waiting on guest code.
|
||||||
|
for process in &upgrade.processes {
|
||||||
|
self.send(
|
||||||
|
&process.pid,
|
||||||
|
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":upgrade.current}}),
|
||||||
|
);
|
||||||
|
}
|
||||||
let host = self.clone();
|
let host = self.clone();
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
if host.0.shutdown.load(Ordering::SeqCst) {
|
if host.0.shutdown.load(Ordering::SeqCst) {
|
||||||
@@ -83,7 +90,7 @@ impl Host {
|
|||||||
"core",
|
"core",
|
||||||
&upgrade.plugin,
|
&upgrade.plugin,
|
||||||
"core",
|
"core",
|
||||||
&format!("upgrade grace {} seconds", upgrade.grace.as_secs_f64()),
|
&format!("upgrade grace {} seconds", UPGRADE_GRACE.as_secs()),
|
||||||
);
|
);
|
||||||
let mut versions: BTreeMap<String, Vec<watch::Receiver<bool>>> = BTreeMap::new();
|
let mut versions: BTreeMap<String, Vec<watch::Receiver<bool>>> = BTreeMap::new();
|
||||||
versions.entry(upgrade.previous).or_default();
|
versions.entry(upgrade.previous).or_default();
|
||||||
@@ -93,23 +100,11 @@ impl Host {
|
|||||||
.or_default()
|
.or_default()
|
||||||
.push(process.ended.clone());
|
.push(process.ended.clone());
|
||||||
let host = host.clone();
|
let host = host.clone();
|
||||||
let grace = upgrade.grace;
|
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
let mut ended = process.ended;
|
let mut ended = process.ended;
|
||||||
tokio::select! {
|
tokio::select! {
|
||||||
_ = ended.wait_for(|ended|*ended) => (),
|
_ = ended.wait_for(|ended|*ended) => (),
|
||||||
_ = tokio::time::sleep(grace) => { host.begin_stop(&process.pid); }
|
_ = tokio::time::sleep(UPGRADE_GRACE) => { host.begin_retire(&process.pid); }
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
|
||||||
for (service, ended) in upgrade.services {
|
|
||||||
let host = host.clone();
|
|
||||||
tokio::spawn(async move {
|
|
||||||
if let Some(mut ended) = ended {
|
|
||||||
let _ = ended.wait_for(|ended| *ended).await;
|
|
||||||
}
|
|
||||||
if !host.0.shutdown.load(Ordering::SeqCst) {
|
|
||||||
host.start_service(service.method, service.params).await;
|
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,115 @@
|
|||||||
|
//! Controlled-clock coverage of both upgrade windows and discovery exclusion.
|
||||||
|
use super::*;
|
||||||
|
use crate::{
|
||||||
|
process::{Process, ProcessInfo},
|
||||||
|
runtime::Runtime,
|
||||||
|
world::World,
|
||||||
|
};
|
||||||
|
use tokio::{
|
||||||
|
sync::mpsc,
|
||||||
|
task::JoinHandle,
|
||||||
|
time::{Instant, advance},
|
||||||
|
};
|
||||||
|
|
||||||
|
/// A process with active work that ignores both superseded and EOF.
|
||||||
|
fn process(
|
||||||
|
host: &Host,
|
||||||
|
pid: &str,
|
||||||
|
method: &str,
|
||||||
|
) -> (mpsc::UnboundedReceiver<serde_json::Value>, JoinHandle<()>) {
|
||||||
|
let (input, incoming) = mpsc::unbounded_channel();
|
||||||
|
let task = tokio::spawn(std::future::pending());
|
||||||
|
host.0.state.lock().unwrap().processes.insert(
|
||||||
|
pid.into(),
|
||||||
|
Process {
|
||||||
|
info: ProcessInfo {
|
||||||
|
plugin: "notes".into(),
|
||||||
|
version: "old".into(),
|
||||||
|
method: method.into(),
|
||||||
|
params: json!({}),
|
||||||
|
parent: None,
|
||||||
|
detached: true,
|
||||||
|
start_time: 0,
|
||||||
|
state: "running".into(),
|
||||||
|
},
|
||||||
|
memory: sausage_vfs::Memory::default(),
|
||||||
|
files: sausage_vfs::Process::default(),
|
||||||
|
store_ended: watch::channel(true).0,
|
||||||
|
input: Some(input),
|
||||||
|
abort: Some(task.abort_handle()),
|
||||||
|
caller: "console".into(),
|
||||||
|
id: None,
|
||||||
|
answer: None,
|
||||||
|
ended: watch::channel(false).0,
|
||||||
|
stopped: false,
|
||||||
|
superseded: false,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
(incoming, task)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 3.7/3.6: five seconds running, then five seconds stopping; discovery is untouched.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn two_five_second_windows_and_discovery_exclusion() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let root = dir.path().join("world");
|
||||||
|
World::create(&root).unwrap();
|
||||||
|
let host = Host::test_world(&root, Runtime::new(&dir.path().join("cache")).unwrap())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let (mut input, work) = process(&host, "old-work", "work");
|
||||||
|
let (mut discovery_input, discovery) = process(&host, "old-discovery", "rpc.discover");
|
||||||
|
let upgrade = host.snapshot_upgrade("notes", "old".into(), "new");
|
||||||
|
assert_eq!(upgrade.processes.len(), 1);
|
||||||
|
tokio::time::pause();
|
||||||
|
let start = Instant::now();
|
||||||
|
host.begin_upgrade(upgrade);
|
||||||
|
assert_eq!(
|
||||||
|
input.recv().await.unwrap(),
|
||||||
|
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":"new"}})
|
||||||
|
);
|
||||||
|
// Run the upgrade dispatcher, then its per-process task, so the deadline is armed.
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
let window = Duration::from_secs(5);
|
||||||
|
assert_eq!(UPGRADE_GRACE, window);
|
||||||
|
// tokio::time::sleep documents millisecond granularity; the paused driver
|
||||||
|
// may round a deadline up by one tick. Check each window independently.
|
||||||
|
let tick = Duration::from_millis(1);
|
||||||
|
let before_deadline = window - tick;
|
||||||
|
advance(before_deadline).await;
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert_eq!(host.processes()["old-work"].state, "running");
|
||||||
|
assert!(!work.is_finished());
|
||||||
|
advance(tick).await;
|
||||||
|
assert!(
|
||||||
|
input.recv().await.is_none(),
|
||||||
|
"EOF at grace, even with active work"
|
||||||
|
);
|
||||||
|
let stopped = Instant::now();
|
||||||
|
assert!((window..=window + tick).contains(&(stopped - start)));
|
||||||
|
assert_eq!(host.processes()["old-work"].state, "stopping");
|
||||||
|
// begin_retire schedules the independent PROTOCOL 3.6 deadline.
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
advance(before_deadline).await;
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert!(
|
||||||
|
!work.is_finished(),
|
||||||
|
"on_stop still has its full five seconds"
|
||||||
|
);
|
||||||
|
advance(tick).await;
|
||||||
|
assert!(work.await.unwrap_err().is_cancelled());
|
||||||
|
assert!((window..=window + tick).contains(&(Instant::now() - stopped)));
|
||||||
|
assert_eq!(host.processes()["old-discovery"].state, "running");
|
||||||
|
assert!(matches!(
|
||||||
|
discovery_input.try_recv(),
|
||||||
|
Err(mpsc::error::TryRecvError::Empty)
|
||||||
|
));
|
||||||
|
assert!(!discovery.is_finished());
|
||||||
|
discovery.abort();
|
||||||
|
let _ = discovery.await;
|
||||||
|
// Synthetic processes have no execution cleanup task; discard only this test table.
|
||||||
|
host.0.shutdown.store(true, Ordering::SeqCst);
|
||||||
|
host.0.state.lock().unwrap().processes.clear();
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
@@ -0,0 +1,178 @@
|
|||||||
|
//! Armed plugin-file watches. A Notify permit coalesces changes without losing a later change.
|
||||||
|
use crate::{Host, RpcResult, error};
|
||||||
|
use sausage_vfs::observe::under;
|
||||||
|
use serde_json::{Value, json};
|
||||||
|
use std::{
|
||||||
|
collections::HashMap,
|
||||||
|
sync::{Arc, Mutex},
|
||||||
|
};
|
||||||
|
use tokio::sync::Notify;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
pub(crate) struct Watch {
|
||||||
|
path: String,
|
||||||
|
changed: Arc<Notify>,
|
||||||
|
}
|
||||||
|
pub(crate) type Watches = Arc<Mutex<HashMap<Uuid, Watch>>>;
|
||||||
|
pub(crate) fn changed(watches: &Watches, path: &str) {
|
||||||
|
let targets: Vec<_> = watches
|
||||||
|
.lock()
|
||||||
|
.unwrap()
|
||||||
|
.values()
|
||||||
|
// Removing/moving an ancestor also invalidates a watch inside that subtree.
|
||||||
|
.filter(|w| under(path, &w.path) || under(&w.path, path))
|
||||||
|
.map(|w| w.changed.clone())
|
||||||
|
.collect();
|
||||||
|
for changed in targets {
|
||||||
|
changed.notify_one();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
struct Registration {
|
||||||
|
watches: Watches,
|
||||||
|
key: Uuid,
|
||||||
|
}
|
||||||
|
impl Drop for Registration {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
self.watches.lock().unwrap().remove(&self.key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fn path(caller: &str, path: &str) -> String {
|
||||||
|
let path = sausage_vfs::resolve("/", path);
|
||||||
|
let path = path.trim_end_matches('/');
|
||||||
|
if path.is_empty() {
|
||||||
|
return "/".into();
|
||||||
|
}
|
||||||
|
if under(path, "/self") {
|
||||||
|
format!("/proc/{caller}{}", &path[5..])
|
||||||
|
} else {
|
||||||
|
path.into()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Host {
|
||||||
|
pub(crate) async fn watch(&self, caller: &str, id: Option<Value>, target: &str) -> RpcResult {
|
||||||
|
let id = id.ok_or_else(|| error(-32600, "watch needs a request ID"))?;
|
||||||
|
let mut ended = self
|
||||||
|
.0
|
||||||
|
.state
|
||||||
|
.lock()
|
||||||
|
.unwrap()
|
||||||
|
.processes
|
||||||
|
.get(caller)
|
||||||
|
.filter(|p| p.info.state == "running")
|
||||||
|
.map(|p| p.ended.subscribe())
|
||||||
|
.ok_or_else(|| error(-32600, "watch needs a running caller process"))?;
|
||||||
|
let changed = Arc::new(Notify::new());
|
||||||
|
let registration = Registration {
|
||||||
|
watches: self.0.watches.clone(),
|
||||||
|
key: Uuid::now_v7(),
|
||||||
|
};
|
||||||
|
self.0.watches.lock().unwrap().insert(
|
||||||
|
registration.key,
|
||||||
|
Watch {
|
||||||
|
path: path(caller, target),
|
||||||
|
changed: changed.clone(),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
self.call_notify(caller, &id, "armed", json!({}));
|
||||||
|
loop {
|
||||||
|
tokio::select! {
|
||||||
|
biased;
|
||||||
|
_ = ended.wait_for(|ended| *ended) => return Ok(Value::Null),
|
||||||
|
_ = changed.notified() => self.call_notify(caller, &id, "changed", json!({})),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
/// Pending changes coalesce; a change after consuming the last permit is retained.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn coalescing_and_cleanup() {
|
||||||
|
let watches = Watches::default();
|
||||||
|
let n = Arc::new(Notify::new());
|
||||||
|
let r = Registration {
|
||||||
|
watches: watches.clone(),
|
||||||
|
key: Uuid::now_v7(),
|
||||||
|
};
|
||||||
|
watches.lock().unwrap().insert(
|
||||||
|
r.key,
|
||||||
|
Watch {
|
||||||
|
path: "/mailbox".into(),
|
||||||
|
changed: n.clone(),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
changed(&watches, "/mailboxes/a");
|
||||||
|
assert!(
|
||||||
|
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
changed(&watches, "/mailbox/a");
|
||||||
|
changed(&watches, "/mailbox/b");
|
||||||
|
n.notified().await;
|
||||||
|
assert!(
|
||||||
|
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
changed(&watches, "/mailbox/c");
|
||||||
|
n.notified().await;
|
||||||
|
drop(r);
|
||||||
|
assert!(watches.lock().unwrap().is_empty());
|
||||||
|
changed(&watches, "/mailbox/d");
|
||||||
|
assert!(
|
||||||
|
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
#[test]
|
||||||
|
fn self_alias_and_normalization() {
|
||||||
|
assert_eq!(path("writer", "/self/../self/a/"), "/proc/writer/a");
|
||||||
|
assert_eq!(path("writer", "/selfish/a"), "/selfish/a");
|
||||||
|
assert_eq!(path("writer", "/"), "/");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod lifetime_tests {
|
||||||
|
use super::*;
|
||||||
|
/// The real caller's ended event releases the registration, without a cancellation RPC.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn caller_death_removes_watch() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let runtime = crate::runtime::Runtime::new(&temp.path().join("cache")).unwrap();
|
||||||
|
let host = Host::test_world(&temp.path().join("world"), runtime)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel();
|
||||||
|
host.register_native(
|
||||||
|
"holder",
|
||||||
|
Arc::new(move |ctx, _, _| {
|
||||||
|
let tx = tx.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
tx.send(ctx.pid.clone()).unwrap();
|
||||||
|
std::future::pending().await
|
||||||
|
})
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
let h = host.clone();
|
||||||
|
let task = tokio::spawn(async move { h.call("holder.hold", json!({})).await });
|
||||||
|
let pid = rx.recv().await.unwrap();
|
||||||
|
let mut watch = Box::pin(host.watch(&pid, Some(json!("watch")), "/mailbox"));
|
||||||
|
assert!(
|
||||||
|
watch
|
||||||
|
.as_mut()
|
||||||
|
.poll(&mut std::task::Context::from_waker(std::task::Waker::noop()))
|
||||||
|
.is_pending()
|
||||||
|
);
|
||||||
|
assert_eq!(host.0.watches.lock().unwrap().len(), 1);
|
||||||
|
host.call("core.kill", json!({"pid":pid})).await.unwrap();
|
||||||
|
watch.await.unwrap();
|
||||||
|
assert!(host.0.watches.lock().unwrap().is_empty());
|
||||||
|
assert!(task.await.unwrap().is_err());
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -92,6 +92,7 @@ mod tests {
|
|||||||
"/config",
|
"/config",
|
||||||
"/state",
|
"/state",
|
||||||
"/state/core/journal.sqlite",
|
"/state/core/journal.sqlite",
|
||||||
|
"/logs/core/journal.sqlite",
|
||||||
] {
|
] {
|
||||||
assert!(!std::path::Path::new(path).exists(), "{path}");
|
assert!(!std::path::Path::new(path).exists(), "{path}");
|
||||||
}
|
}
|
||||||
|
|||||||
+311
-11
@@ -43,6 +43,86 @@ fn fixture(version: usize) -> PathBuf {
|
|||||||
})[version - 1]
|
})[version - 1]
|
||||||
.clone()
|
.clone()
|
||||||
}
|
}
|
||||||
|
fn fixture_hash(version: usize) -> String {
|
||||||
|
use sha2::{Digest, Sha256};
|
||||||
|
format!(
|
||||||
|
"{:x}",
|
||||||
|
Sha256::digest(std::fs::read(fixture(version)).unwrap())
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 10.3: summaries omit even large schemas and report sorted main-method diffs.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn summarizes_method_changes() {
|
||||||
|
let mut old = document("unchanged");
|
||||||
|
old["methods"] = json!([
|
||||||
|
{"name":"z_removed"}, {"name":"a_removed"}, {"name":"direct", "params":[]},
|
||||||
|
{"name":"referenced", "result":{"schema":{"$ref":"#/components/schemas/Node"}}},
|
||||||
|
{"name":"unrelated", "result":{"schema":{"type":"string"}}}
|
||||||
|
]);
|
||||||
|
old["components"]["schemas"] = json!({
|
||||||
|
"Node":{"properties":{"next":{"$ref":"#/components/schemas/Node"}, "value":{"$ref":"#/components/schemas/Value"}}},
|
||||||
|
"Value":{"type":"string"}, "Unused":{"type":"string"}
|
||||||
|
});
|
||||||
|
let mut next = old.clone();
|
||||||
|
next["methods"].as_array_mut().unwrap().drain(..2);
|
||||||
|
next["methods"][0]["params"] = json!([{"name":"limit","schema":{"type":"integer"}}]);
|
||||||
|
next["methods"].as_array_mut().unwrap().extend(
|
||||||
|
json!([
|
||||||
|
{"name":"z_added"}, {"name":"a_added"}
|
||||||
|
])
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.iter()
|
||||||
|
.cloned(),
|
||||||
|
);
|
||||||
|
next["components"]["schemas"]["Value"]["type"] = json!("integer");
|
||||||
|
next["components"]["schemas"]["Unused"]["description"] = json!("x".repeat(220_000));
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.plugin("notes", old, next.clone());
|
||||||
|
let host = world.start().await;
|
||||||
|
let result = activate(&host, "notes").await.unwrap();
|
||||||
|
assert_eq!(result["notes"]["method_count"], 5);
|
||||||
|
assert_eq!(result["notes"]["added"], json!(["a_added", "z_added"]));
|
||||||
|
assert_eq!(
|
||||||
|
result["notes"]["removed"],
|
||||||
|
json!(["a_removed", "z_removed"])
|
||||||
|
);
|
||||||
|
assert_eq!(result["notes"]["changed"], json!(["direct", "referenced"]));
|
||||||
|
assert!(result.to_string().len() < 1024);
|
||||||
|
assert_eq!(call(&host, "notes.rpc.discover", json!({})).await, next);
|
||||||
|
assert_eq!(
|
||||||
|
call(
|
||||||
|
&host,
|
||||||
|
"notes.read",
|
||||||
|
json!({"path":"/proc/core/plugins/notes/discover.json"})
|
||||||
|
)
|
||||||
|
.await,
|
||||||
|
next
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 10.3: first installation has no previous build; a no-op has no method changes.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn summarizes_first_install_and_noop() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.plugin("notes", document("add"), document("add"));
|
||||||
|
std::fs::remove_file(world.0.join("plugins/notes/plugin.wasm")).unwrap();
|
||||||
|
let host = world.start().await;
|
||||||
|
let result = activate(&host, "notes").await.unwrap();
|
||||||
|
assert_eq!(result["notes"]["previous"], Value::Null);
|
||||||
|
assert_eq!(result["notes"]["added"], json!(["add"]));
|
||||||
|
assert_eq!(result["notes"]["hash"], fixture_hash(2));
|
||||||
|
let result = activate(&host, "notes").await.unwrap();
|
||||||
|
assert_eq!(result["notes"]["previous"], fixture_hash(2));
|
||||||
|
for field in ["added", "removed", "changed"] {
|
||||||
|
assert_eq!(result["notes"][field], json!([]));
|
||||||
|
}
|
||||||
|
assert!(!world.0.join("plugins/notes/plugin.prev.wasm").exists());
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
struct TestWorld(PathBuf);
|
struct TestWorld(PathBuf);
|
||||||
impl TestWorld {
|
impl TestWorld {
|
||||||
/// Creates an isolated world with two version documents per plugin.
|
/// Creates an isolated world with two version documents per plugin.
|
||||||
@@ -80,9 +160,9 @@ impl Drop for TestWorld {
|
|||||||
let _ = std::fs::remove_dir_all(&self.0);
|
let _ = std::fs::remove_dir_all(&self.0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/// Builds a method-only document without an upgrade delay.
|
/// Builds a method-only document; all plugins use the host's five-second grace.
|
||||||
fn document(name: &str) -> Value {
|
fn document(name: &str) -> Value {
|
||||||
json!({"methods":[{"name":name,"params":[],"result":{"schema":{"type":"string"}}}],"x-sausage-grace":0})
|
json!({"methods":[{"name":name,"params":[],"result":{"schema":{"type":"string"}}}]})
|
||||||
}
|
}
|
||||||
/// Adds an import of a provider's method list.
|
/// Adds an import of a provider's method list.
|
||||||
fn consumer(provider: &str, expected: Value) -> Value {
|
fn consumer(provider: &str, expected: Value) -> Value {
|
||||||
@@ -404,7 +484,7 @@ async fn review_dropped_activation_completes() {
|
|||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
/// PROTOCOL 10.1: discovery fails after its fixed ten-second deadline and receives EOF.
|
/// PROTOCOL 10.1: discovery fails after its fixed ten-second deadline and receives EOF.
|
||||||
/// PROTOCOL 3.7: old processes retain their declared grace while new calls use new code.
|
/// PROTOCOL 3.7: old processes get five seconds regardless of declarations; new calls use new code.
|
||||||
/// PROTOCOL 3.7: retirement waits for all live admissions, including reverted versions.
|
/// PROTOCOL 3.7: retirement waits for all live admissions, including reverted versions.
|
||||||
/// PROTOCOL 3.7: rejected discovery candidates never receive retirement hooks.
|
/// PROTOCOL 3.7: rejected discovery candidates never receive retirement hooks.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
@@ -481,7 +561,19 @@ async fn activates_copy() {
|
|||||||
world.plugin("notes", document("add"), document("add"));
|
world.plugin("notes", document("add"), document("add"));
|
||||||
let host = world.start().await;
|
let host = world.start().await;
|
||||||
let result = activate(&host, "notes").await.unwrap();
|
let result = activate(&host, "notes").await.unwrap();
|
||||||
assert!(result["notes"]["methods"].is_array());
|
assert_eq!(
|
||||||
|
result,
|
||||||
|
json!({"notes": {
|
||||||
|
"plugin":"notes", "wasm":"/plugins/notes/plugin.wasm",
|
||||||
|
"hash":fixture_hash(2), "previous":fixture_hash(1),
|
||||||
|
"method_count":1, "added":[], "removed":[], "changed":[],
|
||||||
|
"interface_check":"passed"
|
||||||
|
}})
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
call(&host, "notes.rpc.discover", json!({})).await,
|
||||||
|
document("add")
|
||||||
|
);
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
call(&host, "notes.version", json!({})).await["version"],
|
call(&host, "notes.version", json!({})).await["version"],
|
||||||
"2"
|
"2"
|
||||||
@@ -512,7 +604,7 @@ async fn source_changes_do_not_change_candidate() {
|
|||||||
);
|
);
|
||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
/// PROTOCOL 3.7: plugins without a declared grace period retain old calls before the default deadline.
|
/// PROTOCOL 3.7: missing metadata also retains old calls until the uniform deadline.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn default_grace_retains_old_process() {
|
async fn default_grace_retains_old_process() {
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
@@ -531,6 +623,104 @@ async fn default_grace_retains_old_process() {
|
|||||||
assert_eq!(host.processes()[&pid].state, "running");
|
assert_eq!(host.processes()[&pid].state, "running");
|
||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
|
/// PROTOCOL 3.7: metadata cannot shorten or extend the actual grace for active requests.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn uniform_grace_ignores_discovery_declarations() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
let declarations = [
|
||||||
|
("missing", None),
|
||||||
|
("default", Some(json!(60))),
|
||||||
|
("long", Some(json!(3600))),
|
||||||
|
("zero", Some(json!(0))),
|
||||||
|
("negative", Some(json!(-1))),
|
||||||
|
("invalid", Some(json!("long"))),
|
||||||
|
("huge", Some(json!(1e300))),
|
||||||
|
];
|
||||||
|
for (name, grace) in &declarations {
|
||||||
|
let mut old = document("work");
|
||||||
|
if let Some(grace) = grace {
|
||||||
|
old["x-sausage-grace"] = grace.clone();
|
||||||
|
}
|
||||||
|
world.plugin(name, old, document("work"));
|
||||||
|
}
|
||||||
|
let host = world.start().await;
|
||||||
|
let mut ready = host.subscribe();
|
||||||
|
let mut calls = Vec::new();
|
||||||
|
for (name, _) in &declarations {
|
||||||
|
let host = host.clone();
|
||||||
|
let method = format!("{name}.work");
|
||||||
|
calls.push(tokio::spawn(async move {
|
||||||
|
let result = host.call(&method, json!({})).await;
|
||||||
|
(result, std::time::Instant::now())
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
// Guest notifications are events, not process/file polling.
|
||||||
|
for _ in &declarations {
|
||||||
|
let event = tokio::time::timeout(Duration::from_secs(30), ready.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(event["params"]["name"], "ready");
|
||||||
|
}
|
||||||
|
let before = std::time::Instant::now();
|
||||||
|
let plugins: Vec<_> = declarations
|
||||||
|
.iter()
|
||||||
|
.map(|(name, _)| json!({"plugin":name,"wasm":format!("/state/{name}/next.wasm")}))
|
||||||
|
.collect();
|
||||||
|
call(&host, "core.activate", json!({"plugins":plugins})).await;
|
||||||
|
// Use the existing discovery+stop test budget, not a new execution limit.
|
||||||
|
tokio::time::timeout(Duration::from_secs(15), async {
|
||||||
|
for task in calls {
|
||||||
|
let (result, ended) = task.await.unwrap();
|
||||||
|
assert_eq!(result.unwrap_err()["code"], -32002);
|
||||||
|
assert!(
|
||||||
|
ended.duration_since(before) >= Duration::from_secs(5),
|
||||||
|
"every declaration gets the full grace"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 10.4/3.7: a watcher refresh uses the same policy as explicit activation.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn manual_copy_uses_uniform_grace() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
let mut old = document("work");
|
||||||
|
old["x-sausage-grace"] = json!(3600);
|
||||||
|
world.plugin("notes", old, document("work"));
|
||||||
|
let host = world.start().await;
|
||||||
|
// Wait for initial discovery without changing the installed version.
|
||||||
|
call(
|
||||||
|
&host,
|
||||||
|
"core.activate",
|
||||||
|
json!({"plugin":"notes","wasm":"/plugins/notes/plugin.wasm"}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let mut ready = host.subscribe();
|
||||||
|
let work_host = host.clone();
|
||||||
|
let work = tokio::spawn(async move { work_host.call("notes.work", json!({})).await });
|
||||||
|
let event = tokio::time::timeout(Duration::from_secs(30), ready.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(event["params"]["name"], "ready");
|
||||||
|
let before = std::time::Instant::now();
|
||||||
|
std::fs::copy(fixture(2), world.0.join("plugins/notes/plugin.wasm")).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
tokio::time::timeout(Duration::from_secs(15), work)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap()
|
||||||
|
.unwrap_err()["code"],
|
||||||
|
-32002
|
||||||
|
);
|
||||||
|
assert!(before.elapsed() >= Duration::from_secs(5));
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
/// PROTOCOL 10.3: candidate imports must fit installed providers.
|
/// PROTOCOL 10.3: candidate imports must fit installed providers.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn checks_candidate_imports() {
|
async fn checks_candidate_imports() {
|
||||||
@@ -577,7 +767,12 @@ async fn activates_group() {
|
|||||||
consumer("notes", document("other")),
|
consumer("notes", document("other")),
|
||||||
);
|
);
|
||||||
let host = world.start().await;
|
let host = world.start().await;
|
||||||
call(&host,"core.activate",json!({"plugins":[{"plugin":"notes","wasm":"/state/notes/next.wasm"},{"plugin":"agent","wasm":"/state/agent/next.wasm"}]})).await;
|
let result = call(&host,"core.activate",json!({"plugins":[{"plugin":"notes","wasm":"/state/notes/next.wasm"},{"plugin":"agent","wasm":"/state/agent/next.wasm"}]})).await;
|
||||||
|
assert_eq!(result.as_object().unwrap().len(), 2);
|
||||||
|
assert_eq!(result["notes"]["added"], json!(["other"]));
|
||||||
|
assert_eq!(result["notes"]["removed"], json!(["add"]));
|
||||||
|
assert_eq!(result["agent"]["method_count"], 1);
|
||||||
|
assert_eq!(result["agent"]["interface_check"], "passed");
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
call(&host, "agent.version", json!({})).await["version"],
|
call(&host, "agent.version", json!({})).await["version"],
|
||||||
"2"
|
"2"
|
||||||
@@ -594,12 +789,13 @@ async fn force_skips_checks() {
|
|||||||
consumer("absent", document("add")),
|
consumer("absent", document("add")),
|
||||||
);
|
);
|
||||||
let host = world.start().await;
|
let host = world.start().await;
|
||||||
call(
|
let result = call(
|
||||||
&host,
|
&host,
|
||||||
"core.activate",
|
"core.activate",
|
||||||
json!({"plugin":"notes","wasm":"/state/notes/next.wasm","force":true}),
|
json!({"plugin":"notes","wasm":"/state/notes/next.wasm","force":true}),
|
||||||
)
|
)
|
||||||
.await;
|
.await;
|
||||||
|
assert_eq!(result["notes"]["interface_check"], "skipped");
|
||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
/// PROTOCOL 10.3: revert activates the previous copy and preserves the replaced version.
|
/// PROTOCOL 10.3: revert activates the previous copy and preserves the replaced version.
|
||||||
@@ -609,7 +805,11 @@ async fn reverts_previous_version() {
|
|||||||
world.plugin("notes", document("add"), document("add"));
|
world.plugin("notes", document("add"), document("add"));
|
||||||
let host = world.start().await;
|
let host = world.start().await;
|
||||||
activate(&host, "notes").await.unwrap();
|
activate(&host, "notes").await.unwrap();
|
||||||
call(&host, "core.revert", json!({"plugin":"notes"})).await;
|
let result = call(&host, "core.revert", json!({"plugin":"notes"})).await;
|
||||||
|
assert_eq!(result["notes"]["hash"], fixture_hash(1));
|
||||||
|
assert_eq!(result["notes"]["previous"], fixture_hash(2));
|
||||||
|
assert_eq!(result["notes"]["interface_check"], "passed");
|
||||||
|
assert!(result["notes"].get("methods").is_none());
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
call(&host, "notes.version", json!({})).await["version"],
|
call(&host, "notes.version", json!({})).await["version"],
|
||||||
"1"
|
"1"
|
||||||
@@ -716,11 +916,11 @@ async fn graph_missing_import() {
|
|||||||
assert!(dot.as_str().unwrap().contains("absent"));
|
assert!(dot.as_str().unwrap().contains("absent"));
|
||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
/// Checks the declared upgrade grace and dispatch to the new version.
|
/// Checks that an explicit long declaration cannot extend grace; new calls use the new version.
|
||||||
async fn old_process_grace_and_new_calls() {
|
async fn old_process_grace_and_new_calls() {
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
let mut old = document("add");
|
let mut old = document("add");
|
||||||
old["x-sausage-grace"] = json!(0.5);
|
old["x-sausage-grace"] = json!(3600);
|
||||||
world.plugin("notes", old, document("add"));
|
world.plugin("notes", old, document("add"));
|
||||||
let host = world.start().await;
|
let host = world.start().await;
|
||||||
call(&host, "notes.hold", json!({})).await;
|
call(&host, "notes.hold", json!({})).await;
|
||||||
@@ -737,6 +937,76 @@ async fn old_process_grace_and_new_calls() {
|
|||||||
"2"
|
"2"
|
||||||
);
|
);
|
||||||
marker(world.0.join("state/notes/ended-1"), "ended").await;
|
marker(world.0.join("state/notes/ended-1"), "ended").await;
|
||||||
|
let notice: Value = serde_json::from_slice(
|
||||||
|
&std::fs::read(world.0.join(format!("state/notes/superseded-{pid}"))).unwrap(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(notice["params"]["current"], fixture_hash(2));
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
/// PROTOCOL 3.7: every old process gets the new hash while still running, before grace ends.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn superseded_notifies_each_old_process() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
let mut old = document("hold");
|
||||||
|
old["x-sausage-grace"] = json!(60);
|
||||||
|
world.plugin("notes", old, document("hold"));
|
||||||
|
let host = world.start().await;
|
||||||
|
call(&host, "notes.hold", json!({})).await;
|
||||||
|
call(&host, "notes.hold", json!({})).await;
|
||||||
|
let pids: Vec<_> = host
|
||||||
|
.processes()
|
||||||
|
.into_iter()
|
||||||
|
.filter(|(_, process)| process.method == "hold")
|
||||||
|
.map(|(pid, _)| pid)
|
||||||
|
.collect();
|
||||||
|
assert_eq!(pids.len(), 2);
|
||||||
|
activate(&host, "notes").await.unwrap();
|
||||||
|
let expected =
|
||||||
|
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":fixture_hash(2)}})
|
||||||
|
.to_string();
|
||||||
|
for pid in pids {
|
||||||
|
// The notice arrives while the old process is still in its five-second window.
|
||||||
|
marker(
|
||||||
|
world.0.join(format!("state/notes/superseded-{pid}")),
|
||||||
|
&expected,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert_eq!(host.processes()[&pid].state, "running");
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
call(&host, "notes.version", json!({})).await["version"],
|
||||||
|
"2"
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
/// PROTOCOL 3.7: a service that exits on the notice restarts before its grace deadline.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn superseded_service_exits_early() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
let mut old = document("serve");
|
||||||
|
old["x-sausage-grace"] = json!(60);
|
||||||
|
world.plugin("notes", old, document("serve"));
|
||||||
|
std::fs::write(
|
||||||
|
world.0.join("config/boot.yaml"),
|
||||||
|
json!([{"method":"notes.serve","params":{"exit_on_superseded":true}}]).to_string(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let host = world.start().await;
|
||||||
|
marker(world.0.join("state/notes/service"), "1").await;
|
||||||
|
let pid = host
|
||||||
|
.processes()
|
||||||
|
.into_iter()
|
||||||
|
.find(|(_, p)| p.method == "serve")
|
||||||
|
.unwrap()
|
||||||
|
.0;
|
||||||
|
activate(&host, "notes").await.unwrap();
|
||||||
|
marker(world.0.join("state/notes/service"), "2").await;
|
||||||
|
assert!(!host.processes().contains_key(&pid));
|
||||||
|
assert!(
|
||||||
|
!world.0.join("state/notes/ended-1").exists(),
|
||||||
|
"old service must exit on the notice, not EOF"
|
||||||
|
);
|
||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
/// PROTOCOL 3.7: retirement receives the previous hash only after old processes end.
|
/// PROTOCOL 3.7: retirement receives the previous hash only after old processes end.
|
||||||
@@ -821,7 +1091,7 @@ async fn upgrades_enter_the_journal() {
|
|||||||
.as_array()
|
.as_array()
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.iter()
|
.iter()
|
||||||
.any(|e| e["line"] == "upgrade grace 60 seconds")
|
.any(|e| e["line"] == "upgrade grace 5 seconds")
|
||||||
{
|
{
|
||||||
break entries;
|
break entries;
|
||||||
}
|
}
|
||||||
@@ -945,6 +1215,36 @@ async fn old_consumer_can_call_removed_provider_method() {
|
|||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 8: supervise actual WASI exits on either side of the readiness response.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn crashed_boot_service_restarts_before_and_after_ready() {
|
||||||
|
for when in ["before", "after"] {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.plugin("notes", document("serve"), document("serve"));
|
||||||
|
std::fs::write(
|
||||||
|
world.0.join("config/boot.yaml"),
|
||||||
|
json!([{"method":"notes.serve","params":{"crash_once":when}}]).to_string(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let host = world.start().await;
|
||||||
|
marker(world.0.join("state/notes/service"), "1").await;
|
||||||
|
let entries = call(
|
||||||
|
&host,
|
||||||
|
"core.log",
|
||||||
|
json!({"grep":"service start notes.serve"}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert_eq!(entries.as_array().unwrap().len(), 2);
|
||||||
|
let processes = host.processes();
|
||||||
|
let services: Vec<_> = processes.values().filter(|p| p.method == "serve").collect();
|
||||||
|
assert_eq!(services.len(), 1);
|
||||||
|
assert_eq!(services[0].params, json!({"crash_once":when}));
|
||||||
|
let exits = call(&host, "core.log", json!({"plugin":"notes","grep":"exit 1"})).await;
|
||||||
|
assert_eq!(exits.as_array().unwrap().len(), 1);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// PROTOCOL 10.2: activation and revert check interfaces but leave persistent state unchanged.
|
/// PROTOCOL 10.2: activation and revert check interfaces but leave persistent state unchanged.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn state_is_outside_interface_checks() {
|
async fn state_is_outside_interface_checks() {
|
||||||
|
|||||||
+22
-1
@@ -489,7 +489,28 @@ async fn oversized_current_turn_preserves_the_previous_cursor() {
|
|||||||
.unwrap();
|
.unwrap();
|
||||||
let (body, reply) = server.next().await;
|
let (body, reply) = server.next().await;
|
||||||
let serialized = input(&body, api).to_string();
|
let serialized = input(&body, api).to_string();
|
||||||
assert!(serialized.contains("[tool output truncated"));
|
assert!(serialized.contains("Full output: /logs/agent/"));
|
||||||
|
let history = world.history();
|
||||||
|
let result: Value = history
|
||||||
|
.lines()
|
||||||
|
.map(|line| serde_json::from_str::<Value>(line).unwrap())
|
||||||
|
.find(|entry| {
|
||||||
|
entry["output"]
|
||||||
|
.as_str()
|
||||||
|
.is_some_and(|out| out.contains("Full output: "))
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
let output = result["output"].as_str().unwrap();
|
||||||
|
let path = output
|
||||||
|
.split("Full output: ")
|
||||||
|
.nth(1)
|
||||||
|
.unwrap()
|
||||||
|
.split(". ")
|
||||||
|
.next()
|
||||||
|
.unwrap();
|
||||||
|
let archive = std::fs::read_to_string(world.0.join(path.trim_start_matches('/'))).unwrap();
|
||||||
|
assert!(archive.contains(&"x".repeat(200_000)));
|
||||||
|
assert!(output.len() <= 50 * 1024);
|
||||||
assert!(serialized.len() < 70_000);
|
assert!(serialized.len() < 70_000);
|
||||||
reply
|
reply
|
||||||
.send((413, json!({"type":"request_too_large"})))
|
.send((413, json!({"type":"request_too_large"})))
|
||||||
|
|||||||
+413
-12
@@ -12,6 +12,23 @@ impl TestWorld {
|
|||||||
World::create(&path).unwrap();
|
World::create(&path).unwrap();
|
||||||
Self(path)
|
Self(path)
|
||||||
}
|
}
|
||||||
|
/// Short-deadline tests must not queue on another test's Cargo target lock.
|
||||||
|
async fn timeout_host(&self, name: &str) -> Host {
|
||||||
|
Host::start_with_cache(&self.0, &Path::new(CACHE).join("timeouts").join(name))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
/// Creates a dependency-free plugin fixture.
|
||||||
|
fn simple(&self, name: &str, text: &str) {
|
||||||
|
let root = self.0.join(format!("plugins/{name}/source"));
|
||||||
|
std::fs::create_dir_all(root.join("src")).unwrap();
|
||||||
|
std::fs::write(
|
||||||
|
root.join("Cargo.toml"),
|
||||||
|
format!("[package]\nname='{name}'\nversion='0.1.0'\nedition='2024'\n"),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
std::fs::write(root.join("src/main.rs"), text).unwrap();
|
||||||
|
}
|
||||||
/// Copies a fixture into its plugin source folder.
|
/// Copies a fixture into its plugin source folder.
|
||||||
fn source(&self, name: &str, source: &Path) {
|
fn source(&self, name: &str, source: &Path) {
|
||||||
copy(source, &self.0.join(format!("plugins/{name}/source")));
|
copy(source, &self.0.join(format!("plugins/{name}/source")));
|
||||||
@@ -56,8 +73,28 @@ async fn build(world: &TestWorld, name: &str) -> Result<Value, Value> {
|
|||||||
.register(&host);
|
.register(&host);
|
||||||
let result = host.call("cargo.build", json!({"plugin":name})).await;
|
let result = host.call("cargo.build", json!({"plugin":name})).await;
|
||||||
host.stop().await;
|
host.stop().await;
|
||||||
|
let logs = match &result {
|
||||||
|
Ok(output) => &output["logs"],
|
||||||
|
Err(error) => &error["data"]["logs"],
|
||||||
|
};
|
||||||
|
check_logs(world, logs);
|
||||||
result
|
result
|
||||||
}
|
}
|
||||||
|
/// RPC paths point at full raw streams, outside pruned build folders.
|
||||||
|
fn check_logs(world: &TestWorld, logs: &Value) {
|
||||||
|
let mut total = 0;
|
||||||
|
for stream in ["stdout", "stderr"] {
|
||||||
|
let path = logs[stream]["path"].as_str().expect("log path");
|
||||||
|
assert!(path.starts_with("/logs/cargo/"));
|
||||||
|
assert!(path.ends_with(&format!(".{stream}.log")));
|
||||||
|
let bytes = std::fs::metadata(world.0.join(path.trim_start_matches('/')))
|
||||||
|
.unwrap()
|
||||||
|
.len();
|
||||||
|
assert_eq!(logs[stream]["bytes"], bytes);
|
||||||
|
total += bytes;
|
||||||
|
}
|
||||||
|
assert_eq!(logs["total_bytes"], total);
|
||||||
|
}
|
||||||
struct Valid {
|
struct Valid {
|
||||||
world: TestWorld,
|
world: TestWorld,
|
||||||
result: Value,
|
result: Value,
|
||||||
@@ -189,6 +226,20 @@ async fn build_output_and_warnings() {
|
|||||||
.unwrap()
|
.unwrap()
|
||||||
.contains("fixture build warning")
|
.contains("fixture build warning")
|
||||||
);
|
);
|
||||||
|
let logs = &valid.result["logs"];
|
||||||
|
let raw = |stream: &str| {
|
||||||
|
std::fs::read_to_string(
|
||||||
|
valid.world.0.join(
|
||||||
|
logs[stream]["path"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.trim_start_matches('/'),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.unwrap()
|
||||||
|
};
|
||||||
|
assert!(raw("stdout").contains("\"reason\":\"compiler-artifact\""));
|
||||||
|
assert!(raw("stderr").contains("Finished "));
|
||||||
let output = valid.result["tests"]["output"].as_str().unwrap();
|
let output = valid.result["tests"]["output"].as_str().unwrap();
|
||||||
assert!(output.contains("INTEGRATION_CASE"));
|
assert!(output.contains("INTEGRATION_CASE"));
|
||||||
assert!(!output.contains("Compiling "));
|
assert!(!output.contains("Compiling "));
|
||||||
@@ -227,14 +278,7 @@ async fn manifest_and_nested_sources() {
|
|||||||
assert!(manifest.contains("optional = true"));
|
assert!(manifest.contains("optional = true"));
|
||||||
let document: toml_edit::DocumentMut = manifest.parse().unwrap();
|
let document: toml_edit::DocumentMut = manifest.parse().unwrap();
|
||||||
let sdk = Path::new(document["target"]["cfg(target_arch = \"wasm32\")"]["dependencies"]["sausage_plugin"]["path"].as_str().unwrap());
|
let sdk = Path::new(document["target"]["cfg(target_arch = \"wasm32\")"]["dependencies"]["sausage_plugin"]["path"].as_str().unwrap());
|
||||||
assert_eq!(sdk.parent().unwrap(), Path::new(CACHE).join("sdk"));
|
assert_eq!(sdk, valid.world.0.join("plugins/cargo/sdk"));
|
||||||
let hash = sdk.file_name().unwrap().to_str().unwrap();
|
|
||||||
assert_eq!(hash.len(), 64);
|
|
||||||
assert!(hash.bytes().all(|byte| byte.is_ascii_hexdigit()));
|
|
||||||
assert_eq!(
|
|
||||||
std::fs::read(sdk.join("src/lib.rs")).unwrap(),
|
|
||||||
std::fs::read(valid.world.0.join("plugins/cargo/sdk/src/lib.rs")).unwrap()
|
|
||||||
);
|
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
std::fs::read_to_string(root.join("src/target/data")).unwrap(),
|
std::fs::read_to_string(root.join("src/target/data")).unwrap(),
|
||||||
"nested target"
|
"nested target"
|
||||||
@@ -260,6 +304,7 @@ async fn native_discovery_and_error_paths() {
|
|||||||
.await
|
.await
|
||||||
.unwrap_err();
|
.unwrap_err();
|
||||||
assert_eq!(error["code"], -32004);
|
assert_eq!(error["code"], -32004);
|
||||||
|
check_logs(&world, &error["data"]["logs"]);
|
||||||
assert!(
|
assert!(
|
||||||
error["data"]["output"]
|
error["data"]["output"]
|
||||||
.as_str()
|
.as_str()
|
||||||
@@ -431,7 +476,7 @@ async fn source_copy() {
|
|||||||
assert!(
|
assert!(
|
||||||
std::fs::read_to_string(valid.world.0.join("state/cargo/work/Cargo.toml"))
|
std::fs::read_to_string(valid.world.0.join("state/cargo/work/Cargo.toml"))
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.contains(Path::new(CACHE).join("sdk").to_str().unwrap())
|
.contains(valid.world.0.join("plugins/cargo/sdk").to_str().unwrap())
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
/// PROTOCOL 11: compiler failures return compiler diagnostics.
|
/// PROTOCOL 11: compiler failures return compiler diagnostics.
|
||||||
@@ -585,6 +630,96 @@ async fn unchanged_sdk() {
|
|||||||
modified
|
modified
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
/// PROTOCOL 11: an SDK edit is used by the next build on the same running host.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn world_sdk_edits_apply_without_restart() {
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(180), async {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.simple("sdk-probe", "fn main() {}\n#[test]\nfn world_sdk_probe() { assert_eq!(sausage_plugin::world_sdk_probe(), 1); }\n");
|
||||||
|
let manifest = world.0.join("plugins/sdk-probe/source/Cargo.toml");
|
||||||
|
let mut text = std::fs::read_to_string(&manifest).unwrap();
|
||||||
|
text.push_str("\n[dependencies]\nsausage-plugin = '0.1'\n");
|
||||||
|
std::fs::write(manifest, text).unwrap();
|
||||||
|
let host = Host::start_with_cache(&world.0, Path::new(CACHE)).await.unwrap();
|
||||||
|
let library = world.0.join("plugins/cargo/sdk/src/lib.rs");
|
||||||
|
let original = std::fs::read_to_string(&library).unwrap();
|
||||||
|
std::fs::write(&library, format!("{original}\npub fn world_sdk_probe() -> u8 {{ 1 }}\n")).unwrap();
|
||||||
|
let params = json!({"plugin":"sdk-probe", "timeout_secs":120, "test_timeout_secs":10});
|
||||||
|
let first = host.call("cargo.build", params.clone()).await.unwrap();
|
||||||
|
assert_eq!(first["tests"]["passed"], 1);
|
||||||
|
// Only the SDK changes: Cargo must not reuse its old compiled version.
|
||||||
|
std::fs::write(&library, format!("{original}\npub fn world_sdk_probe() -> u8 {{ 2 }}\n")).unwrap();
|
||||||
|
let second = host.call("cargo.build", params).await.unwrap_err();
|
||||||
|
assert_eq!(second["code"], -32004);
|
||||||
|
assert_eq!(second["data"]["tests"]["failed"], 1);
|
||||||
|
assert!(second["data"]["tests"]["output"].as_str().unwrap().contains("world_sdk_probe"));
|
||||||
|
host.stop().await;
|
||||||
|
}).await.expect("SDK edit regression timed out");
|
||||||
|
}
|
||||||
|
/// PROTOCOL 11: startup seeds a missing SDK, but never repairs or overwrites an existing one.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn world_sdk_survives_restart_and_missing_sdk_is_reseeded() {
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(30), async {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
let sdk = world.0.join("plugins/cargo/sdk");
|
||||||
|
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let library = sdk.join("src/lib.rs");
|
||||||
|
let original = std::fs::read(&library).unwrap();
|
||||||
|
let manifest = sdk.join("Cargo.toml");
|
||||||
|
let edited = b"// world SDK edit must survive restart\n";
|
||||||
|
std::fs::write(&library, edited).unwrap();
|
||||||
|
std::fs::write(&manifest, "# edited manifest\n").unwrap();
|
||||||
|
let removed = sdk.join("macros/src/lib.rs");
|
||||||
|
std::fs::remove_file(&removed).unwrap();
|
||||||
|
host.stop().await;
|
||||||
|
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(std::fs::read(&library).unwrap(), edited);
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read_to_string(&manifest).unwrap(),
|
||||||
|
"# edited manifest\n"
|
||||||
|
);
|
||||||
|
assert!(!removed.exists(), "existing SDK must not be repaired");
|
||||||
|
host.stop().await;
|
||||||
|
std::fs::remove_dir_all(&sdk).unwrap();
|
||||||
|
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(std::fs::read(&library).unwrap(), original);
|
||||||
|
assert!(removed.is_file());
|
||||||
|
assert!(sdk.join("interface/src/lib.rs").is_file());
|
||||||
|
host.stop().await;
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.expect("SDK restart regression timed out");
|
||||||
|
}
|
||||||
|
/// PROTOCOL 11: removing the SDK while the host is running re-seeds it on the next build.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn world_sdk_missing_at_build_is_reseeded() {
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(180), async {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.simple("reseed-probe", "fn main() {}\n");
|
||||||
|
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let sdk = world.0.join("plugins/cargo/sdk");
|
||||||
|
let original = std::fs::read(sdk.join("src/lib.rs")).unwrap();
|
||||||
|
std::fs::remove_dir_all(&sdk).unwrap();
|
||||||
|
host.call(
|
||||||
|
"cargo.build",
|
||||||
|
json!({"plugin":"reseed-probe", "timeout_secs":120, "test_timeout_secs":10}),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(std::fs::read(sdk.join("src/lib.rs")).unwrap(), original);
|
||||||
|
host.stop().await;
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.expect("SDK re-seed build regression timed out");
|
||||||
|
}
|
||||||
/// PROTOCOL 11: init can build a plugin without a running host.
|
/// PROTOCOL 11: init can build a plugin without a running host.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn standalone_builder() {
|
async fn standalone_builder() {
|
||||||
@@ -606,6 +741,259 @@ async fn standalone_builder() {
|
|||||||
1
|
1
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
/// a named CPU-bound WASI test times out, while other instances still run.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn named_test_timeout_and_mutex_release() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.simple("test-timeout", "fn main() {} #[test] fn a_hung() { loop { std::hint::black_box(1); } } #[test] fn z_remaining() { eprintln!(\"REMAINING_TEST_RAN\"); }");
|
||||||
|
world.simple("next", "fn main() {} #[test] fn passes() {}");
|
||||||
|
let integration = world.0.join("plugins/test-timeout/source/tests");
|
||||||
|
std::fs::create_dir_all(&integration).unwrap();
|
||||||
|
std::fs::write(
|
||||||
|
integration.join("remaining.rs"),
|
||||||
|
"#[test] fn integration_remaining() { eprintln!(\"REMAINING_BINARY_RAN\"); }",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let host = world.timeout_host("named").await;
|
||||||
|
let error = tokio::time::timeout(
|
||||||
|
std::time::Duration::from_secs(60),
|
||||||
|
host.call(
|
||||||
|
"cargo.build",
|
||||||
|
json!({"plugin":"test-timeout", "test_timeout_secs":1, "timeout_secs":45}),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap_err();
|
||||||
|
assert_eq!(error["code"], -32004);
|
||||||
|
assert_eq!(error["message"], "WASI tests failed");
|
||||||
|
assert_eq!(error["data"]["tests"]["failed"], 1);
|
||||||
|
assert_eq!(error["data"]["tests"]["passed"], 2);
|
||||||
|
let output = error["data"]["output"].as_str().unwrap();
|
||||||
|
assert!(output.contains("test a_hung ... FAILED"), "{output}");
|
||||||
|
assert!(output.contains("timed out after 1 s"), "{output}");
|
||||||
|
assert!(output.contains("REMAINING_TEST_RAN"), "{output}");
|
||||||
|
assert!(output.contains("REMAINING_BINARY_RAN"), "{output}");
|
||||||
|
let next = tokio::time::timeout(
|
||||||
|
std::time::Duration::from_secs(30),
|
||||||
|
host.call("cargo.build", json!({"plugin":"next"})),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(next["tests"]["passed"], 1);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Wait for a build-script child without using timing sleeps.
|
||||||
|
async fn script_child(world: &TestWorld) -> u32 {
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(60), async {
|
||||||
|
loop {
|
||||||
|
if let Ok(text) = std::fs::read_to_string(world.0.join("state/cargo/work/child.pid"))
|
||||||
|
&& let Ok(pid) = text.trim().parse()
|
||||||
|
{
|
||||||
|
break pid;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.expect("build script never started")
|
||||||
|
}
|
||||||
|
/// A killed descendant may briefly remain as a zombie, but cannot keep executing.
|
||||||
|
async fn child_ended(child: u32) {
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(10), async {
|
||||||
|
loop {
|
||||||
|
match std::fs::read_to_string(format!("/proc/{child}/stat")) {
|
||||||
|
Ok(status) if status.split_whitespace().nth(2) == Some("Z") => break,
|
||||||
|
Err(error) if error.kind() == std::io::ErrorKind::NotFound => break,
|
||||||
|
_ => tokio::task::yield_now().await,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.expect("cargo descendant survived cancellation/timeout");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// an explicit small total budget kills build.rs and unblocks a queued build.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn build_timeout_and_queued_build() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.source("hang", &fixture("cargo-hang"));
|
||||||
|
world.simple("next", "fn main() {} #[test] fn passes() {}");
|
||||||
|
let host = world.timeout_host("queued").await;
|
||||||
|
let building = host.clone();
|
||||||
|
let task = tokio::spawn(async move {
|
||||||
|
building
|
||||||
|
.call("cargo.build", json!({"plugin":"hang", "timeout_secs":5}))
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
let child = script_child(&world).await;
|
||||||
|
let queued_host = host.clone();
|
||||||
|
let queued = tokio::spawn(async move {
|
||||||
|
queued_host
|
||||||
|
.call("cargo.build", json!({"plugin":"next", "timeout_secs":3}))
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
let error = task.await.unwrap().unwrap_err();
|
||||||
|
assert_eq!(error["code"], -32004);
|
||||||
|
assert_eq!(error["message"], "timeout");
|
||||||
|
assert_eq!(error["data"]["reason"], "timeout");
|
||||||
|
check_logs(&world, &error["data"]["logs"]);
|
||||||
|
assert_eq!(error["data"]["step"], "tests");
|
||||||
|
assert!(error["data"]["elapsed_secs"].as_f64().unwrap() >= 5.0);
|
||||||
|
assert!(!error["data"]["output"].as_str().unwrap().is_empty());
|
||||||
|
child_ended(child).await;
|
||||||
|
let next = tokio::time::timeout(std::time::Duration::from_secs(20), queued)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(next["tests"]["passed"], 1);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// the same total deadline covers release compilation and retains test results.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn release_build_timeout_retains_tests() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.simple(
|
||||||
|
"release-timeout",
|
||||||
|
r#"fn main() {}
|
||||||
|
#[test] fn prepare_release() {
|
||||||
|
std::fs::write(concat!(env!("CARGO_MANIFEST_DIR"), "/release-marker"), "ready").unwrap();
|
||||||
|
}"#,
|
||||||
|
);
|
||||||
|
std::fs::write(
|
||||||
|
world.0.join("plugins/release-timeout/source/build.rs"),
|
||||||
|
r#"
|
||||||
|
fn main() {
|
||||||
|
println!("cargo:rerun-if-changed=release-marker");
|
||||||
|
if std::path::Path::new("release-marker").exists() {
|
||||||
|
std::process::Command::new("sh")
|
||||||
|
.args(["-c", "echo $$ > child.pid; exec sleep 1000"])
|
||||||
|
.status().unwrap();
|
||||||
|
}
|
||||||
|
}"#,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let host = world.timeout_host("release").await;
|
||||||
|
let building = host.clone();
|
||||||
|
let task = tokio::spawn(async move {
|
||||||
|
building
|
||||||
|
.call(
|
||||||
|
"cargo.build",
|
||||||
|
json!({"plugin":"release-timeout", "timeout_secs":5}),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
let child = script_child(&world).await;
|
||||||
|
let error = tokio::time::timeout(std::time::Duration::from_secs(10), task)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap()
|
||||||
|
.unwrap_err();
|
||||||
|
assert_eq!(error["code"], -32004);
|
||||||
|
assert_eq!(error["data"]["reason"], "timeout");
|
||||||
|
check_logs(&world, &error["data"]["logs"]);
|
||||||
|
assert_eq!(error["data"]["step"], "release build");
|
||||||
|
assert_eq!(error["data"]["tests"]["passed"], 1);
|
||||||
|
assert!(error["data"]["elapsed_secs"].as_f64().unwrap() >= 5.0);
|
||||||
|
child_ended(child).await;
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// omission selects the real 900-second deadline, without a 15-minute test.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn omitted_build_timeout_uses_default() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.source("hang", &fixture("cargo-hang"));
|
||||||
|
let host = world.timeout_host("default").await;
|
||||||
|
let building = host.clone();
|
||||||
|
let task =
|
||||||
|
tokio::spawn(async move { building.call("cargo.build", json!({"plugin":"hang"})).await });
|
||||||
|
let child = script_child(&world).await;
|
||||||
|
tokio::time::pause();
|
||||||
|
tokio::time::advance(std::time::Duration::from_secs(
|
||||||
|
sausage::cargo::DEFAULT_TIMEOUT_SECS,
|
||||||
|
))
|
||||||
|
.await;
|
||||||
|
// Let the awakened deadline observe the advanced clock before resuming it.
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
let error = task.await.unwrap().unwrap_err();
|
||||||
|
tokio::time::resume();
|
||||||
|
assert_eq!(error["data"]["reason"], "timeout");
|
||||||
|
assert!(error["data"]["elapsed_secs"].as_f64().unwrap() >= 900.0);
|
||||||
|
child_ended(child).await;
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// a caller ending stops its attached native build and releases the mutex.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn ended_caller_releases_build_mutex() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
world.source("hang", &fixture("cargo-hang"));
|
||||||
|
world.simple("next", "fn main() {} #[test] fn passes() {}");
|
||||||
|
let host = world.timeout_host("caller").await;
|
||||||
|
let (finish, finished) = tokio::sync::watch::channel(false);
|
||||||
|
host.register_native(
|
||||||
|
"caller",
|
||||||
|
std::sync::Arc::new(move |context, _, _| {
|
||||||
|
let mut finished = finished.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
tokio::spawn(
|
||||||
|
async move { context.call("cargo.build", json!({"plugin":"hang"})).await },
|
||||||
|
);
|
||||||
|
let _ = finished.wait_for(|done| *done).await;
|
||||||
|
Ok(json!({}))
|
||||||
|
})
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
let calling = host.clone();
|
||||||
|
let caller = tokio::spawn(async move { calling.call("caller.run", json!({})).await });
|
||||||
|
let child = script_child(&world).await;
|
||||||
|
let queued_host = host.clone();
|
||||||
|
let queued = tokio::spawn(async move {
|
||||||
|
queued_host
|
||||||
|
.call("cargo.build", json!({"plugin":"next"}))
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
finish.send_replace(true);
|
||||||
|
caller.await.unwrap().unwrap();
|
||||||
|
let next = tokio::time::timeout(std::time::Duration::from_secs(12), queued)
|
||||||
|
.await
|
||||||
|
.expect("ended caller retained mutex")
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(next["tests"]["passed"], 1);
|
||||||
|
child_ended(child).await;
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// discovery exposes non-null defaults; invalid budgets cannot disable limits.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn timeout_params_discovery_and_validation() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let doc = host.call("cargo.rpc.discover", json!({})).await.unwrap();
|
||||||
|
let params = doc["methods"][0]["params"].as_array().unwrap();
|
||||||
|
for (name, default) in [("timeout_secs", 900), ("test_timeout_secs", 120)] {
|
||||||
|
let param = params.iter().find(|param| param["name"] == name).unwrap();
|
||||||
|
assert_eq!(param["schema"]["default"], default);
|
||||||
|
assert_eq!(param["schema"]["minimum"], 1);
|
||||||
|
assert!(!param["description"].as_str().unwrap().is_empty());
|
||||||
|
for value in [json!(0), json!(-1), json!(1.5), json!("1"), json!(null)] {
|
||||||
|
let mut args = json!({"plugin":"missing"});
|
||||||
|
args[name] = value;
|
||||||
|
let error = host.call("cargo.build", args).await.unwrap_err();
|
||||||
|
assert_eq!(error["code"], -32602, "{error}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
/// PROTOCOL 11: killing a native build also kills its build-script descendants.
|
/// PROTOCOL 11: killing a native build also kills its build-script descendants.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn kill_process_group() {
|
async fn kill_process_group() {
|
||||||
@@ -615,9 +1003,7 @@ async fn kill_process_group() {
|
|||||||
);
|
);
|
||||||
let world = TestWorld::new();
|
let world = TestWorld::new();
|
||||||
world.source("hang", &fixture("cargo-hang"));
|
world.source("hang", &fixture("cargo-hang"));
|
||||||
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
|
let host = world.timeout_host("kill").await;
|
||||||
.await
|
|
||||||
.unwrap();
|
|
||||||
let building = host.clone();
|
let building = host.clone();
|
||||||
let task =
|
let task =
|
||||||
tokio::spawn(async move { building.call("cargo.build", json!({"plugin":"hang"})).await });
|
tokio::spawn(async move { building.call("cargo.build", json!({"plugin":"hang"})).await });
|
||||||
@@ -685,8 +1071,23 @@ async fn kill_process_group() {
|
|||||||
.unwrap()
|
.unwrap()
|
||||||
.is_empty()
|
.is_empty()
|
||||||
);
|
);
|
||||||
|
// Queue a build on the SAME builder before cancelling the lock holder.
|
||||||
|
world.simple("next", "fn main() {} #[test] fn passes() {}");
|
||||||
|
let queued_host = host.clone();
|
||||||
|
let queued = tokio::spawn(async move {
|
||||||
|
queued_host
|
||||||
|
.call("cargo.build", json!({"plugin":"next"}))
|
||||||
|
.await
|
||||||
|
});
|
||||||
host.call("core.kill", json!({"pid":pid})).await.unwrap();
|
host.call("core.kill", json!({"pid":pid})).await.unwrap();
|
||||||
assert!(task.await.unwrap().is_err());
|
assert!(task.await.unwrap().is_err());
|
||||||
|
child_ended(ancestor).await;
|
||||||
|
let next = tokio::time::timeout(std::time::Duration::from_secs(10), queued)
|
||||||
|
.await
|
||||||
|
.expect("cancelled build retained mutex")
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(next["tests"]["passed"], 1);
|
||||||
tokio::time::timeout(std::time::Duration::from_secs(10), async {
|
tokio::time::timeout(std::time::Duration::from_secs(10), async {
|
||||||
loop {
|
loop {
|
||||||
let status = std::fs::read_to_string(format!("/proc/{child}/stat"));
|
let status = std::fs::read_to_string(format!("/proc/{child}/stat"));
|
||||||
|
|||||||
+53
-18
@@ -19,6 +19,7 @@ fn binary() -> Command {
|
|||||||
.arg("--cache-dir")
|
.arg("--cache-dir")
|
||||||
.arg(CACHE)
|
.arg(CACHE)
|
||||||
.env_remove("SAUSAGE_HEADLESS")
|
.env_remove("SAUSAGE_HEADLESS")
|
||||||
|
.env_remove("SAUSAGE_LOOP")
|
||||||
.kill_on_drop(true);
|
.kill_on_drop(true);
|
||||||
command
|
command
|
||||||
}
|
}
|
||||||
@@ -35,7 +36,7 @@ async fn status(child: &mut Child) -> std::process::ExitStatus {
|
|||||||
}
|
}
|
||||||
async fn journal_exists(world: &Path) {
|
async fn journal_exists(world: &Path) {
|
||||||
tokio::time::timeout(Duration::from_secs(15), async {
|
tokio::time::timeout(Duration::from_secs(15), async {
|
||||||
while !world.join("state/core/journal.sqlite").exists() {
|
while !world.join("logs/core/journal.sqlite").exists() {
|
||||||
tokio::task::yield_now().await;
|
tokio::task::yield_now().await;
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
@@ -54,9 +55,15 @@ fn install_probe(world: &Path) {
|
|||||||
std::fs::copy(probe(), world.join("plugins/probe/plugin.wasm")).unwrap();
|
std::fs::copy(probe(), world.join("plugins/probe/plugin.wasm")).unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A failed agent build leaves no world. Bare sausage retries and publishes a complete world.
|
/// Failed init leaves no world; successful init seeds the restart policy and a working agent.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn init_builds_agent() {
|
async fn init_builds_agent() {
|
||||||
|
tokio::time::timeout(Duration::from_secs(300), init_builds_agent_inner())
|
||||||
|
.await
|
||||||
|
.expect("init and seeded-agent checks exceeded 300 seconds");
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn init_builds_agent_inner() {
|
||||||
let temporary = tempfile::tempdir().unwrap();
|
let temporary = tempfile::tempdir().unwrap();
|
||||||
let world = temporary.path().join("world");
|
let world = temporary.path().join("world");
|
||||||
// Legacy configuration variables must not affect the templates.
|
// Legacy configuration variables must not affect the templates.
|
||||||
@@ -81,7 +88,7 @@ async fn init_builds_agent() {
|
|||||||
assert!(!failed.status.success());
|
assert!(!failed.status.success());
|
||||||
assert!(String::from_utf8_lossy(&failed.stderr).contains("nonexistent-sausage-rustc"));
|
assert!(String::from_utf8_lossy(&failed.stderr).contains("nonexistent-sausage-rustc"));
|
||||||
assert!(!world.exists());
|
assert!(!world.exists());
|
||||||
let output = configured().output().await.unwrap();
|
let output = configured().arg("init").output().await.unwrap();
|
||||||
assert!(
|
assert!(
|
||||||
output.status.success(),
|
output.status.success(),
|
||||||
"stdout: {}\nstderr: {}",
|
"stdout: {}\nstderr: {}",
|
||||||
@@ -91,6 +98,22 @@ async fn init_builds_agent() {
|
|||||||
assert!(
|
assert!(
|
||||||
String::from_utf8_lossy(&output.stdout).contains("first agent build takes a few minutes")
|
String::from_utf8_lossy(&output.stdout).contains("first agent build takes a few minutes")
|
||||||
);
|
);
|
||||||
|
// Bare sausage reopens the completed world instead of trying to recreate it.
|
||||||
|
assert!(configured().output().await.unwrap().status.success());
|
||||||
|
let prompt = std::fs::read_to_string(world.join("plugins/agent/source/src/prompt.md")).unwrap();
|
||||||
|
assert!(prompt.contains("HARD RULE: the mediator and all agents NEVER call `core.restart`"));
|
||||||
|
assert!(prompt.contains("without the person's explicit approval each time"));
|
||||||
|
assert!(prompt.contains("alias `core.exit`"));
|
||||||
|
for text in [
|
||||||
|
"`#loop`",
|
||||||
|
"SAUSAGE_LOOP=1",
|
||||||
|
"answers before",
|
||||||
|
"exit code 75",
|
||||||
|
] {
|
||||||
|
assert!(prompt.contains(text), "seeded prompt missing {text}");
|
||||||
|
}
|
||||||
|
let boot = std::fs::read_to_string(world.join("config/boot.yaml")).unwrap();
|
||||||
|
assert!(boot.contains("SAUSAGE_LOOP=1"));
|
||||||
let mut config: Value =
|
let mut config: Value =
|
||||||
serde_yaml_ng::from_slice(&std::fs::read(world.join("config/agent.yaml")).unwrap())
|
serde_yaml_ng::from_slice(&std::fs::read(world.join("config/agent.yaml")).unwrap())
|
||||||
.unwrap();
|
.unwrap();
|
||||||
@@ -228,7 +251,7 @@ async fn signals_work_during_boot() {
|
|||||||
tokio::time::timeout(Duration::from_secs(15), async {
|
tokio::time::timeout(Duration::from_secs(15), async {
|
||||||
loop {
|
loop {
|
||||||
let db =
|
let db =
|
||||||
rusqlite::Connection::open(world.join("state/core/journal.sqlite")).unwrap();
|
rusqlite::Connection::open(world.join("logs/core/journal.sqlite")).unwrap();
|
||||||
if let Ok(2) = db.query_row(
|
if let Ok(2) = db.query_row(
|
||||||
"SELECT count(*) FROM journal WHERE line='service start core.lock'",
|
"SELECT count(*) FROM journal WHERE line='service start core.lock'",
|
||||||
[],
|
[],
|
||||||
@@ -249,27 +272,39 @@ async fn signals_work_during_boot() {
|
|||||||
/// Piped input supports commands, quit, and the restart exit code.
|
/// Piped input supports commands, quit, and the restart exit code.
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn piped_console_and_exit_codes() {
|
async fn piped_console_and_exit_codes() {
|
||||||
for (line, expected) in [("help\n", 0), ("quit\n", 0), ("core.exit\n", 75)] {
|
for (line, supervised, expected) in [
|
||||||
|
("help\n", false, 0),
|
||||||
|
("quit\n", false, 0),
|
||||||
|
("core.restart\n", false, 0),
|
||||||
|
("core.exit\n", false, 0),
|
||||||
|
("core.restart\n", true, 75),
|
||||||
|
("core.exit\n", true, 75),
|
||||||
|
] {
|
||||||
let (_folder, world) = world();
|
let (_folder, world) = world();
|
||||||
let mut child = binary()
|
let mut command = binary();
|
||||||
.arg("run")
|
command.arg("run").arg(world);
|
||||||
.arg(world)
|
if supervised {
|
||||||
|
command.env("SAUSAGE_LOOP", "1");
|
||||||
|
}
|
||||||
|
let mut child = command
|
||||||
.stdin(Stdio::piped())
|
.stdin(Stdio::piped())
|
||||||
.stdout(Stdio::piped())
|
.stdout(Stdio::piped())
|
||||||
.spawn()
|
.spawn()
|
||||||
.unwrap();
|
.unwrap();
|
||||||
child
|
// Supervised calls must finish via restart, not a racing console EOF.
|
||||||
.stdin
|
let mut input = child.stdin.take().unwrap();
|
||||||
.take()
|
input.write_all(line.as_bytes()).await.unwrap();
|
||||||
.unwrap()
|
if !supervised {
|
||||||
.write_all(line.as_bytes())
|
drop(input);
|
||||||
.await
|
}
|
||||||
.unwrap();
|
|
||||||
let output = tokio::time::timeout(Duration::from_secs(15), child.wait_with_output())
|
let output = tokio::time::timeout(Duration::from_secs(15), child.wait_with_output())
|
||||||
.await
|
.await
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.unwrap();
|
.unwrap();
|
||||||
assert_eq!(output.status.code(), Some(expected));
|
assert_eq!(output.status.code(), Some(expected));
|
||||||
|
if !supervised && line.starts_with("core.") {
|
||||||
|
assert!(String::from_utf8_lossy(&output.stdout).contains("SAUSAGE_LOOP not set"));
|
||||||
|
}
|
||||||
if expected == 0 {
|
if expected == 0 {
|
||||||
assert!(!output.stdout.is_empty());
|
assert!(!output.stdout.is_empty());
|
||||||
}
|
}
|
||||||
@@ -301,7 +336,7 @@ async fn console_error_stops_host() {
|
|||||||
.await
|
.await
|
||||||
.unwrap();
|
.unwrap();
|
||||||
assert_eq!(status(&mut child).await.code(), Some(1));
|
assert_eq!(status(&mut child).await.code(), Some(1));
|
||||||
let db = rusqlite::Connection::open(world.join("state/core/journal.sqlite")).unwrap();
|
let db = rusqlite::Connection::open(world.join("logs/core/journal.sqlite")).unwrap();
|
||||||
let stopped: i64 = db
|
let stopped: i64 = db
|
||||||
.query_row(
|
.query_row(
|
||||||
"SELECT count(*) FROM journal WHERE plugin='probe' AND line='stop'",
|
"SELECT count(*) FROM journal WHERE plugin='probe' AND line='stop'",
|
||||||
@@ -328,9 +363,9 @@ async fn interrupt_leaves_call_running() {
|
|||||||
input.write_all(b"probe.busy\n").await.unwrap();
|
input.write_all(b"probe.busy\n").await.unwrap();
|
||||||
let pid = tokio::time::timeout(Duration::from_secs(20), async {
|
let pid = tokio::time::timeout(Duration::from_secs(20), async {
|
||||||
loop {
|
loop {
|
||||||
if world.join("state/core/journal.sqlite").exists() {
|
if world.join("logs/core/journal.sqlite").exists() {
|
||||||
let db =
|
let db =
|
||||||
rusqlite::Connection::open(world.join("state/core/journal.sqlite")).unwrap();
|
rusqlite::Connection::open(world.join("logs/core/journal.sqlite")).unwrap();
|
||||||
if let Ok(pid) = db.query_row(
|
if let Ok(pid) = db.query_row(
|
||||||
"SELECT pid FROM journal WHERE plugin='probe' AND line='start busy {}'",
|
"SELECT pid FROM journal WHERE plugin='probe' AND line='start busy {}'",
|
||||||
[],
|
[],
|
||||||
|
|||||||
+267
-49
@@ -129,7 +129,7 @@ async fn stderr_flood_does_not_block_stdin() {
|
|||||||
let payload = json!({"large":"x".repeat(1024*1024)});
|
let payload = json!({"large":"x".repeat(1024*1024)});
|
||||||
assert_eq!(call(&h, "probe.echo", payload.clone()).await, payload);
|
assert_eq!(call(&h, "probe.echo", payload.clone()).await, payload);
|
||||||
h.stop().await;
|
h.stop().await;
|
||||||
let db = rusqlite::Connection::open(w.0.join("state/core/journal.sqlite")).unwrap();
|
let db = rusqlite::Connection::open(w.0.join("logs/core/journal.sqlite")).unwrap();
|
||||||
let count: i64 = db
|
let count: i64 = db
|
||||||
.query_row(
|
.query_row(
|
||||||
"SELECT count(*) FROM journal WHERE line LIKE 'flood %'",
|
"SELECT count(*) FROM journal WHERE line LIKE 'flood %'",
|
||||||
@@ -198,7 +198,7 @@ async fn null_error_is_success() {
|
|||||||
async fn journal_recovers_after_sqlite_write_failure() {
|
async fn journal_recovers_after_sqlite_write_failure() {
|
||||||
let w = TestWorld::new();
|
let w = TestWorld::new();
|
||||||
let h = w.start().await;
|
let h = w.start().await;
|
||||||
let db = rusqlite::Connection::open(w.0.join("state/core/journal.sqlite")).unwrap();
|
let db = rusqlite::Connection::open(w.0.join("logs/core/journal.sqlite")).unwrap();
|
||||||
db.busy_timeout(std::time::Duration::from_secs(5)).unwrap();
|
db.busy_timeout(std::time::Duration::from_secs(5)).unwrap();
|
||||||
db.execute_batch("CREATE TRIGGER reject_writes BEFORE INSERT ON journal BEGIN SELECT RAISE(FAIL,'test failure'); END;").unwrap();
|
db.execute_batch("CREATE TRIGGER reject_writes BEFORE INSERT ON journal BEGIN SELECT RAISE(FAIL,'test failure'); END;").unwrap();
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
@@ -206,6 +206,20 @@ async fn journal_recovers_after_sqlite_write_failure() {
|
|||||||
json!({"failed_log":true})
|
json!({"failed_log":true})
|
||||||
);
|
);
|
||||||
call(&h, "core.log", json!({})).await;
|
call(&h, "core.log", json!({})).await;
|
||||||
|
let batches: Vec<Value> = std::fs::read_dir(w.0.join("logs/core/failed-batches"))
|
||||||
|
.unwrap()
|
||||||
|
.map(Result::unwrap)
|
||||||
|
// The writer atomically publishes .json files; its temporary files can
|
||||||
|
// still be incomplete while later process-exit entries are spooling.
|
||||||
|
.filter(|file| file.path().extension().is_some_and(|ext| ext == "json"))
|
||||||
|
.map(|file| serde_json::from_slice(&std::fs::read(file.path()).unwrap()).unwrap())
|
||||||
|
.collect();
|
||||||
|
assert!(
|
||||||
|
batches
|
||||||
|
.iter()
|
||||||
|
.flat_map(|batch| batch.as_array().unwrap())
|
||||||
|
.any(|entry| entry["line"].as_str().unwrap().contains("failed_log"))
|
||||||
|
);
|
||||||
db.execute_batch("DROP TRIGGER reject_writes;").unwrap();
|
db.execute_batch("DROP TRIGGER reject_writes;").unwrap();
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
call(&h, "probe.echo", json!({"recovered":true})).await,
|
call(&h, "probe.echo", json!({"recovered":true})).await,
|
||||||
@@ -490,14 +504,119 @@ async fn unimplemented_core_methods_return_method_not_found() {
|
|||||||
async fn params_must_be_objects() {
|
async fn params_must_be_objects() {
|
||||||
let w = TestWorld::new();
|
let w = TestWorld::new();
|
||||||
let h = w.start().await;
|
let h = w.start().await;
|
||||||
|
for (params, kind) in [
|
||||||
|
(json!("{\"text\":\"hello\"}"), "a string"),
|
||||||
|
(json!([]), "an array"),
|
||||||
|
(json!(42), "a number"),
|
||||||
|
(json!(null), "null"),
|
||||||
|
(json!(true), "a boolean"),
|
||||||
|
] {
|
||||||
|
let expected = json!({
|
||||||
|
"code": -32602,
|
||||||
|
"message": format!("params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string.")
|
||||||
|
});
|
||||||
|
for method in ["probe.echo", "core.ps"] {
|
||||||
|
assert_eq!(h.call(method, params.clone()).await.unwrap_err(), expected);
|
||||||
|
}
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
h.call("probe.echo", json!([])).await.unwrap_err()["code"],
|
h.call("core.call", json!({"method":"probe.echo", "params":params}))
|
||||||
-32602
|
.await
|
||||||
|
.unwrap_err(),
|
||||||
|
expected
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let object = json!({"text":"hello"});
|
||||||
|
assert_eq!(call(&h, "probe.echo", object.clone()).await, object);
|
||||||
|
assert_eq!(
|
||||||
|
call(
|
||||||
|
&h,
|
||||||
|
"core.call",
|
||||||
|
json!({"method":"probe.echo", "params":object})
|
||||||
|
)
|
||||||
|
.await,
|
||||||
|
object
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
call(&h, "core.call", json!({"method":"probe.echo"})).await,
|
||||||
|
json!({})
|
||||||
);
|
);
|
||||||
|
|
||||||
h.stop().await;
|
h.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 3.1: stdout fast paths reject non-objects without changing defaults.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn stdout_params_must_be_objects() {
|
||||||
|
let w = TestWorld::new();
|
||||||
|
let h = w.start().await;
|
||||||
|
let mut notifications = h.subscribe();
|
||||||
|
for method in ["core.notify", "core.unlock", "probe.echo"] {
|
||||||
|
for (params, kind) in [
|
||||||
|
(json!("{}"), "a string"),
|
||||||
|
(json!([]), "an array"),
|
||||||
|
(json!(1.5), "a number"),
|
||||||
|
(json!(null), "null"),
|
||||||
|
(json!(false), "a boolean"),
|
||||||
|
] {
|
||||||
|
let response = call(
|
||||||
|
&h,
|
||||||
|
"probe.raw_call",
|
||||||
|
json!({
|
||||||
|
"jsonrpc":"2.0", "id":1, "method":method, "params":params
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert_eq!(response["id"], 1);
|
||||||
|
assert_eq!(
|
||||||
|
response["error"],
|
||||||
|
json!({
|
||||||
|
"code": -32602,
|
||||||
|
"message": format!("params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string.")
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let params = match method {
|
||||||
|
"core.notify" => json!({"name":"valid", "params":{"n":1}}),
|
||||||
|
"core.unlock" => json!({"path":"/state/not-held"}),
|
||||||
|
_ => json!({"n":1}),
|
||||||
|
};
|
||||||
|
let response = call(
|
||||||
|
&h,
|
||||||
|
"probe.raw_call",
|
||||||
|
json!({
|
||||||
|
"jsonrpc":"2.0", "id":1, "method":method, "params":params
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert_eq!(
|
||||||
|
response["result"],
|
||||||
|
if method == "probe.echo" {
|
||||||
|
params
|
||||||
|
} else {
|
||||||
|
Value::Null
|
||||||
|
}
|
||||||
|
);
|
||||||
|
assert!(response.get("error").is_none());
|
||||||
|
let omitted = call(
|
||||||
|
&h,
|
||||||
|
"probe.raw_call",
|
||||||
|
json!({
|
||||||
|
"jsonrpc":"2.0", "id":1, "method":method
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
match method {
|
||||||
|
"core.notify" => {
|
||||||
|
assert_eq!(omitted["error"]["message"], "name must be a string");
|
||||||
|
assert_eq!(notifications.try_recv().unwrap()["params"]["name"], "valid");
|
||||||
|
}
|
||||||
|
"core.unlock" => assert_eq!(omitted["error"]["message"], "path must be a string"),
|
||||||
|
_ => assert_eq!(omitted["result"], json!({})),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
h.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
/// PROTOCOL 3.9: exit without answer reports status and last logs.
|
/// PROTOCOL 3.9: exit without answer reports status and last logs.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn exit_without_answer_reports_status_and_last_logs() {
|
async fn exit_without_answer_reports_status_and_last_logs() {
|
||||||
@@ -896,6 +1015,7 @@ async fn stop_deadlines() {
|
|||||||
async fn shutdown_stops_busy_processes_in_parallel_after_five_seconds() {
|
async fn shutdown_stops_busy_processes_in_parallel_after_five_seconds() {
|
||||||
let w = TestWorld::new();
|
let w = TestWorld::new();
|
||||||
let h = w.start().await;
|
let h = w.start().await;
|
||||||
|
let mut notifications = h.subscribe();
|
||||||
|
|
||||||
let mut tasks = Vec::new();
|
let mut tasks = Vec::new();
|
||||||
for _ in 0..2 {
|
for _ in 0..2 {
|
||||||
@@ -905,14 +1025,14 @@ async fn shutdown_stops_busy_processes_in_parallel_after_five_seconds() {
|
|||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
tokio::time::timeout(std::time::Duration::from_secs(20), async {
|
tokio::time::timeout(std::time::Duration::from_secs(20), async {
|
||||||
while h
|
// Admission precedes guest startup. Stopping at admission can close
|
||||||
.processes()
|
// stdin before the guest receives its busy request, so there is then
|
||||||
.values()
|
// no busy work that needs the five-second kill budget.
|
||||||
.filter(|p| p.plugin == "probe" && p.method == "busy")
|
let mut ready = 0;
|
||||||
.count()
|
while ready != tasks.len() {
|
||||||
!= 2
|
if notifications.recv().await.unwrap()["params"]["name"] == "busy-ready" {
|
||||||
{
|
ready += 1;
|
||||||
tokio::task::yield_now().await;
|
}
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
.await
|
.await
|
||||||
@@ -989,17 +1109,16 @@ async fn stop_closes_stdin_and_waits_for_exit() {
|
|||||||
let w = TestWorld::new();
|
let w = TestWorld::new();
|
||||||
let h = w.start().await;
|
let h = w.start().await;
|
||||||
let hc = h.clone();
|
let hc = h.clone();
|
||||||
|
let mut events = h.subscribe();
|
||||||
let pending = tokio::spawn(async move { hc.call("probe.wait_eof", json!({})).await });
|
let pending = tokio::spawn(async move { hc.call("probe.wait_eof", json!({})).await });
|
||||||
|
// Admission is not readiness: stop may otherwise close stdin before the
|
||||||
|
// request reaches the guest, so wait for the handler to enter its EOF loop.
|
||||||
let pid = tokio::time::timeout(std::time::Duration::from_secs(20), async {
|
let pid = tokio::time::timeout(std::time::Duration::from_secs(20), async {
|
||||||
loop {
|
loop {
|
||||||
if let Some((pid, _)) = h
|
let event = events.recv().await.unwrap();
|
||||||
.processes()
|
if event["params"]["name"] == "waiting-eof" {
|
||||||
.into_iter()
|
break event["params"]["params"]["pid"].clone();
|
||||||
.find(|(_, p)| p.method == "wait_eof")
|
|
||||||
{
|
|
||||||
break pid;
|
|
||||||
}
|
}
|
||||||
tokio::task::yield_now().await;
|
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
.await
|
.await
|
||||||
@@ -1014,19 +1133,6 @@ async fn stop_closes_stdin_and_waits_for_exit() {
|
|||||||
h.stop().await;
|
h.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// PROTOCOL 9: core exit requests restart from its handler.
|
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
|
||||||
async fn core_exit_requests_restart_from_its_handler() {
|
|
||||||
let w = TestWorld::new();
|
|
||||||
let h = w.start().await;
|
|
||||||
assert_eq!(call(&h, "core.exit", json!({})).await, json!({}));
|
|
||||||
tokio::time::timeout(std::time::Duration::from_secs(1), h.exit_requested())
|
|
||||||
.await
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
h.stop().await;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// PROTOCOL 3.3: process ids are uuidv7 and versions are sha256.
|
/// PROTOCOL 3.3: process ids are uuidv7 and versions are sha256.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn process_ids_are_uuidv7_and_versions_are_sha256() {
|
async fn process_ids_are_uuidv7_and_versions_are_sha256() {
|
||||||
@@ -1081,7 +1187,7 @@ async fn journal_filters_and_entries_persist_in_sqlite() {
|
|||||||
);
|
);
|
||||||
h.stop().await;
|
h.stop().await;
|
||||||
assert!(
|
assert!(
|
||||||
std::fs::read(w.0.join("state/core/journal.sqlite"))
|
std::fs::read(w.0.join("logs/core/journal.sqlite"))
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.starts_with(b"SQLite format 3")
|
.starts_with(b"SQLite format 3")
|
||||||
);
|
);
|
||||||
@@ -1267,8 +1373,9 @@ async fn journal_entries_have_uuid_time_identity_stream_and_line() {
|
|||||||
async fn journal_has_wal_and_identity_indexes() {
|
async fn journal_has_wal_and_identity_indexes() {
|
||||||
let w = TestWorld::new();
|
let w = TestWorld::new();
|
||||||
let h = w.start().await;
|
let h = w.start().await;
|
||||||
|
assert!(!w.0.join("state/core/journal.sqlite").exists());
|
||||||
h.stop().await;
|
h.stop().await;
|
||||||
let db = rusqlite::Connection::open(w.0.join("state/core/journal.sqlite")).unwrap();
|
let db = rusqlite::Connection::open(w.0.join("logs/core/journal.sqlite")).unwrap();
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
db.query_row("PRAGMA journal_mode", [], |r| r.get::<_, String>(0))
|
db.query_row("PRAGMA journal_mode", [], |r| r.get::<_, String>(0))
|
||||||
.unwrap(),
|
.unwrap(),
|
||||||
@@ -1288,19 +1395,6 @@ async fn journal_has_wal_and_identity_indexes() {
|
|||||||
h.stop().await;
|
h.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// PROTOCOL 9: core exit works through core call.
|
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
|
||||||
async fn core_exit_works_through_core_call() {
|
|
||||||
let w = TestWorld::new();
|
|
||||||
let h = w.start().await;
|
|
||||||
assert_eq!(
|
|
||||||
call(&h, "core.call", json!({"method":"core.exit","params":{}})).await,
|
|
||||||
json!({})
|
|
||||||
);
|
|
||||||
assert!(h.restart_requested());
|
|
||||||
h.stop().await;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// PROTOCOL 6: acquisitions from different processes keep their FIFO positions.
|
/// PROTOCOL 6: acquisitions from different processes keep their FIFO positions.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn lock_acquisitions_from_different_processes_are_fifo() {
|
async fn lock_acquisitions_from_different_processes_are_fifo() {
|
||||||
@@ -1931,9 +2025,67 @@ async fn detached_processes_need_explicit_termination() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// PROTOCOL 9: core.exit from boot ends the host executable with status 75.
|
/// PROTOCOL 9: restart answers the detached caller, then uses normal stop and status 75.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn core_exit_sets_executable_status_75() {
|
async fn core_restart_cleanly_stops_detached_caller() {
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(30), async {
|
||||||
|
let w = TestWorld::new();
|
||||||
|
std::fs::write(
|
||||||
|
w.0.join("config/boot.yaml"),
|
||||||
|
br#"[{"method":"probe.restart","params":{}}]"#,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let output = tokio::process::Command::new(env!("CARGO_BIN_EXE_sausage"))
|
||||||
|
.env("SAUSAGE_LOOP", "1")
|
||||||
|
.arg("--headless")
|
||||||
|
.arg("--cache-dir")
|
||||||
|
.arg(concat!(env!("CARGO_TARGET_TMPDIR"), "/cache"))
|
||||||
|
.arg("run")
|
||||||
|
.arg(&w.0)
|
||||||
|
.stdin(std::process::Stdio::null())
|
||||||
|
.kill_on_drop(true)
|
||||||
|
.output()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
output.status.code(),
|
||||||
|
Some(75),
|
||||||
|
"{}",
|
||||||
|
String::from_utf8_lossy(&output.stderr)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read_to_string(w.0.join("state/restart-answer")).unwrap(),
|
||||||
|
"received"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read_to_string(w.0.join("state/restart-stop")).unwrap(),
|
||||||
|
"persisted after EOF"
|
||||||
|
);
|
||||||
|
let db = rusqlite::Connection::open(w.0.join("logs/core/journal.sqlite")).unwrap();
|
||||||
|
let count: i64 = db
|
||||||
|
.query_row(
|
||||||
|
"SELECT count(*) FROM journal WHERE line = 'restart stop hook finished'",
|
||||||
|
[],
|
||||||
|
|row| row.get(0),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(count, 1);
|
||||||
|
let killed: i64 = db
|
||||||
|
.query_row(
|
||||||
|
"SELECT count(*) FROM journal WHERE line = 'kill'",
|
||||||
|
[],
|
||||||
|
|row| row.get(0),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(killed, 0);
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 9: supervised core.exit from boot uses the restart shutdown path and code 75.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn core_exit_alias_sets_executable_status_75() {
|
||||||
let w = TestWorld::new();
|
let w = TestWorld::new();
|
||||||
std::fs::write(
|
std::fs::write(
|
||||||
w.0.join("config/boot.yaml"),
|
w.0.join("config/boot.yaml"),
|
||||||
@@ -1948,6 +2100,7 @@ async fn core_exit_sets_executable_status_75() {
|
|||||||
.arg(concat!(env!("CARGO_TARGET_TMPDIR"), "/cache"))
|
.arg(concat!(env!("CARGO_TARGET_TMPDIR"), "/cache"))
|
||||||
.arg("run")
|
.arg("run")
|
||||||
.arg(&w.0)
|
.arg(&w.0)
|
||||||
|
.env("SAUSAGE_LOOP", "1")
|
||||||
.stdin(std::process::Stdio::null())
|
.stdin(std::process::Stdio::null())
|
||||||
.kill_on_drop(true)
|
.kill_on_drop(true)
|
||||||
.output(),
|
.output(),
|
||||||
@@ -1962,3 +2115,68 @@ async fn core_exit_sets_executable_status_75() {
|
|||||||
String::from_utf8_lossy(&output.stderr)
|
String::from_utf8_lossy(&output.stderr)
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 7: logs move, state and the old journal do not; core.log merges both.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn journal_location_preserves_legacy_history() {
|
||||||
|
let w = TestWorld::new();
|
||||||
|
let old = w.0.join("state/core/journal.sqlite");
|
||||||
|
let db = rusqlite::Connection::open(&old).unwrap();
|
||||||
|
db.execute_batch("CREATE TABLE journal (id TEXT PRIMARY KEY, time TEXT, pid TEXT, plugin TEXT, stream TEXT, line TEXT);
|
||||||
|
INSERT INTO journal VALUES ('0000','old time','old-pid','old-plugin','core','legacy history');").unwrap();
|
||||||
|
drop(db);
|
||||||
|
let before = std::fs::read(&old).unwrap();
|
||||||
|
let history = w.0.join("state/core/console-history");
|
||||||
|
std::fs::write(&history, "keep console history").unwrap();
|
||||||
|
let h = w.start().await;
|
||||||
|
call(&h, "probe.echo", json!({})).await;
|
||||||
|
assert!(w.0.join("logs/core/journal.sqlite").exists());
|
||||||
|
let entries = call(&h, "core.log", json!({"limit":-1})).await;
|
||||||
|
assert_eq!(entries[0]["line"], "legacy history");
|
||||||
|
assert!(entries.as_array().unwrap().len() > 1);
|
||||||
|
assert_eq!(
|
||||||
|
call(&h, "core.log", json!({"pid":"old-pid"})).await[0]["line"],
|
||||||
|
"legacy history"
|
||||||
|
);
|
||||||
|
h.stop().await;
|
||||||
|
assert_eq!(std::fs::read(&old).unwrap(), before);
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read_to_string(history).unwrap(),
|
||||||
|
"keep console history"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 7: the journal retains the full trap, not just its short status reason.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn full_trap_diagnostic_is_journaled() {
|
||||||
|
let w = TestWorld::new();
|
||||||
|
let h = w.start().await;
|
||||||
|
let error = tokio::time::timeout(
|
||||||
|
std::time::Duration::from_secs(20),
|
||||||
|
h.call("probe.trap", json!({})),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap_err();
|
||||||
|
assert_eq!(error["code"], -32002);
|
||||||
|
assert!(
|
||||||
|
error["data"]["status"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.starts_with("trap: wasm trap:")
|
||||||
|
);
|
||||||
|
let entries = call(
|
||||||
|
&h,
|
||||||
|
"core.log",
|
||||||
|
json!({"plugin":"probe","grep":"wasm backtrace"}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert!(
|
||||||
|
entries.as_array().unwrap().iter().any(|e| {
|
||||||
|
let text = e["line"].as_str().unwrap();
|
||||||
|
text.contains("Caused by:") && text.contains("wasm trap:")
|
||||||
|
}),
|
||||||
|
"{entries}"
|
||||||
|
);
|
||||||
|
h.stop().await;
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,146 @@
|
|||||||
|
//! Generated native discovery, proc projection, and a real SDK import consumer.
|
||||||
|
use sausage::{Host, world::World};
|
||||||
|
use sausage_vfs::{Layer, Open};
|
||||||
|
use serde_json::{Value, json};
|
||||||
|
use std::{path::Path, time::Duration};
|
||||||
|
|
||||||
|
async fn call(host: &Host, method: &str, params: Value) -> Value {
|
||||||
|
// Same call budget as the core/watches host tests.
|
||||||
|
tokio::time::timeout(Duration::from_secs(20), host.call(method, params))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn read(host: &Host, path: &str) -> Value {
|
||||||
|
let fs = host.filesystem().await.unwrap();
|
||||||
|
let file = fs.open(path, Open::default()).await.unwrap();
|
||||||
|
let bytes = file
|
||||||
|
.read(0, file.stat().await.unwrap().size as usize)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
serde_json::from_slice(&bytes).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// No saved core.json: the RPC, proc file, importer and admission use generated types.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn generated_core_import_runs_under_wasi() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let world = dir.path().join("world");
|
||||||
|
World::create(&world).unwrap();
|
||||||
|
let host = Host::start_with_cache(&world, Path::new(super::common::CACHE))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let doc = call(&host, "core.rpc.discover", json!({})).await;
|
||||||
|
assert_eq!(doc["openrpc"], "1.2.6");
|
||||||
|
assert_eq!(doc["info"]["title"], "core");
|
||||||
|
assert_eq!(read(&host, "/proc/core/discover.json").await, doc);
|
||||||
|
let fs = host.filesystem().await.unwrap();
|
||||||
|
assert!(
|
||||||
|
fs.list("/proc/core")
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.iter()
|
||||||
|
.any(|e| e.name == "discover.json")
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
fs.open(
|
||||||
|
"/proc/core/discover.json",
|
||||||
|
Open {
|
||||||
|
write: true,
|
||||||
|
..Open::default()
|
||||||
|
}
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
let schemas = &doc["components"]["schemas"];
|
||||||
|
assert_eq!(schemas["Held"]["properties"]["held"]["type"], "boolean");
|
||||||
|
assert_eq!(schemas["Entry"]["properties"]["line"]["type"], "string");
|
||||||
|
let problems = sausage_interface::compat::check(&doc, &doc);
|
||||||
|
assert!(problems.is_empty(), "{problems:?}");
|
||||||
|
|
||||||
|
let source = dir.path().join("consumer");
|
||||||
|
std::fs::create_dir_all(source.join("imports")).unwrap();
|
||||||
|
std::fs::create_dir_all(source.join("src")).unwrap();
|
||||||
|
// This is exactly the user's copy-from-proc workflow, not a JSON fixture.
|
||||||
|
std::fs::write(source.join("imports/core.json"), doc.to_string()).unwrap();
|
||||||
|
let sdk = Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||||
|
.parent()
|
||||||
|
.unwrap()
|
||||||
|
.join("sdk");
|
||||||
|
std::fs::write(
|
||||||
|
source.join("Cargo.toml"),
|
||||||
|
format!(
|
||||||
|
r#"
|
||||||
|
[package]
|
||||||
|
name = "native-core-consumer"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2024"
|
||||||
|
[workspace]
|
||||||
|
[dependencies]
|
||||||
|
sausage-plugin = {{ path = {sdk:?} }}
|
||||||
|
"#
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
std::fs::write(
|
||||||
|
source.join("src/main.rs"),
|
||||||
|
r#"
|
||||||
|
use sausage_plugin::{Result, rpc, serde_json::json};
|
||||||
|
sausage_plugin::import!("core");
|
||||||
|
struct Consumer;
|
||||||
|
#[rpc]
|
||||||
|
impl Consumer {
|
||||||
|
/// Exercise actual generated parameter and result types.
|
||||||
|
async fn run(&self) -> Result<bool> {
|
||||||
|
let held = core::lock(core::Lock { path: "/typed-lock".into(), wait: Some(false) }).await?;
|
||||||
|
let filters: core::Log = sausage_plugin::serde_json::from_value(json!({"limit":1}))?;
|
||||||
|
for entry in core::log(filters).await? {
|
||||||
|
let _: String = entry.line;
|
||||||
|
}
|
||||||
|
core::unlock(core::Path { path: "/typed-lock".into() }).await?;
|
||||||
|
let doc = core::rpc_discover().await?;
|
||||||
|
Ok(held.held && doc["info"]["title"] == "core")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fn main() { sausage_plugin::run(Consumer) }
|
||||||
|
"#,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let target = Path::new(concat!(env!("CARGO_TARGET_TMPDIR"), "/guests"));
|
||||||
|
assert!(
|
||||||
|
std::process::Command::new("cargo")
|
||||||
|
.args([
|
||||||
|
"build",
|
||||||
|
"--offline",
|
||||||
|
"--target",
|
||||||
|
"wasm32-wasip2",
|
||||||
|
"--manifest-path"
|
||||||
|
])
|
||||||
|
.arg(source.join("Cargo.toml"))
|
||||||
|
.arg("--target-dir")
|
||||||
|
.arg(target)
|
||||||
|
.status()
|
||||||
|
.unwrap()
|
||||||
|
.success()
|
||||||
|
);
|
||||||
|
std::fs::copy(
|
||||||
|
target.join("wasm32-wasip2/debug/native-core-consumer.wasm"),
|
||||||
|
world.join("candidate.wasm"),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let activation = call(
|
||||||
|
&host,
|
||||||
|
"core.activate",
|
||||||
|
json!({"plugin":"consumer","wasm":"/candidate.wasm"}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert_eq!(activation["consumer"]["interface_check"], "passed");
|
||||||
|
assert_eq!(call(&host, "consumer.run", json!({})).await, true);
|
||||||
|
assert_eq!(
|
||||||
|
read(&host, "/proc/core/graph.json").await["plugins"]["consumer"]["imports"]["core"]["status"],
|
||||||
|
"compatible"
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
@@ -6,4 +6,6 @@ mod cli;
|
|||||||
mod common;
|
mod common;
|
||||||
mod console;
|
mod console;
|
||||||
mod core;
|
mod core;
|
||||||
|
mod core_discovery;
|
||||||
mod sdk;
|
mod sdk;
|
||||||
|
mod watches;
|
||||||
|
|||||||
@@ -178,6 +178,56 @@ async fn stop_runs_hook() {
|
|||||||
assert_eq!(std::fs::read(world.0.join("stopped")).unwrap(), b"yes");
|
assert_eq!(std::fs::read(world.0.join("stopped")).unwrap(), b"yes");
|
||||||
host.stop().await;
|
host.stop().await;
|
||||||
}
|
}
|
||||||
|
/// PROTOCOL 3.7: an idle SDK listener opts into drain and exits before EOF/grace.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn superseded_idle_listener_drains_before_grace() {
|
||||||
|
let world = TestWorld::new();
|
||||||
|
let host = world.start().await;
|
||||||
|
let (notice, mut notices) = tokio::sync::mpsc::unbounded_channel();
|
||||||
|
host.register_native(
|
||||||
|
"other",
|
||||||
|
std::sync::Arc::new(move |_, method, _| {
|
||||||
|
let notice = notice.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
assert_eq!(method, "superseded");
|
||||||
|
notice.send(std::time::Instant::now()).unwrap();
|
||||||
|
Ok(json!({}))
|
||||||
|
})
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
let pid = call(&host, "probe.drain_service", json!({})).await;
|
||||||
|
// Replace with a different real component; force only skips unrelated test consumers.
|
||||||
|
std::fs::copy(
|
||||||
|
fixtures().join("notes.wasm"),
|
||||||
|
world.0.join("state/next.wasm"),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
call(
|
||||||
|
&host,
|
||||||
|
"core.activate",
|
||||||
|
json!({"plugin":"probe","wasm":"/state/next.wasm","force":true}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
// Activation preparation/discovery is outside upgrade grace. Time from the
|
||||||
|
// hook's outgoing call, not from the start of candidate compilation.
|
||||||
|
let before = tokio::time::timeout(Duration::from_secs(60), notices.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
// Waiting on its lifetime lock is an exit event, not process-liveness polling.
|
||||||
|
call(&host, "core.lock", json!({"path":"/drain-owner"})).await;
|
||||||
|
assert!(before.elapsed() < Duration::from_secs(5));
|
||||||
|
assert!(!host.processes().contains_key(pid.as_str().unwrap()));
|
||||||
|
assert!(
|
||||||
|
!world
|
||||||
|
.0
|
||||||
|
.join(format!("stopped-{}", pid.as_str().unwrap()))
|
||||||
|
.exists(),
|
||||||
|
"drain must not run on_stop (discovery processes may run their own hooks)"
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
/// SDK framing carries payloads above 64 KiB in both directions.
|
/// SDK framing carries payloads above 64 KiB in both directions.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn large_messages() {
|
async fn large_messages() {
|
||||||
|
|||||||
@@ -0,0 +1,174 @@
|
|||||||
|
//! Real SDK watch callbacks and raw WASI filesystem mutations share the host namespace.
|
||||||
|
use super::common::{CACHE, probe, sdk_probe};
|
||||||
|
use sausage::{Host, world::World};
|
||||||
|
use serde_json::{Value, json};
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
// Same call bound as core.rs; no sleeps or polling.
|
||||||
|
async fn call(h: &Host, method: &str, p: Value) -> Value {
|
||||||
|
tokio::time::timeout(Duration::from_secs(20), h.call(method, p))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
async fn event(h: &Host, rx: &mut tokio::sync::broadcast::Receiver<Value>, name: &str) -> Value {
|
||||||
|
let result = tokio::time::timeout(Duration::from_secs(20), async {
|
||||||
|
loop {
|
||||||
|
let e = rx.recv().await.unwrap();
|
||||||
|
if e["params"]["name"] == name {
|
||||||
|
return e["params"]["params"].clone();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
match result {
|
||||||
|
Ok(v) => v,
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!(
|
||||||
|
"JOURNAL {}",
|
||||||
|
h.call("core.log", json!({"limit":100})).await.unwrap()
|
||||||
|
);
|
||||||
|
panic!("{e}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
async fn start() -> (tempfile::TempDir, Host) {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let world = World::create(&dir.path().join("world")).unwrap();
|
||||||
|
for (name, wasm) in [("probe", probe()), ("sdk", sdk_probe())] {
|
||||||
|
std::fs::create_dir_all(world.plugin_dir(name)).unwrap();
|
||||||
|
std::fs::copy(wasm, world.plugin_dir(name).join("plugin.wasm")).unwrap();
|
||||||
|
}
|
||||||
|
let host = Host::start_with_cache(&world.root, std::path::Path::new(CACHE))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
(dir, host)
|
||||||
|
}
|
||||||
|
fn watch(h: &Host, path: &str) -> tokio::task::JoinHandle<sausage::RpcResult> {
|
||||||
|
let (h, path) = (h.clone(), path.to_owned());
|
||||||
|
tokio::spawn(async move { h.call("sdk.watch", json!({"path":path})).await })
|
||||||
|
}
|
||||||
|
async fn mutate(h: &Host, op: &str, path: &str) {
|
||||||
|
call(h, "probe.mutate", json!({"op":op,"path":path})).await;
|
||||||
|
}
|
||||||
|
/// PROTOCOL 4: pre-arm state is in the first scan; each post-scan mutation wakes
|
||||||
|
/// the SDK without completing the watch. Covers descriptor/stream/truncate paths.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn armed_scan_and_later_changes() {
|
||||||
|
let (_dir, h) = start().await;
|
||||||
|
mutate(&h, "mkdir", "/state/mailbox").await;
|
||||||
|
call(
|
||||||
|
&h,
|
||||||
|
"probe.write",
|
||||||
|
json!({"path":"/state/mailbox/before", "text":"before"}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let mut rx = h.subscribe();
|
||||||
|
let task = watch(&h, "/state/mailbox/../mailbox/");
|
||||||
|
let armed = event(&h, &mut rx, "armed").await;
|
||||||
|
assert_eq!(armed["entries"], json!(["before"]));
|
||||||
|
assert!(!task.is_finished());
|
||||||
|
for (op, expected) in [
|
||||||
|
("descriptor", "descriptor"),
|
||||||
|
("stream", "streamptor"),
|
||||||
|
("append", "streamptorappend"),
|
||||||
|
("size", ""),
|
||||||
|
("descriptor", "descriptor"),
|
||||||
|
("truncate", ""),
|
||||||
|
] {
|
||||||
|
mutate(&h, op, "/state/mailbox/before").await;
|
||||||
|
let mut scan = event(&h, &mut rx, "changed").await;
|
||||||
|
while scan["text"] != expected {
|
||||||
|
scan = event(&h, &mut rx, "changed").await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mutate(&h, "burst", "/state/mailbox").await;
|
||||||
|
let mut scanned = event(&h, &mut rx, "changed").await;
|
||||||
|
while !scanned["entries"].as_array().unwrap().contains(&json!("4")) {
|
||||||
|
scanned = event(&h, &mut rx, "changed").await;
|
||||||
|
}
|
||||||
|
while rx.try_recv().is_ok() {}
|
||||||
|
mutate(&h, "delete", "/state/mailbox/4").await;
|
||||||
|
let mut scanned = event(&h, &mut rx, "changed").await;
|
||||||
|
while scanned["entries"].as_array().unwrap().contains(&json!("4")) {
|
||||||
|
scanned = event(&h, &mut rx, "changed").await;
|
||||||
|
}
|
||||||
|
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
|
||||||
|
assert!(task.await.unwrap().is_err());
|
||||||
|
h.stop().await;
|
||||||
|
}
|
||||||
|
/// PROTOCOL 4: atomic publication, rename source/destination, mounted paths,
|
||||||
|
/// absent paths, and caller-death cleanup. No external host writes are required.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn rename_delete_and_mounts() {
|
||||||
|
let (_dir, h) = start().await;
|
||||||
|
for base in ["/state/inbox", "/run/inbox"] {
|
||||||
|
let mut rx = h.subscribe();
|
||||||
|
let task = watch(&h, base);
|
||||||
|
let armed = event(&h, &mut rx, "armed").await;
|
||||||
|
assert_eq!(armed["entries"], json!([]));
|
||||||
|
let staging = format!("{base}-staging");
|
||||||
|
mutate(&h, "mkdir", &staging).await;
|
||||||
|
call(
|
||||||
|
&h,
|
||||||
|
"probe.mutate",
|
||||||
|
json!({"op":"rename","path":staging,"to":base}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
event(&h, &mut rx, "changed").await;
|
||||||
|
mutate(&h, "atomic", &format!("{base}/published")).await;
|
||||||
|
let mut scan = event(&h, &mut rx, "changed").await;
|
||||||
|
while !scan["entries"]
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.contains(&json!("published"))
|
||||||
|
{
|
||||||
|
scan = event(&h, &mut rx, "changed").await;
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
call(
|
||||||
|
&h,
|
||||||
|
"probe.read",
|
||||||
|
json!({"path":format!("{base}/published")})
|
||||||
|
)
|
||||||
|
.await,
|
||||||
|
"complete"
|
||||||
|
);
|
||||||
|
while rx.try_recv().is_ok() {}
|
||||||
|
call(
|
||||||
|
&h,
|
||||||
|
"probe.mutate",
|
||||||
|
json!({"op":"rename","path":base,"to":format!("{base}-moved")}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let mut scan = event(&h, &mut rx, "changed").await;
|
||||||
|
while scan["entries"] != json!([]) {
|
||||||
|
scan = event(&h, &mut rx, "changed").await;
|
||||||
|
}
|
||||||
|
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
|
||||||
|
assert!(task.await.unwrap().is_err());
|
||||||
|
}
|
||||||
|
// /self watches bind to this caller, and mutations are visible via /proc/<pid>.
|
||||||
|
let mut rx = h.subscribe();
|
||||||
|
let task = watch(&h, "/self");
|
||||||
|
let armed = event(&h, &mut rx, "armed").await;
|
||||||
|
assert!(
|
||||||
|
armed["entries"]
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.contains(&json!("owned"))
|
||||||
|
);
|
||||||
|
event(&h, &mut rx, "changed").await;
|
||||||
|
assert_eq!(
|
||||||
|
call(
|
||||||
|
&h,
|
||||||
|
"probe.read",
|
||||||
|
json!({"path":format!("/proc/{}/owned", armed["pid"].as_str().unwrap())})
|
||||||
|
)
|
||||||
|
.await,
|
||||||
|
"visible via proc"
|
||||||
|
);
|
||||||
|
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
|
||||||
|
assert!(task.await.unwrap().is_err());
|
||||||
|
h.stop().await;
|
||||||
|
}
|
||||||
@@ -84,12 +84,52 @@ fn main() {
|
|||||||
.unwrap();
|
.unwrap();
|
||||||
json!({"retired":version})
|
json!({"retired":version})
|
||||||
}
|
}
|
||||||
|
"work" => {
|
||||||
|
// Keep an actual request unanswered through grace; notices do not drain it.
|
||||||
|
send(
|
||||||
|
json!({"jsonrpc":"2.0","method":"core.notify","params":{"name":"ready","params":{}}}),
|
||||||
|
);
|
||||||
|
for _ in lines {}
|
||||||
|
return;
|
||||||
|
}
|
||||||
"serve" | "hold" => {
|
"serve" | "hold" => {
|
||||||
|
// One genuine process failure, followed by a healthy service, without
|
||||||
|
// timers or liveness polling in either the fixture or supervisor.
|
||||||
|
let crash = request["params"]["crash_once"].as_str();
|
||||||
|
let first = crash.is_some()
|
||||||
|
&& std::fs::OpenOptions::new()
|
||||||
|
.write(true)
|
||||||
|
.create_new(true)
|
||||||
|
.open(format!("/state/{plugin}/crashed-once"))
|
||||||
|
.is_ok();
|
||||||
|
if first && crash == Some("before") {
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
send(json!({"jsonrpc":"2.0","id":request["id"],"result":{"version":version}}));
|
send(json!({"jsonrpc":"2.0","id":request["id"],"result":{"version":version}}));
|
||||||
|
if first && crash == Some("after") {
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
if request["method"] == "serve" {
|
if request["method"] == "serve" {
|
||||||
std::fs::write(format!("/state/{plugin}/service"), version).unwrap();
|
std::fs::write(format!("/state/{plugin}/service"), version).unwrap();
|
||||||
}
|
}
|
||||||
for _ in lines {}
|
for line in lines {
|
||||||
|
let Ok(message) = serde_json::from_str::<Value>(&line.unwrap()) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
if message["method"] == "rpc.superseded" {
|
||||||
|
std::fs::write(
|
||||||
|
format!(
|
||||||
|
"/state/{plugin}/superseded-{}",
|
||||||
|
std::env::var("SAUSAGE_PID").unwrap()
|
||||||
|
),
|
||||||
|
message.to_string(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
if request["params"]["exit_on_superseded"] == true {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
std::fs::write(format!("/state/{plugin}/ended-{version}"), "ended").unwrap();
|
std::fs::write(format!("/state/{plugin}/ended-{version}"), "ended").unwrap();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -83,10 +83,16 @@ fn unknown_id() {
|
|||||||
}
|
}
|
||||||
fn wait_eof(lines: &mut Lines) {
|
fn wait_eof(lines: &mut Lines) {
|
||||||
std::fs::write("/state/waiting", "yes").unwrap();
|
std::fs::write("/state/waiting", "yes").unwrap();
|
||||||
|
send(json!({"jsonrpc":"2.0","method":"core.notify","params":{
|
||||||
|
"name":"waiting-eof", "params":{"pid":std::env::var("SAUSAGE_PID").unwrap()}
|
||||||
|
}}));
|
||||||
for _ in lines {}
|
for _ in lines {}
|
||||||
std::fs::write("/state/stopped", "yes").unwrap();
|
std::fs::write("/state/stopped", "yes").unwrap();
|
||||||
}
|
}
|
||||||
fn busy() -> Value {
|
fn busy() -> Value {
|
||||||
|
send(json!({"jsonrpc":"2.0", "method":"core.notify", "params":{
|
||||||
|
"name":"busy-ready", "params":{}
|
||||||
|
}}));
|
||||||
loop {
|
loop {
|
||||||
std::hint::spin_loop();
|
std::hint::spin_loop();
|
||||||
}
|
}
|
||||||
@@ -376,6 +382,7 @@ fn dispatch(request: &Value, lines: &mut Lines) -> Option<Value> {
|
|||||||
let params = &request["params"];
|
let params = &request["params"];
|
||||||
let result = match request["method"].as_str().unwrap() {
|
let result = match request["method"].as_str().unwrap() {
|
||||||
"batch" => batch(lines),
|
"batch" => batch(lines),
|
||||||
|
"trap" => std::arch::wasm32::unreachable(),
|
||||||
"nested_environment" => nested_environment(lines),
|
"nested_environment" => nested_environment(lines),
|
||||||
"forget" => forget(lines),
|
"forget" => forget(lines),
|
||||||
"forgotten" => {
|
"forgotten" => {
|
||||||
@@ -447,13 +454,33 @@ fn dispatch(request: &Value, lines: &mut Lines) -> Option<Value> {
|
|||||||
burst(request);
|
burst(request);
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
|
"restart" => {
|
||||||
|
assert_eq!(
|
||||||
|
lock(&json!({"path":"/state/restart-stop"}), lines),
|
||||||
|
json!({"held":true})
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
outgoing_call(&json!({"method":"core.restart", "params":{}}), lines),
|
||||||
|
json!({})
|
||||||
|
);
|
||||||
|
std::fs::write("/state/restart-answer", "received").unwrap();
|
||||||
|
serve(request, lines);
|
||||||
|
std::fs::write("/state/restart-stop", "persisted after EOF").unwrap();
|
||||||
|
eprintln!("restart stop hook finished");
|
||||||
|
return None;
|
||||||
|
}
|
||||||
"serve" => {
|
"serve" => {
|
||||||
serve(request, lines);
|
serve(request, lines);
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
"network" => network(params),
|
"network" => network(params),
|
||||||
|
"mutate" => mutate(params),
|
||||||
"read" => read_file(params),
|
"read" => read_file(params),
|
||||||
"write" => write_file(params),
|
"write" => write_file(params),
|
||||||
|
"raw_call" => {
|
||||||
|
send(params.clone());
|
||||||
|
receive(lines)
|
||||||
|
}
|
||||||
"call" => outgoing_call(params, lines),
|
"call" => outgoing_call(params, lines),
|
||||||
"lock" => lock(params, lines),
|
"lock" => lock(params, lines),
|
||||||
_ => {
|
_ => {
|
||||||
@@ -498,3 +525,68 @@ fn main() {
|
|||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Separate mutation calls let the host await a scan before each subsequent change.
|
||||||
|
fn mutate(p: &Value) -> Value {
|
||||||
|
use wasip2::filesystem::{
|
||||||
|
preopens,
|
||||||
|
types::{DescriptorFlags as D, OpenFlags as O, PathFlags as P},
|
||||||
|
};
|
||||||
|
let path = p["path"].as_str().unwrap();
|
||||||
|
match p["op"].as_str().unwrap() {
|
||||||
|
"mkdir" => std::fs::create_dir_all(path).unwrap(),
|
||||||
|
"rename" => std::fs::rename(path, p["to"].as_str().unwrap()).unwrap(),
|
||||||
|
"delete" => std::fs::remove_file(path).unwrap(),
|
||||||
|
"rmdir" => std::fs::remove_dir(path).unwrap(),
|
||||||
|
"atomic" => {
|
||||||
|
let temp = format!("{path}.tmp");
|
||||||
|
std::fs::write(&temp, "complete").unwrap();
|
||||||
|
std::fs::rename(temp, path).unwrap();
|
||||||
|
}
|
||||||
|
"burst" => {
|
||||||
|
for i in 0..5 {
|
||||||
|
std::fs::write(format!("{path}/{i}"), "complete").unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
op => {
|
||||||
|
let root = preopens::get_directories().remove(0).0;
|
||||||
|
let file = root
|
||||||
|
.open_at(
|
||||||
|
P::empty(),
|
||||||
|
path.trim_start_matches('/'),
|
||||||
|
O::empty(),
|
||||||
|
D::READ | D::WRITE,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
match op {
|
||||||
|
"descriptor" => {
|
||||||
|
file.write(b"descriptor", 0).unwrap();
|
||||||
|
}
|
||||||
|
"stream" => {
|
||||||
|
file.write_via_stream(0)
|
||||||
|
.unwrap()
|
||||||
|
.blocking_write_and_flush(b"stream")
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
"append" => {
|
||||||
|
file.append_via_stream()
|
||||||
|
.unwrap()
|
||||||
|
.blocking_write_and_flush(b"append")
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
"size" => file.set_size(0).unwrap(),
|
||||||
|
"truncate" => {
|
||||||
|
root.open_at(
|
||||||
|
P::empty(),
|
||||||
|
path.trim_start_matches('/'),
|
||||||
|
O::TRUNCATE,
|
||||||
|
D::WRITE,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
_ => panic!("unknown mutation"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
json!({})
|
||||||
|
}
|
||||||
|
|||||||
@@ -80,7 +80,7 @@
|
|||||||
# `nix run .#loop -- <world>` runs the newest `sausage run <world>`
|
# `nix run .#loop -- <world>` runs the newest `sausage run <world>`
|
||||||
# (without a folder: bare `sausage`, i.e. ./world, created if needed)
|
# (without a folder: bare `sausage`, i.e. ./world, created if needed)
|
||||||
# from the public repo, again and again: after a crash or a call of
|
# from the public repo, again and again: after a crash or a call of
|
||||||
# `core.exit` it refreshes the flake and starts the host again, so
|
# `core.restart` (exit code 75) it refreshes the flake and starts the host again, so
|
||||||
# the world updates itself. `quit` (exit code 0) ends the loop.
|
# the world updates itself. `quit` (exit code 0) ends the loop.
|
||||||
# SAUSAGE_FLAKE overrides where it fetches from (any flake URL,
|
# SAUSAGE_FLAKE overrides where it fetches from (any flake URL,
|
||||||
# e.g. a fork or a local checkout).
|
# e.g. a fork or a local checkout).
|
||||||
@@ -90,7 +90,7 @@
|
|||||||
# No folder: bare `sausage` (creates ./world first if needed).
|
# No folder: bare `sausage` (creates ./world first if needed).
|
||||||
if [ $# -gt 0 ]; then set -- run "$@"; fi
|
if [ $# -gt 0 ]; then set -- run "$@"; fi
|
||||||
while true; do
|
while true; do
|
||||||
nix run --refresh "$url#default" -- "$@"
|
SAUSAGE_LOOP=1 nix run --refresh "$url#default" -- "$@"
|
||||||
code=$?
|
code=$?
|
||||||
if [ "$code" -eq 0 ]; then exit 0; fi
|
if [ "$code" -eq 0 ]; then exit 0; fi
|
||||||
echo "sausage-loop: sausage exited with $code, starting the newest version in 2 s …" >&2
|
echo "sausage-loop: sausage exited with $code, starting the newest version in 2 s …" >&2
|
||||||
|
|||||||
@@ -48,8 +48,9 @@ pub fn client_rs(name: &str, doc: &Value) -> Result<String, String> {
|
|||||||
let mut result = vec![];
|
let mut result = vec![];
|
||||||
for method in methods.as_array().into_iter().flatten() {
|
for method in methods.as_array().into_iter().flatten() {
|
||||||
let wire = method["name"].as_str().ok_or("method name missing")?;
|
let wire = method["name"].as_str().ok_or("method name missing")?;
|
||||||
let ident = syn::parse_str::<syn::Ident>(wire)
|
let rust_name = wire.replace('.', "_");
|
||||||
.or_else(|_| syn::parse_str::<syn::Ident>(&format!("r#{wire}")))
|
let ident = syn::parse_str::<syn::Ident>(&rust_name)
|
||||||
|
.or_else(|_| syn::parse_str::<syn::Ident>(&format!("r#{rust_name}")))
|
||||||
.map_err(|e| e.to_string())?;
|
.map_err(|e| e.to_string())?;
|
||||||
let mut rs = method["result"]["schema"].clone();
|
let mut rs = method["result"]["schema"].clone();
|
||||||
local_refs(&mut rs);
|
local_refs(&mut rs);
|
||||||
|
|||||||
+46
-1
@@ -204,7 +204,7 @@ fn supported(sender: &Value, document: &Value, seen: &mut HashSet<usize>) -> Res
|
|||||||
supported(child, document, seen)?;
|
supported(child, document, seen)?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
"additionalProperties" if value.is_boolean() => (),
|
"additionalProperties" => supported(value, document, seen)?,
|
||||||
"enum"
|
"enum"
|
||||||
if value
|
if value
|
||||||
.as_array()
|
.as_array()
|
||||||
@@ -447,6 +447,15 @@ fn compare_variants(
|
|||||||
{
|
{
|
||||||
let error = if let Some(other) = receiver["properties"].get(key) {
|
let error = if let Some(other) = receiver["properties"].get(key) {
|
||||||
subset(child, other, sender_doc, receiver_doc, active).err()
|
subset(child, other, sender_doc, receiver_doc, active).err()
|
||||||
|
} else if receiver["additionalProperties"].is_object() {
|
||||||
|
subset(
|
||||||
|
child,
|
||||||
|
&receiver["additionalProperties"],
|
||||||
|
sender_doc,
|
||||||
|
receiver_doc,
|
||||||
|
active,
|
||||||
|
)
|
||||||
|
.err()
|
||||||
} else if receiver["additionalProperties"] == false {
|
} else if receiver["additionalProperties"] == false {
|
||||||
Some((
|
Some((
|
||||||
format!(
|
format!(
|
||||||
@@ -471,6 +480,42 @@ fn compare_variants(
|
|||||||
));
|
));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// A typed dictionary can send arbitrary keys, unlike a struct's
|
||||||
|
// closed set of output fields. Check both receiver named fields
|
||||||
|
// and its catch-all value contract.
|
||||||
|
if let Some(extra) = sender.get("additionalProperties").filter(|v| v.is_object()) {
|
||||||
|
let mut key = "extra".to_string();
|
||||||
|
while sender["properties"].get(&key).is_some()
|
||||||
|
|| receiver["properties"].get(&key).is_some()
|
||||||
|
{
|
||||||
|
key.push('!');
|
||||||
|
}
|
||||||
|
let catch_all = receiver
|
||||||
|
.get("additionalProperties")
|
||||||
|
.unwrap_or(&Value::Bool(true));
|
||||||
|
let targets = receiver["properties"]
|
||||||
|
.as_object()
|
||||||
|
.into_iter()
|
||||||
|
.flatten()
|
||||||
|
.filter(|(key, _)| sender["properties"].get(*key).is_none())
|
||||||
|
.chain(std::iter::once((&key, catch_all)));
|
||||||
|
for (key, target) in targets {
|
||||||
|
if let Err((message, example)) =
|
||||||
|
subset(extra, target, sender_doc, receiver_doc, active)
|
||||||
|
{
|
||||||
|
let mut object = sample(sender, sender_doc, &mut HashSet::new());
|
||||||
|
if let (Some(Value::Object(object)), Some(value)) = (&mut object, example) {
|
||||||
|
object.insert(key.clone(), value);
|
||||||
|
} else {
|
||||||
|
object = None;
|
||||||
|
}
|
||||||
|
return Err((
|
||||||
|
format!("the dictionary field {key:?} is incompatible: {message}"),
|
||||||
|
object,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
"array" => {
|
"array" => {
|
||||||
if let Err((message, example)) = subset(
|
if let Err((message, example)) = subset(
|
||||||
|
|||||||
@@ -0,0 +1,57 @@
|
|||||||
|
//! Typed dictionary results retain their value contracts in both directions.
|
||||||
|
use sausage_interface::compat::check;
|
||||||
|
use serde_json::{Value, json};
|
||||||
|
|
||||||
|
fn document(schema: Value) -> Value {
|
||||||
|
json!({"methods":[{"name":"values","params":[],"result":{"schema":schema}}]})
|
||||||
|
}
|
||||||
|
fn map(value: Value) -> Value {
|
||||||
|
json!({"type":"object","additionalProperties":value})
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn dictionary_values_are_covariant() {
|
||||||
|
let strings = document(map(json!({"type":"string"})));
|
||||||
|
let numbers = document(map(json!({"type":"integer"})));
|
||||||
|
let short = document(map(json!({"type":"string","enum":["ok"]})));
|
||||||
|
assert!(check(&strings, &strings).is_empty());
|
||||||
|
assert!(check(&strings, &short).is_empty());
|
||||||
|
assert!(!check(&short, &strings).is_empty());
|
||||||
|
let errors = check(&strings, &numbers);
|
||||||
|
assert_eq!(errors.len(), 1);
|
||||||
|
assert_eq!(errors[0].counterexample, Some(json!({"extra":0})));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn dictionaries_and_named_fields_check_each_other() {
|
||||||
|
let strings = document(map(json!({"type":"string"})));
|
||||||
|
let named_string = document(json!({"type":"object","properties":{"name":{"type":"string"}}}));
|
||||||
|
let named_number = document(json!({"type":"object","properties":{"name":{"type":"integer"}}}));
|
||||||
|
assert!(check(&strings, &named_string).is_empty());
|
||||||
|
assert!(check(&named_string, &strings).is_empty());
|
||||||
|
assert!(!check(&strings, &named_number).is_empty());
|
||||||
|
assert!(!check(&named_number, &strings).is_empty());
|
||||||
|
let closed = document(json!({"type":"object","additionalProperties":false}));
|
||||||
|
assert!(!check(&closed, &strings).is_empty());
|
||||||
|
assert!(check(&strings, &closed).is_empty());
|
||||||
|
let required = document(
|
||||||
|
json!({"type":"object","properties":{"name":{"type":"string"}},"required":["name"]}),
|
||||||
|
);
|
||||||
|
assert!(!check(&required, &strings).is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn dictionary_references_are_checked_not_ignored() {
|
||||||
|
let mut expected = document(map(json!({"$ref":"#/components/schemas/Value"})));
|
||||||
|
expected["components"]["schemas"]["Value"] = json!({"type":"string"});
|
||||||
|
assert!(check(&expected, &expected).is_empty());
|
||||||
|
let mut actual = expected.clone();
|
||||||
|
actual["components"]["schemas"]["Value"] = json!({"type":"boolean"});
|
||||||
|
assert!(!check(&expected, &actual).is_empty());
|
||||||
|
actual["components"]["schemas"]["Value"] = json!({"pattern":"x"});
|
||||||
|
assert!(
|
||||||
|
check(&expected, &actual)[0]
|
||||||
|
.message
|
||||||
|
.contains("unsupported schema construct pattern")
|
||||||
|
);
|
||||||
|
}
|
||||||
+27
-7
@@ -153,6 +153,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
|
|||||||
let mut stop = quote!();
|
let mut stop = quote!();
|
||||||
let mut has_stop = false;
|
let mut has_stop = false;
|
||||||
let mut has_retired = false;
|
let mut has_retired = false;
|
||||||
|
let mut has_superseded = false;
|
||||||
for member in &mut item.items {
|
for member in &mut item.items {
|
||||||
let ImplItem::Fn(f) = member else { continue };
|
let ImplItem::Fn(f) = member else { continue };
|
||||||
let retired = f.attrs.iter().any(|a| {
|
let retired = f.attrs.iter().any(|a| {
|
||||||
@@ -161,6 +162,12 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
|
|||||||
.last()
|
.last()
|
||||||
.is_some_and(|s| s.ident == "on_retired")
|
.is_some_and(|s| s.ident == "on_retired")
|
||||||
});
|
});
|
||||||
|
let superseded = f.attrs.iter().any(|a| {
|
||||||
|
a.path()
|
||||||
|
.segments
|
||||||
|
.last()
|
||||||
|
.is_some_and(|s| s.ident == "on_superseded")
|
||||||
|
});
|
||||||
let stopping = f.attrs.iter().any(|a| {
|
let stopping = f.attrs.iter().any(|a| {
|
||||||
a.path()
|
a.path()
|
||||||
.segments
|
.segments
|
||||||
@@ -179,11 +186,16 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
|
|||||||
"only one on_retired method is allowed",
|
"only one on_retired method is allowed",
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
if superseded && std::mem::replace(&mut has_superseded, true) {
|
||||||
|
return Err(syn::Error::new(
|
||||||
|
f.span(),
|
||||||
|
"only one on_superseded method is allowed",
|
||||||
|
));
|
||||||
|
}
|
||||||
f.attrs.retain(|a| {
|
f.attrs.retain(|a| {
|
||||||
!a.path()
|
!a.path().segments.last().is_some_and(|s| {
|
||||||
.segments
|
s.ident == "on_retired" || s.ident == "on_stop" || s.ident == "on_superseded"
|
||||||
.last()
|
})
|
||||||
.is_some_and(|s| s.ident == "on_retired" || s.ident == "on_stop")
|
|
||||||
});
|
});
|
||||||
documented(&f.attrs, f.span())?;
|
documented(&f.attrs, f.span())?;
|
||||||
if f.sig.inputs.is_empty()
|
if f.sig.inputs.is_empty()
|
||||||
@@ -224,7 +236,9 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
|
|||||||
stop = quote!(fn stop(&self)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=()> + '_>> {Box::pin(async move {let _=self.#name() #await_;})});
|
stop = quote!(fn stop(&self)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=()> + '_>> {Box::pin(async move {let _=self.#name() #await_;})});
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let wire = if retired {
|
let wire = if superseded {
|
||||||
|
"rpc.superseded".to_string()
|
||||||
|
} else if retired {
|
||||||
"rpc.retired".to_string()
|
"rpc.retired".to_string()
|
||||||
} else {
|
} else {
|
||||||
name.unraw().to_string()
|
name.unraw().to_string()
|
||||||
@@ -253,7 +267,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
|
|||||||
};
|
};
|
||||||
let doc = docs(&f.attrs);
|
let doc = docs(&f.attrs);
|
||||||
arms.push(quote!(#wire => { fn params_type<T: ::sausage_plugin::Params>() {} params_type::<#p>(); let result=self.#name(#(#args),*) #await_ ?; Ok(::sausage_plugin::serde_json::to_value(result)?) }));
|
arms.push(quote!(#wire => { fn params_type<T: ::sausage_plugin::Params>() {} params_type::<#p>(); let result=self.#name(#(#args),*) #await_ ?; Ok(::sausage_plugin::serde_json::to_value(result)?) }));
|
||||||
if !retired {
|
if !retired && !superseded {
|
||||||
entries.push(quote!(::sausage_plugin::method::<#p,#output>(&mut doc,#wire,#doc);));
|
entries.push(quote!(::sausage_plugin::method::<#p,#output>(&mut doc,#wire,#doc);));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -270,7 +284,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
|
|||||||
fn methods_document()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace ::sausage_plugin::finish_document(doc)}
|
fn methods_document()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace ::sausage_plugin::finish_document(doc)}
|
||||||
fn describe()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace #(doc.endpoint::<#types>(#names);)* ::sausage_plugin::finish_document(doc)}
|
fn describe()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace #(doc.endpoint::<#types>(#names);)* ::sausage_plugin::finish_document(doc)}
|
||||||
fn discover(&self)->::sausage_plugin::serde_json::Value {<Self as ::sausage_plugin::Interface>::describe()}
|
fn discover(&self)->::sausage_plugin::serde_json::Value {<Self as ::sausage_plugin::Interface>::describe()}
|
||||||
fn dispatch<'a>(&'a self, method:&'a str, params: ::sausage_plugin::serde_json::Value,ctx:&'a ::sausage_plugin::Ctx)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=::sausage_plugin::Result<::sausage_plugin::serde_json::Value>>+'a>> {Box::pin(async move {if !params.is_object() {return Err(::sausage_plugin::Error::new(-32602,"params must be an object"));} match method {#(#arms,)* _=>Err(::sausage_plugin::Error::new(-32601,"method not found"))}})}
|
fn dispatch<'a>(&'a self, method:&'a str, params: ::sausage_plugin::serde_json::Value,ctx:&'a ::sausage_plugin::Ctx)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=::sausage_plugin::Result<::sausage_plugin::serde_json::Value>>+'a>> {Box::pin(async move {if !params.is_object() {return Err(::sausage_plugin::Error::params_not_object(¶ms));} match method {#(#arms,)* _=>Err(::sausage_plugin::Error::new(-32601,"method not found"))}})}
|
||||||
#stop
|
#stop
|
||||||
}})
|
}})
|
||||||
})();
|
})();
|
||||||
@@ -328,6 +342,12 @@ pub fn import(input: TokenStream) -> TokenStream {
|
|||||||
pub fn on_retired(_: TokenStream, input: TokenStream) -> TokenStream {
|
pub fn on_retired(_: TokenStream, input: TokenStream) -> TokenStream {
|
||||||
input
|
input
|
||||||
}
|
}
|
||||||
|
/// Mark the method that handles `rpc.superseded` at the start of upgrade grace.
|
||||||
|
/// Calls still work here. Use `sausage_plugin::drain` to finish active work and exit.
|
||||||
|
#[proc_macro_attribute]
|
||||||
|
pub fn on_superseded(_: TokenStream, input: TokenStream) -> TokenStream {
|
||||||
|
input
|
||||||
|
}
|
||||||
/// Mark the hook that runs at stdin EOF without a live call transport.
|
/// Mark the hook that runs at stdin EOF without a live call transport.
|
||||||
#[proc_macro_attribute]
|
#[proc_macro_attribute]
|
||||||
pub fn on_stop(_: TokenStream, input: TokenStream) -> TokenStream {
|
pub fn on_stop(_: TokenStream, input: TokenStream) -> TokenStream {
|
||||||
|
|||||||
+184
-2
@@ -1,5 +1,19 @@
|
|||||||
//! Outgoing core calls and notification callbacks.
|
//! Outgoing core calls and notification callbacks.
|
||||||
use super::*;
|
use super::*;
|
||||||
|
/// Removes a dropped call's local routing state, without cancelling host work.
|
||||||
|
struct PendingCall(u64);
|
||||||
|
impl Drop for PendingCall {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let pending = STATE.with(|s| s.borrow_mut().pending.remove(&self.0));
|
||||||
|
if let Some(mut pending) = pending {
|
||||||
|
// A lock may still be granted: release it when its late response arrives.
|
||||||
|
if let Some(lease) = pending.unlock.take() {
|
||||||
|
STATE.with(|s| s.borrow_mut().abandoned_locks.insert(self.0, lease));
|
||||||
|
}
|
||||||
|
// Drop user-owned callback captures outside the STATE borrow.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
/// An outgoing request builder.
|
/// An outgoing request builder.
|
||||||
#[must_use = "await the call to send it"]
|
#[must_use = "await the call to send it"]
|
||||||
pub struct Call {
|
pub struct Call {
|
||||||
@@ -26,6 +40,21 @@ impl Call {
|
|||||||
self
|
self
|
||||||
}
|
}
|
||||||
/// Receive progress notifications before the result arrives.
|
/// Receive progress notifications before the result arrives.
|
||||||
|
/// Dropping the call future removes its local pending entry and callback;
|
||||||
|
/// later notifications and responses are ignored. It does not cancel host work.
|
||||||
|
///
|
||||||
|
/// For `core.watch`, poll the call future alongside work in the handler, then
|
||||||
|
/// drop it when the handler returns. Do not put it in `background()` just to
|
||||||
|
/// keep the watch alive: background tasks keep the process alive, while the
|
||||||
|
/// watch only ends when the process ends. Dropping the future does not cancel
|
||||||
|
/// the host watch, but it lets an otherwise idle process exit normally. A
|
||||||
|
/// long-lived service should keep one watch per purpose for its lifetime,
|
||||||
|
/// rather than repeatedly opening and dropping watches on the host.
|
||||||
|
///
|
||||||
|
/// Scan only after `armed`, and rescan on `changed`. Events may coalesce, but
|
||||||
|
/// are not rate-limited: watch the narrowest useful path to avoid rescanning
|
||||||
|
/// for unrelated writes (for example, watch a mailbox, not all session data).
|
||||||
|
/// Do not use endpoint connect retries to wait for file publication.
|
||||||
pub fn on_notify(mut self, f: impl FnMut(&str, Value) + 'static) -> Self {
|
pub fn on_notify(mut self, f: impl FnMut(&str, Value) + 'static) -> Self {
|
||||||
self.notify = Some(Box::new(f));
|
self.notify = Some(Box::new(f));
|
||||||
self
|
self
|
||||||
@@ -37,7 +66,7 @@ impl IntoFuture for Call {
|
|||||||
fn into_future(self) -> Self::IntoFuture {
|
fn into_future(self) -> Self::IntoFuture {
|
||||||
Box::pin(async move {
|
Box::pin(async move {
|
||||||
if !self.params.is_object() {
|
if !self.params.is_object() {
|
||||||
return Err(Error::new(-32602, "params must be an object"));
|
return Err(Error::params_not_object(&self.params));
|
||||||
}
|
}
|
||||||
let (tx, rx) = oneshot::channel();
|
let (tx, rx) = oneshot::channel();
|
||||||
let id = STATE.with(|s| {
|
let id = STATE.with(|s| {
|
||||||
@@ -60,6 +89,7 @@ impl IntoFuture for Call {
|
|||||||
);
|
);
|
||||||
Ok(id)
|
Ok(id)
|
||||||
})?;
|
})?;
|
||||||
|
let _pending = PendingCall(id);
|
||||||
let (method, params) = if self.detach {
|
let (method, params) = if self.detach {
|
||||||
(
|
(
|
||||||
"core.call".to_string(),
|
"core.call".to_string(),
|
||||||
@@ -69,7 +99,8 @@ impl IntoFuture for Call {
|
|||||||
(self.method, self.params)
|
(self.method, self.params)
|
||||||
};
|
};
|
||||||
if let Err(e) = send(json!({"jsonrpc":"2.0","id":id,"method":method,"params":params})) {
|
if let Err(e) = send(json!({"jsonrpc":"2.0","id":id,"method":method,"params":params})) {
|
||||||
STATE.with(|s| s.borrow_mut().pending.remove(&id));
|
let pending = STATE.with(|s| s.borrow_mut().pending.remove(&id));
|
||||||
|
drop(pending);
|
||||||
return Err(e);
|
return Err(e);
|
||||||
}
|
}
|
||||||
rx.await
|
rx.await
|
||||||
@@ -77,3 +108,154 @@ impl IntoFuture for Call {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::{STATE, State, block_on, call, clear_pending, lock, receive};
|
||||||
|
use crate::Result;
|
||||||
|
use futures::{FutureExt, future::LocalBoxFuture};
|
||||||
|
use serde_json::{Value, json};
|
||||||
|
use std::{cell::RefCell, future::IntoFuture, rc::Rc, task::Context};
|
||||||
|
|
||||||
|
fn activate() {
|
||||||
|
STATE.with(|s| {
|
||||||
|
*s.borrow_mut() = State {
|
||||||
|
active: true,
|
||||||
|
..State::default()
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
fn start(
|
||||||
|
mut call: LocalBoxFuture<'static, Result<Value>>,
|
||||||
|
) -> (u64, LocalBoxFuture<'static, Result<Value>>) {
|
||||||
|
assert!(
|
||||||
|
call.as_mut()
|
||||||
|
.poll(&mut Context::from_waker(std::task::Waker::noop()))
|
||||||
|
.is_pending()
|
||||||
|
);
|
||||||
|
(STATE.with(|s| s.borrow().next), call)
|
||||||
|
}
|
||||||
|
fn notification(id: u64, name: &str) {
|
||||||
|
receive(json!({"method":"core.notify","params":{"call":id,"name":name,"params":{}}}));
|
||||||
|
}
|
||||||
|
fn response(id: u64, result: Value) {
|
||||||
|
receive(json!({"id":id,"result":result}));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn dropped_call_removes_pending_and_callback() {
|
||||||
|
activate();
|
||||||
|
let events = Rc::new(RefCell::new(Vec::new()));
|
||||||
|
let captured = events.clone();
|
||||||
|
let (id, future) = start(
|
||||||
|
call("core.watch", json!({"path":"/mailbox"}))
|
||||||
|
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
|
||||||
|
.into_future(),
|
||||||
|
);
|
||||||
|
notification(id, "armed");
|
||||||
|
assert_eq!(&*events.borrow(), &["armed"]);
|
||||||
|
assert_eq!(Rc::strong_count(&events), 2);
|
||||||
|
drop(future);
|
||||||
|
STATE.with(|s| {
|
||||||
|
assert!(s.borrow().pending.is_empty());
|
||||||
|
assert!(s.borrow().abandoned_locks.is_empty());
|
||||||
|
});
|
||||||
|
assert_eq!(Rc::strong_count(&events), 1);
|
||||||
|
notification(id, "changed");
|
||||||
|
response(id, Value::Null);
|
||||||
|
receive(json!({"id":id,"error":{"code":-32000,"message":"late error"}}));
|
||||||
|
assert_eq!(&*events.borrow(), &["armed"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn kept_watch_delivers_until_result() {
|
||||||
|
activate();
|
||||||
|
let events = Rc::new(RefCell::new(Vec::new()));
|
||||||
|
let captured = events.clone();
|
||||||
|
let (id, future) = start(
|
||||||
|
call("core.watch", json!({"path":"/mailbox"}))
|
||||||
|
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
|
||||||
|
.into_future(),
|
||||||
|
);
|
||||||
|
notification(id, "armed");
|
||||||
|
notification(id, "changed");
|
||||||
|
notification(id, "changed");
|
||||||
|
response(id, json!({"done":true}));
|
||||||
|
assert_eq!(block_on(future).unwrap(), json!({"done":true}));
|
||||||
|
notification(id, "late");
|
||||||
|
assert_eq!(&*events.borrow(), &["armed", "changed", "changed"]);
|
||||||
|
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
|
||||||
|
assert_eq!(Rc::strong_count(&events), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn callback_can_drop_its_own_call_without_being_restored() {
|
||||||
|
activate();
|
||||||
|
let holder = Rc::new(RefCell::new(None));
|
||||||
|
let captured = holder.clone();
|
||||||
|
let (id, future) = start(
|
||||||
|
call("core.watch", json!({"path":"/mailbox"}))
|
||||||
|
.on_notify(move |_, _| {
|
||||||
|
// Panics on a second invocation; the first must remove the entry.
|
||||||
|
drop(captured.borrow_mut().take().expect("live call"));
|
||||||
|
})
|
||||||
|
.into_future(),
|
||||||
|
);
|
||||||
|
*holder.borrow_mut() = Some(future);
|
||||||
|
notification(id, "armed");
|
||||||
|
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
|
||||||
|
notification(id, "changed");
|
||||||
|
response(id, Value::Null);
|
||||||
|
assert_eq!(Rc::strong_count(&holder), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn eof_cleanup_can_drop_callback_owned_futures() {
|
||||||
|
activate();
|
||||||
|
let holder = Rc::new(RefCell::new(None));
|
||||||
|
let weak = Rc::downgrade(&holder);
|
||||||
|
let captured = holder.clone();
|
||||||
|
let (_, future) = start(
|
||||||
|
call("core.watch", json!({"path":"/mailbox"}))
|
||||||
|
.on_notify(move |_, _| {
|
||||||
|
let _ = &captured;
|
||||||
|
})
|
||||||
|
.into_future(),
|
||||||
|
);
|
||||||
|
*holder.borrow_mut() = Some(future);
|
||||||
|
drop(holder);
|
||||||
|
STATE.with(|s| s.borrow_mut().stopping = true);
|
||||||
|
clear_pending();
|
||||||
|
assert!(weak.upgrade().is_none());
|
||||||
|
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn unpolled_call_never_registers() {
|
||||||
|
activate();
|
||||||
|
drop(call("core.watch", json!({"path":"/mailbox"})).into_future());
|
||||||
|
STATE.with(|s| {
|
||||||
|
assert!(s.borrow().pending.is_empty());
|
||||||
|
assert_eq!(s.borrow().next, 0);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn dropped_lock_keeps_only_late_grant_cleanup() {
|
||||||
|
activate();
|
||||||
|
let (id, future) = start(
|
||||||
|
lock("/late")
|
||||||
|
.into_future()
|
||||||
|
.map(|r| r.map(|_| Value::Null))
|
||||||
|
.boxed_local(),
|
||||||
|
);
|
||||||
|
drop(future);
|
||||||
|
STATE.with(|s| {
|
||||||
|
assert!(s.borrow().pending.is_empty());
|
||||||
|
assert_eq!(s.borrow().abandoned_locks.len(), 1);
|
||||||
|
});
|
||||||
|
notification(id, "ignored");
|
||||||
|
response(id, json!({"held":true}));
|
||||||
|
STATE.with(|s| assert!(s.borrow().abandoned_locks.is_empty()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -65,7 +65,7 @@ impl<T: serde::de::DeserializeOwned + 'static> IntoFuture for EndpointCall<T> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Publish a declared interface and keep its listener alive until stdin EOF.
|
/// Publish a declared interface and keep its listener alive until stdin EOF or [`drain`].
|
||||||
#[cfg(target_os = "wasi")]
|
#[cfg(target_os = "wasi")]
|
||||||
pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Result<()> {
|
pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Result<()> {
|
||||||
use wstd::iter::AsyncIterator;
|
use wstd::iter::AsyncIterator;
|
||||||
@@ -80,7 +80,7 @@ pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Resul
|
|||||||
while let Some(Ok(stream)) = incoming.next().await {
|
while let Some(Ok(stream)) = incoming.next().await {
|
||||||
let value = value.clone();
|
let value = value.clone();
|
||||||
let name = name.clone();
|
let name = name.clone();
|
||||||
crate::transport::endpoint_task(async move {
|
background(async move {
|
||||||
let _ = serve_connection(stream, value, name).await;
|
let _ = serve_connection(stream, value, name).await;
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -211,7 +211,11 @@ async fn serve_connection<T: Interface>(
|
|||||||
use futures::{FutureExt, StreamExt};
|
use futures::{FutureExt, StreamExt};
|
||||||
let (mut reader, mut writer) = stream.split();
|
let (mut reader, mut writer) = stream.split();
|
||||||
let mut lines = crate::lines::Lines::default();
|
let mut lines = crate::lines::Lines::default();
|
||||||
while let Some(request) = lines.rpc(&mut reader).await? {
|
while let Some(request) = crate::transport::until_drained(lines.rpc(&mut reader))
|
||||||
|
.await
|
||||||
|
.transpose()?
|
||||||
|
.flatten()
|
||||||
|
{
|
||||||
let method = request["method"].as_str().unwrap_or_default();
|
let method = request["method"].as_str().unwrap_or_default();
|
||||||
let (tx, mut rx) = futures::channel::mpsc::unbounded();
|
let (tx, mut rx) = futures::channel::mpsc::unbounded();
|
||||||
let ctx = Ctx {
|
let ctx = Ctx {
|
||||||
|
|||||||
@@ -15,6 +15,25 @@ pub struct Error {
|
|||||||
pub data: Option<Value>,
|
pub data: Option<Value>,
|
||||||
}
|
}
|
||||||
impl Error {
|
impl Error {
|
||||||
|
/// Build the object-params diagnostic used by calls and generated dispatch.
|
||||||
|
#[doc(hidden)]
|
||||||
|
pub fn params_not_object(params: &Value) -> Self {
|
||||||
|
let kind = match params {
|
||||||
|
Value::String(_) => "a string",
|
||||||
|
Value::Array(_) => "an array",
|
||||||
|
Value::Number(_) => "a number",
|
||||||
|
Value::Null => "null",
|
||||||
|
Value::Bool(_) => "a boolean",
|
||||||
|
Value::Object(_) => "an object",
|
||||||
|
};
|
||||||
|
Self::new(
|
||||||
|
-32602,
|
||||||
|
format!(
|
||||||
|
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
/// Construct an error without structured details.
|
/// Construct an error without structured details.
|
||||||
pub fn new(code: i64, message: impl Into<String>) -> Self {
|
pub fn new(code: i64, message: impl Into<String>) -> Self {
|
||||||
Self {
|
Self {
|
||||||
|
|||||||
+24
-3
@@ -356,13 +356,20 @@
|
|||||||
//!
|
//!
|
||||||
//! The interface check covers calls, not state. Old and new processes share files in `/state`.
|
//! The interface check covers calls, not state. Old and new processes share files in `/state`.
|
||||||
//! `core.revert` restores code, not state. Preserve compatibility or migrate the state.
|
//! `core.revert` restores code, not state. Preserve compatibility or migrate the state.
|
||||||
//! `#[rpc(grace_seconds = N)]` declares the grace period for old processes. The default is 60 seconds.
|
//! Core gives every old process 5 seconds of upgrade grace, then closes stdin and allows
|
||||||
|
//! a separate 5-second stop budget before killing it. Discovery-only processes are excluded.
|
||||||
|
//! `#[rpc(grace_seconds = N)]` still emits legacy `x-sausage-grace` metadata, but core ignores it.
|
||||||
//! New calls use the new version immediately. Old processes can still use old imports during their grace period.
|
//! New calls use the new version immediately. Old processes can still use old imports during their grace period.
|
||||||
|
//! [`macro@on_superseded`] handles the notice at grace start, with normal calls still available.
|
||||||
|
//! It is opt-in: without a hook the process continues as before until core stops it.
|
||||||
|
//! Call [`drain`] in this hook to close SDK endpoint listeners and idle connections,
|
||||||
|
//! let active handlers and background tasks finish, and exit without waiting for grace.
|
||||||
|
//! Custom listeners and long-lived background tasks must be ended by the plugin itself.
|
||||||
//! [`macro@on_retired`] handles `rpc.retired` after the last process of an old version ends.
|
//! [`macro@on_retired`] handles `rpc.retired` after the last process of an old version ends.
|
||||||
//! This hook is best effort. If the host stops first, the hook does not run.
|
//! This hook is best effort. If the host stops first, the hook does not run.
|
||||||
//! New processes must handle unmigrated state. Check a migration marker and migrate if it is absent.
|
//! New processes must handle unmigrated state. Check a migration marker and migrate if it is absent.
|
||||||
//! See section 3.7 of `/proc/core/PROTOCOL.md`.
|
//! See section 3.7 of `/proc/core/PROTOCOL.md`.
|
||||||
//! Declare at most one stop hook and one retirement hook per interface.
|
//! Declare at most one hook of each kind per interface.
|
||||||
//!
|
//!
|
||||||
//! ```no_run
|
//! ```no_run
|
||||||
//! use sausage_plugin::{rpc, schema, state_dir, write_atomic, Result};
|
//! use sausage_plugin::{rpc, schema, state_dir, write_atomic, Result};
|
||||||
@@ -374,8 +381,21 @@
|
|||||||
//! /// The SHA-256 of the previous WASM file.
|
//! /// The SHA-256 of the previous WASM file.
|
||||||
//! previous: String,
|
//! previous: String,
|
||||||
//! }
|
//! }
|
||||||
|
//! /// The version now receiving new calls.
|
||||||
|
//! #[schema]
|
||||||
|
//! struct SupersededParams {
|
||||||
|
//! /// The SHA-256 of the newly active WASM file.
|
||||||
|
//! current: String,
|
||||||
|
//! }
|
||||||
//! #[rpc]
|
//! #[rpc]
|
||||||
//! impl Migrator {
|
//! impl Migrator {
|
||||||
|
//! /// Finish current work and exit early during upgrade grace.
|
||||||
|
//! #[sausage_plugin::on_superseded]
|
||||||
|
//! fn superseded(&self, p: SupersededParams) -> Result<()> {
|
||||||
|
//! eprintln!("replaced by {}", p.current);
|
||||||
|
//! sausage_plugin::drain();
|
||||||
|
//! Ok(())
|
||||||
|
//! }
|
||||||
//! /// Record a retired version. Normal methods must also handle missing markers.
|
//! /// Record a retired version. Normal methods must also handle missing markers.
|
||||||
//! #[sausage_plugin::on_retired]
|
//! #[sausage_plugin::on_retired]
|
||||||
//! fn retired(&self, p: RetiredParams) -> Result<()> {
|
//! fn retired(&self, p: RetiredParams) -> Result<()> {
|
||||||
@@ -392,6 +412,7 @@ mod error;
|
|||||||
mod helpers;
|
mod helpers;
|
||||||
mod imports;
|
mod imports;
|
||||||
mod lines;
|
mod lines;
|
||||||
|
mod sqlite;
|
||||||
mod transport;
|
mod transport;
|
||||||
pub use discovery::*;
|
pub use discovery::*;
|
||||||
pub use endpoints::*;
|
pub use endpoints::*;
|
||||||
@@ -400,6 +421,6 @@ pub use helpers::*;
|
|||||||
pub use imports::*;
|
pub use imports::*;
|
||||||
#[doc(hidden)]
|
#[doc(hidden)]
|
||||||
pub use inventory;
|
pub use inventory;
|
||||||
pub use sausage_plugin_macros::{import, on_retired, on_stop, rpc, schema, test};
|
pub use sausage_plugin_macros::{import, on_retired, on_stop, on_superseded, rpc, schema, test};
|
||||||
pub use transport::*;
|
pub use transport::*;
|
||||||
pub use {schemars, serde, serde_json};
|
pub use {schemars, serde, serde_json};
|
||||||
|
|||||||
@@ -0,0 +1,43 @@
|
|||||||
|
//! Shared SQLite open policy, expanded against the plugin's own rusqlite version.
|
||||||
|
|
||||||
|
/// Open SQLite under an already-held Sausage host lock. The caller must depend
|
||||||
|
/// on `rusqlite` (bundled) and hold the SAME host lock for EVERY connection's
|
||||||
|
/// entire lifetime, including reads and old/new plugin versions. This macro does
|
||||||
|
/// not acquire the lock. Do not use it for the host journal or databases accessed
|
||||||
|
/// outside Sausage. Native builds retain OS locks; WASI selects `unix-none`, so
|
||||||
|
/// crash-leftover dotfile lock directories cannot block recovery.
|
||||||
|
///
|
||||||
|
/// Temporary tables/sorts stay in memory before any schema access. Writable opens
|
||||||
|
/// use DELETE journals; SQLite rolls back hot journals itself. Never delete a
|
||||||
|
/// leftover journal. Read-only opens cannot recover a hot journal: recover with
|
||||||
|
/// a writable open first.
|
||||||
|
///
|
||||||
|
/// The flags form preserves the caller's open flags. A macro avoids coupling the
|
||||||
|
/// SDK to a particular rusqlite/libsqlite3-sys version (the host and plugins use
|
||||||
|
/// different versions, which cannot coexist as dependencies due to `links`).
|
||||||
|
///
|
||||||
|
/// ```ignore
|
||||||
|
/// let _guard = sausage_plugin::lock("/state/example/db.lock").await?;
|
||||||
|
/// let c = sausage_plugin::sqlite_open!("/state/example/db.sqlite")?;
|
||||||
|
/// // Query here; c must drop before _guard.
|
||||||
|
/// ```
|
||||||
|
#[macro_export]
|
||||||
|
macro_rules! sqlite_open {
|
||||||
|
($path:expr) => {
|
||||||
|
$crate::sqlite_open!($path, ::rusqlite::OpenFlags::default())
|
||||||
|
};
|
||||||
|
($path:expr, $flags:expr) => {{
|
||||||
|
(|| -> ::rusqlite::Result<::rusqlite::Connection> {
|
||||||
|
let flags = $flags;
|
||||||
|
#[cfg(target_os = "wasi")]
|
||||||
|
let c = ::rusqlite::Connection::open_with_flags_and_vfs($path, flags, "unix-none")?;
|
||||||
|
#[cfg(not(target_os = "wasi"))]
|
||||||
|
let c = ::rusqlite::Connection::open_with_flags($path, flags)?;
|
||||||
|
c.pragma_update(None, "temp_store", "MEMORY")?;
|
||||||
|
if !flags.contains(::rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY) {
|
||||||
|
c.pragma_update(None, "journal_mode", "DELETE")?;
|
||||||
|
}
|
||||||
|
Ok(c)
|
||||||
|
})()
|
||||||
|
}};
|
||||||
|
}
|
||||||
+63
-8
@@ -96,8 +96,11 @@ struct State {
|
|||||||
active: bool,
|
active: bool,
|
||||||
endpoints: HashMap<std::any::TypeId, String>,
|
endpoints: HashMap<std::any::TypeId, String>,
|
||||||
stopping: bool,
|
stopping: bool,
|
||||||
|
draining: bool,
|
||||||
next: u64,
|
next: u64,
|
||||||
pending: HashMap<u64, Pending>,
|
pending: HashMap<u64, Pending>,
|
||||||
|
// Only leases, never callbacks or response senders, survive dropped lock acquisitions.
|
||||||
|
abandoned_locks: HashMap<u64, std::rc::Rc<Lease>>,
|
||||||
#[cfg(target_os = "wasi")]
|
#[cfg(target_os = "wasi")]
|
||||||
tasks: usize,
|
tasks: usize,
|
||||||
#[cfg(target_os = "wasi")]
|
#[cfg(target_os = "wasi")]
|
||||||
@@ -108,6 +111,18 @@ struct State {
|
|||||||
listeners: HashMap<u64, futures::future::AbortHandle>,
|
listeners: HashMap<u64, futures::future::AbortHandle>,
|
||||||
}
|
}
|
||||||
thread_local! { static STATE: RefCell<State> = RefCell::new(State::default()); }
|
thread_local! { static STATE: RefCell<State> = RefCell::new(State::default()); }
|
||||||
|
#[cfg(any(target_os = "wasi", test))]
|
||||||
|
fn clear_pending() {
|
||||||
|
let pending = STATE.with(|s| {
|
||||||
|
let mut s = s.borrow_mut();
|
||||||
|
(
|
||||||
|
std::mem::take(&mut s.pending),
|
||||||
|
std::mem::take(&mut s.abandoned_locks),
|
||||||
|
)
|
||||||
|
});
|
||||||
|
// Callback captures may own other call futures whose drop guards use STATE.
|
||||||
|
drop(pending);
|
||||||
|
}
|
||||||
pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
|
pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
|
||||||
STATE
|
STATE
|
||||||
.with(|s| {
|
.with(|s| {
|
||||||
@@ -125,17 +140,48 @@ pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
|
|||||||
}
|
}
|
||||||
#[cfg(target_os = "wasi")]
|
#[cfg(target_os = "wasi")]
|
||||||
pub(crate) fn endpoint_task(future: impl Future<Output = ()> + 'static) {
|
pub(crate) fn endpoint_task(future: impl Future<Output = ()> + 'static) {
|
||||||
|
background(async move {
|
||||||
|
let _ = until_drained(future).await;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
/// Cancel an accept/read wait when draining, but never an active handler.
|
||||||
|
#[cfg(target_os = "wasi")]
|
||||||
|
pub(crate) async fn until_drained<F: Future>(future: F) -> Option<F::Output> {
|
||||||
let (handle, registration) = futures::future::AbortHandle::new_pair();
|
let (handle, registration) = futures::future::AbortHandle::new_pair();
|
||||||
let id = STATE.with(|s| {
|
let id = STATE.with(|s| {
|
||||||
let mut s = s.borrow_mut();
|
let mut s = s.borrow_mut();
|
||||||
|
if s.draining {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
s.next += 1;
|
s.next += 1;
|
||||||
let id = s.next;
|
let id = s.next;
|
||||||
s.listeners.insert(id, handle);
|
s.listeners.insert(id, handle);
|
||||||
id
|
Some(id)
|
||||||
});
|
})?;
|
||||||
background(async move {
|
let result = futures::future::Abortable::new(future, registration)
|
||||||
let _ = futures::future::Abortable::new(future, registration).await;
|
.await
|
||||||
|
.ok();
|
||||||
STATE.with(|s| s.borrow_mut().listeners.remove(&id));
|
STATE.with(|s| s.borrow_mut().listeners.remove(&id));
|
||||||
|
result
|
||||||
|
}
|
||||||
|
/// Stop accepting SDK endpoint work, then exit when handlers and background tasks finish.
|
||||||
|
///
|
||||||
|
/// Calls and notifications still work. This does not run the EOF stop hook or cancel
|
||||||
|
/// background tasks: services with their own listeners/tasks must arrange for those
|
||||||
|
/// to finish too. Repeated calls are harmless. Core's upgrade grace remains the bound.
|
||||||
|
pub fn drain() {
|
||||||
|
STATE.with(|s| {
|
||||||
|
let mut s = s.borrow_mut();
|
||||||
|
s.draining = true;
|
||||||
|
#[cfg(target_os = "wasi")]
|
||||||
|
{
|
||||||
|
for (_, listener) in s.listeners.drain() {
|
||||||
|
listener.abort();
|
||||||
|
}
|
||||||
|
if let Some(w) = s.wake.take() {
|
||||||
|
w.wake();
|
||||||
|
}
|
||||||
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
#[cfg(target_os = "wasi")]
|
#[cfg(target_os = "wasi")]
|
||||||
@@ -192,16 +238,16 @@ pub fn test_run<F: Future>(f: F) -> F::Output {
|
|||||||
STATE.with(|s| {
|
STATE.with(|s| {
|
||||||
let mut s = s.borrow_mut();
|
let mut s = s.borrow_mut();
|
||||||
s.stopping = true;
|
s.stopping = true;
|
||||||
s.pending.clear();
|
|
||||||
});
|
});
|
||||||
|
clear_pending();
|
||||||
});
|
});
|
||||||
let result = f.await;
|
let result = f.await;
|
||||||
drop(reader);
|
drop(reader);
|
||||||
STATE.with(|s| {
|
STATE.with(|s| {
|
||||||
let mut s = s.borrow_mut();
|
let mut s = s.borrow_mut();
|
||||||
s.active = false;
|
s.active = false;
|
||||||
s.pending.clear();
|
|
||||||
});
|
});
|
||||||
|
clear_pending();
|
||||||
result
|
result
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -273,10 +319,19 @@ pub fn receive(value: Value) {
|
|||||||
.set(result.as_ref().is_ok_and(|v| v["held"] == true));
|
.set(result.as_ref().is_ok_and(|v| v["held"] == true));
|
||||||
}
|
}
|
||||||
let _ = p.tx.send(result);
|
let _ = p.tx.send(result);
|
||||||
|
} else {
|
||||||
|
let lease = value["id"]
|
||||||
|
.as_u64()
|
||||||
|
.and_then(|id| STATE.with(|s| s.borrow_mut().abandoned_locks.remove(&id)));
|
||||||
|
if let Some(lease) = lease {
|
||||||
|
lease
|
||||||
|
.held
|
||||||
|
.set(value.get("error").is_none() && value["result"]["held"] == true);
|
||||||
} else {
|
} else {
|
||||||
eprintln!("note: response with unknown ID {}", value["id"]);
|
eprintln!("note: response with unknown ID {}", value["id"]);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Serve the main interface until its work ends or stdin reaches EOF.
|
/// Serve the main interface until its work ends or stdin reaches EOF.
|
||||||
#[cfg(target_os = "wasi")]
|
#[cfg(target_os = "wasi")]
|
||||||
@@ -324,11 +379,11 @@ async fn serve<T: Interface>(value: T) {
|
|||||||
STATE.with(|s| {
|
STATE.with(|s| {
|
||||||
let mut s = s.borrow_mut();
|
let mut s = s.borrow_mut();
|
||||||
s.stopping = true;
|
s.stopping = true;
|
||||||
s.pending.clear();
|
|
||||||
for (_, listener) in s.listeners.drain() {
|
for (_, listener) in s.listeners.drain() {
|
||||||
listener.abort();
|
listener.abort();
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
clear_pending();
|
||||||
value.stop().await;
|
value.stop().await;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -379,8 +434,8 @@ async fn serve<T: Interface>(value: T) {
|
|||||||
STATE.with(|s| {
|
STATE.with(|s| {
|
||||||
let mut s = s.borrow_mut();
|
let mut s = s.borrow_mut();
|
||||||
s.active = false;
|
s.active = false;
|
||||||
s.pending.clear();
|
|
||||||
});
|
});
|
||||||
|
clear_pending();
|
||||||
let _ = std::io::stdout().flush();
|
let _ = std::io::stdout().flush();
|
||||||
std::process::exit(0);
|
std::process::exit(0);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,6 +2,12 @@
|
|||||||
use sausage_plugin::serde_json::{Value, json};
|
use sausage_plugin::serde_json::{Value, json};
|
||||||
use sausage_plugin::{Ctx, Result, call, lock, rpc, schema};
|
use sausage_plugin::{Ctx, Result, call, lock, rpc, schema};
|
||||||
struct Probe;
|
struct Probe;
|
||||||
|
/// A virtual path to observe.
|
||||||
|
#[schema]
|
||||||
|
struct WatchPath {
|
||||||
|
/// Directory or file, not necessarily present yet.
|
||||||
|
path: String,
|
||||||
|
}
|
||||||
/// The values to add.
|
/// The values to add.
|
||||||
#[schema]
|
#[schema]
|
||||||
struct Add {
|
struct Add {
|
||||||
@@ -16,6 +22,12 @@ struct Retired {
|
|||||||
/// The previous hash.
|
/// The previous hash.
|
||||||
previous: String,
|
previous: String,
|
||||||
}
|
}
|
||||||
|
/// The replacement version.
|
||||||
|
#[schema]
|
||||||
|
struct Superseded {
|
||||||
|
/// The current hash.
|
||||||
|
current: String,
|
||||||
|
}
|
||||||
/// A large payload.
|
/// A large payload.
|
||||||
#[schema]
|
#[schema]
|
||||||
struct Echo {
|
struct Echo {
|
||||||
@@ -24,6 +36,98 @@ struct Echo {
|
|||||||
}
|
}
|
||||||
#[rpc(grace_seconds = 12,endpoints(control=Control))]
|
#[rpc(grace_seconds = 12,endpoints(control=Control))]
|
||||||
impl Probe {
|
impl Probe {
|
||||||
|
/// Scan only after the watch is armed; keep the call open until process death.
|
||||||
|
async fn watch(&self, p: WatchPath) -> Result<()> {
|
||||||
|
let path = p.path.clone();
|
||||||
|
call("core.watch", json!({"path":p.path}))
|
||||||
|
.on_notify(move |name, _| {
|
||||||
|
if name == "armed" && path == "/self" {
|
||||||
|
std::fs::write("/self/owned", "visible via proc").unwrap();
|
||||||
|
}
|
||||||
|
let mut entries: Vec<_> = std::fs::read_dir(&path)
|
||||||
|
.into_iter()
|
||||||
|
.flatten()
|
||||||
|
.filter_map(std::result::Result::ok)
|
||||||
|
.map(|e| e.file_name().into_string().unwrap())
|
||||||
|
.collect();
|
||||||
|
entries.sort();
|
||||||
|
let text = std::fs::read_to_string(format!("{path}/before")).ok();
|
||||||
|
Ctx::default()
|
||||||
|
.notify(
|
||||||
|
name,
|
||||||
|
json!({"pid":sausage_plugin::pid(), "entries":entries, "text":text}),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
})
|
||||||
|
.await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Poll the watch in the handler, then return idle without a background task.
|
||||||
|
async fn watch_once(&self, p: WatchPath) -> Result<()> {
|
||||||
|
use std::{
|
||||||
|
cell::RefCell,
|
||||||
|
rc::Rc,
|
||||||
|
task::{Poll, Waker},
|
||||||
|
};
|
||||||
|
let state = Rc::new(RefCell::new((false, None::<Waker>)));
|
||||||
|
let received = state.clone();
|
||||||
|
let mut watch = std::pin::pin!(std::future::IntoFuture::into_future(
|
||||||
|
call("core.watch", json!({"path":p.path})).on_notify(move |name, _| {
|
||||||
|
if name == "armed" {
|
||||||
|
let mut state = received.borrow_mut();
|
||||||
|
state.0 = true;
|
||||||
|
if let Some(waker) = state.1.take() {
|
||||||
|
waker.wake();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
));
|
||||||
|
std::future::poll_fn(|cx| {
|
||||||
|
if let Poll::Ready(result) = watch.as_mut().poll(cx) {
|
||||||
|
return Poll::Ready(result.map(|_| ()));
|
||||||
|
}
|
||||||
|
let mut state = state.borrow_mut();
|
||||||
|
state.1 = Some(cx.waker().clone());
|
||||||
|
if state.0 {
|
||||||
|
Poll::Ready(Ok(()))
|
||||||
|
} else {
|
||||||
|
Poll::Pending
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Drop one watch while another still delivers, in a process that remains alive.
|
||||||
|
async fn drop_watch(&self) -> Result<Vec<String>> {
|
||||||
|
use std::{cell::RefCell, future::IntoFuture, rc::Rc, task::Poll};
|
||||||
|
let events = Rc::new(RefCell::new(Vec::new()));
|
||||||
|
let captured = events.clone();
|
||||||
|
let mut dropped = call("core.watch", json!({"path":"/dropped"}))
|
||||||
|
.on_notify(move |_, _| captured.borrow_mut().push("dropped".to_owned()))
|
||||||
|
.into_future();
|
||||||
|
std::future::poll_fn(|cx| {
|
||||||
|
assert!(dropped.as_mut().poll(cx).is_pending());
|
||||||
|
Poll::Ready(())
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
drop(dropped);
|
||||||
|
let captured = events.clone();
|
||||||
|
let mut kept = call("core.watch", json!({"path":"/kept"}))
|
||||||
|
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
|
||||||
|
.into_future();
|
||||||
|
std::future::poll_fn(|cx| {
|
||||||
|
assert!(kept.as_mut().poll(cx).is_pending());
|
||||||
|
Poll::Ready(())
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
// The harness sends late traffic and kept-watch events before answering.
|
||||||
|
call("other.barrier", json!({})).await?;
|
||||||
|
drop(kept);
|
||||||
|
let result = events.borrow().clone();
|
||||||
|
Ok(result)
|
||||||
|
}
|
||||||
|
|
||||||
/// Publish a control endpoint until EOF.
|
/// Publish a control endpoint until EOF.
|
||||||
async fn serve(&self) -> Result<()> {
|
async fn serve(&self) -> Result<()> {
|
||||||
sausage_plugin::listen("/endpoint", std::rc::Rc::new(Control)).await
|
sausage_plugin::listen("/endpoint", std::rc::Rc::new(Control)).await
|
||||||
@@ -54,6 +158,11 @@ impl Probe {
|
|||||||
sausage_plugin::listen("/state/control", std::rc::Rc::new(Control)).await?;
|
sausage_plugin::listen("/state/control", std::rc::Rc::new(Control)).await?;
|
||||||
Ok(sausage_plugin::pid())
|
Ok(sausage_plugin::pid())
|
||||||
}
|
}
|
||||||
|
/// Keep an idle listener and ownership lock until opt-in drain ends this process.
|
||||||
|
async fn drain_service(&self) -> Result<String> {
|
||||||
|
lock("/drain-owner").await?.keep();
|
||||||
|
self.service().await
|
||||||
|
}
|
||||||
/// Add two numbers.
|
/// Add two numbers.
|
||||||
///
|
///
|
||||||
/// The result is their sum.
|
/// The result is their sum.
|
||||||
@@ -111,6 +220,15 @@ impl Probe {
|
|||||||
fn retired(&self, p: Retired) -> Result<String> {
|
fn retired(&self, p: Retired) -> Result<String> {
|
||||||
Ok(p.previous)
|
Ok(p.previous)
|
||||||
}
|
}
|
||||||
|
/// Drain on replacement while calls are still available.
|
||||||
|
#[sausage_plugin::on_superseded]
|
||||||
|
async fn superseded(&self, p: Superseded) -> Result<()> {
|
||||||
|
call("other.superseded", json!({"current":p.current})).await?;
|
||||||
|
sausage_plugin::drain();
|
||||||
|
sausage_plugin::drain();
|
||||||
|
Ctx::default().notify("draining", json!({}))?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
/// Wait until EOF, then run the stop hook.
|
/// Wait until EOF, then run the stop hook.
|
||||||
async fn wait(&self) -> Result<String> {
|
async fn wait(&self) -> Result<String> {
|
||||||
call("other.wait", json!({})).await?;
|
call("other.wait", json!({})).await?;
|
||||||
@@ -155,6 +273,7 @@ impl Probe {
|
|||||||
#[sausage_plugin::on_stop]
|
#[sausage_plugin::on_stop]
|
||||||
async fn stop(&self) {
|
async fn stop(&self) {
|
||||||
std::fs::write("/stopped", b"yes").unwrap();
|
std::fs::write("/stopped", b"yes").unwrap();
|
||||||
|
std::fs::write(format!("/stopped-{}", sausage_plugin::pid()), b"yes").unwrap();
|
||||||
Ctx::default().notify("stopped", json!({})).unwrap();
|
Ctx::default().notify("stopped", json!({})).unwrap();
|
||||||
assert!(call("other.no", json!({})).await.is_err());
|
assert!(call("other.no", json!({})).await.is_err());
|
||||||
}
|
}
|
||||||
@@ -193,6 +312,11 @@ impl Control {
|
|||||||
ctx.notify("endpoint-progress", json!({"value":7}))?;
|
ctx.notify("endpoint-progress", json!({"value":7}))?;
|
||||||
Ok(7)
|
Ok(7)
|
||||||
}
|
}
|
||||||
|
/// Finish an in-flight call after draining begins.
|
||||||
|
async fn wait(&self) -> Result<String> {
|
||||||
|
call("other.inflight", json!({})).await?;
|
||||||
|
Ok("finished".into())
|
||||||
|
}
|
||||||
/// End the endpoint process after the response.
|
/// End the endpoint process after the response.
|
||||||
fn stop(&self, ctx: &Ctx) -> Result<()> {
|
fn stop(&self, ctx: &Ctx) -> Result<()> {
|
||||||
ctx.exit_after_response();
|
ctx.exit_after_response();
|
||||||
|
|||||||
@@ -116,6 +116,53 @@ fn discovery_and_dispatch() {
|
|||||||
.into();
|
.into();
|
||||||
assert_eq!(internal.code, -32603);
|
assert_eq!(internal.code, -32603);
|
||||||
}
|
}
|
||||||
|
#[sausage_plugin::test]
|
||||||
|
async fn params_must_be_objects() {
|
||||||
|
for (params, kind) in [
|
||||||
|
(json!("{\"a\":2,\"b\":3}"), "a string"),
|
||||||
|
(json!([]), "an array"),
|
||||||
|
(json!(42), "a number"),
|
||||||
|
(json!(null), "null"),
|
||||||
|
(json!(true), "a boolean"),
|
||||||
|
] {
|
||||||
|
let message = format!(
|
||||||
|
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
|
||||||
|
);
|
||||||
|
let outgoing = sausage_plugin::call("plugin.add", params.clone())
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert_eq!(outgoing.code, -32602);
|
||||||
|
assert_eq!(outgoing.message, message);
|
||||||
|
for method in ["add", "choice", "unknown"] {
|
||||||
|
let incoming = Plugin
|
||||||
|
.dispatch(method, params.clone(), &Ctx::default())
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert_eq!(incoming.code, -32602);
|
||||||
|
assert_eq!(incoming.message, message);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
Plugin
|
||||||
|
.dispatch("add", json!({"a":2,"b":3}), &Ctx::default())
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
5
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
Plugin
|
||||||
|
.dispatch("defaults", json!({}), &Ctx::default())
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
json!({"value":""})
|
||||||
|
);
|
||||||
|
// An object reaches transport validation rather than the params rejection.
|
||||||
|
let outgoing = sausage_plugin::call("plugin.add", json!({"a":2,"b":3}))
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(outgoing.message.contains("no core transport"));
|
||||||
|
}
|
||||||
|
|
||||||
#[sausage_plugin::test]
|
#[sausage_plugin::test]
|
||||||
async fn missing_core_fails_without_wait() {
|
async fn missing_core_fails_without_wait() {
|
||||||
assert!(
|
assert!(
|
||||||
|
|||||||
@@ -0,0 +1,11 @@
|
|||||||
|
struct Supersedes;
|
||||||
|
#[sausage_plugin::rpc]
|
||||||
|
impl Supersedes {
|
||||||
|
/// First superseded hook.
|
||||||
|
#[sausage_plugin::on_superseded]
|
||||||
|
fn a(&self) -> sausage_plugin::Result<()> { Ok(()) }
|
||||||
|
/// Second superseded hook.
|
||||||
|
#[sausage_plugin::on_superseded]
|
||||||
|
fn b(&self) -> sausage_plugin::Result<()> { Ok(()) }
|
||||||
|
}
|
||||||
|
fn main() {}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
error: only one on_superseded method is allowed
|
||||||
|
--> tests/ui/superseded.rs:7:5
|
||||||
|
|
|
||||||
|
7 | /// Second superseded hook.
|
||||||
|
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
+127
-4
@@ -34,10 +34,8 @@ impl Process {
|
|||||||
let mut s = String::new();
|
let mut s = String::new();
|
||||||
if self.output.read_line(&mut s).unwrap() == 0 {
|
if self.output.read_line(&mut s).unwrap() == 0 {
|
||||||
let mut stderr = String::new();
|
let mut stderr = String::new();
|
||||||
let _ = std::io::Read::read_to_string(
|
let _ =
|
||||||
self.child.stderr.as_mut().unwrap(),
|
std::io::Read::read_to_string(self.child.stderr.as_mut().unwrap(), &mut stderr);
|
||||||
&mut stderr,
|
|
||||||
);
|
|
||||||
panic!("unexpected EOF from wasmtime. Its stderr:\n{stderr}");
|
panic!("unexpected EOF from wasmtime. Its stderr:\n{stderr}");
|
||||||
}
|
}
|
||||||
if let Ok(v) = sausage_plugin::serde_json::from_str::<Value>(&s)
|
if let Ok(v) = sausage_plugin::serde_json::from_str::<Value>(&s)
|
||||||
@@ -110,6 +108,34 @@ fn protocol() {
|
|||||||
}
|
}
|
||||||
p.finishes();
|
p.finishes();
|
||||||
}
|
}
|
||||||
|
// A handler-owned watch does not keep an idle process alive after return,
|
||||||
|
// even though the host never answers the watch and stdin remains open.
|
||||||
|
let mut p = start(&wasm, &dir);
|
||||||
|
p.request("watch_once", json!({"path":"/mailbox"}));
|
||||||
|
let watch = p.read();
|
||||||
|
assert_eq!(watch["method"], "core.watch");
|
||||||
|
p.send(json!({"jsonrpc":"2.0","method":"core.notify","params":{"call":watch["id"],"name":"armed","params":{}}}));
|
||||||
|
assert_eq!(p.read()["result"], Value::Null);
|
||||||
|
p.finishes();
|
||||||
|
// Keep the process alive after dropping a call: late traffic is ignored,
|
||||||
|
// while the kept watch continues to deliver every notification.
|
||||||
|
let mut p = start(&wasm, &dir);
|
||||||
|
p.request("drop_watch", json!({}));
|
||||||
|
let dropped = p.read();
|
||||||
|
let kept = p.read();
|
||||||
|
let barrier = p.read();
|
||||||
|
assert_eq!(dropped["params"]["path"], "/dropped");
|
||||||
|
assert_eq!(kept["params"]["path"], "/kept");
|
||||||
|
assert_eq!(barrier["method"], "other.barrier");
|
||||||
|
for watch in [&dropped, &kept] {
|
||||||
|
for name in ["armed", "changed", "changed"] {
|
||||||
|
p.send(json!({"jsonrpc":"2.0","method":"core.notify","params":{"call":watch["id"],"name":name,"params":{}}}));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
p.answer(&dropped, Value::Null);
|
||||||
|
p.answer(&barrier, json!({}));
|
||||||
|
assert_eq!(p.read()["result"], json!(["armed", "changed", "changed"]));
|
||||||
|
p.finishes();
|
||||||
let mut p = start(&wasm, &dir);
|
let mut p = start(&wasm, &dir);
|
||||||
p.request("rpc.discover", json!({}));
|
p.request("rpc.discover", json!({}));
|
||||||
let d = p.read();
|
let d = p.read();
|
||||||
@@ -203,6 +229,103 @@ fn protocol() {
|
|||||||
assert_eq!(outgoing["method"], "other.background");
|
assert_eq!(outgoing["method"], "other.background");
|
||||||
p.answer(&outgoing, json!({}));
|
p.answer(&outgoing, json!({}));
|
||||||
assert!(p.child.wait().unwrap().success());
|
assert!(p.child.wait().unwrap().success());
|
||||||
|
// Replacement is a notification, not EOF. Drain keeps both endpoint handlers
|
||||||
|
// and background calls alive, while closing idle connections and the listener.
|
||||||
|
assert!(
|
||||||
|
!d["result"]["methods"]
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.iter()
|
||||||
|
.any(|m| m["name"] == "rpc.superseded" || m["name"] == "superseded")
|
||||||
|
);
|
||||||
|
std::fs::remove_file(dir.join("stopped")).unwrap();
|
||||||
|
let mut provider = start(&wasm, &dir);
|
||||||
|
provider.request("serve", json!({}));
|
||||||
|
assert_eq!(provider.read()["result"], Value::Null);
|
||||||
|
provider.request("start", json!({}));
|
||||||
|
let first = provider.read();
|
||||||
|
let second = provider.read();
|
||||||
|
let background = if first["method"] == "other.background" {
|
||||||
|
first
|
||||||
|
} else {
|
||||||
|
second
|
||||||
|
};
|
||||||
|
assert_eq!(background["method"], "other.background");
|
||||||
|
let address = std::fs::read_to_string(dir.join("endpoint"))
|
||||||
|
.unwrap()
|
||||||
|
.split_whitespace()
|
||||||
|
.next()
|
||||||
|
.unwrap()
|
||||||
|
.trim_start_matches("tcp:")
|
||||||
|
.to_owned();
|
||||||
|
let mut idle = std::net::TcpStream::connect(&address).unwrap();
|
||||||
|
idle.set_read_timeout(Some(std::time::Duration::from_secs(4)))
|
||||||
|
.unwrap();
|
||||||
|
writeln!(
|
||||||
|
idle,
|
||||||
|
"{}",
|
||||||
|
json!({"jsonrpc":"2.0","id":0,"method":"rpc.endpoint","params":{}})
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let mut idle = BufReader::new(idle);
|
||||||
|
let mut line = String::new();
|
||||||
|
idle.read_line(&mut line).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
sausage_plugin::serde_json::from_str::<Value>(&line).unwrap()["id"],
|
||||||
|
0
|
||||||
|
);
|
||||||
|
let mut active = std::net::TcpStream::connect(&address).unwrap();
|
||||||
|
active
|
||||||
|
.set_read_timeout(Some(std::time::Duration::from_secs(4)))
|
||||||
|
.unwrap();
|
||||||
|
writeln!(
|
||||||
|
active,
|
||||||
|
"{}",
|
||||||
|
json!({"jsonrpc":"2.0","id":1,"method":"wait","params":{}})
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let inflight = provider.read();
|
||||||
|
assert_eq!(inflight["method"], "other.inflight");
|
||||||
|
provider
|
||||||
|
.send(json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":"new-hash"}}));
|
||||||
|
let hook = provider.read();
|
||||||
|
assert_eq!(hook["method"], "other.superseded");
|
||||||
|
assert_eq!(hook["params"]["current"], "new-hash");
|
||||||
|
provider.answer(&hook, json!({}));
|
||||||
|
assert_eq!(provider.read()["params"]["name"], "draining");
|
||||||
|
line.clear();
|
||||||
|
assert_eq!(idle.read_line(&mut line).unwrap(), 0);
|
||||||
|
// drain() wakes independent accept/read cancellations. Idle EOF does not
|
||||||
|
// order listener destruction: TCP may still complete a handshake, but that
|
||||||
|
// connection must close without serving work. Do not retry the connect.
|
||||||
|
if let Ok(mut late) = std::net::TcpStream::connect(&address) {
|
||||||
|
eprintln!("TCP handshake raced listener cancellation; waiting for close");
|
||||||
|
// Reuse the four-second socket bound used by idle/active above.
|
||||||
|
late.set_read_timeout(Some(std::time::Duration::from_secs(4)))
|
||||||
|
.unwrap();
|
||||||
|
let closed = std::io::Read::read(&mut late, &mut [0]);
|
||||||
|
assert!(
|
||||||
|
matches!(closed, Ok(0))
|
||||||
|
|| matches!(&closed, Err(e) if e.kind() == std::io::ErrorKind::ConnectionReset),
|
||||||
|
"late connection survived drain: {closed:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert!(provider.child.try_wait().unwrap().is_none());
|
||||||
|
provider.answer(&inflight, json!({}));
|
||||||
|
line.clear();
|
||||||
|
BufReader::new(active).read_line(&mut line).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
sausage_plugin::serde_json::from_str::<Value>(&line).unwrap()["result"],
|
||||||
|
"finished"
|
||||||
|
);
|
||||||
|
assert!(provider.child.try_wait().unwrap().is_none());
|
||||||
|
provider.answer(&background, json!({}));
|
||||||
|
provider.finishes();
|
||||||
|
assert!(std::net::TcpStream::connect(&address).is_err());
|
||||||
|
assert!(
|
||||||
|
!dir.join("stopped").exists(),
|
||||||
|
"drain is not the EOF stop hook"
|
||||||
|
);
|
||||||
let provider_doc = &d["result"];
|
let provider_doc = &d["result"];
|
||||||
let imports = target.join("imports");
|
let imports = target.join("imports");
|
||||||
std::fs::create_dir_all(&imports).unwrap();
|
std::fs::create_dir_all(&imports).unwrap();
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ pub mod generated;
|
|||||||
pub mod host;
|
pub mod host;
|
||||||
pub mod memory;
|
pub mod memory;
|
||||||
pub mod mounts;
|
pub mod mounts;
|
||||||
|
pub mod observe;
|
||||||
pub mod process;
|
pub mod process;
|
||||||
mod readonly;
|
mod readonly;
|
||||||
mod streams;
|
mod streams;
|
||||||
|
|||||||
@@ -0,0 +1,615 @@
|
|||||||
|
//! Observes successful namespace and open-file mutations, even if their caller is cancelled.
|
||||||
|
use crate::*;
|
||||||
|
use std::{
|
||||||
|
collections::BTreeMap,
|
||||||
|
sync::{Mutex, Weak},
|
||||||
|
};
|
||||||
|
use tokio::sync::{OwnedRwLockReadGuard, OwnedRwLockWriteGuard, RwLock};
|
||||||
|
|
||||||
|
type Namespace = Mutex<BTreeMap<String, Weak<RwLock<()>>>>;
|
||||||
|
enum Held {
|
||||||
|
Read { _guard: OwnedRwLockReadGuard<()> },
|
||||||
|
Write { _guard: OwnedRwLockWriteGuard<()> },
|
||||||
|
}
|
||||||
|
|
||||||
|
type TrackedPath = Mutex<Option<String>>;
|
||||||
|
type Path = Arc<TrackedPath>;
|
||||||
|
type Callback = dyn Fn(&str) + Send + Sync;
|
||||||
|
/// Shared observation for mount namespaces. Paths supplied to the callback are virtual paths.
|
||||||
|
/// Callbacks run after filesystem locks have been released; they must not block.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct Observer {
|
||||||
|
changed: Arc<Callback>,
|
||||||
|
paths: Arc<Mutex<Vec<Weak<TrackedPath>>>>,
|
||||||
|
// Only open-registration versus rename/remove needs ordering. Lock the affected
|
||||||
|
// subtrees, not all IO: a blocked open must not stall unrelated namespaces.
|
||||||
|
namespace: Arc<Namespace>,
|
||||||
|
}
|
||||||
|
impl Observer {
|
||||||
|
pub fn new(changed: impl Fn(&str) + Send + Sync + 'static) -> Self {
|
||||||
|
Self {
|
||||||
|
changed: Arc::new(changed),
|
||||||
|
paths: Arc::default(),
|
||||||
|
namespace: Arc::default(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
/// Wrap a layer at its canonical namespace prefix (e.g. `/run`).
|
||||||
|
pub fn layer(&self, layer: Arc<dyn Layer>, prefix: &str, process: Process) -> Arc<dyn Layer> {
|
||||||
|
Arc::new(Observed {
|
||||||
|
layer,
|
||||||
|
prefix: prefix.trim_end_matches('/').into(),
|
||||||
|
observer: self.clone(),
|
||||||
|
process,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
/// Opens share their path and ancestors; rename/remove exclusively hold their
|
||||||
|
/// endpoints, sharing ancestors. Sorted acquisition avoids cross-rename deadlocks.
|
||||||
|
/// No descriptor IO or notification holds these guards.
|
||||||
|
async fn order(&self, endpoints: &[(&str, bool)]) -> Vec<Held> {
|
||||||
|
let mut modes = BTreeMap::new();
|
||||||
|
for &(path, exclusive) in endpoints {
|
||||||
|
modes
|
||||||
|
.entry(path.to_owned())
|
||||||
|
.and_modify(|mode| *mode |= exclusive)
|
||||||
|
.or_insert(exclusive);
|
||||||
|
let mut ancestor = path;
|
||||||
|
while let Some((parent, _)) = ancestor.rsplit_once('/') {
|
||||||
|
let parent = if parent.is_empty() { "/" } else { parent };
|
||||||
|
modes.entry(parent.to_owned()).or_insert(false);
|
||||||
|
if parent == "/" {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
ancestor = parent;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let locks: Vec<_> = {
|
||||||
|
let mut namespace = self.namespace.lock().unwrap();
|
||||||
|
namespace.retain(|_, lock| lock.strong_count() != 0);
|
||||||
|
modes
|
||||||
|
.into_iter()
|
||||||
|
.map(|(path, exclusive)| {
|
||||||
|
let lock = namespace.entry(path).or_default();
|
||||||
|
let strong = lock.upgrade().unwrap_or_else(|| {
|
||||||
|
let strong = Arc::new(RwLock::new(()));
|
||||||
|
*lock = Arc::downgrade(&strong);
|
||||||
|
strong
|
||||||
|
});
|
||||||
|
(strong, exclusive)
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
};
|
||||||
|
let mut held = Vec::new();
|
||||||
|
for (lock, exclusive) in locks {
|
||||||
|
held.push(if exclusive {
|
||||||
|
Held::Write {
|
||||||
|
_guard: lock.write_owned().await,
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
Held::Read {
|
||||||
|
_guard: lock.read_owned().await,
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
held
|
||||||
|
}
|
||||||
|
fn path(&self, path: String) -> Path {
|
||||||
|
let path = Arc::new(Mutex::new(Some(path)));
|
||||||
|
let mut paths = self.paths.lock().unwrap();
|
||||||
|
paths.retain(|p| p.strong_count() != 0);
|
||||||
|
paths.push(Arc::downgrade(&path));
|
||||||
|
path
|
||||||
|
}
|
||||||
|
fn moved(&self, from: &str, to: Option<&str>) {
|
||||||
|
let mut paths = self.paths.lock().unwrap();
|
||||||
|
paths.retain(|p| {
|
||||||
|
let Some(p) = p.upgrade() else { return false };
|
||||||
|
let mut path = p.lock().unwrap();
|
||||||
|
if let Some(old) = path.as_deref() {
|
||||||
|
if under(old, from) {
|
||||||
|
*path = to.map(|to| format!("{to}{}", &old[from.len()..]));
|
||||||
|
} else if to.is_some_and(|to| under(old, to)) {
|
||||||
|
// A replaced destination's old descriptor no longer names a visible file.
|
||||||
|
*path = None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
true
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
/// Component-aware descendant matching, including the path itself.
|
||||||
|
pub fn under(path: &str, base: &str) -> bool {
|
||||||
|
base == "/" || path == base || path.strip_prefix(base).is_some_and(|s| s.starts_with('/'))
|
||||||
|
}
|
||||||
|
struct Observed {
|
||||||
|
layer: Arc<dyn Layer>,
|
||||||
|
prefix: String,
|
||||||
|
observer: Observer,
|
||||||
|
process: Process,
|
||||||
|
}
|
||||||
|
impl Observed {
|
||||||
|
fn path(&self, path: &str) -> String {
|
||||||
|
let path = format!("{}{}", self.prefix, path.trim_end_matches('/'));
|
||||||
|
if path.is_empty() { "/".into() } else { path }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
struct Handle {
|
||||||
|
file: Arc<dyn File>,
|
||||||
|
path: Path,
|
||||||
|
observer: Observer,
|
||||||
|
process: Process,
|
||||||
|
}
|
||||||
|
impl Handle {
|
||||||
|
async fn mutate<T: Send + 'static>(
|
||||||
|
&self,
|
||||||
|
f: impl Future<Output = Result<T>> + Send + 'static,
|
||||||
|
) -> Result<T> {
|
||||||
|
let (path, observer) = (self.path.clone(), self.observer.clone());
|
||||||
|
self.process
|
||||||
|
.operation(async move {
|
||||||
|
let result = f.await?;
|
||||||
|
let path = path.lock().unwrap().clone();
|
||||||
|
if let Some(path) = path {
|
||||||
|
(observer.changed)(&path);
|
||||||
|
}
|
||||||
|
Ok(result)
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#[async_trait]
|
||||||
|
impl Layer for Observed {
|
||||||
|
async fn stat(&self, path: &str, follow: bool) -> Result<Stat> {
|
||||||
|
self.layer.stat(path, follow).await
|
||||||
|
}
|
||||||
|
async fn readlink(&self, path: &str) -> Result<String> {
|
||||||
|
self.layer.readlink(path).await
|
||||||
|
}
|
||||||
|
async fn list(&self, path: &str) -> Result<Vec<Entry>> {
|
||||||
|
self.layer.list(path).await
|
||||||
|
}
|
||||||
|
async fn open(&self, path: &str, options: Open) -> Result<Arc<dyn File>> {
|
||||||
|
let (layer, local, path, observer, process) = (
|
||||||
|
self.layer.clone(),
|
||||||
|
path.to_owned(),
|
||||||
|
self.path(path),
|
||||||
|
self.observer.clone(),
|
||||||
|
self.process.clone(),
|
||||||
|
);
|
||||||
|
self.process
|
||||||
|
.operation(async move {
|
||||||
|
let order = observer.order(&[(&path, false)]).await;
|
||||||
|
let file = layer.open(&local, options).await?;
|
||||||
|
let handle = Arc::new(Handle {
|
||||||
|
file,
|
||||||
|
path: observer.path(path.clone()),
|
||||||
|
observer: observer.clone(),
|
||||||
|
process,
|
||||||
|
});
|
||||||
|
drop(order);
|
||||||
|
if options.create || options.truncate {
|
||||||
|
(observer.changed)(&path);
|
||||||
|
}
|
||||||
|
Ok(handle as Arc<dyn File>)
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
async fn mkdir(&self, path: &str) -> Result<()> {
|
||||||
|
let (layer, local, path, observer) = (
|
||||||
|
self.layer.clone(),
|
||||||
|
path.to_owned(),
|
||||||
|
self.path(path),
|
||||||
|
self.observer.clone(),
|
||||||
|
);
|
||||||
|
self.process
|
||||||
|
.operation(async move {
|
||||||
|
layer.mkdir(&local).await?;
|
||||||
|
(observer.changed)(&path);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
async fn remove_file(&self, path: &str) -> Result<()> {
|
||||||
|
self.remove(path, false).await
|
||||||
|
}
|
||||||
|
async fn remove_dir(&self, path: &str) -> Result<()> {
|
||||||
|
self.remove(path, true).await
|
||||||
|
}
|
||||||
|
async fn rename(&self, from: &str, to: &str) -> Result<()> {
|
||||||
|
let (layer, a, b, from, to, observer) = (
|
||||||
|
self.layer.clone(),
|
||||||
|
from.to_owned(),
|
||||||
|
to.to_owned(),
|
||||||
|
self.path(from),
|
||||||
|
self.path(to),
|
||||||
|
self.observer.clone(),
|
||||||
|
);
|
||||||
|
self.process
|
||||||
|
.operation(async move {
|
||||||
|
let order = observer.order(&[(&from, true), (&to, true)]).await;
|
||||||
|
layer.rename(&a, &b).await?;
|
||||||
|
if from != to {
|
||||||
|
observer.moved(&from, Some(&to));
|
||||||
|
}
|
||||||
|
drop(order);
|
||||||
|
(observer.changed)(&from);
|
||||||
|
(observer.changed)(&to);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Observed {
|
||||||
|
async fn remove(&self, path: &str, directory: bool) -> Result<()> {
|
||||||
|
let (layer, local, path, observer) = (
|
||||||
|
self.layer.clone(),
|
||||||
|
path.to_owned(),
|
||||||
|
self.path(path),
|
||||||
|
self.observer.clone(),
|
||||||
|
);
|
||||||
|
self.process
|
||||||
|
.operation(async move {
|
||||||
|
let order = observer.order(&[(&path, true)]).await;
|
||||||
|
if directory {
|
||||||
|
layer.remove_dir(&local).await?;
|
||||||
|
} else {
|
||||||
|
layer.remove_file(&local).await?;
|
||||||
|
}
|
||||||
|
observer.moved(&path, None);
|
||||||
|
drop(order);
|
||||||
|
(observer.changed)(&path);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#[async_trait]
|
||||||
|
impl File for Handle {
|
||||||
|
async fn stat(&self) -> Result<Stat> {
|
||||||
|
self.file.stat().await
|
||||||
|
}
|
||||||
|
async fn read(&self, offset: u64, length: usize) -> Result<Vec<u8>> {
|
||||||
|
self.file.read(offset, length).await
|
||||||
|
}
|
||||||
|
async fn write(&self, offset: u64, bytes: Vec<u8>) -> Result<usize> {
|
||||||
|
let f = self.file.clone();
|
||||||
|
self.mutate(async move { f.write(offset, bytes).await })
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
async fn append(&self, bytes: Vec<u8>) -> Result<usize> {
|
||||||
|
let f = self.file.clone();
|
||||||
|
self.mutate(async move { f.append(bytes).await }).await
|
||||||
|
}
|
||||||
|
async fn set_size(&self, size: u64) -> Result<()> {
|
||||||
|
let f = self.file.clone();
|
||||||
|
self.mutate(async move { f.set_size(size).await }).await
|
||||||
|
}
|
||||||
|
async fn sync(&self) -> Result<()> {
|
||||||
|
self.file.sync().await
|
||||||
|
}
|
||||||
|
async fn set_times(
|
||||||
|
&self,
|
||||||
|
accessed: Option<SystemTime>,
|
||||||
|
modified: Option<SystemTime>,
|
||||||
|
) -> Result<()> {
|
||||||
|
let f = self.file.clone();
|
||||||
|
self.mutate(async move { f.set_times(accessed, modified).await })
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
#[test]
|
||||||
|
fn component_matching() {
|
||||||
|
assert!(under("/mailbox/a", "/mailbox"));
|
||||||
|
assert!(under("/mailbox", "/mailbox"));
|
||||||
|
assert!(under("/anything", "/"));
|
||||||
|
assert!(!under("/mailboxes/a", "/mailbox"));
|
||||||
|
}
|
||||||
|
/// Every visible mutation is delivered after the layer call, including rebased
|
||||||
|
/// open handles in another process's namespace. Removed handles stay usable.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn handles_follow_rename_and_unlink() {
|
||||||
|
for host in [false, true] {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let process = Process::default();
|
||||||
|
let layer: Arc<dyn Layer> = if host {
|
||||||
|
Arc::new(Host::new(temp.path(), process.clone()).unwrap())
|
||||||
|
} else {
|
||||||
|
Arc::new(Memory::default())
|
||||||
|
};
|
||||||
|
let (tx, mut rx) = mpsc::unbounded_channel();
|
||||||
|
let observer = Observer::new(move |p| {
|
||||||
|
tx.send(p.to_owned()).unwrap();
|
||||||
|
});
|
||||||
|
let a = observer.layer(layer.clone(), "/run", process.clone());
|
||||||
|
let b = observer.layer(layer, "/run", process);
|
||||||
|
a.mkdir("/dir").await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/dir");
|
||||||
|
let f = a
|
||||||
|
.open(
|
||||||
|
"/dir/a",
|
||||||
|
Open {
|
||||||
|
create: true,
|
||||||
|
write: true,
|
||||||
|
..Open::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/dir/a");
|
||||||
|
b.rename("/dir", "/moved").await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/dir");
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/moved");
|
||||||
|
f.write(0, b"new".to_vec()).await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/moved/a");
|
||||||
|
assert_eq!(
|
||||||
|
b.open("/moved/a", Open::default())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.read(0, 3)
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
b"new"
|
||||||
|
);
|
||||||
|
let old = b
|
||||||
|
.open(
|
||||||
|
"/old",
|
||||||
|
Open {
|
||||||
|
create: true,
|
||||||
|
write: true,
|
||||||
|
..Open::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
rx.recv().await.unwrap();
|
||||||
|
b.rename("/moved/a", "/old").await.unwrap();
|
||||||
|
rx.recv().await.unwrap();
|
||||||
|
rx.recv().await.unwrap();
|
||||||
|
old.write(0, b"invisible".to_vec()).await.unwrap();
|
||||||
|
assert!(rx.try_recv().is_err());
|
||||||
|
f.append(b"!".to_vec()).await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/old");
|
||||||
|
b.remove_file("/old").await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/old");
|
||||||
|
f.set_size(0).await.unwrap();
|
||||||
|
assert!(rx.try_recv().is_err());
|
||||||
|
assert!(b.remove_file("/missing").await.is_err());
|
||||||
|
assert!(rx.try_recv().is_err());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
/// A pending exclusive namespace change must not block a sibling's opens,
|
||||||
|
/// even though Tokio's RwLock prefers queued writers over new readers.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn namespace_order_is_scoped_and_shared() {
|
||||||
|
async fn ready<T>(future: impl Future<Output = T>) -> T {
|
||||||
|
let mut future = std::pin::pin!(future);
|
||||||
|
std::future::poll_fn(|cx| match future.as_mut().poll(cx) {
|
||||||
|
std::task::Poll::Ready(value) => std::task::Poll::Ready(value),
|
||||||
|
std::task::Poll::Pending => panic!("unrelated namespace operation blocked"),
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
let observer = Observer::new(|_| {});
|
||||||
|
let open = observer.order(&[("/run/slow/file", false)]).await;
|
||||||
|
let concurrent_open = ready(observer.order(&[("/run/slow/file", false)])).await;
|
||||||
|
let mut rename =
|
||||||
|
std::pin::pin!(observer.order(&[("/run/slow", true), ("/run/moved", true)]));
|
||||||
|
std::future::poll_fn(|cx| {
|
||||||
|
assert!(rename.as_mut().poll(cx).is_pending());
|
||||||
|
std::task::Poll::Ready(())
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
// Same mounted layer, different subtree; also a different mounted layer.
|
||||||
|
let sibling_open = ready(observer.order(&[("/run/sibling/file", false)])).await;
|
||||||
|
let host_open = ready(observer.order(&[("/state/file", false)])).await;
|
||||||
|
drop(sibling_open);
|
||||||
|
let sibling_rename =
|
||||||
|
ready(observer.order(&[("/run/sibling", true), ("/run/other", true)])).await;
|
||||||
|
drop((sibling_rename, host_open, concurrent_open, open));
|
||||||
|
let renamed = ready(rename).await;
|
||||||
|
// A descendant open waits for its ancestor's rename until rebasing is done.
|
||||||
|
let mut descendant = std::pin::pin!(observer.order(&[("/run/moved/file", false)]));
|
||||||
|
std::future::poll_fn(|cx| {
|
||||||
|
assert!(descendant.as_mut().poll(cx).is_pending());
|
||||||
|
std::task::Poll::Ready(())
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
drop(renamed);
|
||||||
|
drop(ready(descendant).await);
|
||||||
|
drop(
|
||||||
|
observer
|
||||||
|
.order(&[("/run/moved/file", true), ("/run/moved", true)])
|
||||||
|
.await,
|
||||||
|
);
|
||||||
|
drop(
|
||||||
|
observer
|
||||||
|
.order(&[("/run/moved", true), ("/run/moved", true)])
|
||||||
|
.await,
|
||||||
|
);
|
||||||
|
// Dead weak entries are pruned, rather than retaining every path ever opened.
|
||||||
|
observer.order(&[("/", false)]).await;
|
||||||
|
assert_eq!(observer.namespace.lock().unwrap().len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pause an open after obtaining its real identity, before registering its
|
||||||
|
/// observed handle, like a slow host syscall or /self/log query.
|
||||||
|
struct SlowOpen {
|
||||||
|
memory: Arc<Memory>,
|
||||||
|
gate: Mutex<
|
||||||
|
Option<(
|
||||||
|
tokio::sync::oneshot::Sender<()>,
|
||||||
|
tokio::sync::oneshot::Receiver<()>,
|
||||||
|
)>,
|
||||||
|
>,
|
||||||
|
}
|
||||||
|
#[async_trait]
|
||||||
|
impl Layer for SlowOpen {
|
||||||
|
async fn stat(&self, p: &str, follow: bool) -> Result<Stat> {
|
||||||
|
self.memory.stat(p, follow).await
|
||||||
|
}
|
||||||
|
async fn list(&self, p: &str) -> Result<Vec<Entry>> {
|
||||||
|
self.memory.list(p).await
|
||||||
|
}
|
||||||
|
async fn open(&self, p: &str, options: Open) -> Result<Arc<dyn File>> {
|
||||||
|
let f = self.memory.open(p, options).await?;
|
||||||
|
let gate = self.gate.lock().unwrap().take();
|
||||||
|
if let Some((started, release)) = gate {
|
||||||
|
started.send(()).unwrap();
|
||||||
|
release.await.unwrap();
|
||||||
|
}
|
||||||
|
Ok(f)
|
||||||
|
}
|
||||||
|
async fn mkdir(&self, p: &str) -> Result<()> {
|
||||||
|
self.memory.mkdir(p).await
|
||||||
|
}
|
||||||
|
async fn remove_file(&self, p: &str) -> Result<()> {
|
||||||
|
self.memory.remove_file(p).await
|
||||||
|
}
|
||||||
|
async fn remove_dir(&self, p: &str) -> Result<()> {
|
||||||
|
self.memory.remove_dir(p).await
|
||||||
|
}
|
||||||
|
async fn rename(&self, a: &str, b: &str) -> Result<()> {
|
||||||
|
self.memory.rename(a, b).await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn blocked_io_does_not_stall_other_processes() {
|
||||||
|
let memory = Arc::new(Memory::default());
|
||||||
|
memory.mkdir("/slow").await.unwrap();
|
||||||
|
let options = Open {
|
||||||
|
create: true,
|
||||||
|
write: true,
|
||||||
|
..Open::default()
|
||||||
|
};
|
||||||
|
let raw = memory.open("/slow/file", options).await.unwrap();
|
||||||
|
let (started, opening) = tokio::sync::oneshot::channel();
|
||||||
|
let (release, released) = tokio::sync::oneshot::channel();
|
||||||
|
let (tx, mut rx) = mpsc::unbounded_channel();
|
||||||
|
let observer = Observer::new(move |p| {
|
||||||
|
tx.send(p.to_owned()).unwrap();
|
||||||
|
});
|
||||||
|
let slow = observer.layer(
|
||||||
|
Arc::new(SlowOpen {
|
||||||
|
memory: memory.clone(),
|
||||||
|
gate: Mutex::new(Some((started, released))),
|
||||||
|
}),
|
||||||
|
"/run",
|
||||||
|
Process::default(),
|
||||||
|
);
|
||||||
|
let other = observer.layer(memory, "/run", Process::default());
|
||||||
|
let opening_task = tokio::spawn(async move {
|
||||||
|
slow.open(
|
||||||
|
"/slow/file",
|
||||||
|
Open {
|
||||||
|
write: true,
|
||||||
|
..Open::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
opening.await.unwrap();
|
||||||
|
let renamer = other.clone();
|
||||||
|
let renamed = tokio::spawn(async move { renamer.rename("/slow", "/moved").await });
|
||||||
|
// These must complete while the first process's open is still blocked,
|
||||||
|
// and a conflicting rename is waiting to rebase its registered handle.
|
||||||
|
other
|
||||||
|
.open(
|
||||||
|
"/",
|
||||||
|
Open {
|
||||||
|
directory: true,
|
||||||
|
..Open::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let f = other.open("/other", options).await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/other");
|
||||||
|
f.write(0, b"visible".to_vec()).await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/other");
|
||||||
|
other.mkdir("/sibling").await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/sibling");
|
||||||
|
other.rename("/other", "/sibling/file").await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/other");
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/sibling/file");
|
||||||
|
other.remove_file("/sibling/file").await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/sibling/file");
|
||||||
|
assert!(!renamed.is_finished());
|
||||||
|
release.send(()).unwrap();
|
||||||
|
let opened = opening_task.await.unwrap().unwrap();
|
||||||
|
renamed.await.unwrap().unwrap();
|
||||||
|
rx.recv().await.unwrap();
|
||||||
|
rx.recv().await.unwrap();
|
||||||
|
opened.set_size(0).await.unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/moved/file");
|
||||||
|
|
||||||
|
// No namespace guard surrounds descriptor IO: a rename completes while
|
||||||
|
// this write is held in flight. Its notification uses the rebased path.
|
||||||
|
let handle = Handle {
|
||||||
|
file: raw.clone(),
|
||||||
|
path: observer.path("/run/moved/file".into()),
|
||||||
|
observer,
|
||||||
|
process: Process::default(),
|
||||||
|
};
|
||||||
|
let (started, writing) = tokio::sync::oneshot::channel();
|
||||||
|
let (release, released) = tokio::sync::oneshot::channel();
|
||||||
|
let write = tokio::spawn(async move {
|
||||||
|
handle
|
||||||
|
.mutate(async move {
|
||||||
|
started.send(()).unwrap();
|
||||||
|
released.await.unwrap();
|
||||||
|
raw.write(0, b"after rename".to_vec()).await
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
writing.await.unwrap();
|
||||||
|
other.rename("/moved", "/again").await.unwrap();
|
||||||
|
rx.recv().await.unwrap();
|
||||||
|
rx.recv().await.unwrap();
|
||||||
|
other.open("/again/file", Open::default()).await.unwrap();
|
||||||
|
release.send(()).unwrap();
|
||||||
|
write.await.unwrap().unwrap();
|
||||||
|
assert_eq!(rx.recv().await.unwrap(), "/run/again/file");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Killing a waiting writer does not discard observation of a completed mutation;
|
||||||
|
/// the process's idle barrier includes the callback, not only the system call.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn cancellation_keeps_mutation_and_observation() {
|
||||||
|
let (tx, mut rx) = mpsc::unbounded_channel();
|
||||||
|
let observer = Observer::new(move |p| {
|
||||||
|
tx.send(p.to_owned()).unwrap();
|
||||||
|
});
|
||||||
|
let process = Process::default();
|
||||||
|
let layer = observer.layer(Arc::new(Memory::default()), "/", process.clone());
|
||||||
|
let order = observer.order(&[("/published", true)]).await;
|
||||||
|
let (started, ready) = tokio::sync::oneshot::channel();
|
||||||
|
let p = process.clone();
|
||||||
|
let task = tokio::spawn(async move {
|
||||||
|
// Signal from inside the tracked operation, not before its guard exists.
|
||||||
|
p.operation(async move {
|
||||||
|
started.send(()).unwrap();
|
||||||
|
layer
|
||||||
|
.open(
|
||||||
|
"/published",
|
||||||
|
Open {
|
||||||
|
create: true,
|
||||||
|
write: true,
|
||||||
|
..Open::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
ready.await.unwrap();
|
||||||
|
task.abort();
|
||||||
|
let _ = task.await;
|
||||||
|
drop(order);
|
||||||
|
process.idle().await;
|
||||||
|
assert_eq!(rx.try_recv().unwrap(), "/published");
|
||||||
|
}
|
||||||
|
}
|
||||||
+16
-1
@@ -36,7 +36,22 @@ impl Process {
|
|||||||
.await
|
.await
|
||||||
.map_err(std::io::Error::other)?
|
.map_err(std::io::Error::other)?
|
||||||
}
|
}
|
||||||
/// Wait for all outstanding system calls.
|
/// Complete an asynchronous mutation and its observation even if the guest is cancelled.
|
||||||
|
/// Its guard participates in idle(), so no mutation outlives process cleanup.
|
||||||
|
pub async fn operation<T: Send + 'static>(
|
||||||
|
&self,
|
||||||
|
operation: impl std::future::Future<Output = std::io::Result<T>> + Send + 'static,
|
||||||
|
) -> std::io::Result<T> {
|
||||||
|
self.0.count.fetch_add(1, Ordering::SeqCst);
|
||||||
|
let guard = Guard(self.0.clone());
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let _guard = guard;
|
||||||
|
operation.await
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.map_err(std::io::Error::other)?
|
||||||
|
}
|
||||||
|
/// Wait for all outstanding system calls and observed mutations.
|
||||||
/// First abort the guest task. Then await its JoinHandle, or drop its store.
|
/// First abort the guest task. Then await its JoinHandle, or drop its store.
|
||||||
/// Only then await idle before releasing locks. Abort alone does not stop a running poll.
|
/// Only then await idle before releasing locks. Abort alone does not stop a running poll.
|
||||||
/// This ordering also prevents later memory writes, which need no blocking guard.
|
/// This ordering also prevents later memory writes, which need no blocking guard.
|
||||||
|
|||||||
Reference in New Issue
Block a user