Compare commits

...
10 Commits
Author SHA1 Message Date
Sausage ee88adf6c6 Remove private-world pointer from service timing comment
ci / check (push) Failing after 3m18s
2026-11-01 22:17:04 +00:00
Sausage 6dc21cd8dc Remove card labels from reusable cargo deadline tests 2026-11-01 22:17:04 +00:00
Sausage 5be5b3b397 Clean up dropped SDK calls and make drain assertions order-independent 2026-11-01 22:17:04 +00:00
Sausage a95b5287cf Add armed file watches without serializing unrelated IO 2026-11-01 22:17:04 +00:00
Sausage 959ddf46ff Supervise boot services with uniform five-second upgrade grace 2026-11-01 20:51:08 +00:00
Sausage 4da8a28392 Notify superseded plugin processes and support SDK draining 2026-11-01 20:51:08 +00:00
SAU-317 implementer c54edea596 Name the received JSON type in non-object params errors
ci / check (push) Failing after 2m38s
Keep -32602 and existing validation order; tell callers to send the object itself.
Cover host routing, stdout fast paths, SDK calls and generated dispatch.

Card: SAU-317
2026-10-28 19:16:39 +00:00
Sausage Worker 9ef73e6bc7 Use the world SDK for cargo builds and seed it only when missing 2026-10-28 08:39:36 +00:00
SAU-249 implementer dc81f78c20 Add shared SQLite open macro with WASI crash recovery and in-memory temp storage
ci / check (push) Failing after 3m53s
Card: SAU-249
2026-10-28 00:33:33 +00:00
SAU-158 implementer 9a8c143856 Return compact activation summaries instead of discovery schemas
ci / check (push) Failing after 2m40s
Card: SAU-158
2026-10-26 22:24:43 +00:00
39 changed files with 3359 additions and 225 deletions
+101 -13
View File
@@ -19,7 +19,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. | | call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. | | interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. | | endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
| service | A call in `/config/boot.yaml`. The core starts it at boot and again after an upgrade of its plugin. | | service | A call in `/config/boot.yaml`. The core starts it at boot, restarts it after a crash, and starts it again after an upgrade of its plugin. |
| import | An interface of another plugin that a plugin uses. | | import | An interface of another plugin that a plugin uses. |
| provider, consumer | The plugin that has an interface, and a plugin that imports it. | | provider, consumer | The plugin that has an interface, and a plugin that imports it. |
| journal | The central log of the world. | | journal | The central log of the world. |
@@ -70,7 +70,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
`notes.rpc.discover` sends it `rpc.discover`. `notes.rpc.discover` sends it `rpc.discover`.
- `core.<method>` is a method of the core (section 9). - `core.<method>` is a method of the core (section 9).
- Method names that start with `rpc.` are reserved: `rpc.discover`, - Method names that start with `rpc.` are reserved: `rpc.discover`,
`rpc.retired`, and `rpc.endpoint`. `rpc.superseded`, `rpc.retired`, and `rpc.endpoint`.
### 3.3 What a call does ### 3.3 What a call does
@@ -153,9 +153,17 @@ A process is an instance of one plugin version. When a new version becomes
active (section 10.3): active (section 10.3):
1. New calls get the new version at once. 1. New calls get the new version at once.
2. Each process of an older version gets a grace period to finish its work. 2. At grace start, the core sends each process of an older version the
The default is 60 seconds. A plugin can declare its own value in JSON-RPC notification `rpc.superseded {"current": "<sha256 of the new file>"}`
`rpc.discover` (`x-sausage-grace`). Then the core stops the process. on stdin, without an ID. The process is still running: calls and responses
continue to work. It can stop accepting work, finish its in-flight calls,
and exit early. Ignoring the notice changes nothing. The grace period is
the upper bound before the core stops the process (section 3.6); its
duration is 5 seconds for every plugin. Discovery metadata, including an
explicit `x-sausage-grace`, does not change it. After grace, stdin closes
and the separate 5-second stop budget applies (section 3.6): at most two
5-second windows. Discovery-only processes do not receive this notice or
enter upgrade grace; they follow the discovery lifetime (section 10.1).
3. When the old process of a service has ended, the core starts the service 3. When the old process of a service has ended, the core starts the service
again with the new version. again with the new version.
4. When the last process of an older version has ended, the core starts a 4. When the last process of an older version has ended, the core starts a
@@ -241,6 +249,40 @@ Rules:
Principle: **inspect through files, act through calls.** Principle: **inspect through files, act through calls.**
### File watching
`core.watch {"path": "/state/session/mailbox"}` stays open for the caller
process's lifetime. It needs a request ID and a running plugin caller.
Its first `core.notify` is `"armed"`, with empty params. Only then scan the
files. Writes before arming are visible in that scan; writes after arming
produce `"changed"` notifications (also with empty params). Rescan after
each notification; files, not event counts, are the truth.
Notifications follow successful plugin-originated file mutations: creation,
truncation, descriptor/stream writes, rename (both source and destination),
and deletion, including mounted `/run` and `/self` files. Paths are normalized
in the virtual namespace; `/self` means the caller's `/proc/<pid>` folder.
A watch matches itself and descendants, not sibling prefixes. Moving or
removing an ancestor also invalidates it. Paths need not exist when armed.
Open file writes follow plugin renames; unlinked/replaced handles no longer
name visible files. External host writes are outside this contract.
Changes may coalesce, including a change already visible in the initial
scan, but a change after the last notification always produces another.
Delivery holds no filesystem locks. Observation does not serialize descriptor
IO; only overlapping opens and rename/removal wait for one another.
Watch the narrowest useful path: delivery is not rate-limited, so watching a
busy parent can cause a rescan for every write.
The host-side watch ends only when its caller process exits, not on a change
notification; there is no cancellation API or polling/retry loop. SDK callers
should poll the watch future alongside their handler's work and drop it on
return, not keep it in a background task that prevents an idle process from
exiting. Dropping the future removes the SDK's pending entry and callback, so
late notifications and responses are ignored; it does not cancel the host
watch. A long-lived service keeps one watch per purpose for its lifetime,
rather than repeatedly opening and dropping watches.
## 5. Endpoints ## 5. Endpoints
An endpoint is a JSON-RPC interface that a running process serves over TCP, An endpoint is a JSON-RPC interface that a running process serves over TCP,
@@ -317,10 +359,29 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
- A service is a detached call with the parent `core`. The core waits for - A service is a detached call with the parent `core`. The core waits for
its result before it starts the next one. The result shows that the its result before it starts the next one. The result shows that the
service is ready. service is ready.
- If a service fails, the core writes the error to the journal and continues. - If a service fails before answering, the core writes the error to the
- After an upgrade of the plugin of a service, the core starts the service journal and continues booting. A failure to admit a process (for example,
again with the new version (section 3.7). a missing plugin or invalid parameters) is logged, not retried in a loop.
- The core does not restart other detached processes. - The core supervises the process lifetime, not its readiness response. A
nonzero exit, trap or failed host task restarts the same service with its
original method and parameters, including a crash before readiness. A
readiness response alone never triggers a restart; a clean exit does not
count as a crash or trigger an immediate restart.
- More than 5 crashes within 10 seconds gives up on that service and writes
`service gave up` to the journal. Each service has an independent sliding
window; crashes exactly 10 seconds old expire. There is no retry delay or
periodic liveness check. Giving up is latched for this host lifetime:
neither elapsed time nor an upgrade re-arms it.
- After an upgrade of the plugin of a service, the same supervisor starts
it again with the current version after its old process ends (section
3.7). Superseded exits are retirement, not crashes. Admission and cleanup
reserve a single owner/attempt, so an upgrade racing a crash cannot start
two owners. A matching boot-service start from `rpc.retired` is ignored
(returns `null`), even while restart is pending or after giving up; older
plugins' fallback starts do not bypass supervision.
- An explicit `core.stop` or `core.kill` disables automatic attempts for
that service, including during an upgrade. Host shutdown starts no new
processes. The core does not restart other detached processes.
- A singleton is a service with a fixed endpoint in `/run`. Only the core - A singleton is a service with a fixed endpoint in `/run`. Only the core
starts it, so two instances cannot start at the same time. starts it, so two instances cannot start at the same time.
@@ -329,12 +390,13 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
| Method | Params | Result | | Method | Params | Result |
| --- | --- | --- | | --- | --- | --- |
| `core.call` | `method`, `params`, `detach`? | the result of the method | | `core.call` | `method`, `params`, `detach`? | the result of the method |
| `core.watch` | `path` | stays open; `armed`, then `changed` notifications (section 4) |
| `core.notify` | `name`, `params` | none (a notification) | | `core.notify` | `name`, `params` | none (a notification) |
| `core.stop` | `pid` | `{}` when the process has ended | | `core.stop` | `pid` | `{}` when the process has ended |
| `core.kill` | `pid` | `{}` | | `core.kill` | `pid` | `{}` |
| `core.lock` | `path`, `wait`? | `{"held": bool}` | | `core.lock` | `path`, `wait`? | `{"held": bool}` |
| `core.unlock` | `path` | none (a notification) | | `core.unlock` | `path` | none (a notification) |
| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents | | `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | short activation summaries, keyed by plugin |
| `core.revert` | `plugin`, `force`? | like `core.activate` | | `core.revert` | `plugin`, `force`? | like `core.activate` |
| `core.log` | filters (section 7) | journal entries | | `core.log` | filters (section 7) | journal entries |
| `core.restart` | none | `{}`. With `SAUSAGE_LOOP=1`, schedule clean shutdown and exit code 75. | | `core.restart` | none | `{}`. With `SAUSAGE_LOOP=1`, schedule clean shutdown and exit code 75. |
@@ -386,7 +448,8 @@ Params are by name. The document can have these extension fields:
`methods`. `methods`.
- `x-sausage-imports`: for each provider, the part of its document that the - `x-sausage-imports`: for each provider, the part of its document that the
plugin uses. plugin uses.
- `x-sausage-grace`: the grace period in seconds. - `x-sausage-grace`: legacy grace declaration in seconds. The core ignores
this field; upgrade grace is always 5 seconds (section 3.7).
The schema of params describes what the provider can read. The schema of a The schema of params describes what the provider can read. The schema of a
result describes what the provider writes. Descriptions are part of the result describes what the provider writes. Descriptions are part of the
@@ -434,6 +497,26 @@ imported method:
If a check fails, nothing changes, and the error -32003 lists each problem, If a check fails, nothing changes, and the error -32003 lists each problem,
with a counterexample where possible. with a counterexample where possible.
The result is an object keyed by plugin name. Each value contains:
- `plugin`: the plugin name.
- `wasm`: the installed world path (`/plugins/<name>/plugin.wasm`).
- `hash`: the SHA-256 of the installed copy.
- `previous`: the replaced version's SHA-256, or `null` on first installation.
For an identical-hash no-op, this is the current hash.
- `method_count`: the number of methods in the new main interface.
- `added`, `removed`, `changed`: sorted method-name arrays compared with the
previous discovery document. A change to a method description or any
transitively referenced local schema counts as changed. Unrelated schemas
and endpoint interfaces do not affect these arrays. With no previous
document, all methods are added.
- `interface_check`: `"passed"` after successful checks, or `"skipped"` with
`force: true` (not a claim of compatibility).
The result does not include schemas. The full document remains available
through `<plugin>.rpc.discover` and `/proc/core/plugins/<name>/discover.json`.
An identical-hash no-op and `core.revert` return the same summary shape.
- The core runs one activation at a time. - The core runs one activation at a time.
- `plugins: [...]` activates several plugins together. The checks then use - `plugins: [...]` activates several plugins together. The checks then use
the new set. the new set.
@@ -496,8 +579,13 @@ builds `/plugins/notes/source/`:
its caller ends (under the normal process stopping deadline). its caller ends (under the normal process stopping deadline).
- A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal - A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal
entries, like other calls. entries, like other calls.
- `/plugins/cargo/sdk/` shows the source of the SDK. The dependency - `/plugins/cargo/sdk/` is the SDK used by builds. The dependency
`sausage-plugin = "0.1"` points to the same files in the host's cache. `sausage-plugin = "0.1"` points directly to this folder, including its macros
and interface crate. Edits apply to the next build without a host restart.
- The host seeds a missing SDK folder from its embedded copy at startup or
before a build. An existing folder is never overwritten or repaired, even
after a host upgrade. To reset to the host's bundled SDK, remove the whole
folder; the next startup or build re-seeds it. Back up local edits first.
- Compiled crates stay in the host's cache, outside the world, and all worlds - Compiled crates stay in the host's cache, outside the world, and all worlds
share them. So a second build compiles only what changed. share them. So a second build compiles only what changed.
- Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2` - Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2`
+13 -1
View File
@@ -93,10 +93,22 @@ plugins/<name>/plugin.wasm the current version of a plugin
plugins/<name>/source/ its source plugins/<name>/source/ its source
state/<plugin>/ the state of a plugin, by convention state/<plugin>/ the state of a plugin, by convention
state/core/journal.sqlite the journal: stderr of all processes, and core events state/core/journal.sqlite the journal: stderr of all processes, and core events
config/boot.yaml the services that start with the world config/boot.yaml ordered boot services, supervised for crashes (PROTOCOL §8)
config/agent.yaml the configuration of the agent config/agent.yaml the configuration of the agent
``` ```
Boot services restart on process crashes, not on readiness responses. More
than 5 crashes within 10 seconds stops automatic attempts for that service
and journals `service gave up`; an upgrade does not reset that decision.
Explicitly stopping or killing a service also leaves it stopped for this
host lifetime. See [Boot and services](PROTOCOL.md#8-boot-and-services).
Rust builds use the SDK in `plugins/cargo/sdk/` directly. The host seeds this
folder only when it is missing; restarts and host upgrades preserve existing
files. SDK edits take effect on the next `cargo.build`, without a restart.
To reset to the host's bundled SDK, back up any edits and remove the whole
folder; the next startup or build re-seeds it.
Each process also sees `/proc` (the processes and the state of the core), Each process also sees `/proc` (the processes and the state of the core),
`/self` (its own folder in `/proc`), and `/run` (shared files in memory). `/self` (its own folder in `/proc`), and `/run` (shared files in memory).
+94 -13
View File
@@ -6,6 +6,44 @@ use crate::{
use serde_json::{Value, json}; use serde_json::{Value, json};
use std::{collections::BTreeMap, path::PathBuf}; use std::{collections::BTreeMap, path::PathBuf};
/// Method descriptions plus their reachable local references, without expanding cycles.
fn methods(document: &Value) -> BTreeMap<String, Value> {
fn references(value: &Value, document: &Value, found: &mut BTreeMap<String, Value>) {
match value {
Value::Object(object) => {
if let Some(reference) = object.get("$ref").and_then(Value::as_str)
&& let Some(pointer) = reference.strip_prefix('#')
&& !found.contains_key(reference)
&& let Some(target) = document.pointer(pointer)
{
found.insert(reference.into(), target.clone());
references(target, document, found);
}
for child in object.values() {
references(child, document, found);
}
}
Value::Array(array) => {
for child in array {
references(child, document, found);
}
}
_ => {}
}
}
document["methods"]
.as_array()
.into_iter()
.flatten()
.filter_map(|method| {
let name = method["name"].as_str()?;
let mut found = BTreeMap::new();
references(method, document, &mut found);
Some((name.into(), json!({"method":method,"references":found})))
})
.collect()
}
struct Candidate { struct Candidate {
plugin: String, plugin: String,
copy: PathBuf, copy: PathBuf,
@@ -113,7 +151,31 @@ impl Host {
.map(|candidate| candidate.plugin.clone()) .map(|candidate| candidate.plugin.clone())
.collect(); .collect();
self.refresh_graph_except(&names).await?; self.refresh_graph_except(&names).await?;
let previous = self.0.graph.lock().unwrap().plugins.clone(); let mut previous = self.0.graph.lock().unwrap().plugins.clone();
// Startup discovery may not yet have visited a replaced plugin. Discover
// its immutable old copy for the diff, but never prevent repairing it.
for candidate in candidates.iter() {
if let Some(old) = previous.get_mut(&candidate.plugin)
&& old.document.is_null()
&& old.error.is_none()
{
if old.hash == candidate.interface.hash {
old.document = candidate.interface.document.clone();
continue;
}
let directory = self.0.world.plugin_dir(&candidate.plugin);
if let Ok((bytes, hash)) = read_hashed(directory.join("plugin.wasm")).await {
let copy = directory.join(format!("discover-{}.wasm", uuid::Uuid::now_v7()));
if tokio::fs::write(&copy, bytes).await.is_ok()
&& let Ok(interface) =
self.discover_copy(&candidate.plugin, &copy, hash).await
{
*old = interface;
}
let _ = tokio::fs::remove_file(copy).await;
}
}
}
let mut proposed = previous.clone(); let mut proposed = previous.clone();
for candidate in candidates.iter() { for candidate in candidates.iter() {
proposed.insert(candidate.plugin.clone(), candidate.interface.clone()); proposed.insert(candidate.plugin.clone(), candidate.interface.clone());
@@ -148,7 +210,7 @@ impl Host {
); );
} }
} }
self.install_candidates(candidates, previous, proposed) self.install_candidates(candidates, previous, proposed, params["force"] != true)
.await .await
} }
/// Installs staged files and snapshots all old admissions under the shared write gate. /// Installs staged files and snapshots all old admissions under the shared write gate.
@@ -157,6 +219,7 @@ impl Host {
candidates: &[Candidate], candidates: &[Candidate],
previous: BTreeMap<String, Interface>, previous: BTreeMap<String, Interface>,
proposed: BTreeMap<String, Interface>, proposed: BTreeMap<String, Interface>,
checked: bool,
) -> RpcResult { ) -> RpcResult {
let admission = self.0.admission.write().await; let admission = self.0.admission.write().await;
// Every candidate already has a temporary name before any installed file changes. // Every candidate already has a temporary name before any installed file changes.
@@ -183,25 +246,43 @@ impl Host {
.map_err(io_error)?; .map_err(io_error)?;
} }
let mut upgrades = Vec::new(); let mut upgrades = Vec::new();
let mut documents = serde_json::Map::new(); let mut summaries = serde_json::Map::new();
for candidate in candidates { for candidate in candidates {
documents.insert( let old = previous.get(&candidate.plugin);
candidate.plugin.clone(), let before = methods(old.map(|old| &old.document).unwrap_or(&Value::Null));
candidate.interface.document.clone(), let after = methods(&candidate.interface.document);
); let added: Vec<_> = after
.keys()
.filter(|name| !before.contains_key(*name))
.collect();
let removed: Vec<_> = before
.keys()
.filter(|name| !after.contains_key(*name))
.collect();
let changed: Vec<_> = after
.iter()
.filter(|(name, method)| before.get(*name).is_some_and(|old| old != *method))
.map(|(name, _)| name)
.collect();
summaries.insert(candidate.plugin.clone(), json!({
"plugin": candidate.plugin,
"wasm": format!("/plugins/{}/plugin.wasm", candidate.plugin),
"hash": candidate.interface.hash,
"previous": old.map(|old| &old.hash).filter(|hash| !hash.is_empty()),
"method_count": candidate.interface.document["methods"].as_array().map_or(0, Vec::len),
"added": added,
"removed": removed,
"changed": changed,
"interface_check": if checked { "passed" } else { "skipped" },
}));
if let Some(old) = previous if let Some(old) = previous
.get(&candidate.plugin) .get(&candidate.plugin)
.filter(|old| !old.hash.is_empty() && old.hash != candidate.interface.hash) .filter(|old| !old.hash.is_empty() && old.hash != candidate.interface.hash)
{ {
let grace = old.document["x-sausage-grace"]
.as_f64()
.filter(|value| value.is_finite() && *value >= 0.0)
.unwrap_or(60.0);
upgrades.push(self.snapshot_upgrade( upgrades.push(self.snapshot_upgrade(
&candidate.plugin, &candidate.plugin,
old.hash.clone(), old.hash.clone(),
&candidate.interface.hash, &candidate.interface.hash,
grace,
)); ));
} }
self.0.journal.write( self.0.journal.write(
@@ -216,6 +297,6 @@ impl Host {
for upgrade in upgrades { for upgrade in upgrades {
self.begin_upgrade(upgrade); self.begin_upgrade(upgrade);
} }
Ok(Value::Object(documents)) Ok(Value::Object(summaries))
} }
} }
+33 -43
View File
@@ -3,7 +3,6 @@ use crate::{Host, world::World};
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
use include_dir::{Dir, include_dir}; use include_dir::{Dir, include_dir};
use serde_json::{Value, json}; use serde_json::{Value, json};
use sha2::{Digest, Sha256};
use std::time::Duration; use std::time::Duration;
use std::{ use std::{
path::{Path, PathBuf}, path::{Path, PathBuf},
@@ -169,10 +168,9 @@ pub struct Cargo {
runner: PathBuf, runner: PathBuf,
target: PathBuf, target: PathBuf,
sdk: PathBuf, sdk: PathBuf,
sdk_files: Arc<std::collections::BTreeMap<PathBuf, Vec<u8>>>,
} }
impl Cargo { impl Cargo {
/// Installs changed SDK files and selects the current sausage executable as runner. /// Seeds a missing world SDK and selects the current sausage executable as runner.
pub fn new(world: World, cache: &Path) -> Result<Self> { pub fn new(world: World, cache: &Path) -> Result<Self> {
let mut runner = std::env::current_exe()?; let mut runner = std::env::current_exe()?;
if runner if runner
@@ -183,9 +181,23 @@ impl Cargo {
} }
Self::with_runner(world, runner, cache) Self::with_runner(world, runner, cache)
} }
/// Installs the SDK and accepts an explicit sausage binary, including for init integrations. /// Seeds a missing world SDK and accepts an explicit sausage binary, including for init integrations.
pub fn with_runner(world: World, runner: PathBuf, cache: &Path) -> Result<Self> { pub fn with_runner(world: World, runner: PathBuf, cache: &Path) -> Result<Self> {
let sdk = world.plugin_dir(ID).join("sdk"); let sdk = world.plugin_dir(ID).join("sdk");
Self::seed_sdk(&sdk)?;
Ok(Self {
sdk,
target: std::path::absolute(cache)?.join("cargo-target"),
world,
runner,
builds: Arc::new(tokio::sync::Mutex::new(())),
})
}
/// Seeds only an absent SDK; an existing world owns all of its SDK files.
fn seed_sdk(sdk: &Path) -> Result<()> {
if sdk.try_exists()? {
return Ok(());
}
let mut files = std::collections::BTreeMap::new(); let mut files = std::collections::BTreeMap::new();
collect_files(&SDK, Path::new("src"), &mut files); collect_files(&SDK, Path::new("src"), &mut files);
collect_files(&MACROS, Path::new("macros/src"), &mut files); collect_files(&MACROS, Path::new("macros/src"), &mut files);
@@ -223,26 +235,23 @@ impl Cargo {
} }
files.insert(destination, document.to_string().into_bytes()); files.insert(destination, document.to_string().into_bytes());
} }
let mut hash = Sha256::new(); let parent = sdk.parent().context("SDK has no parent")?;
std::fs::create_dir_all(parent).with_context(|| format!("create {}", parent.display()))?;
let staging = tempfile::Builder::new()
.prefix("seed-sdk-")
.tempdir_in(parent)
.with_context(|| format!("stage SDK in {}", parent.display()))?;
for (path, bytes) in &files { for (path, bytes) in &files {
let name = path.to_str().unwrap().as_bytes(); write_changed(&staging.path().join(path), bytes)?;
hash.update((name.len() as u64).to_le_bytes());
hash.update(name);
hash.update((bytes.len() as u64).to_le_bytes());
hash.update(bytes);
write_changed(&sdk.join(path), bytes)?;
} }
let cached = std::path::absolute(cache)? // Another builder may have seeded the world while we prepared the files.
.join("sdk") if !sdk.try_exists()?
.join(format!("{:x}", hash.finalize())); && let Err(error) = std::fs::rename(staging.path(), sdk)
Ok(Self { && !sdk.try_exists()?
sdk: cached, {
sdk_files: Arc::new(files), return Err(error).with_context(|| format!("seed {}", sdk.display()));
target: std::path::absolute(cache)?.join("cargo-target"), }
world, Ok(())
runner,
builds: Arc::new(tokio::sync::Mutex::new(())),
})
} }
/// Registers cargo.build as a cancellable native process. /// Registers cargo.build as a cancellable native process.
pub fn register(self, host: &Host) { pub fn register(self, host: &Host) {
@@ -310,27 +319,8 @@ impl Cargo {
let source = source.to_path_buf(); let source = source.to_path_buf();
let copy = work.clone(); let copy = work.clone();
let sdk = self.sdk.clone(); let sdk = self.sdk.clone();
let sdk_files = self.sdk_files.clone();
let build_guard = tokio::task::spawn_blocking(move || -> Result<_> { let build_guard = tokio::task::spawn_blocking(move || -> Result<_> {
// Host startup installs only the readable world copy. Builds populate the cache. Self::seed_sdk(&sdk)?;
if !sdk.is_dir() {
let parent = sdk.parent().unwrap();
std::fs::create_dir_all(parent)
.with_context(|| format!("create {}", parent.display()))?;
let staging = tempfile::Builder::new()
.prefix("install-")
.tempdir_in(parent)
.with_context(|| format!("stage SDK in {}", parent.display()))?;
for (path, bytes) in sdk_files.iter() {
write_changed(&staging.path().join(path), bytes)?;
}
// Another builder can publish the same complete SDK first.
if let Err(error) = std::fs::rename(staging.path(), &sdk)
&& !sdk.is_dir()
{
return Err(error).with_context(|| format!("install {}", sdk.display()));
}
}
if cleanup.exists() { if cleanup.exists() {
std::fs::remove_dir_all(&cleanup) std::fs::remove_dir_all(&cleanup)
.with_context(|| format!("remove {}", cleanup.display()))?; .with_context(|| format!("remove {}", cleanup.display()))?;
@@ -659,7 +649,7 @@ fn write_changed(path: &Path, bytes: &[u8]) -> Result<()> {
} }
Ok(()) Ok(())
} }
/// Collects embedded source files for deterministic hashing and installation. /// Collects embedded source files for seeding a missing world SDK.
fn collect_files( fn collect_files(
directory: &Dir<'_>, directory: &Dir<'_>,
destination: &Path, destination: &Path,
+1
View File
@@ -12,6 +12,7 @@ impl Host {
p: Value, p: Value,
) -> RpcResult { ) -> RpcResult {
match method { match method {
"watch" => self.watch(caller, id, required_str(&p, "path")?).await,
"call" => { "call" => {
self.route( self.route(
caller.into(), caller.into(),
+20 -3
View File
@@ -47,23 +47,40 @@ impl Host {
tracker: Process, tracker: Process,
) -> anyhow::Result<Mounts> { ) -> anyhow::Result<Mounts> {
let root = self.0.world.root.clone(); let root = self.0.world.root.clone();
let operations = tracker.clone();
let host = tracker let host = tracker
.clone() .clone()
.blocking(move || sausage_vfs::Host::new(root, tracker)) .blocking(move || sausage_vfs::Host::new(root, tracker))
.await?; .await?;
let mut mounts = Mounts::default(); let mut mounts = Mounts::default();
mounts.mount("/", Arc::new(host), false); mounts.mount(
"/",
self.0
.observer
.layer(Arc::new(host), "/", operations.clone()),
false,
);
mounts.mount("/proc", Arc::new(Proc(self.clone())), true); mounts.mount("/proc", Arc::new(Proc(self.clone())), true);
mounts.mount( mounts.mount(
"/self", "/self",
self.0.observer.layer(
Arc::new(SelfFolder { Arc::new(SelfFolder {
host: self.clone(), host: self.clone(),
pid, pid: pid.clone(),
memory, memory,
}), }),
&format!("/proc/{}", pid.as_deref().unwrap_or("console")),
operations.clone(),
),
false,
);
mounts.mount(
"/run",
self.0
.observer
.layer(Arc::new(self.0.run.clone()), "/run", operations),
false, false,
); );
mounts.mount("/run", Arc::new(self.0.run.clone()), false);
Ok(mounts) Ok(mounts)
} }
fn core_files(&self) -> Generated { fn core_files(&self) -> Generated {
+1 -10
View File
@@ -223,16 +223,7 @@ impl Host {
if let Some(old) = old_plugins.get(plugin).filter(|old| { if let Some(old) = old_plugins.get(plugin).filter(|old| {
!old.hash.is_empty() && old.hash != current.hash && current.error.is_none() !old.hash.is_empty() && old.hash != current.hash && current.error.is_none()
}) { }) {
let grace = old.document["x-sausage-grace"] upgrades.push(self.snapshot_upgrade(plugin, old.hash.clone(), &current.hash));
.as_f64()
.filter(|value| value.is_finite() && *value >= 0.0)
.unwrap_or(60.0);
upgrades.push(self.snapshot_upgrade(
plugin,
old.hash.clone(),
&current.hash,
grace,
));
} }
} }
self.publish_graph(plugins); self.publish_graph(plugins);
+8
View File
@@ -37,6 +37,8 @@ pub(crate) struct Inner {
/// processes, and upgrade tasks do nothing. /// processes, and upgrade tasks do nothing.
pub(crate) shutdown: std::sync::atomic::AtomicBool, pub(crate) shutdown: std::sync::atomic::AtomicBool,
pub(crate) run: sausage_vfs::Memory, pub(crate) run: sausage_vfs::Memory,
pub(crate) watches: crate::watches::Watches,
pub(crate) observer: sausage_vfs::observe::Observer,
pub(crate) graph: Mutex<crate::graph::Graph>, pub(crate) graph: Mutex<crate::graph::Graph>,
pub(crate) activations: tokio::sync::Mutex<()>, pub(crate) activations: tokio::sync::Mutex<()>,
pub(crate) watcher: Mutex<Option<notify::RecommendedWatcher>>, pub(crate) watcher: Mutex<Option<notify::RecommendedWatcher>>,
@@ -91,6 +93,10 @@ impl Host {
)) ))
} }
fn with_parts(world: World, runtime: Arc<Runtime>, journal: journal::Journal) -> Self { fn with_parts(world: World, runtime: Arc<Runtime>, journal: journal::Journal) -> Self {
let watches = crate::watches::Watches::default();
let events = watches.clone();
let observer =
sausage_vfs::observe::Observer::new(move |path| crate::watches::changed(&events, path));
Self(Arc::new(Inner { Self(Arc::new(Inner {
world, world,
runtime, runtime,
@@ -109,6 +115,8 @@ impl Host {
admission: tokio::sync::RwLock::new(()), admission: tokio::sync::RwLock::new(()),
shutdown: std::sync::atomic::AtomicBool::new(false), shutdown: std::sync::atomic::AtomicBool::new(false),
run: sausage_vfs::Memory::default(), run: sausage_vfs::Memory::default(),
watches,
observer,
graph: Mutex::new(crate::graph::Graph::default()), graph: Mutex::new(crate::graph::Graph::default()),
activations: tokio::sync::Mutex::new(()), activations: tokio::sync::Mutex::new(()),
watcher: Mutex::new(None), watcher: Mutex::new(None),
+1
View File
@@ -18,6 +18,7 @@ pub mod runtime;
mod services; mod services;
mod upgrades; mod upgrades;
mod watcher; mod watcher;
mod watches;
pub mod world; pub mod world;
pub use host::Host; pub use host::Host;
pub use native::{NativeContext, NativeFuture, NativePlugin}; pub use native::{NativeContext, NativeFuture, NativePlugin};
+22 -1
View File
@@ -33,6 +33,8 @@ pub(crate) struct Process {
pub(crate) id: Option<Value>, pub(crate) id: Option<Value>,
pub(crate) answer: Option<oneshot::Sender<RpcResult>>, pub(crate) answer: Option<oneshot::Sender<RpcResult>>,
pub(crate) ended: watch::Sender<bool>, pub(crate) ended: watch::Sender<bool>,
pub(crate) stopped: bool,
pub(crate) superseded: bool,
} }
impl Host { impl Host {
/// Called only after the execution task ends. Wait for the store and file operations before releasing locks. /// Called only after the execution task ends. Wait for the store and file operations before releasing locks.
@@ -57,6 +59,7 @@ impl Host {
.map(|_| self.0.journal.query_later(json!({"pid":pid,"limit":20}))); .map(|_| self.0.journal.query_later(json!({"pid":pid,"limit":20})));
self.0.journal.forget(pid); self.0.journal.forget(pid);
Self::release_locks(&mut state, pid); Self::release_locks(&mut state, pid);
self.service_ended(&mut state, pid, status, p.stopped, p.superseded);
let status = status.to_string(); let status = status.to_string();
tokio::spawn(async move { tokio::spawn(async move {
if let Some(answer) = p.answer { if let Some(answer) = p.answer {
@@ -84,13 +87,24 @@ impl Host {
} }
/// Close stdin and start the same five-second deadline for each attached child. /// Close stdin and start the same five-second deadline for each attached child.
pub(crate) fn begin_stop(&self, pid: &str) -> Option<watch::Receiver<bool>> { pub(crate) fn begin_stop(&self, pid: &str) -> Option<watch::Receiver<bool>> {
self.stop_process(pid, false)
}
/// Grace expiry is retirement, not an operator stopping the service.
pub(crate) fn begin_retire(&self, pid: &str) -> Option<watch::Receiver<bool>> {
self.stop_process(pid, true)
}
fn stop_process(&self, pid: &str, retirement: bool) -> Option<watch::Receiver<bool>> {
let mut state = self.0.state.lock().unwrap(); let mut state = self.0.state.lock().unwrap();
let p = state.processes.get_mut(pid)?; let p = state.processes.get_mut(pid)?;
if !retirement {
p.superseded = false;
}
let ended = p.ended.subscribe(); let ended = p.ended.subscribe();
if p.info.state == "stopping" { if p.info.state == "stopping" {
return Some(ended); return Some(ended);
} }
p.info.state = "stopping".into(); p.info.state = "stopping".into();
p.stopped = true;
p.input.take(); p.input.take();
self.0.journal.write(pid, &p.info.plugin, "core", "stop"); self.0.journal.write(pid, &p.info.plugin, "core", "stop");
for lock in state.locks.values_mut() { for lock in state.locks.values_mut() {
@@ -110,17 +124,24 @@ impl Host {
let pid = pid.to_string(); let pid = pid.to_string();
tokio::spawn(async move { tokio::spawn(async move {
tokio::time::sleep(std::time::Duration::from_secs(5)).await; tokio::time::sleep(std::time::Duration::from_secs(5)).await;
h.kill(&pid); h.kill_process(&pid, retirement);
}); });
Some(ended) Some(ended)
} }
/// Abort a process task and perform its ended-state cleanup. /// Abort a process task and perform its ended-state cleanup.
pub(crate) fn kill(&self, pid: &str) { pub(crate) fn kill(&self, pid: &str) {
self.kill_process(pid, false);
}
fn kill_process(&self, pid: &str, retirement: bool) {
let mut state = self.0.state.lock().unwrap(); let mut state = self.0.state.lock().unwrap();
for lock in state.locks.values_mut() { for lock in state.locks.values_mut() {
lock.queue.retain(|(queued, _)| queued != pid); lock.queue.retain(|(queued, _)| queued != pid);
} }
if let Some(p) = state.processes.get_mut(pid) { if let Some(p) = state.processes.get_mut(pid) {
if !retirement {
p.superseded = false;
}
p.stopped = true;
p.info.state = "stopping".into(); p.info.state = "stopping".into();
p.input.take(); p.input.take();
self.0.journal.write(pid, &p.info.plugin, "core", "kill"); self.0.journal.write(pid, &p.info.plugin, "core", "kill");
+60 -9
View File
@@ -7,6 +7,23 @@ use serde_json::{Value, json};
use tokio::sync::{mpsc, oneshot, watch}; use tokio::sync::{mpsc, oneshot, watch};
use uuid::Uuid; use uuid::Uuid;
fn params_not_object(params: &Value) -> Value {
let kind = match params {
Value::String(_) => "a string",
Value::Array(_) => "an array",
Value::Number(_) => "a number",
Value::Null => "null",
Value::Bool(_) => "a boolean",
Value::Object(_) => "an object",
};
error(
-32602,
&format!(
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
),
)
}
impl Host { impl Host {
/// Queue an incoming response or notification for its destination. /// Queue an incoming response or notification for its destination.
pub(crate) fn send(&self, pid: &str, value: Value) { pub(crate) fn send(&self, pid: &str, value: Value) {
@@ -34,15 +51,20 @@ impl Host {
.processes .processes
.get(pid) .get(pid)
.and_then(|p| p.answer.as_ref().map(|_| (p.caller.clone(), p.id.clone()))); .and_then(|p| p.answer.as_ref().map(|_| (p.caller.clone(), p.id.clone())));
if let Some((caller, id)) = target { if let Some((caller, Some(id))) = target {
let notification = json!({ self.call_notify(&caller, &id, name, params);
"jsonrpc": "2.0",
"method": "core.notify",
"params": { "call": id, "name": name, "params": params }
});
self.send(&caller, notification);
} }
} }
/// Deliver a notification on an open core call using the same envelope as plugin progress.
pub(crate) fn call_notify(&self, caller: &str, id: &Value, name: &str, params: Value) {
self.send(
caller,
json!({
"jsonrpc":"2.0", "method":"core.notify",
"params": {"call":id, "name":name, "params":params}
}),
);
}
/// Validate and route a request, then start its native or WASI process. /// Validate and route a request, then start its native or WASI process.
pub(crate) fn route( pub(crate) fn route(
&self, &self,
@@ -69,7 +91,7 @@ impl Host {
let host = self.clone(); let host = self.clone();
Box::pin(async move { Box::pin(async move {
if !params.is_object() { if !params.is_object() {
return Err(error(-32602, "params must be an object")); return Err(params_not_object(&params));
} }
if caller != "console" && caller != "core" { if caller != "console" && caller != "core" {
let state = host.0.state.lock().unwrap(); let state = host.0.state.lock().unwrap();
@@ -149,6 +171,33 @@ impl Host {
{ {
return Err(error(-32600, "caller stopped or ended")); return Err(error(-32600, "caller stopped or ended"));
} }
// Older plugins may still try to start their boot service from
// rpc.retired. The supervisor owns it even while a restart is
// pending, or after it has given up. Do not create a second owner.
if state
.processes
.get(&caller)
.is_some_and(|p| p.info.method == "rpc.retired")
&& state
.services
.iter()
.any(|s| s.method == format!("{plugin}.{method}") && s.params == params)
{
return Ok(Value::Null);
}
if caller == "core"
&& let Some(service) = state
.services
.iter_mut()
.find(|s| Some(&s.request) == id.as_ref())
{
if !service.enabled || service.owner.is_some() {
return Err(error(-32600, "service attempt is no longer current"));
}
service.owner = Some(pid.clone());
service.pending = false;
service.version = Some(info.version.clone());
}
host.0 host.0
.journal .journal
.write(&pid, plugin, "core", &format!("start {method} {params}")); .write(&pid, plugin, "core", &format!("start {method} {params}"));
@@ -165,6 +214,8 @@ impl Host {
id: id.clone(), id: id.clone(),
answer: if id.is_some() { Some(tx) } else { None }, answer: if id.is_some() { Some(tx) } else { None },
ended: watch::channel(false).0, ended: watch::channel(false).0,
stopped: false,
superseded: false,
}, },
); );
} }
@@ -238,7 +289,7 @@ impl Host {
// Deliver progress before a following response removes its destination. // Deliver progress before a following response removes its destination.
let params = v.get("params").cloned().unwrap_or(json!({})); let params = v.get("params").cloned().unwrap_or(json!({}));
let result = if !params.is_object() { let result = if !params.is_object() {
Err(error(-32602, "params must be an object")) Err(params_not_object(&params))
} else if method == "core.unlock" { } else if method == "core.unlock" {
if let Some(path) = params["path"].as_str() { if let Some(path) = params["path"].as_str() {
self.unlock(pid, path); self.unlock(pid, path);
+129 -16
View File
@@ -1,12 +1,36 @@
//! Ordered boot services and their identities for upgrade restarts (PROTOCOL.md section 8). //! Ordered boot calls and the single restart owner for each service (PROTOCOL.md section 8).
use crate::Host; use crate::{Host, host::State};
use serde_json::{Value, json}; use serde_json::{Value, json};
#[derive(Clone)] use std::{collections::VecDeque, sync::atomic::Ordering, time::Duration};
use tokio::time::Instant;
// Systemd-derived values:
// https://www.man7.org/linux/man-pages/man5/systemd-system.conf.5.html.
const CRASH_LIMIT: usize = 5;
const CRASH_WINDOW: Duration = Duration::from_secs(10);
pub(crate) struct Service { pub(crate) struct Service {
pub(crate) method: String, pub(crate) method: String,
pub(crate) params: Value, pub(crate) params: Value,
pub(crate) request: Value, pub(crate) request: Value,
pub(crate) starting: std::sync::Arc<std::sync::atomic::AtomicBool>, pub(crate) owner: Option<String>,
pub(crate) pending: bool,
pub(crate) version: Option<String>,
pub(crate) enabled: bool,
crashes: VecDeque<Instant>,
}
impl Service {
fn crashed(&mut self, now: Instant) -> bool {
while self
.crashes
.front()
.is_some_and(|t| now.duration_since(*t) >= CRASH_WINDOW)
{
self.crashes.pop_front();
}
self.crashes.push_back(now);
self.crashes.len() <= CRASH_LIMIT
}
} }
impl Host { impl Host {
/// Starts discovery in the background, then waits for each boot service in order. /// Starts discovery in the background, then waits for each boot service in order.
@@ -24,30 +48,119 @@ impl Host {
} }
Ok(()) Ok(())
} }
/// Records a unique request identity before the service can answer or exit. /// Register once, before admission or readiness. Restarts reuse this registration.
pub(crate) async fn start_service(&self, method: String, params: Value) { pub(crate) async fn start_service(&self, method: String, params: Value) {
if self.0.shutdown.load(std::sync::atomic::Ordering::SeqCst) { let request = json!(uuid::Uuid::now_v7().to_string());
{
let mut state = self.0.state.lock().unwrap();
if self.0.shutdown.load(Ordering::SeqCst) {
return; return;
} }
let request = json!(uuid::Uuid::now_v7().to_string()); state.services.push(Service {
let starting = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(true)); method,
self.0.state.lock().unwrap().services.push(Service { params,
method: method.clone(),
params: params.clone(),
request: request.clone(), request: request.clone(),
starting: starting.clone(), owner: None,
enabled: true,
pending: true,
version: None,
crashes: VecDeque::new(),
}); });
}
self.run_service(request).await;
}
/// Await readiness only; process cleanup, not this response, decides whether to restart.
async fn run_service(&self, request: Value) {
let (method, params) = {
let state = self.0.state.lock().unwrap();
if self.0.shutdown.load(Ordering::SeqCst) {
return;
}
let Some(service) = state
.services
.iter()
.find(|s| s.request == request && s.enabled)
else {
return;
};
(service.method.clone(), service.params.clone())
};
self.0 self.0
.journal .journal
.write("core", "core", "core", &format!("service start {method}")); .write("core", "core", "core", &format!("service start {method}"));
if let Err(error) = self let result = self
.route("core".into(), Some(request), method, params, true) .route("core".into(), Some(request.clone()), method, params, true)
.await .await;
{ {
let mut state = self.0.state.lock().unwrap();
if let Some(service) = state.services.iter_mut().find(|s| s.request == request) {
service.pending = false;
}
}
if let Err(error) = result {
// Admission failures have no process lifetime to supervise. A process crash
// before readiness is handled by finish just like one after readiness.
self.0 self.0
.journal .journal
.write("core", "core", "core", &format!("service failed {error}")); .write("core", "core", "core", &format!("service failed {error}"));
} }
starting.store(false, std::sync::atomic::Ordering::SeqCst); }
/// Called under the state lock after the old owner and its files/locks are gone.
pub(crate) fn service_ended(
&self,
state: &mut State,
pid: &str,
status: &str,
stopped: bool,
superseded: bool,
) {
let Some(service) = state
.services
.iter_mut()
.find(|s| s.owner.as_deref() == Some(pid))
else {
return;
};
service.owner = None;
if self.0.shutdown.load(Ordering::SeqCst) || !service.enabled {
return;
}
if !superseded {
if stopped {
service.enabled = false;
return;
}
if status == "exit 0" {
return;
}
if !service.crashed(Instant::now()) {
service.enabled = false;
self.0.journal.write(
"core",
"core",
"core",
&format!(
"service gave up {}: more than {CRASH_LIMIT} crashes within {} seconds",
service.method,
CRASH_WINDOW.as_secs(),
),
);
return;
}
}
self.queue_service(service);
}
/// Reserve the next attempt before spawning it; upgrades and crashes share this owner.
pub(crate) fn queue_service(&self, service: &mut Service) {
service.request = json!(uuid::Uuid::now_v7().to_string());
service.pending = true;
let request = service.request.clone();
let host = self.clone();
tokio::spawn(async move {
host.run_service(request).await;
});
} }
} }
#[cfg(test)]
mod tests;
+356
View File
@@ -0,0 +1,356 @@
//! Deterministic service supervision tests: native tasks stand in for guest lifetimes.
use super::*;
use crate::{NativePlugin, runtime::Runtime};
use std::sync::Arc;
use tokio::sync::{mpsc, oneshot, watch};
struct Rig {
host: Host,
starts: mpsc::UnboundedReceiver<(String, oneshot::Sender<()>)>,
_world: tempfile::TempDir,
}
impl Rig {
async fn new() -> Self {
let world = tempfile::tempdir().unwrap();
let runtime = Runtime::new(world.path()).unwrap();
let host = Host::test_world(world.path(), runtime).await.unwrap();
let (tx, starts) = mpsc::unbounded_channel();
let plugin: NativePlugin = Arc::new(move |ctx, method, _| {
let tx = tx.clone();
Box::pin(async move {
if method != "serve" {
return Ok(Value::Null);
}
let (end, ended) = oneshot::channel();
tx.send((ctx.pid, end)).unwrap();
let _ = ended.await;
Ok(Value::Null)
})
});
host.register_native("demo", plugin.clone());
host.register_native("other", plugin);
Self {
host,
starts,
_world: world,
}
}
async fn start(&mut self, method: &str) -> (String, oneshot::Sender<()>) {
let host = self.host.clone();
let method = method.to_string();
tokio::spawn(async move {
host.start_service(method, json!({"kept":true})).await;
});
self.next().await
}
async fn next(&mut self) -> (String, oneshot::Sender<()>) {
// Reuse the host integration call bound (activation.rs::call).
tokio::time::timeout(Duration::from_secs(60), self.starts.recv())
.await
.unwrap()
.unwrap()
}
fn ended(&self, pid: &str) -> watch::Receiver<bool> {
self.host.0.state.lock().unwrap().processes[pid]
.ended
.subscribe()
}
async fn crash(&self, pid: &str) {
let mut ended = self.ended(pid);
// Abort execution without asking the supervisor for an intentional stop.
self.host.0.state.lock().unwrap().processes[pid]
.abort
.as_ref()
.unwrap()
.abort();
ended.wait_for(|done| *done).await.unwrap();
}
async fn clean(&self, pid: &str, end: oneshot::Sender<()>) {
let mut ended = self.ended(pid);
end.send(()).unwrap();
ended.wait_for(|done| *done).await.unwrap();
}
fn assert_single(&self, pid: &str, registrations: usize) {
let state = self.host.0.state.lock().unwrap();
assert_eq!(state.services.len(), registrations);
let p = &state.processes[pid];
assert_eq!(p.info.params, json!({"kept":true}));
assert!(p.info.detached);
assert_eq!(p.caller, "core");
assert_eq!(
state
.services
.iter()
.filter(|s| s.owner.as_deref() == Some(pid))
.count(),
1
);
}
fn no_start(&mut self) {
assert!(self.starts.try_recv().is_err());
}
}
#[tokio::test(start_paused = true)]
async fn crash_before_and_after_readiness_observes_process_not_response() {
let mut rig = Rig::new().await;
let (pid, _end) = rig.start("demo.serve").await;
rig.assert_single(&pid, 1);
rig.crash(&pid).await; // No ready RPC response was sent.
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
rig.host.answer(&pid, Ok(json!({"ready":true})));
tokio::task::yield_now().await;
rig.no_start(); // Ready does not end the service lifetime.
rig.crash(&pid).await;
drop(end);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
rig.clean(&pid, end).await;
rig.no_start(); // A clean, intentional return is not a crash.
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn fifth_restarts_sixth_gives_up_and_journals_once() {
let mut rig = Rig::new().await;
let (mut pid, mut end) = rig.start("demo.serve").await;
for crash in 1..=CRASH_LIMIT {
rig.crash(&pid).await;
drop(end);
(pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
assert_eq!(
rig.host.0.state.lock().unwrap().services[0].crashes.len(),
crash
);
}
rig.crash(&pid).await;
drop(end);
rig.no_start();
let logs = rig
.host
.call("core.log", json!({"grep":"service gave up"}))
.await
.unwrap();
assert_eq!(logs.as_array().unwrap().len(), 1);
assert!(
logs[0]["line"]
.as_str()
.unwrap()
.contains("more than 5 crashes within 10 seconds")
);
assert!(!rig.host.0.state.lock().unwrap().services[0].enabled);
// A retired hook cannot bypass give-up while its supervisor has no owner.
rig.host.register_native(
"fallback",
Arc::new(move |ctx, _, _| {
Box::pin(async move {
ctx.call(
"core.call",
json!({"method":"demo.serve","params":{"kept":true},"detach":true}),
)
.await
})
}),
);
assert_eq!(
rig.host
.call("fallback.rpc.retired", json!({}))
.await
.unwrap(),
Value::Null
);
rig.no_start();
// Neither time passing nor an upgrade re-arms a given-up service.
tokio::time::advance(CRASH_WINDOW).await;
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
rig.host.begin_upgrade(upgrade);
tokio::task::yield_now().await;
rig.no_start();
rig.host.stop().await;
}
#[test]
fn sliding_window_boundary_and_progress() {
let now = Instant::now();
let service = || Service {
method: "demo.serve".into(),
params: json!({}),
request: json!("request"),
owner: None,
enabled: true,
pending: false,
version: None,
crashes: VecDeque::new(),
};
let mut before = service();
let mut exact = service();
let mut after = service();
for _ in 0..CRASH_LIMIT {
assert!(before.crashed(now));
assert!(exact.crashed(now));
assert!(after.crashed(now));
}
assert!(!before.crashed(now + CRASH_WINDOW - Duration::from_nanos(1)));
assert!(exact.crashed(now + CRASH_WINDOW));
assert_eq!(exact.crashes.len(), 1);
assert!(after.crashed(now + CRASH_WINDOW + Duration::from_nanos(1)));
// Repeated progress beyond the window never accumulates old failures.
for step in 2..=CRASH_LIMIT as u32 + 1 {
assert!(exact.crashed(now + CRASH_WINDOW * step));
assert_eq!(exact.crashes.len(), 1);
}
}
#[tokio::test(start_paused = true)]
async fn services_have_independent_crash_budgets() {
let mut rig = Rig::new().await;
let (mut a, mut a_end) = rig.start("demo.serve").await;
let (b, b_end) = rig.start("other.serve").await;
for _ in 0..CRASH_LIMIT {
rig.crash(&a).await;
drop(a_end);
(a, a_end) = rig.next().await;
}
rig.crash(&a).await;
drop(a_end);
rig.crash(&b).await;
drop(b_end);
let (b, b_end) = rig.next().await;
rig.assert_single(&b, 2);
rig.clean(&b, b_end).await;
rig.no_start();
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn upgrade_crash_race_has_one_restart_and_retired_fallback_cannot_duplicate_it() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
rig.crash(&pid).await; // Exits between snapshot and begin_upgrade.
drop(end);
rig.host.begin_upgrade(upgrade);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
assert!(
rig.host.0.state.lock().unwrap().services[0]
.crashes
.is_empty()
);
// A native retired hook exercises the same routing path as a guest fallback.
rig.host.register_native(
"fallback",
Arc::new(move |ctx, _, _| {
Box::pin(async move {
ctx.call(
"core.call",
json!({"method":"demo.serve","params":{"kept":true},"detach":true}),
)
.await
})
}),
);
assert_eq!(
rig.host
.call("fallback.rpc.retired", json!({}))
.await
.unwrap(),
Value::Null
);
rig.no_start();
rig.clean(&pid, end).await;
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn crash_before_upgrade_snapshot_also_has_one_restart() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
rig.crash(&pid).await;
drop(end);
// Restart may be pending admission or already admitted: snapshot handles both.
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "native");
rig.host.begin_upgrade(upgrade);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
rig.no_start();
rig.clean(&pid, end).await;
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn pending_admission_is_single_owned_across_upgrade() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
let host = rig.host.clone();
let admission = host.0.admission.write().await;
rig.crash(&pid).await;
drop(end);
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "native");
rig.host.begin_upgrade(upgrade);
assert!(rig.host.0.state.lock().unwrap().services[0].pending);
drop(admission);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
rig.no_start();
rig.clean(&pid, end).await;
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn shutdown_rejects_restart_already_waiting_for_admission() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
let host = rig.host.clone();
let admission = host.0.admission.write().await;
rig.crash(&pid).await;
drop(end);
rig.host.stop().await;
drop(admission);
tokio::task::yield_now().await;
rig.no_start();
assert!(rig.host.processes().is_empty());
}
#[tokio::test(start_paused = true)]
async fn clean_service_stays_dormant_until_upgrade_and_explicit_stop_stays_disabled() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
rig.clean(&pid, end).await;
rig.no_start();
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
rig.host.begin_upgrade(upgrade);
let (pid, end) = rig.next().await;
rig.assert_single(&pid, 1);
// An explicit stop stays disabled even if the task subsequently exits cleanly.
let mut ended = rig.host.begin_stop(&pid).unwrap();
end.send(()).unwrap();
ended.wait_for(|done| *done).await.unwrap();
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "newer");
rig.host.begin_upgrade(upgrade);
tokio::task::yield_now().await;
rig.no_start();
rig.host.stop().await;
}
#[tokio::test(start_paused = true)]
async fn intentional_stop_and_shutdown_never_restart_even_during_upgrade() {
let mut rig = Rig::new().await;
let (pid, end) = rig.start("demo.serve").await;
let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
// An explicit kill after supersession overrides the automatic upgrade restart.
rig.host
.call("core.kill", json!({"pid":pid}))
.await
.unwrap();
drop(end);
rig.host.begin_upgrade(upgrade);
tokio::task::yield_now().await;
rig.no_start();
let (_pid, end) = rig.start("other.serve").await;
rig.host.stop().await;
drop(end);
rig.no_start();
}
+35 -40
View File
@@ -1,9 +1,15 @@
//! Grace periods, service restarts, and retirement of replaced versions (PROTOCOL.md section 3.7). //! Grace periods, service restarts, and retirement of replaced versions (PROTOCOL.md section 3.7).
use crate::{Host, services::Service}; use crate::Host;
use serde_json::json; use serde_json::json;
use std::{collections::BTreeMap, sync::atomic::Ordering, time::Duration}; use std::{collections::BTreeMap, sync::atomic::Ordering, time::Duration};
use tokio::sync::watch; use tokio::sync::watch;
/// Uniform host policy (PROTOCOL 3.7), independent of discovery metadata.
const UPGRADE_GRACE: Duration = Duration::from_secs(5);
#[cfg(test)]
mod tests;
pub(crate) struct OldProcess { pub(crate) struct OldProcess {
pid: String, pid: String,
version: String, version: String,
@@ -12,9 +18,8 @@ pub(crate) struct OldProcess {
pub(crate) struct Upgrade { pub(crate) struct Upgrade {
plugin: String, plugin: String,
previous: String, previous: String,
grace: Duration, current: String,
processes: Vec<OldProcess>, processes: Vec<OldProcess>,
services: Vec<(Service, Option<watch::Receiver<bool>>)>,
} }
impl Host { impl Host {
/// Snapshots old admissions while activation holds the admission write guard. /// Snapshots old admissions while activation holds the admission write guard.
@@ -23,57 +28,59 @@ impl Host {
plugin: &str, plugin: &str,
previous: String, previous: String,
current: &str, current: &str,
grace: f64,
) -> Upgrade { ) -> Upgrade {
let mut state = self.0.state.lock().unwrap(); let mut state = self.0.state.lock().unwrap();
let processes = state let processes = state
.processes .processes
.iter() .iter_mut()
.filter(|(_, process)| { .filter(|(_, process)| {
process.info.plugin == plugin process.info.plugin == plugin
&& process.info.version != current && process.info.version != current
&& process.info.method != "rpc.discover" && process.info.method != "rpc.discover"
}) })
.map(|(pid, process)| OldProcess { .map(|(pid, process)| {
// The service supervisor owns the restart, including an exit on
// the superseded notice before the grace timer is installed.
process.superseded = !process.stopped;
OldProcess {
pid: pid.clone(), pid: pid.clone(),
version: process.info.version.clone(), version: process.info.version.clone(),
ended: process.ended.subscribe(), ended: process.ended.subscribe(),
}
}) })
.collect(); .collect();
let mut services = Vec::new(); // A cleanly returned service is dormant until an upgrade. A pending
let mut retained = Vec::new(); // admission will choose the current version itself. Disabled services
for service in std::mem::take(&mut state.services) { // (explicit stop or crash-limit give-up) must never be re-armed here.
if service for service in &mut state.services {
if service.enabled
&& !service.pending
&& service.owner.is_none()
&& service.version.as_deref() != Some(current)
&& service
.method .method
.split_once('.') .split_once('.')
.is_some_and(|(name, _)| name == plugin) .is_some_and(|(name, _)| name == plugin)
{ {
let process = state self.queue_service(service);
.processes
.values()
.find(|process| process.id.as_ref() == Some(&service.request));
if process.is_some_and(|process| process.info.version == current)
|| (process.is_none() && service.starting.load(Ordering::SeqCst))
{
retained.push(service);
} else {
services.push((service, process.map(|process| process.ended.subscribe())));
}
} else {
retained.push(service);
} }
} }
state.services = retained;
Upgrade { Upgrade {
plugin: plugin.into(), plugin: plugin.into(),
previous, previous,
grace: Duration::from_secs_f64(grace), current: current.into(),
processes, processes,
services,
} }
} }
/// Lets old calls finish, stops them after grace, and starts each lifecycle hook once. /// Lets old calls finish, stops them after grace, and starts each lifecycle hook once.
pub(crate) fn begin_upgrade(&self, upgrade: Upgrade) { pub(crate) fn begin_upgrade(&self, upgrade: Upgrade) {
// Queue the notice before returning from activation, without waiting on guest code.
for process in &upgrade.processes {
self.send(
&process.pid,
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":upgrade.current}}),
);
}
let host = self.clone(); let host = self.clone();
tokio::spawn(async move { tokio::spawn(async move {
if host.0.shutdown.load(Ordering::SeqCst) { if host.0.shutdown.load(Ordering::SeqCst) {
@@ -83,7 +90,7 @@ impl Host {
"core", "core",
&upgrade.plugin, &upgrade.plugin,
"core", "core",
&format!("upgrade grace {} seconds", upgrade.grace.as_secs_f64()), &format!("upgrade grace {} seconds", UPGRADE_GRACE.as_secs()),
); );
let mut versions: BTreeMap<String, Vec<watch::Receiver<bool>>> = BTreeMap::new(); let mut versions: BTreeMap<String, Vec<watch::Receiver<bool>>> = BTreeMap::new();
versions.entry(upgrade.previous).or_default(); versions.entry(upgrade.previous).or_default();
@@ -93,23 +100,11 @@ impl Host {
.or_default() .or_default()
.push(process.ended.clone()); .push(process.ended.clone());
let host = host.clone(); let host = host.clone();
let grace = upgrade.grace;
tokio::spawn(async move { tokio::spawn(async move {
let mut ended = process.ended; let mut ended = process.ended;
tokio::select! { tokio::select! {
_ = ended.wait_for(|ended|*ended) => (), _ = ended.wait_for(|ended|*ended) => (),
_ = tokio::time::sleep(grace) => { host.begin_stop(&process.pid); } _ = tokio::time::sleep(UPGRADE_GRACE) => { host.begin_retire(&process.pid); }
}
});
}
for (service, ended) in upgrade.services {
let host = host.clone();
tokio::spawn(async move {
if let Some(mut ended) = ended {
let _ = ended.wait_for(|ended| *ended).await;
}
if !host.0.shutdown.load(Ordering::SeqCst) {
host.start_service(service.method, service.params).await;
} }
}); });
} }
+115
View File
@@ -0,0 +1,115 @@
//! Controlled-clock coverage of both upgrade windows and discovery exclusion.
use super::*;
use crate::{
process::{Process, ProcessInfo},
runtime::Runtime,
world::World,
};
use tokio::{
sync::mpsc,
task::JoinHandle,
time::{Instant, advance},
};
/// A process with active work that ignores both superseded and EOF.
fn process(
host: &Host,
pid: &str,
method: &str,
) -> (mpsc::UnboundedReceiver<serde_json::Value>, JoinHandle<()>) {
let (input, incoming) = mpsc::unbounded_channel();
let task = tokio::spawn(std::future::pending());
host.0.state.lock().unwrap().processes.insert(
pid.into(),
Process {
info: ProcessInfo {
plugin: "notes".into(),
version: "old".into(),
method: method.into(),
params: json!({}),
parent: None,
detached: true,
start_time: 0,
state: "running".into(),
},
memory: sausage_vfs::Memory::default(),
files: sausage_vfs::Process::default(),
store_ended: watch::channel(true).0,
input: Some(input),
abort: Some(task.abort_handle()),
caller: "console".into(),
id: None,
answer: None,
ended: watch::channel(false).0,
stopped: false,
superseded: false,
},
);
(incoming, task)
}
/// PROTOCOL 3.7/3.6: five seconds running, then five seconds stopping; discovery is untouched.
#[tokio::test]
async fn two_five_second_windows_and_discovery_exclusion() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path().join("world");
World::create(&root).unwrap();
let host = Host::test_world(&root, Runtime::new(&dir.path().join("cache")).unwrap())
.await
.unwrap();
let (mut input, work) = process(&host, "old-work", "work");
let (mut discovery_input, discovery) = process(&host, "old-discovery", "rpc.discover");
let upgrade = host.snapshot_upgrade("notes", "old".into(), "new");
assert_eq!(upgrade.processes.len(), 1);
tokio::time::pause();
let start = Instant::now();
host.begin_upgrade(upgrade);
assert_eq!(
input.recv().await.unwrap(),
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":"new"}})
);
// Run the upgrade dispatcher, then its per-process task, so the deadline is armed.
tokio::task::yield_now().await;
tokio::task::yield_now().await;
let window = Duration::from_secs(5);
assert_eq!(UPGRADE_GRACE, window);
// tokio::time::sleep documents millisecond granularity; the paused driver
// may round a deadline up by one tick. Check each window independently.
let tick = Duration::from_millis(1);
let before_deadline = window - tick;
advance(before_deadline).await;
tokio::task::yield_now().await;
assert_eq!(host.processes()["old-work"].state, "running");
assert!(!work.is_finished());
advance(tick).await;
assert!(
input.recv().await.is_none(),
"EOF at grace, even with active work"
);
let stopped = Instant::now();
assert!((window..=window + tick).contains(&(stopped - start)));
assert_eq!(host.processes()["old-work"].state, "stopping");
// begin_retire schedules the independent PROTOCOL 3.6 deadline.
tokio::task::yield_now().await;
advance(before_deadline).await;
tokio::task::yield_now().await;
assert!(
!work.is_finished(),
"on_stop still has its full five seconds"
);
advance(tick).await;
assert!(work.await.unwrap_err().is_cancelled());
assert!((window..=window + tick).contains(&(Instant::now() - stopped)));
assert_eq!(host.processes()["old-discovery"].state, "running");
assert!(matches!(
discovery_input.try_recv(),
Err(mpsc::error::TryRecvError::Empty)
));
assert!(!discovery.is_finished());
discovery.abort();
let _ = discovery.await;
// Synthetic processes have no execution cleanup task; discard only this test table.
host.0.shutdown.store(true, Ordering::SeqCst);
host.0.state.lock().unwrap().processes.clear();
host.stop().await;
}
+178
View File
@@ -0,0 +1,178 @@
//! Armed plugin-file watches. A Notify permit coalesces changes without losing a later change.
use crate::{Host, RpcResult, error};
use sausage_vfs::observe::under;
use serde_json::{Value, json};
use std::{
collections::HashMap,
sync::{Arc, Mutex},
};
use tokio::sync::Notify;
use uuid::Uuid;
pub(crate) struct Watch {
path: String,
changed: Arc<Notify>,
}
pub(crate) type Watches = Arc<Mutex<HashMap<Uuid, Watch>>>;
pub(crate) fn changed(watches: &Watches, path: &str) {
let targets: Vec<_> = watches
.lock()
.unwrap()
.values()
// Removing/moving an ancestor also invalidates a watch inside that subtree.
.filter(|w| under(path, &w.path) || under(&w.path, path))
.map(|w| w.changed.clone())
.collect();
for changed in targets {
changed.notify_one();
}
}
struct Registration {
watches: Watches,
key: Uuid,
}
impl Drop for Registration {
fn drop(&mut self) {
self.watches.lock().unwrap().remove(&self.key);
}
}
fn path(caller: &str, path: &str) -> String {
let path = sausage_vfs::resolve("/", path);
let path = path.trim_end_matches('/');
if path.is_empty() {
return "/".into();
}
if under(path, "/self") {
format!("/proc/{caller}{}", &path[5..])
} else {
path.into()
}
}
impl Host {
pub(crate) async fn watch(&self, caller: &str, id: Option<Value>, target: &str) -> RpcResult {
let id = id.ok_or_else(|| error(-32600, "watch needs a request ID"))?;
let mut ended = self
.0
.state
.lock()
.unwrap()
.processes
.get(caller)
.filter(|p| p.info.state == "running")
.map(|p| p.ended.subscribe())
.ok_or_else(|| error(-32600, "watch needs a running caller process"))?;
let changed = Arc::new(Notify::new());
let registration = Registration {
watches: self.0.watches.clone(),
key: Uuid::now_v7(),
};
self.0.watches.lock().unwrap().insert(
registration.key,
Watch {
path: path(caller, target),
changed: changed.clone(),
},
);
self.call_notify(caller, &id, "armed", json!({}));
loop {
tokio::select! {
biased;
_ = ended.wait_for(|ended| *ended) => return Ok(Value::Null),
_ = changed.notified() => self.call_notify(caller, &id, "changed", json!({})),
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Pending changes coalesce; a change after consuming the last permit is retained.
#[tokio::test]
async fn coalescing_and_cleanup() {
let watches = Watches::default();
let n = Arc::new(Notify::new());
let r = Registration {
watches: watches.clone(),
key: Uuid::now_v7(),
};
watches.lock().unwrap().insert(
r.key,
Watch {
path: "/mailbox".into(),
changed: n.clone(),
},
);
changed(&watches, "/mailboxes/a");
assert!(
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
.await
.is_err()
);
changed(&watches, "/mailbox/a");
changed(&watches, "/mailbox/b");
n.notified().await;
assert!(
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
.await
.is_err()
);
changed(&watches, "/mailbox/c");
n.notified().await;
drop(r);
assert!(watches.lock().unwrap().is_empty());
changed(&watches, "/mailbox/d");
assert!(
tokio::time::timeout(std::time::Duration::ZERO, n.notified())
.await
.is_err()
);
}
#[test]
fn self_alias_and_normalization() {
assert_eq!(path("writer", "/self/../self/a/"), "/proc/writer/a");
assert_eq!(path("writer", "/selfish/a"), "/selfish/a");
assert_eq!(path("writer", "/"), "/");
}
}
#[cfg(test)]
mod lifetime_tests {
use super::*;
/// The real caller's ended event releases the registration, without a cancellation RPC.
#[tokio::test]
async fn caller_death_removes_watch() {
let temp = tempfile::tempdir().unwrap();
let runtime = crate::runtime::Runtime::new(&temp.path().join("cache")).unwrap();
let host = Host::test_world(&temp.path().join("world"), runtime)
.await
.unwrap();
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel();
host.register_native(
"holder",
Arc::new(move |ctx, _, _| {
let tx = tx.clone();
Box::pin(async move {
tx.send(ctx.pid.clone()).unwrap();
std::future::pending().await
})
}),
);
let h = host.clone();
let task = tokio::spawn(async move { h.call("holder.hold", json!({})).await });
let pid = rx.recv().await.unwrap();
let mut watch = Box::pin(host.watch(&pid, Some(json!("watch")), "/mailbox"));
assert!(
watch
.as_mut()
.poll(&mut std::task::Context::from_waker(std::task::Waker::noop()))
.is_pending()
);
assert_eq!(host.0.watches.lock().unwrap().len(), 1);
host.call("core.kill", json!({"pid":pid})).await.unwrap();
watch.await.unwrap();
assert!(host.0.watches.lock().unwrap().is_empty());
assert!(task.await.unwrap().is_err());
host.stop().await;
}
}
+311 -11
View File
@@ -43,6 +43,86 @@ fn fixture(version: usize) -> PathBuf {
})[version - 1] })[version - 1]
.clone() .clone()
} }
fn fixture_hash(version: usize) -> String {
use sha2::{Digest, Sha256};
format!(
"{:x}",
Sha256::digest(std::fs::read(fixture(version)).unwrap())
)
}
/// PROTOCOL 10.3: summaries omit even large schemas and report sorted main-method diffs.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn summarizes_method_changes() {
let mut old = document("unchanged");
old["methods"] = json!([
{"name":"z_removed"}, {"name":"a_removed"}, {"name":"direct", "params":[]},
{"name":"referenced", "result":{"schema":{"$ref":"#/components/schemas/Node"}}},
{"name":"unrelated", "result":{"schema":{"type":"string"}}}
]);
old["components"]["schemas"] = json!({
"Node":{"properties":{"next":{"$ref":"#/components/schemas/Node"}, "value":{"$ref":"#/components/schemas/Value"}}},
"Value":{"type":"string"}, "Unused":{"type":"string"}
});
let mut next = old.clone();
next["methods"].as_array_mut().unwrap().drain(..2);
next["methods"][0]["params"] = json!([{"name":"limit","schema":{"type":"integer"}}]);
next["methods"].as_array_mut().unwrap().extend(
json!([
{"name":"z_added"}, {"name":"a_added"}
])
.as_array()
.unwrap()
.iter()
.cloned(),
);
next["components"]["schemas"]["Value"]["type"] = json!("integer");
next["components"]["schemas"]["Unused"]["description"] = json!("x".repeat(220_000));
let world = TestWorld::new();
world.plugin("notes", old, next.clone());
let host = world.start().await;
let result = activate(&host, "notes").await.unwrap();
assert_eq!(result["notes"]["method_count"], 5);
assert_eq!(result["notes"]["added"], json!(["a_added", "z_added"]));
assert_eq!(
result["notes"]["removed"],
json!(["a_removed", "z_removed"])
);
assert_eq!(result["notes"]["changed"], json!(["direct", "referenced"]));
assert!(result.to_string().len() < 1024);
assert_eq!(call(&host, "notes.rpc.discover", json!({})).await, next);
assert_eq!(
call(
&host,
"notes.read",
json!({"path":"/proc/core/plugins/notes/discover.json"})
)
.await,
next
);
host.stop().await;
}
/// PROTOCOL 10.3: first installation has no previous build; a no-op has no method changes.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn summarizes_first_install_and_noop() {
let world = TestWorld::new();
world.plugin("notes", document("add"), document("add"));
std::fs::remove_file(world.0.join("plugins/notes/plugin.wasm")).unwrap();
let host = world.start().await;
let result = activate(&host, "notes").await.unwrap();
assert_eq!(result["notes"]["previous"], Value::Null);
assert_eq!(result["notes"]["added"], json!(["add"]));
assert_eq!(result["notes"]["hash"], fixture_hash(2));
let result = activate(&host, "notes").await.unwrap();
assert_eq!(result["notes"]["previous"], fixture_hash(2));
for field in ["added", "removed", "changed"] {
assert_eq!(result["notes"][field], json!([]));
}
assert!(!world.0.join("plugins/notes/plugin.prev.wasm").exists());
host.stop().await;
}
struct TestWorld(PathBuf); struct TestWorld(PathBuf);
impl TestWorld { impl TestWorld {
/// Creates an isolated world with two version documents per plugin. /// Creates an isolated world with two version documents per plugin.
@@ -80,9 +160,9 @@ impl Drop for TestWorld {
let _ = std::fs::remove_dir_all(&self.0); let _ = std::fs::remove_dir_all(&self.0);
} }
} }
/// Builds a method-only document without an upgrade delay. /// Builds a method-only document; all plugins use the host's five-second grace.
fn document(name: &str) -> Value { fn document(name: &str) -> Value {
json!({"methods":[{"name":name,"params":[],"result":{"schema":{"type":"string"}}}],"x-sausage-grace":0}) json!({"methods":[{"name":name,"params":[],"result":{"schema":{"type":"string"}}}]})
} }
/// Adds an import of a provider's method list. /// Adds an import of a provider's method list.
fn consumer(provider: &str, expected: Value) -> Value { fn consumer(provider: &str, expected: Value) -> Value {
@@ -404,7 +484,7 @@ async fn review_dropped_activation_completes() {
host.stop().await; host.stop().await;
} }
/// PROTOCOL 10.1: discovery fails after its fixed ten-second deadline and receives EOF. /// PROTOCOL 10.1: discovery fails after its fixed ten-second deadline and receives EOF.
/// PROTOCOL 3.7: old processes retain their declared grace while new calls use new code. /// PROTOCOL 3.7: old processes get five seconds regardless of declarations; new calls use new code.
/// PROTOCOL 3.7: retirement waits for all live admissions, including reverted versions. /// PROTOCOL 3.7: retirement waits for all live admissions, including reverted versions.
/// PROTOCOL 3.7: rejected discovery candidates never receive retirement hooks. /// PROTOCOL 3.7: rejected discovery candidates never receive retirement hooks.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -481,7 +561,19 @@ async fn activates_copy() {
world.plugin("notes", document("add"), document("add")); world.plugin("notes", document("add"), document("add"));
let host = world.start().await; let host = world.start().await;
let result = activate(&host, "notes").await.unwrap(); let result = activate(&host, "notes").await.unwrap();
assert!(result["notes"]["methods"].is_array()); assert_eq!(
result,
json!({"notes": {
"plugin":"notes", "wasm":"/plugins/notes/plugin.wasm",
"hash":fixture_hash(2), "previous":fixture_hash(1),
"method_count":1, "added":[], "removed":[], "changed":[],
"interface_check":"passed"
}})
);
assert_eq!(
call(&host, "notes.rpc.discover", json!({})).await,
document("add")
);
assert_eq!( assert_eq!(
call(&host, "notes.version", json!({})).await["version"], call(&host, "notes.version", json!({})).await["version"],
"2" "2"
@@ -512,7 +604,7 @@ async fn source_changes_do_not_change_candidate() {
); );
host.stop().await; host.stop().await;
} }
/// PROTOCOL 3.7: plugins without a declared grace period retain old calls before the default deadline. /// PROTOCOL 3.7: missing metadata also retains old calls until the uniform deadline.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn default_grace_retains_old_process() { async fn default_grace_retains_old_process() {
let world = TestWorld::new(); let world = TestWorld::new();
@@ -531,6 +623,104 @@ async fn default_grace_retains_old_process() {
assert_eq!(host.processes()[&pid].state, "running"); assert_eq!(host.processes()[&pid].state, "running");
host.stop().await; host.stop().await;
} }
/// PROTOCOL 3.7: metadata cannot shorten or extend the actual grace for active requests.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn uniform_grace_ignores_discovery_declarations() {
let world = TestWorld::new();
let declarations = [
("missing", None),
("default", Some(json!(60))),
("long", Some(json!(3600))),
("zero", Some(json!(0))),
("negative", Some(json!(-1))),
("invalid", Some(json!("long"))),
("huge", Some(json!(1e300))),
];
for (name, grace) in &declarations {
let mut old = document("work");
if let Some(grace) = grace {
old["x-sausage-grace"] = grace.clone();
}
world.plugin(name, old, document("work"));
}
let host = world.start().await;
let mut ready = host.subscribe();
let mut calls = Vec::new();
for (name, _) in &declarations {
let host = host.clone();
let method = format!("{name}.work");
calls.push(tokio::spawn(async move {
let result = host.call(&method, json!({})).await;
(result, std::time::Instant::now())
}));
}
// Guest notifications are events, not process/file polling.
for _ in &declarations {
let event = tokio::time::timeout(Duration::from_secs(30), ready.recv())
.await
.unwrap()
.unwrap();
assert_eq!(event["params"]["name"], "ready");
}
let before = std::time::Instant::now();
let plugins: Vec<_> = declarations
.iter()
.map(|(name, _)| json!({"plugin":name,"wasm":format!("/state/{name}/next.wasm")}))
.collect();
call(&host, "core.activate", json!({"plugins":plugins})).await;
// Use the existing discovery+stop test budget, not a new execution limit.
tokio::time::timeout(Duration::from_secs(15), async {
for task in calls {
let (result, ended) = task.await.unwrap();
assert_eq!(result.unwrap_err()["code"], -32002);
assert!(
ended.duration_since(before) >= Duration::from_secs(5),
"every declaration gets the full grace"
);
}
})
.await
.unwrap();
host.stop().await;
}
/// PROTOCOL 10.4/3.7: a watcher refresh uses the same policy as explicit activation.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn manual_copy_uses_uniform_grace() {
let world = TestWorld::new();
let mut old = document("work");
old["x-sausage-grace"] = json!(3600);
world.plugin("notes", old, document("work"));
let host = world.start().await;
// Wait for initial discovery without changing the installed version.
call(
&host,
"core.activate",
json!({"plugin":"notes","wasm":"/plugins/notes/plugin.wasm"}),
)
.await;
let mut ready = host.subscribe();
let work_host = host.clone();
let work = tokio::spawn(async move { work_host.call("notes.work", json!({})).await });
let event = tokio::time::timeout(Duration::from_secs(30), ready.recv())
.await
.unwrap()
.unwrap();
assert_eq!(event["params"]["name"], "ready");
let before = std::time::Instant::now();
std::fs::copy(fixture(2), world.0.join("plugins/notes/plugin.wasm")).unwrap();
assert_eq!(
tokio::time::timeout(Duration::from_secs(15), work)
.await
.unwrap()
.unwrap()
.unwrap_err()["code"],
-32002
);
assert!(before.elapsed() >= Duration::from_secs(5));
host.stop().await;
}
/// PROTOCOL 10.3: candidate imports must fit installed providers. /// PROTOCOL 10.3: candidate imports must fit installed providers.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn checks_candidate_imports() { async fn checks_candidate_imports() {
@@ -577,7 +767,12 @@ async fn activates_group() {
consumer("notes", document("other")), consumer("notes", document("other")),
); );
let host = world.start().await; let host = world.start().await;
call(&host,"core.activate",json!({"plugins":[{"plugin":"notes","wasm":"/state/notes/next.wasm"},{"plugin":"agent","wasm":"/state/agent/next.wasm"}]})).await; let result = call(&host,"core.activate",json!({"plugins":[{"plugin":"notes","wasm":"/state/notes/next.wasm"},{"plugin":"agent","wasm":"/state/agent/next.wasm"}]})).await;
assert_eq!(result.as_object().unwrap().len(), 2);
assert_eq!(result["notes"]["added"], json!(["other"]));
assert_eq!(result["notes"]["removed"], json!(["add"]));
assert_eq!(result["agent"]["method_count"], 1);
assert_eq!(result["agent"]["interface_check"], "passed");
assert_eq!( assert_eq!(
call(&host, "agent.version", json!({})).await["version"], call(&host, "agent.version", json!({})).await["version"],
"2" "2"
@@ -594,12 +789,13 @@ async fn force_skips_checks() {
consumer("absent", document("add")), consumer("absent", document("add")),
); );
let host = world.start().await; let host = world.start().await;
call( let result = call(
&host, &host,
"core.activate", "core.activate",
json!({"plugin":"notes","wasm":"/state/notes/next.wasm","force":true}), json!({"plugin":"notes","wasm":"/state/notes/next.wasm","force":true}),
) )
.await; .await;
assert_eq!(result["notes"]["interface_check"], "skipped");
host.stop().await; host.stop().await;
} }
/// PROTOCOL 10.3: revert activates the previous copy and preserves the replaced version. /// PROTOCOL 10.3: revert activates the previous copy and preserves the replaced version.
@@ -609,7 +805,11 @@ async fn reverts_previous_version() {
world.plugin("notes", document("add"), document("add")); world.plugin("notes", document("add"), document("add"));
let host = world.start().await; let host = world.start().await;
activate(&host, "notes").await.unwrap(); activate(&host, "notes").await.unwrap();
call(&host, "core.revert", json!({"plugin":"notes"})).await; let result = call(&host, "core.revert", json!({"plugin":"notes"})).await;
assert_eq!(result["notes"]["hash"], fixture_hash(1));
assert_eq!(result["notes"]["previous"], fixture_hash(2));
assert_eq!(result["notes"]["interface_check"], "passed");
assert!(result["notes"].get("methods").is_none());
assert_eq!( assert_eq!(
call(&host, "notes.version", json!({})).await["version"], call(&host, "notes.version", json!({})).await["version"],
"1" "1"
@@ -716,11 +916,11 @@ async fn graph_missing_import() {
assert!(dot.as_str().unwrap().contains("absent")); assert!(dot.as_str().unwrap().contains("absent"));
host.stop().await; host.stop().await;
} }
/// Checks the declared upgrade grace and dispatch to the new version. /// Checks that an explicit long declaration cannot extend grace; new calls use the new version.
async fn old_process_grace_and_new_calls() { async fn old_process_grace_and_new_calls() {
let world = TestWorld::new(); let world = TestWorld::new();
let mut old = document("add"); let mut old = document("add");
old["x-sausage-grace"] = json!(0.5); old["x-sausage-grace"] = json!(3600);
world.plugin("notes", old, document("add")); world.plugin("notes", old, document("add"));
let host = world.start().await; let host = world.start().await;
call(&host, "notes.hold", json!({})).await; call(&host, "notes.hold", json!({})).await;
@@ -737,6 +937,76 @@ async fn old_process_grace_and_new_calls() {
"2" "2"
); );
marker(world.0.join("state/notes/ended-1"), "ended").await; marker(world.0.join("state/notes/ended-1"), "ended").await;
let notice: Value = serde_json::from_slice(
&std::fs::read(world.0.join(format!("state/notes/superseded-{pid}"))).unwrap(),
)
.unwrap();
assert_eq!(notice["params"]["current"], fixture_hash(2));
host.stop().await;
}
/// PROTOCOL 3.7: every old process gets the new hash while still running, before grace ends.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn superseded_notifies_each_old_process() {
let world = TestWorld::new();
let mut old = document("hold");
old["x-sausage-grace"] = json!(60);
world.plugin("notes", old, document("hold"));
let host = world.start().await;
call(&host, "notes.hold", json!({})).await;
call(&host, "notes.hold", json!({})).await;
let pids: Vec<_> = host
.processes()
.into_iter()
.filter(|(_, process)| process.method == "hold")
.map(|(pid, _)| pid)
.collect();
assert_eq!(pids.len(), 2);
activate(&host, "notes").await.unwrap();
let expected =
json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":fixture_hash(2)}})
.to_string();
for pid in pids {
// The notice arrives while the old process is still in its five-second window.
marker(
world.0.join(format!("state/notes/superseded-{pid}")),
&expected,
)
.await;
assert_eq!(host.processes()[&pid].state, "running");
}
assert_eq!(
call(&host, "notes.version", json!({})).await["version"],
"2"
);
host.stop().await;
}
/// PROTOCOL 3.7: a service that exits on the notice restarts before its grace deadline.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn superseded_service_exits_early() {
let world = TestWorld::new();
let mut old = document("serve");
old["x-sausage-grace"] = json!(60);
world.plugin("notes", old, document("serve"));
std::fs::write(
world.0.join("config/boot.yaml"),
json!([{"method":"notes.serve","params":{"exit_on_superseded":true}}]).to_string(),
)
.unwrap();
let host = world.start().await;
marker(world.0.join("state/notes/service"), "1").await;
let pid = host
.processes()
.into_iter()
.find(|(_, p)| p.method == "serve")
.unwrap()
.0;
activate(&host, "notes").await.unwrap();
marker(world.0.join("state/notes/service"), "2").await;
assert!(!host.processes().contains_key(&pid));
assert!(
!world.0.join("state/notes/ended-1").exists(),
"old service must exit on the notice, not EOF"
);
host.stop().await; host.stop().await;
} }
/// PROTOCOL 3.7: retirement receives the previous hash only after old processes end. /// PROTOCOL 3.7: retirement receives the previous hash only after old processes end.
@@ -821,7 +1091,7 @@ async fn upgrades_enter_the_journal() {
.as_array() .as_array()
.unwrap() .unwrap()
.iter() .iter()
.any(|e| e["line"] == "upgrade grace 60 seconds") .any(|e| e["line"] == "upgrade grace 5 seconds")
{ {
break entries; break entries;
} }
@@ -945,6 +1215,36 @@ async fn old_consumer_can_call_removed_provider_method() {
host.stop().await; host.stop().await;
} }
/// PROTOCOL 8: supervise actual WASI exits on either side of the readiness response.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn crashed_boot_service_restarts_before_and_after_ready() {
for when in ["before", "after"] {
let world = TestWorld::new();
world.plugin("notes", document("serve"), document("serve"));
std::fs::write(
world.0.join("config/boot.yaml"),
json!([{"method":"notes.serve","params":{"crash_once":when}}]).to_string(),
)
.unwrap();
let host = world.start().await;
marker(world.0.join("state/notes/service"), "1").await;
let entries = call(
&host,
"core.log",
json!({"grep":"service start notes.serve"}),
)
.await;
assert_eq!(entries.as_array().unwrap().len(), 2);
let processes = host.processes();
let services: Vec<_> = processes.values().filter(|p| p.method == "serve").collect();
assert_eq!(services.len(), 1);
assert_eq!(services[0].params, json!({"crash_once":when}));
let exits = call(&host, "core.log", json!({"plugin":"notes","grep":"exit 1"})).await;
assert_eq!(exits.as_array().unwrap().len(), 1);
host.stop().await;
}
}
/// PROTOCOL 10.2: activation and revert check interfaces but leave persistent state unchanged. /// PROTOCOL 10.2: activation and revert check interfaces but leave persistent state unchanged.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn state_is_outside_interface_checks() { async fn state_is_outside_interface_checks() {
+98 -15
View File
@@ -278,14 +278,7 @@ async fn manifest_and_nested_sources() {
assert!(manifest.contains("optional = true")); assert!(manifest.contains("optional = true"));
let document: toml_edit::DocumentMut = manifest.parse().unwrap(); let document: toml_edit::DocumentMut = manifest.parse().unwrap();
let sdk = Path::new(document["target"]["cfg(target_arch = \"wasm32\")"]["dependencies"]["sausage_plugin"]["path"].as_str().unwrap()); let sdk = Path::new(document["target"]["cfg(target_arch = \"wasm32\")"]["dependencies"]["sausage_plugin"]["path"].as_str().unwrap());
assert_eq!(sdk.parent().unwrap(), Path::new(CACHE).join("sdk")); assert_eq!(sdk, valid.world.0.join("plugins/cargo/sdk"));
let hash = sdk.file_name().unwrap().to_str().unwrap();
assert_eq!(hash.len(), 64);
assert!(hash.bytes().all(|byte| byte.is_ascii_hexdigit()));
assert_eq!(
std::fs::read(sdk.join("src/lib.rs")).unwrap(),
std::fs::read(valid.world.0.join("plugins/cargo/sdk/src/lib.rs")).unwrap()
);
assert_eq!( assert_eq!(
std::fs::read_to_string(root.join("src/target/data")).unwrap(), std::fs::read_to_string(root.join("src/target/data")).unwrap(),
"nested target" "nested target"
@@ -483,7 +476,7 @@ async fn source_copy() {
assert!( assert!(
std::fs::read_to_string(valid.world.0.join("state/cargo/work/Cargo.toml")) std::fs::read_to_string(valid.world.0.join("state/cargo/work/Cargo.toml"))
.unwrap() .unwrap()
.contains(Path::new(CACHE).join("sdk").to_str().unwrap()) .contains(valid.world.0.join("plugins/cargo/sdk").to_str().unwrap())
); );
} }
/// PROTOCOL 11: compiler failures return compiler diagnostics. /// PROTOCOL 11: compiler failures return compiler diagnostics.
@@ -637,6 +630,96 @@ async fn unchanged_sdk() {
modified modified
); );
} }
/// PROTOCOL 11: an SDK edit is used by the next build on the same running host.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn world_sdk_edits_apply_without_restart() {
tokio::time::timeout(std::time::Duration::from_secs(180), async {
let world = TestWorld::new();
world.simple("sdk-probe", "fn main() {}\n#[test]\nfn world_sdk_probe() { assert_eq!(sausage_plugin::world_sdk_probe(), 1); }\n");
let manifest = world.0.join("plugins/sdk-probe/source/Cargo.toml");
let mut text = std::fs::read_to_string(&manifest).unwrap();
text.push_str("\n[dependencies]\nsausage-plugin = '0.1'\n");
std::fs::write(manifest, text).unwrap();
let host = Host::start_with_cache(&world.0, Path::new(CACHE)).await.unwrap();
let library = world.0.join("plugins/cargo/sdk/src/lib.rs");
let original = std::fs::read_to_string(&library).unwrap();
std::fs::write(&library, format!("{original}\npub fn world_sdk_probe() -> u8 {{ 1 }}\n")).unwrap();
let params = json!({"plugin":"sdk-probe", "timeout_secs":120, "test_timeout_secs":10});
let first = host.call("cargo.build", params.clone()).await.unwrap();
assert_eq!(first["tests"]["passed"], 1);
// Only the SDK changes: Cargo must not reuse its old compiled version.
std::fs::write(&library, format!("{original}\npub fn world_sdk_probe() -> u8 {{ 2 }}\n")).unwrap();
let second = host.call("cargo.build", params).await.unwrap_err();
assert_eq!(second["code"], -32004);
assert_eq!(second["data"]["tests"]["failed"], 1);
assert!(second["data"]["tests"]["output"].as_str().unwrap().contains("world_sdk_probe"));
host.stop().await;
}).await.expect("SDK edit regression timed out");
}
/// PROTOCOL 11: startup seeds a missing SDK, but never repairs or overwrites an existing one.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn world_sdk_survives_restart_and_missing_sdk_is_reseeded() {
tokio::time::timeout(std::time::Duration::from_secs(30), async {
let world = TestWorld::new();
let sdk = world.0.join("plugins/cargo/sdk");
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
let library = sdk.join("src/lib.rs");
let original = std::fs::read(&library).unwrap();
let manifest = sdk.join("Cargo.toml");
let edited = b"// world SDK edit must survive restart\n";
std::fs::write(&library, edited).unwrap();
std::fs::write(&manifest, "# edited manifest\n").unwrap();
let removed = sdk.join("macros/src/lib.rs");
std::fs::remove_file(&removed).unwrap();
host.stop().await;
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
assert_eq!(std::fs::read(&library).unwrap(), edited);
assert_eq!(
std::fs::read_to_string(&manifest).unwrap(),
"# edited manifest\n"
);
assert!(!removed.exists(), "existing SDK must not be repaired");
host.stop().await;
std::fs::remove_dir_all(&sdk).unwrap();
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
assert_eq!(std::fs::read(&library).unwrap(), original);
assert!(removed.is_file());
assert!(sdk.join("interface/src/lib.rs").is_file());
host.stop().await;
})
.await
.expect("SDK restart regression timed out");
}
/// PROTOCOL 11: removing the SDK while the host is running re-seeds it on the next build.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn world_sdk_missing_at_build_is_reseeded() {
tokio::time::timeout(std::time::Duration::from_secs(180), async {
let world = TestWorld::new();
world.simple("reseed-probe", "fn main() {}\n");
let host = Host::start_with_cache(&world.0, Path::new(CACHE))
.await
.unwrap();
let sdk = world.0.join("plugins/cargo/sdk");
let original = std::fs::read(sdk.join("src/lib.rs")).unwrap();
std::fs::remove_dir_all(&sdk).unwrap();
host.call(
"cargo.build",
json!({"plugin":"reseed-probe", "timeout_secs":120, "test_timeout_secs":10}),
)
.await
.unwrap();
assert_eq!(std::fs::read(sdk.join("src/lib.rs")).unwrap(), original);
host.stop().await;
})
.await
.expect("SDK re-seed build regression timed out");
}
/// PROTOCOL 11: init can build a plugin without a running host. /// PROTOCOL 11: init can build a plugin without a running host.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn standalone_builder() { async fn standalone_builder() {
@@ -658,7 +741,7 @@ async fn standalone_builder() {
1 1
); );
} }
/// SAU-126: a named CPU-bound WASI test times out, while other instances still run. /// a named CPU-bound WASI test times out, while other instances still run.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn named_test_timeout_and_mutex_release() { async fn named_test_timeout_and_mutex_release() {
let world = TestWorld::new(); let world = TestWorld::new();
@@ -732,7 +815,7 @@ async fn child_ended(child: u32) {
.expect("cargo descendant survived cancellation/timeout"); .expect("cargo descendant survived cancellation/timeout");
} }
/// SAU-126: an explicit small total budget kills build.rs and unblocks a queued build. /// an explicit small total budget kills build.rs and unblocks a queued build.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn build_timeout_and_queued_build() { async fn build_timeout_and_queued_build() {
let world = TestWorld::new(); let world = TestWorld::new();
@@ -770,7 +853,7 @@ async fn build_timeout_and_queued_build() {
host.stop().await; host.stop().await;
} }
/// SAU-126: the same total deadline covers release compilation and retains test results. /// the same total deadline covers release compilation and retains test results.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn release_build_timeout_retains_tests() { async fn release_build_timeout_retains_tests() {
let world = TestWorld::new(); let world = TestWorld::new();
@@ -820,7 +903,7 @@ fn main() {
host.stop().await; host.stop().await;
} }
/// SAU-126: omission selects the real 900-second deadline, without a 15-minute test. /// omission selects the real 900-second deadline, without a 15-minute test.
#[tokio::test] #[tokio::test]
async fn omitted_build_timeout_uses_default() { async fn omitted_build_timeout_uses_default() {
let world = TestWorld::new(); let world = TestWorld::new();
@@ -845,7 +928,7 @@ async fn omitted_build_timeout_uses_default() {
host.stop().await; host.stop().await;
} }
/// SAU-126: a caller ending stops its attached native build and releases the mutex. /// a caller ending stops its attached native build and releases the mutex.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn ended_caller_releases_build_mutex() { async fn ended_caller_releases_build_mutex() {
let world = TestWorld::new(); let world = TestWorld::new();
@@ -887,7 +970,7 @@ async fn ended_caller_releases_build_mutex() {
host.stop().await; host.stop().await;
} }
/// SAU-126: discovery exposes non-null defaults; invalid budgets cannot disable limits. /// discovery exposes non-null defaults; invalid budgets cannot disable limits.
#[tokio::test] #[tokio::test]
async fn timeout_params_discovery_and_validation() { async fn timeout_params_discovery_and_validation() {
let world = TestWorld::new(); let world = TestWorld::new();
+118 -10
View File
@@ -208,7 +208,11 @@ async fn journal_recovers_after_sqlite_write_failure() {
call(&h, "core.log", json!({})).await; call(&h, "core.log", json!({})).await;
let batches: Vec<Value> = std::fs::read_dir(w.0.join("logs/core/failed-batches")) let batches: Vec<Value> = std::fs::read_dir(w.0.join("logs/core/failed-batches"))
.unwrap() .unwrap()
.map(|file| serde_json::from_slice(&std::fs::read(file.unwrap().path()).unwrap()).unwrap()) .map(Result::unwrap)
// The writer atomically publishes .json files; its temporary files can
// still be incomplete while later process-exit entries are spooling.
.filter(|file| file.path().extension().is_some_and(|ext| ext == "json"))
.map(|file| serde_json::from_slice(&std::fs::read(file.path()).unwrap()).unwrap())
.collect(); .collect();
assert!( assert!(
batches batches
@@ -500,14 +504,119 @@ async fn unimplemented_core_methods_return_method_not_found() {
async fn params_must_be_objects() { async fn params_must_be_objects() {
let w = TestWorld::new(); let w = TestWorld::new();
let h = w.start().await; let h = w.start().await;
for (params, kind) in [
(json!("{\"text\":\"hello\"}"), "a string"),
(json!([]), "an array"),
(json!(42), "a number"),
(json!(null), "null"),
(json!(true), "a boolean"),
] {
let expected = json!({
"code": -32602,
"message": format!("params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string.")
});
for method in ["probe.echo", "core.ps"] {
assert_eq!(h.call(method, params.clone()).await.unwrap_err(), expected);
}
assert_eq!( assert_eq!(
h.call("probe.echo", json!([])).await.unwrap_err()["code"], h.call("core.call", json!({"method":"probe.echo", "params":params}))
-32602 .await
.unwrap_err(),
expected
);
}
let object = json!({"text":"hello"});
assert_eq!(call(&h, "probe.echo", object.clone()).await, object);
assert_eq!(
call(
&h,
"core.call",
json!({"method":"probe.echo", "params":object})
)
.await,
object
);
assert_eq!(
call(&h, "core.call", json!({"method":"probe.echo"})).await,
json!({})
); );
h.stop().await; h.stop().await;
} }
/// PROTOCOL 3.1: stdout fast paths reject non-objects without changing defaults.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn stdout_params_must_be_objects() {
let w = TestWorld::new();
let h = w.start().await;
let mut notifications = h.subscribe();
for method in ["core.notify", "core.unlock", "probe.echo"] {
for (params, kind) in [
(json!("{}"), "a string"),
(json!([]), "an array"),
(json!(1.5), "a number"),
(json!(null), "null"),
(json!(false), "a boolean"),
] {
let response = call(
&h,
"probe.raw_call",
json!({
"jsonrpc":"2.0", "id":1, "method":method, "params":params
}),
)
.await;
assert_eq!(response["id"], 1);
assert_eq!(
response["error"],
json!({
"code": -32602,
"message": format!("params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string.")
})
);
}
let params = match method {
"core.notify" => json!({"name":"valid", "params":{"n":1}}),
"core.unlock" => json!({"path":"/state/not-held"}),
_ => json!({"n":1}),
};
let response = call(
&h,
"probe.raw_call",
json!({
"jsonrpc":"2.0", "id":1, "method":method, "params":params
}),
)
.await;
assert_eq!(
response["result"],
if method == "probe.echo" {
params
} else {
Value::Null
}
);
assert!(response.get("error").is_none());
let omitted = call(
&h,
"probe.raw_call",
json!({
"jsonrpc":"2.0", "id":1, "method":method
}),
)
.await;
match method {
"core.notify" => {
assert_eq!(omitted["error"]["message"], "name must be a string");
assert_eq!(notifications.try_recv().unwrap()["params"]["name"], "valid");
}
"core.unlock" => assert_eq!(omitted["error"]["message"], "path must be a string"),
_ => assert_eq!(omitted["result"], json!({})),
}
}
h.stop().await;
}
/// PROTOCOL 3.9: exit without answer reports status and last logs. /// PROTOCOL 3.9: exit without answer reports status and last logs.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn exit_without_answer_reports_status_and_last_logs() { async fn exit_without_answer_reports_status_and_last_logs() {
@@ -999,17 +1108,16 @@ async fn stop_closes_stdin_and_waits_for_exit() {
let w = TestWorld::new(); let w = TestWorld::new();
let h = w.start().await; let h = w.start().await;
let hc = h.clone(); let hc = h.clone();
let mut events = h.subscribe();
let pending = tokio::spawn(async move { hc.call("probe.wait_eof", json!({})).await }); let pending = tokio::spawn(async move { hc.call("probe.wait_eof", json!({})).await });
// Admission is not readiness: stop may otherwise close stdin before the
// request reaches the guest, so wait for the handler to enter its EOF loop.
let pid = tokio::time::timeout(std::time::Duration::from_secs(20), async { let pid = tokio::time::timeout(std::time::Duration::from_secs(20), async {
loop { loop {
if let Some((pid, _)) = h let event = events.recv().await.unwrap();
.processes() if event["params"]["name"] == "waiting-eof" {
.into_iter() break event["params"]["params"]["pid"].clone();
.find(|(_, p)| p.method == "wait_eof")
{
break pid;
} }
tokio::task::yield_now().await;
} }
}) })
.await .await
+1
View File
@@ -7,3 +7,4 @@ mod common;
mod console; mod console;
mod core; mod core;
mod sdk; mod sdk;
mod watches;
+50
View File
@@ -178,6 +178,56 @@ async fn stop_runs_hook() {
assert_eq!(std::fs::read(world.0.join("stopped")).unwrap(), b"yes"); assert_eq!(std::fs::read(world.0.join("stopped")).unwrap(), b"yes");
host.stop().await; host.stop().await;
} }
/// PROTOCOL 3.7: an idle SDK listener opts into drain and exits before EOF/grace.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn superseded_idle_listener_drains_before_grace() {
let world = TestWorld::new();
let host = world.start().await;
let (notice, mut notices) = tokio::sync::mpsc::unbounded_channel();
host.register_native(
"other",
std::sync::Arc::new(move |_, method, _| {
let notice = notice.clone();
Box::pin(async move {
assert_eq!(method, "superseded");
notice.send(std::time::Instant::now()).unwrap();
Ok(json!({}))
})
}),
);
let pid = call(&host, "probe.drain_service", json!({})).await;
// Replace with a different real component; force only skips unrelated test consumers.
std::fs::copy(
fixtures().join("notes.wasm"),
world.0.join("state/next.wasm"),
)
.unwrap();
call(
&host,
"core.activate",
json!({"plugin":"probe","wasm":"/state/next.wasm","force":true}),
)
.await;
// Activation preparation/discovery is outside upgrade grace. Time from the
// hook's outgoing call, not from the start of candidate compilation.
let before = tokio::time::timeout(Duration::from_secs(60), notices.recv())
.await
.unwrap()
.unwrap();
// Waiting on its lifetime lock is an exit event, not process-liveness polling.
call(&host, "core.lock", json!({"path":"/drain-owner"})).await;
assert!(before.elapsed() < Duration::from_secs(5));
assert!(!host.processes().contains_key(pid.as_str().unwrap()));
assert!(
!world
.0
.join(format!("stopped-{}", pid.as_str().unwrap()))
.exists(),
"drain must not run on_stop (discovery processes may run their own hooks)"
);
host.stop().await;
}
/// SDK framing carries payloads above 64 KiB in both directions. /// SDK framing carries payloads above 64 KiB in both directions.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn large_messages() { async fn large_messages() {
+174
View File
@@ -0,0 +1,174 @@
//! Real SDK watch callbacks and raw WASI filesystem mutations share the host namespace.
use super::common::{CACHE, probe, sdk_probe};
use sausage::{Host, world::World};
use serde_json::{Value, json};
use std::time::Duration;
// Same call bound as core.rs; no sleeps or polling.
async fn call(h: &Host, method: &str, p: Value) -> Value {
tokio::time::timeout(Duration::from_secs(20), h.call(method, p))
.await
.unwrap()
.unwrap()
}
async fn event(h: &Host, rx: &mut tokio::sync::broadcast::Receiver<Value>, name: &str) -> Value {
let result = tokio::time::timeout(Duration::from_secs(20), async {
loop {
let e = rx.recv().await.unwrap();
if e["params"]["name"] == name {
return e["params"]["params"].clone();
}
}
})
.await;
match result {
Ok(v) => v,
Err(e) => {
eprintln!(
"JOURNAL {}",
h.call("core.log", json!({"limit":100})).await.unwrap()
);
panic!("{e}");
}
}
}
async fn start() -> (tempfile::TempDir, Host) {
let dir = tempfile::tempdir().unwrap();
let world = World::create(&dir.path().join("world")).unwrap();
for (name, wasm) in [("probe", probe()), ("sdk", sdk_probe())] {
std::fs::create_dir_all(world.plugin_dir(name)).unwrap();
std::fs::copy(wasm, world.plugin_dir(name).join("plugin.wasm")).unwrap();
}
let host = Host::start_with_cache(&world.root, std::path::Path::new(CACHE))
.await
.unwrap();
(dir, host)
}
fn watch(h: &Host, path: &str) -> tokio::task::JoinHandle<sausage::RpcResult> {
let (h, path) = (h.clone(), path.to_owned());
tokio::spawn(async move { h.call("sdk.watch", json!({"path":path})).await })
}
async fn mutate(h: &Host, op: &str, path: &str) {
call(h, "probe.mutate", json!({"op":op,"path":path})).await;
}
/// PROTOCOL 4: pre-arm state is in the first scan; each post-scan mutation wakes
/// the SDK without completing the watch. Covers descriptor/stream/truncate paths.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn armed_scan_and_later_changes() {
let (_dir, h) = start().await;
mutate(&h, "mkdir", "/state/mailbox").await;
call(
&h,
"probe.write",
json!({"path":"/state/mailbox/before", "text":"before"}),
)
.await;
let mut rx = h.subscribe();
let task = watch(&h, "/state/mailbox/../mailbox/");
let armed = event(&h, &mut rx, "armed").await;
assert_eq!(armed["entries"], json!(["before"]));
assert!(!task.is_finished());
for (op, expected) in [
("descriptor", "descriptor"),
("stream", "streamptor"),
("append", "streamptorappend"),
("size", ""),
("descriptor", "descriptor"),
("truncate", ""),
] {
mutate(&h, op, "/state/mailbox/before").await;
let mut scan = event(&h, &mut rx, "changed").await;
while scan["text"] != expected {
scan = event(&h, &mut rx, "changed").await;
}
}
mutate(&h, "burst", "/state/mailbox").await;
let mut scanned = event(&h, &mut rx, "changed").await;
while !scanned["entries"].as_array().unwrap().contains(&json!("4")) {
scanned = event(&h, &mut rx, "changed").await;
}
while rx.try_recv().is_ok() {}
mutate(&h, "delete", "/state/mailbox/4").await;
let mut scanned = event(&h, &mut rx, "changed").await;
while scanned["entries"].as_array().unwrap().contains(&json!("4")) {
scanned = event(&h, &mut rx, "changed").await;
}
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
assert!(task.await.unwrap().is_err());
h.stop().await;
}
/// PROTOCOL 4: atomic publication, rename source/destination, mounted paths,
/// absent paths, and caller-death cleanup. No external host writes are required.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn rename_delete_and_mounts() {
let (_dir, h) = start().await;
for base in ["/state/inbox", "/run/inbox"] {
let mut rx = h.subscribe();
let task = watch(&h, base);
let armed = event(&h, &mut rx, "armed").await;
assert_eq!(armed["entries"], json!([]));
let staging = format!("{base}-staging");
mutate(&h, "mkdir", &staging).await;
call(
&h,
"probe.mutate",
json!({"op":"rename","path":staging,"to":base}),
)
.await;
event(&h, &mut rx, "changed").await;
mutate(&h, "atomic", &format!("{base}/published")).await;
let mut scan = event(&h, &mut rx, "changed").await;
while !scan["entries"]
.as_array()
.unwrap()
.contains(&json!("published"))
{
scan = event(&h, &mut rx, "changed").await;
}
assert_eq!(
call(
&h,
"probe.read",
json!({"path":format!("{base}/published")})
)
.await,
"complete"
);
while rx.try_recv().is_ok() {}
call(
&h,
"probe.mutate",
json!({"op":"rename","path":base,"to":format!("{base}-moved")}),
)
.await;
let mut scan = event(&h, &mut rx, "changed").await;
while scan["entries"] != json!([]) {
scan = event(&h, &mut rx, "changed").await;
}
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
assert!(task.await.unwrap().is_err());
}
// /self watches bind to this caller, and mutations are visible via /proc/<pid>.
let mut rx = h.subscribe();
let task = watch(&h, "/self");
let armed = event(&h, &mut rx, "armed").await;
assert!(
armed["entries"]
.as_array()
.unwrap()
.contains(&json!("owned"))
);
event(&h, &mut rx, "changed").await;
assert_eq!(
call(
&h,
"probe.read",
json!({"path":format!("/proc/{}/owned", armed["pid"].as_str().unwrap())})
)
.await,
"visible via proc"
);
call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
assert!(task.await.unwrap().is_err());
h.stop().await;
}
+41 -1
View File
@@ -84,12 +84,52 @@ fn main() {
.unwrap(); .unwrap();
json!({"retired":version}) json!({"retired":version})
} }
"work" => {
// Keep an actual request unanswered through grace; notices do not drain it.
send(
json!({"jsonrpc":"2.0","method":"core.notify","params":{"name":"ready","params":{}}}),
);
for _ in lines {}
return;
}
"serve" | "hold" => { "serve" | "hold" => {
// One genuine process failure, followed by a healthy service, without
// timers or liveness polling in either the fixture or supervisor.
let crash = request["params"]["crash_once"].as_str();
let first = crash.is_some()
&& std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(format!("/state/{plugin}/crashed-once"))
.is_ok();
if first && crash == Some("before") {
std::process::exit(1);
}
send(json!({"jsonrpc":"2.0","id":request["id"],"result":{"version":version}})); send(json!({"jsonrpc":"2.0","id":request["id"],"result":{"version":version}}));
if first && crash == Some("after") {
std::process::exit(1);
}
if request["method"] == "serve" { if request["method"] == "serve" {
std::fs::write(format!("/state/{plugin}/service"), version).unwrap(); std::fs::write(format!("/state/{plugin}/service"), version).unwrap();
} }
for _ in lines {} for line in lines {
let Ok(message) = serde_json::from_str::<Value>(&line.unwrap()) else {
continue;
};
if message["method"] == "rpc.superseded" {
std::fs::write(
format!(
"/state/{plugin}/superseded-{}",
std::env::var("SAUSAGE_PID").unwrap()
),
message.to_string(),
)
.unwrap();
if request["params"]["exit_on_superseded"] == true {
return;
}
}
}
std::fs::write(format!("/state/{plugin}/ended-{version}"), "ended").unwrap(); std::fs::write(format!("/state/{plugin}/ended-{version}"), "ended").unwrap();
return; return;
} }
+73
View File
@@ -83,6 +83,9 @@ fn unknown_id() {
} }
fn wait_eof(lines: &mut Lines) { fn wait_eof(lines: &mut Lines) {
std::fs::write("/state/waiting", "yes").unwrap(); std::fs::write("/state/waiting", "yes").unwrap();
send(json!({"jsonrpc":"2.0","method":"core.notify","params":{
"name":"waiting-eof", "params":{"pid":std::env::var("SAUSAGE_PID").unwrap()}
}}));
for _ in lines {} for _ in lines {}
std::fs::write("/state/stopped", "yes").unwrap(); std::fs::write("/state/stopped", "yes").unwrap();
} }
@@ -468,8 +471,13 @@ fn dispatch(request: &Value, lines: &mut Lines) -> Option<Value> {
return None; return None;
} }
"network" => network(params), "network" => network(params),
"mutate" => mutate(params),
"read" => read_file(params), "read" => read_file(params),
"write" => write_file(params), "write" => write_file(params),
"raw_call" => {
send(params.clone());
receive(lines)
}
"call" => outgoing_call(params, lines), "call" => outgoing_call(params, lines),
"lock" => lock(params, lines), "lock" => lock(params, lines),
_ => { _ => {
@@ -514,3 +522,68 @@ fn main() {
})); }));
} }
} }
/// Separate mutation calls let the host await a scan before each subsequent change.
fn mutate(p: &Value) -> Value {
use wasip2::filesystem::{
preopens,
types::{DescriptorFlags as D, OpenFlags as O, PathFlags as P},
};
let path = p["path"].as_str().unwrap();
match p["op"].as_str().unwrap() {
"mkdir" => std::fs::create_dir_all(path).unwrap(),
"rename" => std::fs::rename(path, p["to"].as_str().unwrap()).unwrap(),
"delete" => std::fs::remove_file(path).unwrap(),
"rmdir" => std::fs::remove_dir(path).unwrap(),
"atomic" => {
let temp = format!("{path}.tmp");
std::fs::write(&temp, "complete").unwrap();
std::fs::rename(temp, path).unwrap();
}
"burst" => {
for i in 0..5 {
std::fs::write(format!("{path}/{i}"), "complete").unwrap();
}
}
op => {
let root = preopens::get_directories().remove(0).0;
let file = root
.open_at(
P::empty(),
path.trim_start_matches('/'),
O::empty(),
D::READ | D::WRITE,
)
.unwrap();
match op {
"descriptor" => {
file.write(b"descriptor", 0).unwrap();
}
"stream" => {
file.write_via_stream(0)
.unwrap()
.blocking_write_and_flush(b"stream")
.unwrap();
}
"append" => {
file.append_via_stream()
.unwrap()
.blocking_write_and_flush(b"append")
.unwrap();
}
"size" => file.set_size(0).unwrap(),
"truncate" => {
root.open_at(
P::empty(),
path.trim_start_matches('/'),
O::TRUNCATE,
D::WRITE,
)
.unwrap();
}
_ => panic!("unknown mutation"),
}
}
}
json!({})
}
+27 -7
View File
@@ -153,6 +153,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
let mut stop = quote!(); let mut stop = quote!();
let mut has_stop = false; let mut has_stop = false;
let mut has_retired = false; let mut has_retired = false;
let mut has_superseded = false;
for member in &mut item.items { for member in &mut item.items {
let ImplItem::Fn(f) = member else { continue }; let ImplItem::Fn(f) = member else { continue };
let retired = f.attrs.iter().any(|a| { let retired = f.attrs.iter().any(|a| {
@@ -161,6 +162,12 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
.last() .last()
.is_some_and(|s| s.ident == "on_retired") .is_some_and(|s| s.ident == "on_retired")
}); });
let superseded = f.attrs.iter().any(|a| {
a.path()
.segments
.last()
.is_some_and(|s| s.ident == "on_superseded")
});
let stopping = f.attrs.iter().any(|a| { let stopping = f.attrs.iter().any(|a| {
a.path() a.path()
.segments .segments
@@ -179,11 +186,16 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
"only one on_retired method is allowed", "only one on_retired method is allowed",
)); ));
} }
if superseded && std::mem::replace(&mut has_superseded, true) {
return Err(syn::Error::new(
f.span(),
"only one on_superseded method is allowed",
));
}
f.attrs.retain(|a| { f.attrs.retain(|a| {
!a.path() !a.path().segments.last().is_some_and(|s| {
.segments s.ident == "on_retired" || s.ident == "on_stop" || s.ident == "on_superseded"
.last() })
.is_some_and(|s| s.ident == "on_retired" || s.ident == "on_stop")
}); });
documented(&f.attrs, f.span())?; documented(&f.attrs, f.span())?;
if f.sig.inputs.is_empty() if f.sig.inputs.is_empty()
@@ -224,7 +236,9 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
stop = quote!(fn stop(&self)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=()> + '_>> {Box::pin(async move {let _=self.#name() #await_;})}); stop = quote!(fn stop(&self)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=()> + '_>> {Box::pin(async move {let _=self.#name() #await_;})});
continue; continue;
} }
let wire = if retired { let wire = if superseded {
"rpc.superseded".to_string()
} else if retired {
"rpc.retired".to_string() "rpc.retired".to_string()
} else { } else {
name.unraw().to_string() name.unraw().to_string()
@@ -253,7 +267,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
}; };
let doc = docs(&f.attrs); let doc = docs(&f.attrs);
arms.push(quote!(#wire => { fn params_type<T: ::sausage_plugin::Params>() {} params_type::<#p>(); let result=self.#name(#(#args),*) #await_ ?; Ok(::sausage_plugin::serde_json::to_value(result)?) })); arms.push(quote!(#wire => { fn params_type<T: ::sausage_plugin::Params>() {} params_type::<#p>(); let result=self.#name(#(#args),*) #await_ ?; Ok(::sausage_plugin::serde_json::to_value(result)?) }));
if !retired { if !retired && !superseded {
entries.push(quote!(::sausage_plugin::method::<#p,#output>(&mut doc,#wire,#doc);)); entries.push(quote!(::sausage_plugin::method::<#p,#output>(&mut doc,#wire,#doc);));
} }
} }
@@ -270,7 +284,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
fn methods_document()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace ::sausage_plugin::finish_document(doc)} fn methods_document()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace ::sausage_plugin::finish_document(doc)}
fn describe()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace #(doc.endpoint::<#types>(#names);)* ::sausage_plugin::finish_document(doc)} fn describe()->::sausage_plugin::serde_json::Value {let mut doc=::sausage_plugin::document(env!("CARGO_PKG_NAME"),env!("CARGO_PKG_VERSION")); Self::add_methods(&mut doc); #grace #(doc.endpoint::<#types>(#names);)* ::sausage_plugin::finish_document(doc)}
fn discover(&self)->::sausage_plugin::serde_json::Value {<Self as ::sausage_plugin::Interface>::describe()} fn discover(&self)->::sausage_plugin::serde_json::Value {<Self as ::sausage_plugin::Interface>::describe()}
fn dispatch<'a>(&'a self, method:&'a str, params: ::sausage_plugin::serde_json::Value,ctx:&'a ::sausage_plugin::Ctx)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=::sausage_plugin::Result<::sausage_plugin::serde_json::Value>>+'a>> {Box::pin(async move {if !params.is_object() {return Err(::sausage_plugin::Error::new(-32602,"params must be an object"));} match method {#(#arms,)* _=>Err(::sausage_plugin::Error::new(-32601,"method not found"))}})} fn dispatch<'a>(&'a self, method:&'a str, params: ::sausage_plugin::serde_json::Value,ctx:&'a ::sausage_plugin::Ctx)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=::sausage_plugin::Result<::sausage_plugin::serde_json::Value>>+'a>> {Box::pin(async move {if !params.is_object() {return Err(::sausage_plugin::Error::params_not_object(&params));} match method {#(#arms,)* _=>Err(::sausage_plugin::Error::new(-32601,"method not found"))}})}
#stop #stop
}}) }})
})(); })();
@@ -328,6 +342,12 @@ pub fn import(input: TokenStream) -> TokenStream {
pub fn on_retired(_: TokenStream, input: TokenStream) -> TokenStream { pub fn on_retired(_: TokenStream, input: TokenStream) -> TokenStream {
input input
} }
/// Mark the method that handles `rpc.superseded` at the start of upgrade grace.
/// Calls still work here. Use `sausage_plugin::drain` to finish active work and exit.
#[proc_macro_attribute]
pub fn on_superseded(_: TokenStream, input: TokenStream) -> TokenStream {
input
}
/// Mark the hook that runs at stdin EOF without a live call transport. /// Mark the hook that runs at stdin EOF without a live call transport.
#[proc_macro_attribute] #[proc_macro_attribute]
pub fn on_stop(_: TokenStream, input: TokenStream) -> TokenStream { pub fn on_stop(_: TokenStream, input: TokenStream) -> TokenStream {
+184 -2
View File
@@ -1,5 +1,19 @@
//! Outgoing core calls and notification callbacks. //! Outgoing core calls and notification callbacks.
use super::*; use super::*;
/// Removes a dropped call's local routing state, without cancelling host work.
struct PendingCall(u64);
impl Drop for PendingCall {
fn drop(&mut self) {
let pending = STATE.with(|s| s.borrow_mut().pending.remove(&self.0));
if let Some(mut pending) = pending {
// A lock may still be granted: release it when its late response arrives.
if let Some(lease) = pending.unlock.take() {
STATE.with(|s| s.borrow_mut().abandoned_locks.insert(self.0, lease));
}
// Drop user-owned callback captures outside the STATE borrow.
}
}
}
/// An outgoing request builder. /// An outgoing request builder.
#[must_use = "await the call to send it"] #[must_use = "await the call to send it"]
pub struct Call { pub struct Call {
@@ -26,6 +40,21 @@ impl Call {
self self
} }
/// Receive progress notifications before the result arrives. /// Receive progress notifications before the result arrives.
/// Dropping the call future removes its local pending entry and callback;
/// later notifications and responses are ignored. It does not cancel host work.
///
/// For `core.watch`, poll the call future alongside work in the handler, then
/// drop it when the handler returns. Do not put it in `background()` just to
/// keep the watch alive: background tasks keep the process alive, while the
/// watch only ends when the process ends. Dropping the future does not cancel
/// the host watch, but it lets an otherwise idle process exit normally. A
/// long-lived service should keep one watch per purpose for its lifetime,
/// rather than repeatedly opening and dropping watches on the host.
///
/// Scan only after `armed`, and rescan on `changed`. Events may coalesce, but
/// are not rate-limited: watch the narrowest useful path to avoid rescanning
/// for unrelated writes (for example, watch a mailbox, not all session data).
/// Do not use endpoint connect retries to wait for file publication.
pub fn on_notify(mut self, f: impl FnMut(&str, Value) + 'static) -> Self { pub fn on_notify(mut self, f: impl FnMut(&str, Value) + 'static) -> Self {
self.notify = Some(Box::new(f)); self.notify = Some(Box::new(f));
self self
@@ -37,7 +66,7 @@ impl IntoFuture for Call {
fn into_future(self) -> Self::IntoFuture { fn into_future(self) -> Self::IntoFuture {
Box::pin(async move { Box::pin(async move {
if !self.params.is_object() { if !self.params.is_object() {
return Err(Error::new(-32602, "params must be an object")); return Err(Error::params_not_object(&self.params));
} }
let (tx, rx) = oneshot::channel(); let (tx, rx) = oneshot::channel();
let id = STATE.with(|s| { let id = STATE.with(|s| {
@@ -60,6 +89,7 @@ impl IntoFuture for Call {
); );
Ok(id) Ok(id)
})?; })?;
let _pending = PendingCall(id);
let (method, params) = if self.detach { let (method, params) = if self.detach {
( (
"core.call".to_string(), "core.call".to_string(),
@@ -69,7 +99,8 @@ impl IntoFuture for Call {
(self.method, self.params) (self.method, self.params)
}; };
if let Err(e) = send(json!({"jsonrpc":"2.0","id":id,"method":method,"params":params})) { if let Err(e) = send(json!({"jsonrpc":"2.0","id":id,"method":method,"params":params})) {
STATE.with(|s| s.borrow_mut().pending.remove(&id)); let pending = STATE.with(|s| s.borrow_mut().pending.remove(&id));
drop(pending);
return Err(e); return Err(e);
} }
rx.await rx.await
@@ -77,3 +108,154 @@ impl IntoFuture for Call {
}) })
} }
} }
#[cfg(test)]
mod tests {
use super::{STATE, State, block_on, call, clear_pending, lock, receive};
use crate::Result;
use futures::{FutureExt, future::LocalBoxFuture};
use serde_json::{Value, json};
use std::{cell::RefCell, future::IntoFuture, rc::Rc, task::Context};
fn activate() {
STATE.with(|s| {
*s.borrow_mut() = State {
active: true,
..State::default()
};
});
}
fn start(
mut call: LocalBoxFuture<'static, Result<Value>>,
) -> (u64, LocalBoxFuture<'static, Result<Value>>) {
assert!(
call.as_mut()
.poll(&mut Context::from_waker(std::task::Waker::noop()))
.is_pending()
);
(STATE.with(|s| s.borrow().next), call)
}
fn notification(id: u64, name: &str) {
receive(json!({"method":"core.notify","params":{"call":id,"name":name,"params":{}}}));
}
fn response(id: u64, result: Value) {
receive(json!({"id":id,"result":result}));
}
#[test]
fn dropped_call_removes_pending_and_callback() {
activate();
let events = Rc::new(RefCell::new(Vec::new()));
let captured = events.clone();
let (id, future) = start(
call("core.watch", json!({"path":"/mailbox"}))
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
.into_future(),
);
notification(id, "armed");
assert_eq!(&*events.borrow(), &["armed"]);
assert_eq!(Rc::strong_count(&events), 2);
drop(future);
STATE.with(|s| {
assert!(s.borrow().pending.is_empty());
assert!(s.borrow().abandoned_locks.is_empty());
});
assert_eq!(Rc::strong_count(&events), 1);
notification(id, "changed");
response(id, Value::Null);
receive(json!({"id":id,"error":{"code":-32000,"message":"late error"}}));
assert_eq!(&*events.borrow(), &["armed"]);
}
#[test]
fn kept_watch_delivers_until_result() {
activate();
let events = Rc::new(RefCell::new(Vec::new()));
let captured = events.clone();
let (id, future) = start(
call("core.watch", json!({"path":"/mailbox"}))
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
.into_future(),
);
notification(id, "armed");
notification(id, "changed");
notification(id, "changed");
response(id, json!({"done":true}));
assert_eq!(block_on(future).unwrap(), json!({"done":true}));
notification(id, "late");
assert_eq!(&*events.borrow(), &["armed", "changed", "changed"]);
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
assert_eq!(Rc::strong_count(&events), 1);
}
#[test]
fn callback_can_drop_its_own_call_without_being_restored() {
activate();
let holder = Rc::new(RefCell::new(None));
let captured = holder.clone();
let (id, future) = start(
call("core.watch", json!({"path":"/mailbox"}))
.on_notify(move |_, _| {
// Panics on a second invocation; the first must remove the entry.
drop(captured.borrow_mut().take().expect("live call"));
})
.into_future(),
);
*holder.borrow_mut() = Some(future);
notification(id, "armed");
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
notification(id, "changed");
response(id, Value::Null);
assert_eq!(Rc::strong_count(&holder), 1);
}
#[test]
fn eof_cleanup_can_drop_callback_owned_futures() {
activate();
let holder = Rc::new(RefCell::new(None));
let weak = Rc::downgrade(&holder);
let captured = holder.clone();
let (_, future) = start(
call("core.watch", json!({"path":"/mailbox"}))
.on_notify(move |_, _| {
let _ = &captured;
})
.into_future(),
);
*holder.borrow_mut() = Some(future);
drop(holder);
STATE.with(|s| s.borrow_mut().stopping = true);
clear_pending();
assert!(weak.upgrade().is_none());
STATE.with(|s| assert!(s.borrow().pending.is_empty()));
}
#[test]
fn unpolled_call_never_registers() {
activate();
drop(call("core.watch", json!({"path":"/mailbox"})).into_future());
STATE.with(|s| {
assert!(s.borrow().pending.is_empty());
assert_eq!(s.borrow().next, 0);
});
}
#[test]
fn dropped_lock_keeps_only_late_grant_cleanup() {
activate();
let (id, future) = start(
lock("/late")
.into_future()
.map(|r| r.map(|_| Value::Null))
.boxed_local(),
);
drop(future);
STATE.with(|s| {
assert!(s.borrow().pending.is_empty());
assert_eq!(s.borrow().abandoned_locks.len(), 1);
});
notification(id, "ignored");
response(id, json!({"held":true}));
STATE.with(|s| assert!(s.borrow().abandoned_locks.is_empty()));
}
}
+7 -3
View File
@@ -65,7 +65,7 @@ impl<T: serde::de::DeserializeOwned + 'static> IntoFuture for EndpointCall<T> {
} }
} }
/// Publish a declared interface and keep its listener alive until stdin EOF. /// Publish a declared interface and keep its listener alive until stdin EOF or [`drain`].
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Result<()> { pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Result<()> {
use wstd::iter::AsyncIterator; use wstd::iter::AsyncIterator;
@@ -80,7 +80,7 @@ pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Resul
while let Some(Ok(stream)) = incoming.next().await { while let Some(Ok(stream)) = incoming.next().await {
let value = value.clone(); let value = value.clone();
let name = name.clone(); let name = name.clone();
crate::transport::endpoint_task(async move { background(async move {
let _ = serve_connection(stream, value, name).await; let _ = serve_connection(stream, value, name).await;
}); });
} }
@@ -211,7 +211,11 @@ async fn serve_connection<T: Interface>(
use futures::{FutureExt, StreamExt}; use futures::{FutureExt, StreamExt};
let (mut reader, mut writer) = stream.split(); let (mut reader, mut writer) = stream.split();
let mut lines = crate::lines::Lines::default(); let mut lines = crate::lines::Lines::default();
while let Some(request) = lines.rpc(&mut reader).await? { while let Some(request) = crate::transport::until_drained(lines.rpc(&mut reader))
.await
.transpose()?
.flatten()
{
let method = request["method"].as_str().unwrap_or_default(); let method = request["method"].as_str().unwrap_or_default();
let (tx, mut rx) = futures::channel::mpsc::unbounded(); let (tx, mut rx) = futures::channel::mpsc::unbounded();
let ctx = Ctx { let ctx = Ctx {
+19
View File
@@ -15,6 +15,25 @@ pub struct Error {
pub data: Option<Value>, pub data: Option<Value>,
} }
impl Error { impl Error {
/// Build the object-params diagnostic used by calls and generated dispatch.
#[doc(hidden)]
pub fn params_not_object(params: &Value) -> Self {
let kind = match params {
Value::String(_) => "a string",
Value::Array(_) => "an array",
Value::Number(_) => "a number",
Value::Null => "null",
Value::Bool(_) => "a boolean",
Value::Object(_) => "an object",
};
Self::new(
-32602,
format!(
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
),
)
}
/// Construct an error without structured details. /// Construct an error without structured details.
pub fn new(code: i64, message: impl Into<String>) -> Self { pub fn new(code: i64, message: impl Into<String>) -> Self {
Self { Self {
+24 -3
View File
@@ -356,13 +356,20 @@
//! //!
//! The interface check covers calls, not state. Old and new processes share files in `/state`. //! The interface check covers calls, not state. Old and new processes share files in `/state`.
//! `core.revert` restores code, not state. Preserve compatibility or migrate the state. //! `core.revert` restores code, not state. Preserve compatibility or migrate the state.
//! `#[rpc(grace_seconds = N)]` declares the grace period for old processes. The default is 60 seconds. //! Core gives every old process 5 seconds of upgrade grace, then closes stdin and allows
//! a separate 5-second stop budget before killing it. Discovery-only processes are excluded.
//! `#[rpc(grace_seconds = N)]` still emits legacy `x-sausage-grace` metadata, but core ignores it.
//! New calls use the new version immediately. Old processes can still use old imports during their grace period. //! New calls use the new version immediately. Old processes can still use old imports during their grace period.
//! [`macro@on_superseded`] handles the notice at grace start, with normal calls still available.
//! It is opt-in: without a hook the process continues as before until core stops it.
//! Call [`drain`] in this hook to close SDK endpoint listeners and idle connections,
//! let active handlers and background tasks finish, and exit without waiting for grace.
//! Custom listeners and long-lived background tasks must be ended by the plugin itself.
//! [`macro@on_retired`] handles `rpc.retired` after the last process of an old version ends. //! [`macro@on_retired`] handles `rpc.retired` after the last process of an old version ends.
//! This hook is best effort. If the host stops first, the hook does not run. //! This hook is best effort. If the host stops first, the hook does not run.
//! New processes must handle unmigrated state. Check a migration marker and migrate if it is absent. //! New processes must handle unmigrated state. Check a migration marker and migrate if it is absent.
//! See section 3.7 of `/proc/core/PROTOCOL.md`. //! See section 3.7 of `/proc/core/PROTOCOL.md`.
//! Declare at most one stop hook and one retirement hook per interface. //! Declare at most one hook of each kind per interface.
//! //!
//! ```no_run //! ```no_run
//! use sausage_plugin::{rpc, schema, state_dir, write_atomic, Result}; //! use sausage_plugin::{rpc, schema, state_dir, write_atomic, Result};
@@ -374,8 +381,21 @@
//! /// The SHA-256 of the previous WASM file. //! /// The SHA-256 of the previous WASM file.
//! previous: String, //! previous: String,
//! } //! }
//! /// The version now receiving new calls.
//! #[schema]
//! struct SupersededParams {
//! /// The SHA-256 of the newly active WASM file.
//! current: String,
//! }
//! #[rpc] //! #[rpc]
//! impl Migrator { //! impl Migrator {
//! /// Finish current work and exit early during upgrade grace.
//! #[sausage_plugin::on_superseded]
//! fn superseded(&self, p: SupersededParams) -> Result<()> {
//! eprintln!("replaced by {}", p.current);
//! sausage_plugin::drain();
//! Ok(())
//! }
//! /// Record a retired version. Normal methods must also handle missing markers. //! /// Record a retired version. Normal methods must also handle missing markers.
//! #[sausage_plugin::on_retired] //! #[sausage_plugin::on_retired]
//! fn retired(&self, p: RetiredParams) -> Result<()> { //! fn retired(&self, p: RetiredParams) -> Result<()> {
@@ -392,6 +412,7 @@ mod error;
mod helpers; mod helpers;
mod imports; mod imports;
mod lines; mod lines;
mod sqlite;
mod transport; mod transport;
pub use discovery::*; pub use discovery::*;
pub use endpoints::*; pub use endpoints::*;
@@ -400,6 +421,6 @@ pub use helpers::*;
pub use imports::*; pub use imports::*;
#[doc(hidden)] #[doc(hidden)]
pub use inventory; pub use inventory;
pub use sausage_plugin_macros::{import, on_retired, on_stop, rpc, schema, test}; pub use sausage_plugin_macros::{import, on_retired, on_stop, on_superseded, rpc, schema, test};
pub use transport::*; pub use transport::*;
pub use {schemars, serde, serde_json}; pub use {schemars, serde, serde_json};
+43
View File
@@ -0,0 +1,43 @@
//! Shared SQLite open policy, expanded against the plugin's own rusqlite version.
/// Open SQLite under an already-held Sausage host lock. The caller must depend
/// on `rusqlite` (bundled) and hold the SAME host lock for EVERY connection's
/// entire lifetime, including reads and old/new plugin versions. This macro does
/// not acquire the lock. Do not use it for the host journal or databases accessed
/// outside Sausage. Native builds retain OS locks; WASI selects `unix-none`, so
/// crash-leftover dotfile lock directories cannot block recovery.
///
/// Temporary tables/sorts stay in memory before any schema access. Writable opens
/// use DELETE journals; SQLite rolls back hot journals itself. Never delete a
/// leftover journal. Read-only opens cannot recover a hot journal: recover with
/// a writable open first.
///
/// The flags form preserves the caller's open flags. A macro avoids coupling the
/// SDK to a particular rusqlite/libsqlite3-sys version (the host and plugins use
/// different versions, which cannot coexist as dependencies due to `links`).
///
/// ```ignore
/// let _guard = sausage_plugin::lock("/state/example/db.lock").await?;
/// let c = sausage_plugin::sqlite_open!("/state/example/db.sqlite")?;
/// // Query here; c must drop before _guard.
/// ```
#[macro_export]
macro_rules! sqlite_open {
($path:expr) => {
$crate::sqlite_open!($path, ::rusqlite::OpenFlags::default())
};
($path:expr, $flags:expr) => {{
(|| -> ::rusqlite::Result<::rusqlite::Connection> {
let flags = $flags;
#[cfg(target_os = "wasi")]
let c = ::rusqlite::Connection::open_with_flags_and_vfs($path, flags, "unix-none")?;
#[cfg(not(target_os = "wasi"))]
let c = ::rusqlite::Connection::open_with_flags($path, flags)?;
c.pragma_update(None, "temp_store", "MEMORY")?;
if !flags.contains(::rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY) {
c.pragma_update(None, "journal_mode", "DELETE")?;
}
Ok(c)
})()
}};
}
+63 -8
View File
@@ -96,8 +96,11 @@ struct State {
active: bool, active: bool,
endpoints: HashMap<std::any::TypeId, String>, endpoints: HashMap<std::any::TypeId, String>,
stopping: bool, stopping: bool,
draining: bool,
next: u64, next: u64,
pending: HashMap<u64, Pending>, pending: HashMap<u64, Pending>,
// Only leases, never callbacks or response senders, survive dropped lock acquisitions.
abandoned_locks: HashMap<u64, std::rc::Rc<Lease>>,
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
tasks: usize, tasks: usize,
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
@@ -108,6 +111,18 @@ struct State {
listeners: HashMap<u64, futures::future::AbortHandle>, listeners: HashMap<u64, futures::future::AbortHandle>,
} }
thread_local! { static STATE: RefCell<State> = RefCell::new(State::default()); } thread_local! { static STATE: RefCell<State> = RefCell::new(State::default()); }
#[cfg(any(target_os = "wasi", test))]
fn clear_pending() {
let pending = STATE.with(|s| {
let mut s = s.borrow_mut();
(
std::mem::take(&mut s.pending),
std::mem::take(&mut s.abandoned_locks),
)
});
// Callback captures may own other call futures whose drop guards use STATE.
drop(pending);
}
pub(crate) fn endpoint_name<T: 'static>() -> Result<String> { pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
STATE STATE
.with(|s| { .with(|s| {
@@ -125,17 +140,48 @@ pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
} }
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
pub(crate) fn endpoint_task(future: impl Future<Output = ()> + 'static) { pub(crate) fn endpoint_task(future: impl Future<Output = ()> + 'static) {
background(async move {
let _ = until_drained(future).await;
});
}
/// Cancel an accept/read wait when draining, but never an active handler.
#[cfg(target_os = "wasi")]
pub(crate) async fn until_drained<F: Future>(future: F) -> Option<F::Output> {
let (handle, registration) = futures::future::AbortHandle::new_pair(); let (handle, registration) = futures::future::AbortHandle::new_pair();
let id = STATE.with(|s| { let id = STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
if s.draining {
return None;
}
s.next += 1; s.next += 1;
let id = s.next; let id = s.next;
s.listeners.insert(id, handle); s.listeners.insert(id, handle);
id Some(id)
}); })?;
background(async move { let result = futures::future::Abortable::new(future, registration)
let _ = futures::future::Abortable::new(future, registration).await; .await
.ok();
STATE.with(|s| s.borrow_mut().listeners.remove(&id)); STATE.with(|s| s.borrow_mut().listeners.remove(&id));
result
}
/// Stop accepting SDK endpoint work, then exit when handlers and background tasks finish.
///
/// Calls and notifications still work. This does not run the EOF stop hook or cancel
/// background tasks: services with their own listeners/tasks must arrange for those
/// to finish too. Repeated calls are harmless. Core's upgrade grace remains the bound.
pub fn drain() {
STATE.with(|s| {
let mut s = s.borrow_mut();
s.draining = true;
#[cfg(target_os = "wasi")]
{
for (_, listener) in s.listeners.drain() {
listener.abort();
}
if let Some(w) = s.wake.take() {
w.wake();
}
}
}); });
} }
#[cfg(target_os = "wasi")] #[cfg(target_os = "wasi")]
@@ -192,16 +238,16 @@ pub fn test_run<F: Future>(f: F) -> F::Output {
STATE.with(|s| { STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
s.stopping = true; s.stopping = true;
s.pending.clear();
}); });
clear_pending();
}); });
let result = f.await; let result = f.await;
drop(reader); drop(reader);
STATE.with(|s| { STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
s.active = false; s.active = false;
s.pending.clear();
}); });
clear_pending();
result result
}) })
} }
@@ -273,9 +319,18 @@ pub fn receive(value: Value) {
.set(result.as_ref().is_ok_and(|v| v["held"] == true)); .set(result.as_ref().is_ok_and(|v| v["held"] == true));
} }
let _ = p.tx.send(result); let _ = p.tx.send(result);
} else {
let lease = value["id"]
.as_u64()
.and_then(|id| STATE.with(|s| s.borrow_mut().abandoned_locks.remove(&id)));
if let Some(lease) = lease {
lease
.held
.set(value.get("error").is_none() && value["result"]["held"] == true);
} else { } else {
eprintln!("note: response with unknown ID {}", value["id"]); eprintln!("note: response with unknown ID {}", value["id"]);
} }
}
} }
/// Serve the main interface until its work ends or stdin reaches EOF. /// Serve the main interface until its work ends or stdin reaches EOF.
@@ -324,11 +379,11 @@ async fn serve<T: Interface>(value: T) {
STATE.with(|s| { STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
s.stopping = true; s.stopping = true;
s.pending.clear();
for (_, listener) in s.listeners.drain() { for (_, listener) in s.listeners.drain() {
listener.abort(); listener.abort();
} }
}); });
clear_pending();
value.stop().await; value.stop().await;
break; break;
} }
@@ -379,8 +434,8 @@ async fn serve<T: Interface>(value: T) {
STATE.with(|s| { STATE.with(|s| {
let mut s = s.borrow_mut(); let mut s = s.borrow_mut();
s.active = false; s.active = false;
s.pending.clear();
}); });
clear_pending();
let _ = std::io::stdout().flush(); let _ = std::io::stdout().flush();
std::process::exit(0); std::process::exit(0);
} }
+124
View File
@@ -2,6 +2,12 @@
use sausage_plugin::serde_json::{Value, json}; use sausage_plugin::serde_json::{Value, json};
use sausage_plugin::{Ctx, Result, call, lock, rpc, schema}; use sausage_plugin::{Ctx, Result, call, lock, rpc, schema};
struct Probe; struct Probe;
/// A virtual path to observe.
#[schema]
struct WatchPath {
/// Directory or file, not necessarily present yet.
path: String,
}
/// The values to add. /// The values to add.
#[schema] #[schema]
struct Add { struct Add {
@@ -16,6 +22,12 @@ struct Retired {
/// The previous hash. /// The previous hash.
previous: String, previous: String,
} }
/// The replacement version.
#[schema]
struct Superseded {
/// The current hash.
current: String,
}
/// A large payload. /// A large payload.
#[schema] #[schema]
struct Echo { struct Echo {
@@ -24,6 +36,98 @@ struct Echo {
} }
#[rpc(grace_seconds = 12,endpoints(control=Control))] #[rpc(grace_seconds = 12,endpoints(control=Control))]
impl Probe { impl Probe {
/// Scan only after the watch is armed; keep the call open until process death.
async fn watch(&self, p: WatchPath) -> Result<()> {
let path = p.path.clone();
call("core.watch", json!({"path":p.path}))
.on_notify(move |name, _| {
if name == "armed" && path == "/self" {
std::fs::write("/self/owned", "visible via proc").unwrap();
}
let mut entries: Vec<_> = std::fs::read_dir(&path)
.into_iter()
.flatten()
.filter_map(std::result::Result::ok)
.map(|e| e.file_name().into_string().unwrap())
.collect();
entries.sort();
let text = std::fs::read_to_string(format!("{path}/before")).ok();
Ctx::default()
.notify(
name,
json!({"pid":sausage_plugin::pid(), "entries":entries, "text":text}),
)
.unwrap();
})
.await?;
Ok(())
}
/// Poll the watch in the handler, then return idle without a background task.
async fn watch_once(&self, p: WatchPath) -> Result<()> {
use std::{
cell::RefCell,
rc::Rc,
task::{Poll, Waker},
};
let state = Rc::new(RefCell::new((false, None::<Waker>)));
let received = state.clone();
let mut watch = std::pin::pin!(std::future::IntoFuture::into_future(
call("core.watch", json!({"path":p.path})).on_notify(move |name, _| {
if name == "armed" {
let mut state = received.borrow_mut();
state.0 = true;
if let Some(waker) = state.1.take() {
waker.wake();
}
}
})
));
std::future::poll_fn(|cx| {
if let Poll::Ready(result) = watch.as_mut().poll(cx) {
return Poll::Ready(result.map(|_| ()));
}
let mut state = state.borrow_mut();
state.1 = Some(cx.waker().clone());
if state.0 {
Poll::Ready(Ok(()))
} else {
Poll::Pending
}
})
.await
}
/// Drop one watch while another still delivers, in a process that remains alive.
async fn drop_watch(&self) -> Result<Vec<String>> {
use std::{cell::RefCell, future::IntoFuture, rc::Rc, task::Poll};
let events = Rc::new(RefCell::new(Vec::new()));
let captured = events.clone();
let mut dropped = call("core.watch", json!({"path":"/dropped"}))
.on_notify(move |_, _| captured.borrow_mut().push("dropped".to_owned()))
.into_future();
std::future::poll_fn(|cx| {
assert!(dropped.as_mut().poll(cx).is_pending());
Poll::Ready(())
})
.await;
drop(dropped);
let captured = events.clone();
let mut kept = call("core.watch", json!({"path":"/kept"}))
.on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
.into_future();
std::future::poll_fn(|cx| {
assert!(kept.as_mut().poll(cx).is_pending());
Poll::Ready(())
})
.await;
// The harness sends late traffic and kept-watch events before answering.
call("other.barrier", json!({})).await?;
drop(kept);
let result = events.borrow().clone();
Ok(result)
}
/// Publish a control endpoint until EOF. /// Publish a control endpoint until EOF.
async fn serve(&self) -> Result<()> { async fn serve(&self) -> Result<()> {
sausage_plugin::listen("/endpoint", std::rc::Rc::new(Control)).await sausage_plugin::listen("/endpoint", std::rc::Rc::new(Control)).await
@@ -54,6 +158,11 @@ impl Probe {
sausage_plugin::listen("/state/control", std::rc::Rc::new(Control)).await?; sausage_plugin::listen("/state/control", std::rc::Rc::new(Control)).await?;
Ok(sausage_plugin::pid()) Ok(sausage_plugin::pid())
} }
/// Keep an idle listener and ownership lock until opt-in drain ends this process.
async fn drain_service(&self) -> Result<String> {
lock("/drain-owner").await?.keep();
self.service().await
}
/// Add two numbers. /// Add two numbers.
/// ///
/// The result is their sum. /// The result is their sum.
@@ -111,6 +220,15 @@ impl Probe {
fn retired(&self, p: Retired) -> Result<String> { fn retired(&self, p: Retired) -> Result<String> {
Ok(p.previous) Ok(p.previous)
} }
/// Drain on replacement while calls are still available.
#[sausage_plugin::on_superseded]
async fn superseded(&self, p: Superseded) -> Result<()> {
call("other.superseded", json!({"current":p.current})).await?;
sausage_plugin::drain();
sausage_plugin::drain();
Ctx::default().notify("draining", json!({}))?;
Ok(())
}
/// Wait until EOF, then run the stop hook. /// Wait until EOF, then run the stop hook.
async fn wait(&self) -> Result<String> { async fn wait(&self) -> Result<String> {
call("other.wait", json!({})).await?; call("other.wait", json!({})).await?;
@@ -155,6 +273,7 @@ impl Probe {
#[sausage_plugin::on_stop] #[sausage_plugin::on_stop]
async fn stop(&self) { async fn stop(&self) {
std::fs::write("/stopped", b"yes").unwrap(); std::fs::write("/stopped", b"yes").unwrap();
std::fs::write(format!("/stopped-{}", sausage_plugin::pid()), b"yes").unwrap();
Ctx::default().notify("stopped", json!({})).unwrap(); Ctx::default().notify("stopped", json!({})).unwrap();
assert!(call("other.no", json!({})).await.is_err()); assert!(call("other.no", json!({})).await.is_err());
} }
@@ -193,6 +312,11 @@ impl Control {
ctx.notify("endpoint-progress", json!({"value":7}))?; ctx.notify("endpoint-progress", json!({"value":7}))?;
Ok(7) Ok(7)
} }
/// Finish an in-flight call after draining begins.
async fn wait(&self) -> Result<String> {
call("other.inflight", json!({})).await?;
Ok("finished".into())
}
/// End the endpoint process after the response. /// End the endpoint process after the response.
fn stop(&self, ctx: &Ctx) -> Result<()> { fn stop(&self, ctx: &Ctx) -> Result<()> {
ctx.exit_after_response(); ctx.exit_after_response();
+47
View File
@@ -116,6 +116,53 @@ fn discovery_and_dispatch() {
.into(); .into();
assert_eq!(internal.code, -32603); assert_eq!(internal.code, -32603);
} }
#[sausage_plugin::test]
async fn params_must_be_objects() {
for (params, kind) in [
(json!("{\"a\":2,\"b\":3}"), "a string"),
(json!([]), "an array"),
(json!(42), "a number"),
(json!(null), "null"),
(json!(true), "a boolean"),
] {
let message = format!(
"params must be a JSON object, got {kind}. Send the object itself, not a JSON-encoded string."
);
let outgoing = sausage_plugin::call("plugin.add", params.clone())
.await
.unwrap_err();
assert_eq!(outgoing.code, -32602);
assert_eq!(outgoing.message, message);
for method in ["add", "choice", "unknown"] {
let incoming = Plugin
.dispatch(method, params.clone(), &Ctx::default())
.await
.unwrap_err();
assert_eq!(incoming.code, -32602);
assert_eq!(incoming.message, message);
}
}
assert_eq!(
Plugin
.dispatch("add", json!({"a":2,"b":3}), &Ctx::default())
.await
.unwrap(),
5
);
assert_eq!(
Plugin
.dispatch("defaults", json!({}), &Ctx::default())
.await
.unwrap(),
json!({"value":""})
);
// An object reaches transport validation rather than the params rejection.
let outgoing = sausage_plugin::call("plugin.add", json!({"a":2,"b":3}))
.await
.unwrap_err();
assert!(outgoing.message.contains("no core transport"));
}
#[sausage_plugin::test] #[sausage_plugin::test]
async fn missing_core_fails_without_wait() { async fn missing_core_fails_without_wait() {
assert!( assert!(
+11
View File
@@ -0,0 +1,11 @@
struct Supersedes;
#[sausage_plugin::rpc]
impl Supersedes {
/// First superseded hook.
#[sausage_plugin::on_superseded]
fn a(&self) -> sausage_plugin::Result<()> { Ok(()) }
/// Second superseded hook.
#[sausage_plugin::on_superseded]
fn b(&self) -> sausage_plugin::Result<()> { Ok(()) }
}
fn main() {}
+5
View File
@@ -0,0 +1,5 @@
error: only one on_superseded method is allowed
--> tests/ui/superseded.rs:7:5
|
7 | /// Second superseded hook.
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
+125
View File
@@ -108,6 +108,34 @@ fn protocol() {
} }
p.finishes(); p.finishes();
} }
// A handler-owned watch does not keep an idle process alive after return,
// even though the host never answers the watch and stdin remains open.
let mut p = start(&wasm, &dir);
p.request("watch_once", json!({"path":"/mailbox"}));
let watch = p.read();
assert_eq!(watch["method"], "core.watch");
p.send(json!({"jsonrpc":"2.0","method":"core.notify","params":{"call":watch["id"],"name":"armed","params":{}}}));
assert_eq!(p.read()["result"], Value::Null);
p.finishes();
// Keep the process alive after dropping a call: late traffic is ignored,
// while the kept watch continues to deliver every notification.
let mut p = start(&wasm, &dir);
p.request("drop_watch", json!({}));
let dropped = p.read();
let kept = p.read();
let barrier = p.read();
assert_eq!(dropped["params"]["path"], "/dropped");
assert_eq!(kept["params"]["path"], "/kept");
assert_eq!(barrier["method"], "other.barrier");
for watch in [&dropped, &kept] {
for name in ["armed", "changed", "changed"] {
p.send(json!({"jsonrpc":"2.0","method":"core.notify","params":{"call":watch["id"],"name":name,"params":{}}}));
}
}
p.answer(&dropped, Value::Null);
p.answer(&barrier, json!({}));
assert_eq!(p.read()["result"], json!(["armed", "changed", "changed"]));
p.finishes();
let mut p = start(&wasm, &dir); let mut p = start(&wasm, &dir);
p.request("rpc.discover", json!({})); p.request("rpc.discover", json!({}));
let d = p.read(); let d = p.read();
@@ -201,6 +229,103 @@ fn protocol() {
assert_eq!(outgoing["method"], "other.background"); assert_eq!(outgoing["method"], "other.background");
p.answer(&outgoing, json!({})); p.answer(&outgoing, json!({}));
assert!(p.child.wait().unwrap().success()); assert!(p.child.wait().unwrap().success());
// Replacement is a notification, not EOF. Drain keeps both endpoint handlers
// and background calls alive, while closing idle connections and the listener.
assert!(
!d["result"]["methods"]
.as_array()
.unwrap()
.iter()
.any(|m| m["name"] == "rpc.superseded" || m["name"] == "superseded")
);
std::fs::remove_file(dir.join("stopped")).unwrap();
let mut provider = start(&wasm, &dir);
provider.request("serve", json!({}));
assert_eq!(provider.read()["result"], Value::Null);
provider.request("start", json!({}));
let first = provider.read();
let second = provider.read();
let background = if first["method"] == "other.background" {
first
} else {
second
};
assert_eq!(background["method"], "other.background");
let address = std::fs::read_to_string(dir.join("endpoint"))
.unwrap()
.split_whitespace()
.next()
.unwrap()
.trim_start_matches("tcp:")
.to_owned();
let mut idle = std::net::TcpStream::connect(&address).unwrap();
idle.set_read_timeout(Some(std::time::Duration::from_secs(4)))
.unwrap();
writeln!(
idle,
"{}",
json!({"jsonrpc":"2.0","id":0,"method":"rpc.endpoint","params":{}})
)
.unwrap();
let mut idle = BufReader::new(idle);
let mut line = String::new();
idle.read_line(&mut line).unwrap();
assert_eq!(
sausage_plugin::serde_json::from_str::<Value>(&line).unwrap()["id"],
0
);
let mut active = std::net::TcpStream::connect(&address).unwrap();
active
.set_read_timeout(Some(std::time::Duration::from_secs(4)))
.unwrap();
writeln!(
active,
"{}",
json!({"jsonrpc":"2.0","id":1,"method":"wait","params":{}})
)
.unwrap();
let inflight = provider.read();
assert_eq!(inflight["method"], "other.inflight");
provider
.send(json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":"new-hash"}}));
let hook = provider.read();
assert_eq!(hook["method"], "other.superseded");
assert_eq!(hook["params"]["current"], "new-hash");
provider.answer(&hook, json!({}));
assert_eq!(provider.read()["params"]["name"], "draining");
line.clear();
assert_eq!(idle.read_line(&mut line).unwrap(), 0);
// drain() wakes independent accept/read cancellations. Idle EOF does not
// order listener destruction: TCP may still complete a handshake, but that
// connection must close without serving work. Do not retry the connect.
if let Ok(mut late) = std::net::TcpStream::connect(&address) {
eprintln!("TCP handshake raced listener cancellation; waiting for close");
// Reuse the four-second socket bound used by idle/active above.
late.set_read_timeout(Some(std::time::Duration::from_secs(4)))
.unwrap();
let closed = std::io::Read::read(&mut late, &mut [0]);
assert!(
matches!(closed, Ok(0))
|| matches!(&closed, Err(e) if e.kind() == std::io::ErrorKind::ConnectionReset),
"late connection survived drain: {closed:?}"
);
}
assert!(provider.child.try_wait().unwrap().is_none());
provider.answer(&inflight, json!({}));
line.clear();
BufReader::new(active).read_line(&mut line).unwrap();
assert_eq!(
sausage_plugin::serde_json::from_str::<Value>(&line).unwrap()["result"],
"finished"
);
assert!(provider.child.try_wait().unwrap().is_none());
provider.answer(&background, json!({}));
provider.finishes();
assert!(std::net::TcpStream::connect(&address).is_err());
assert!(
!dir.join("stopped").exists(),
"drain is not the EOF stop hook"
);
let provider_doc = &d["result"]; let provider_doc = &d["result"];
let imports = target.join("imports"); let imports = target.join("imports");
std::fs::create_dir_all(&imports).unwrap(); std::fs::create_dir_all(&imports).unwrap();
+1
View File
@@ -5,6 +5,7 @@ pub mod generated;
pub mod host; pub mod host;
pub mod memory; pub mod memory;
pub mod mounts; pub mod mounts;
pub mod observe;
pub mod process; pub mod process;
mod readonly; mod readonly;
mod streams; mod streams;
+615
View File
@@ -0,0 +1,615 @@
//! Observes successful namespace and open-file mutations, even if their caller is cancelled.
use crate::*;
use std::{
collections::BTreeMap,
sync::{Mutex, Weak},
};
use tokio::sync::{OwnedRwLockReadGuard, OwnedRwLockWriteGuard, RwLock};
type Namespace = Mutex<BTreeMap<String, Weak<RwLock<()>>>>;
enum Held {
Read { _guard: OwnedRwLockReadGuard<()> },
Write { _guard: OwnedRwLockWriteGuard<()> },
}
type TrackedPath = Mutex<Option<String>>;
type Path = Arc<TrackedPath>;
type Callback = dyn Fn(&str) + Send + Sync;
/// Shared observation for mount namespaces. Paths supplied to the callback are virtual paths.
/// Callbacks run after filesystem locks have been released; they must not block.
#[derive(Clone)]
pub struct Observer {
changed: Arc<Callback>,
paths: Arc<Mutex<Vec<Weak<TrackedPath>>>>,
// Only open-registration versus rename/remove needs ordering. Lock the affected
// subtrees, not all IO: a blocked open must not stall unrelated namespaces.
namespace: Arc<Namespace>,
}
impl Observer {
pub fn new(changed: impl Fn(&str) + Send + Sync + 'static) -> Self {
Self {
changed: Arc::new(changed),
paths: Arc::default(),
namespace: Arc::default(),
}
}
/// Wrap a layer at its canonical namespace prefix (e.g. `/run`).
pub fn layer(&self, layer: Arc<dyn Layer>, prefix: &str, process: Process) -> Arc<dyn Layer> {
Arc::new(Observed {
layer,
prefix: prefix.trim_end_matches('/').into(),
observer: self.clone(),
process,
})
}
/// Opens share their path and ancestors; rename/remove exclusively hold their
/// endpoints, sharing ancestors. Sorted acquisition avoids cross-rename deadlocks.
/// No descriptor IO or notification holds these guards.
async fn order(&self, endpoints: &[(&str, bool)]) -> Vec<Held> {
let mut modes = BTreeMap::new();
for &(path, exclusive) in endpoints {
modes
.entry(path.to_owned())
.and_modify(|mode| *mode |= exclusive)
.or_insert(exclusive);
let mut ancestor = path;
while let Some((parent, _)) = ancestor.rsplit_once('/') {
let parent = if parent.is_empty() { "/" } else { parent };
modes.entry(parent.to_owned()).or_insert(false);
if parent == "/" {
break;
}
ancestor = parent;
}
}
let locks: Vec<_> = {
let mut namespace = self.namespace.lock().unwrap();
namespace.retain(|_, lock| lock.strong_count() != 0);
modes
.into_iter()
.map(|(path, exclusive)| {
let lock = namespace.entry(path).or_default();
let strong = lock.upgrade().unwrap_or_else(|| {
let strong = Arc::new(RwLock::new(()));
*lock = Arc::downgrade(&strong);
strong
});
(strong, exclusive)
})
.collect()
};
let mut held = Vec::new();
for (lock, exclusive) in locks {
held.push(if exclusive {
Held::Write {
_guard: lock.write_owned().await,
}
} else {
Held::Read {
_guard: lock.read_owned().await,
}
});
}
held
}
fn path(&self, path: String) -> Path {
let path = Arc::new(Mutex::new(Some(path)));
let mut paths = self.paths.lock().unwrap();
paths.retain(|p| p.strong_count() != 0);
paths.push(Arc::downgrade(&path));
path
}
fn moved(&self, from: &str, to: Option<&str>) {
let mut paths = self.paths.lock().unwrap();
paths.retain(|p| {
let Some(p) = p.upgrade() else { return false };
let mut path = p.lock().unwrap();
if let Some(old) = path.as_deref() {
if under(old, from) {
*path = to.map(|to| format!("{to}{}", &old[from.len()..]));
} else if to.is_some_and(|to| under(old, to)) {
// A replaced destination's old descriptor no longer names a visible file.
*path = None;
}
}
true
});
}
}
/// Component-aware descendant matching, including the path itself.
pub fn under(path: &str, base: &str) -> bool {
base == "/" || path == base || path.strip_prefix(base).is_some_and(|s| s.starts_with('/'))
}
struct Observed {
layer: Arc<dyn Layer>,
prefix: String,
observer: Observer,
process: Process,
}
impl Observed {
fn path(&self, path: &str) -> String {
let path = format!("{}{}", self.prefix, path.trim_end_matches('/'));
if path.is_empty() { "/".into() } else { path }
}
}
struct Handle {
file: Arc<dyn File>,
path: Path,
observer: Observer,
process: Process,
}
impl Handle {
async fn mutate<T: Send + 'static>(
&self,
f: impl Future<Output = Result<T>> + Send + 'static,
) -> Result<T> {
let (path, observer) = (self.path.clone(), self.observer.clone());
self.process
.operation(async move {
let result = f.await?;
let path = path.lock().unwrap().clone();
if let Some(path) = path {
(observer.changed)(&path);
}
Ok(result)
})
.await
}
}
#[async_trait]
impl Layer for Observed {
async fn stat(&self, path: &str, follow: bool) -> Result<Stat> {
self.layer.stat(path, follow).await
}
async fn readlink(&self, path: &str) -> Result<String> {
self.layer.readlink(path).await
}
async fn list(&self, path: &str) -> Result<Vec<Entry>> {
self.layer.list(path).await
}
async fn open(&self, path: &str, options: Open) -> Result<Arc<dyn File>> {
let (layer, local, path, observer, process) = (
self.layer.clone(),
path.to_owned(),
self.path(path),
self.observer.clone(),
self.process.clone(),
);
self.process
.operation(async move {
let order = observer.order(&[(&path, false)]).await;
let file = layer.open(&local, options).await?;
let handle = Arc::new(Handle {
file,
path: observer.path(path.clone()),
observer: observer.clone(),
process,
});
drop(order);
if options.create || options.truncate {
(observer.changed)(&path);
}
Ok(handle as Arc<dyn File>)
})
.await
}
async fn mkdir(&self, path: &str) -> Result<()> {
let (layer, local, path, observer) = (
self.layer.clone(),
path.to_owned(),
self.path(path),
self.observer.clone(),
);
self.process
.operation(async move {
layer.mkdir(&local).await?;
(observer.changed)(&path);
Ok(())
})
.await
}
async fn remove_file(&self, path: &str) -> Result<()> {
self.remove(path, false).await
}
async fn remove_dir(&self, path: &str) -> Result<()> {
self.remove(path, true).await
}
async fn rename(&self, from: &str, to: &str) -> Result<()> {
let (layer, a, b, from, to, observer) = (
self.layer.clone(),
from.to_owned(),
to.to_owned(),
self.path(from),
self.path(to),
self.observer.clone(),
);
self.process
.operation(async move {
let order = observer.order(&[(&from, true), (&to, true)]).await;
layer.rename(&a, &b).await?;
if from != to {
observer.moved(&from, Some(&to));
}
drop(order);
(observer.changed)(&from);
(observer.changed)(&to);
Ok(())
})
.await
}
}
impl Observed {
async fn remove(&self, path: &str, directory: bool) -> Result<()> {
let (layer, local, path, observer) = (
self.layer.clone(),
path.to_owned(),
self.path(path),
self.observer.clone(),
);
self.process
.operation(async move {
let order = observer.order(&[(&path, true)]).await;
if directory {
layer.remove_dir(&local).await?;
} else {
layer.remove_file(&local).await?;
}
observer.moved(&path, None);
drop(order);
(observer.changed)(&path);
Ok(())
})
.await
}
}
#[async_trait]
impl File for Handle {
async fn stat(&self) -> Result<Stat> {
self.file.stat().await
}
async fn read(&self, offset: u64, length: usize) -> Result<Vec<u8>> {
self.file.read(offset, length).await
}
async fn write(&self, offset: u64, bytes: Vec<u8>) -> Result<usize> {
let f = self.file.clone();
self.mutate(async move { f.write(offset, bytes).await })
.await
}
async fn append(&self, bytes: Vec<u8>) -> Result<usize> {
let f = self.file.clone();
self.mutate(async move { f.append(bytes).await }).await
}
async fn set_size(&self, size: u64) -> Result<()> {
let f = self.file.clone();
self.mutate(async move { f.set_size(size).await }).await
}
async fn sync(&self) -> Result<()> {
self.file.sync().await
}
async fn set_times(
&self,
accessed: Option<SystemTime>,
modified: Option<SystemTime>,
) -> Result<()> {
let f = self.file.clone();
self.mutate(async move { f.set_times(accessed, modified).await })
.await
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::sync::mpsc;
#[test]
fn component_matching() {
assert!(under("/mailbox/a", "/mailbox"));
assert!(under("/mailbox", "/mailbox"));
assert!(under("/anything", "/"));
assert!(!under("/mailboxes/a", "/mailbox"));
}
/// Every visible mutation is delivered after the layer call, including rebased
/// open handles in another process's namespace. Removed handles stay usable.
#[tokio::test]
async fn handles_follow_rename_and_unlink() {
for host in [false, true] {
let temp = tempfile::tempdir().unwrap();
let process = Process::default();
let layer: Arc<dyn Layer> = if host {
Arc::new(Host::new(temp.path(), process.clone()).unwrap())
} else {
Arc::new(Memory::default())
};
let (tx, mut rx) = mpsc::unbounded_channel();
let observer = Observer::new(move |p| {
tx.send(p.to_owned()).unwrap();
});
let a = observer.layer(layer.clone(), "/run", process.clone());
let b = observer.layer(layer, "/run", process);
a.mkdir("/dir").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/dir");
let f = a
.open(
"/dir/a",
Open {
create: true,
write: true,
..Open::default()
},
)
.await
.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/dir/a");
b.rename("/dir", "/moved").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/dir");
assert_eq!(rx.recv().await.unwrap(), "/run/moved");
f.write(0, b"new".to_vec()).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/moved/a");
assert_eq!(
b.open("/moved/a", Open::default())
.await
.unwrap()
.read(0, 3)
.await
.unwrap(),
b"new"
);
let old = b
.open(
"/old",
Open {
create: true,
write: true,
..Open::default()
},
)
.await
.unwrap();
rx.recv().await.unwrap();
b.rename("/moved/a", "/old").await.unwrap();
rx.recv().await.unwrap();
rx.recv().await.unwrap();
old.write(0, b"invisible".to_vec()).await.unwrap();
assert!(rx.try_recv().is_err());
f.append(b"!".to_vec()).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/old");
b.remove_file("/old").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/old");
f.set_size(0).await.unwrap();
assert!(rx.try_recv().is_err());
assert!(b.remove_file("/missing").await.is_err());
assert!(rx.try_recv().is_err());
}
}
/// A pending exclusive namespace change must not block a sibling's opens,
/// even though Tokio's RwLock prefers queued writers over new readers.
#[tokio::test]
async fn namespace_order_is_scoped_and_shared() {
async fn ready<T>(future: impl Future<Output = T>) -> T {
let mut future = std::pin::pin!(future);
std::future::poll_fn(|cx| match future.as_mut().poll(cx) {
std::task::Poll::Ready(value) => std::task::Poll::Ready(value),
std::task::Poll::Pending => panic!("unrelated namespace operation blocked"),
})
.await
}
let observer = Observer::new(|_| {});
let open = observer.order(&[("/run/slow/file", false)]).await;
let concurrent_open = ready(observer.order(&[("/run/slow/file", false)])).await;
let mut rename =
std::pin::pin!(observer.order(&[("/run/slow", true), ("/run/moved", true)]));
std::future::poll_fn(|cx| {
assert!(rename.as_mut().poll(cx).is_pending());
std::task::Poll::Ready(())
})
.await;
// Same mounted layer, different subtree; also a different mounted layer.
let sibling_open = ready(observer.order(&[("/run/sibling/file", false)])).await;
let host_open = ready(observer.order(&[("/state/file", false)])).await;
drop(sibling_open);
let sibling_rename =
ready(observer.order(&[("/run/sibling", true), ("/run/other", true)])).await;
drop((sibling_rename, host_open, concurrent_open, open));
let renamed = ready(rename).await;
// A descendant open waits for its ancestor's rename until rebasing is done.
let mut descendant = std::pin::pin!(observer.order(&[("/run/moved/file", false)]));
std::future::poll_fn(|cx| {
assert!(descendant.as_mut().poll(cx).is_pending());
std::task::Poll::Ready(())
})
.await;
drop(renamed);
drop(ready(descendant).await);
drop(
observer
.order(&[("/run/moved/file", true), ("/run/moved", true)])
.await,
);
drop(
observer
.order(&[("/run/moved", true), ("/run/moved", true)])
.await,
);
// Dead weak entries are pruned, rather than retaining every path ever opened.
observer.order(&[("/", false)]).await;
assert_eq!(observer.namespace.lock().unwrap().len(), 1);
}
/// Pause an open after obtaining its real identity, before registering its
/// observed handle, like a slow host syscall or /self/log query.
struct SlowOpen {
memory: Arc<Memory>,
gate: Mutex<
Option<(
tokio::sync::oneshot::Sender<()>,
tokio::sync::oneshot::Receiver<()>,
)>,
>,
}
#[async_trait]
impl Layer for SlowOpen {
async fn stat(&self, p: &str, follow: bool) -> Result<Stat> {
self.memory.stat(p, follow).await
}
async fn list(&self, p: &str) -> Result<Vec<Entry>> {
self.memory.list(p).await
}
async fn open(&self, p: &str, options: Open) -> Result<Arc<dyn File>> {
let f = self.memory.open(p, options).await?;
let gate = self.gate.lock().unwrap().take();
if let Some((started, release)) = gate {
started.send(()).unwrap();
release.await.unwrap();
}
Ok(f)
}
async fn mkdir(&self, p: &str) -> Result<()> {
self.memory.mkdir(p).await
}
async fn remove_file(&self, p: &str) -> Result<()> {
self.memory.remove_file(p).await
}
async fn remove_dir(&self, p: &str) -> Result<()> {
self.memory.remove_dir(p).await
}
async fn rename(&self, a: &str, b: &str) -> Result<()> {
self.memory.rename(a, b).await
}
}
#[tokio::test]
async fn blocked_io_does_not_stall_other_processes() {
let memory = Arc::new(Memory::default());
memory.mkdir("/slow").await.unwrap();
let options = Open {
create: true,
write: true,
..Open::default()
};
let raw = memory.open("/slow/file", options).await.unwrap();
let (started, opening) = tokio::sync::oneshot::channel();
let (release, released) = tokio::sync::oneshot::channel();
let (tx, mut rx) = mpsc::unbounded_channel();
let observer = Observer::new(move |p| {
tx.send(p.to_owned()).unwrap();
});
let slow = observer.layer(
Arc::new(SlowOpen {
memory: memory.clone(),
gate: Mutex::new(Some((started, released))),
}),
"/run",
Process::default(),
);
let other = observer.layer(memory, "/run", Process::default());
let opening_task = tokio::spawn(async move {
slow.open(
"/slow/file",
Open {
write: true,
..Open::default()
},
)
.await
});
opening.await.unwrap();
let renamer = other.clone();
let renamed = tokio::spawn(async move { renamer.rename("/slow", "/moved").await });
// These must complete while the first process's open is still blocked,
// and a conflicting rename is waiting to rebase its registered handle.
other
.open(
"/",
Open {
directory: true,
..Open::default()
},
)
.await
.unwrap();
let f = other.open("/other", options).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/other");
f.write(0, b"visible".to_vec()).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/other");
other.mkdir("/sibling").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/sibling");
other.rename("/other", "/sibling/file").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/other");
assert_eq!(rx.recv().await.unwrap(), "/run/sibling/file");
other.remove_file("/sibling/file").await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/sibling/file");
assert!(!renamed.is_finished());
release.send(()).unwrap();
let opened = opening_task.await.unwrap().unwrap();
renamed.await.unwrap().unwrap();
rx.recv().await.unwrap();
rx.recv().await.unwrap();
opened.set_size(0).await.unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/moved/file");
// No namespace guard surrounds descriptor IO: a rename completes while
// this write is held in flight. Its notification uses the rebased path.
let handle = Handle {
file: raw.clone(),
path: observer.path("/run/moved/file".into()),
observer,
process: Process::default(),
};
let (started, writing) = tokio::sync::oneshot::channel();
let (release, released) = tokio::sync::oneshot::channel();
let write = tokio::spawn(async move {
handle
.mutate(async move {
started.send(()).unwrap();
released.await.unwrap();
raw.write(0, b"after rename".to_vec()).await
})
.await
});
writing.await.unwrap();
other.rename("/moved", "/again").await.unwrap();
rx.recv().await.unwrap();
rx.recv().await.unwrap();
other.open("/again/file", Open::default()).await.unwrap();
release.send(()).unwrap();
write.await.unwrap().unwrap();
assert_eq!(rx.recv().await.unwrap(), "/run/again/file");
}
/// Killing a waiting writer does not discard observation of a completed mutation;
/// the process's idle barrier includes the callback, not only the system call.
#[tokio::test]
async fn cancellation_keeps_mutation_and_observation() {
let (tx, mut rx) = mpsc::unbounded_channel();
let observer = Observer::new(move |p| {
tx.send(p.to_owned()).unwrap();
});
let process = Process::default();
let layer = observer.layer(Arc::new(Memory::default()), "/", process.clone());
let order = observer.order(&[("/published", true)]).await;
let (started, ready) = tokio::sync::oneshot::channel();
let p = process.clone();
let task = tokio::spawn(async move {
// Signal from inside the tracked operation, not before its guard exists.
p.operation(async move {
started.send(()).unwrap();
layer
.open(
"/published",
Open {
create: true,
write: true,
..Open::default()
},
)
.await
})
.await
});
ready.await.unwrap();
task.abort();
let _ = task.await;
drop(order);
process.idle().await;
assert_eq!(rx.try_recv().unwrap(), "/published");
}
}
+16 -1
View File
@@ -36,7 +36,22 @@ impl Process {
.await .await
.map_err(std::io::Error::other)? .map_err(std::io::Error::other)?
} }
/// Wait for all outstanding system calls. /// Complete an asynchronous mutation and its observation even if the guest is cancelled.
/// Its guard participates in idle(), so no mutation outlives process cleanup.
pub async fn operation<T: Send + 'static>(
&self,
operation: impl std::future::Future<Output = std::io::Result<T>> + Send + 'static,
) -> std::io::Result<T> {
self.0.count.fetch_add(1, Ordering::SeqCst);
let guard = Guard(self.0.clone());
tokio::spawn(async move {
let _guard = guard;
operation.await
})
.await
.map_err(std::io::Error::other)?
}
/// Wait for all outstanding system calls and observed mutations.
/// First abort the guest task. Then await its JoinHandle, or drop its store. /// First abort the guest task. Then await its JoinHandle, or drop its store.
/// Only then await idle before releasing locks. Abort alone does not stop a running poll. /// Only then await idle before releasing locks. Abort alone does not stop a running poll.
/// This ordering also prevents later memory writes, which need no blocking guard. /// This ordering also prevents later memory writes, which need no blocking guard.