sausage/PROTOCOL.md

# The Sausage protocol

This is the contract between the core and the plugins. A plugin in any
language that follows it works in a Sausage world. The core shows this file
at `/proc/core/PROTOCOL.md`. `DECISIONS.md` in the repository says why the
contract is the way it is.

Rust plugins use the SDK `sausage-plugin`. Its source is at
`/plugins/cargo/sdk/`, and its crate docs explain each item.

## 1. Terms

| Term | Meaning |
| --- | --- |
| world | The folder that a `sausage` host runs. It is `/` for every process. |
| core | The host. It runs processes, routes calls, serves the virtual files, and keeps the journal. |
| plugin | A folder `/plugins/<name>/` with a `plugin.wasm`. A plugin does not run by itself. |
| process | One run of a plugin version. Its ID is `<plugin>-<uuidv7>`, for example `notes-0197f3a2-7c1e-7d2b-9a41-5f0c2e8b1d34`. |
| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
| service | A call in `/config/boot.json`. The core starts it at boot and again after an upgrade of its plugin. |
| import | An interface of another plugin that a plugin uses. |
| provider, consumer | The plugin that has an interface, and a plugin that imports it. |
| journal | The central log of the world. |

## 2. The world folder

```text
<world>/
  plugins/<name>/
    plugin.wasm               the current version
    plugin.prev.wasm          the version before, for `core.revert`
    source/                   the source. `cargo.build` builds it from here.
      imports/<plugin>.json   descriptions of the imported plugins
  state/<plugin>/             persistent state of a plugin, by convention
  state/core/journal.sqlite   the journal
  config/                     global configuration
    boot.json                 the services
    agent.json                API, base URL, model, and token of the agent
```

- Every process can read and write all of it. There are no permissions.
- Plugin names use `a-z`, `0-9`, `-`, and `_`. The name `core` is reserved.
- The world folder must not contain `proc`, `run`, or `self`. These names are
  mount points (section 4). The host does not start a world that has them.

## 3. Calls and processes

### 3.1 The wire protocol

- A plugin is a `wasi:cli/command` component for `wasm32-wasip2`.
- stdin and stdout carry JSON-RPC 2.0, one message per line.
- A line is a message only if it is a JSON object with `"jsonrpc": "2.0"`.
  The core discards every other line on stdout. For the first discarded line
  of a process, it writes one warning to the journal.
- Each line on stderr goes to the journal. Log to stderr.
- `params` is always an object. Batches are not supported.
- Write each message in one write, and write a newline before it. Then a
  `print!` without a newline cannot damage a message.

### 3.2 Method names

- A method name has the form `<target>.<method>`. The core splits it at the
  first `.`.
- `notes.add` starts a new process of `notes` and sends it the request `add`.
  `notes.rpc.discover` sends it `rpc.discover`.
- `core.<method>` is a method of the core (section 9).
- Method names that start with `rpc.` are reserved: `rpc.discover`,
  `rpc.retired`, and `rpc.endpoint`.

### 3.3 What a call does

1. The core gives the call a UUIDv7 and starts the process `<plugin>-<uuid>`
   with the current version of the plugin.
2. The core writes the request to the stdin of the process. The request ID is
   the process ID, and the method has no plugin prefix.
3. The process writes the response to stdout. The core sends the response to
   the caller, with the caller's request ID.
4. If the process ends before it answers, the caller gets the error -32002.
   Its `data` has the exit status and the last journal lines of the process.
   Before the core decides this, it reads the stdout of the process to the
   end.

- A process can write calls to stdout at any time, to the core and to
  plugins. The responses come on stdin. A process can have many calls in
  flight.
- The core keys each pending request by the process and the exact JSON ID.
  The core drops a response with an unknown or late ID.
- A notification to `plugin.method` (without an ID) starts an attached
  process. That process gets the notification without an ID, and nobody gets
  a result.

### 3.4 Detached calls

`core.call {"method": "www.start", "params": {...}, "detach": true}` makes the
call and returns its result to the caller. The new process has no parent, so
it does not end with its caller.

Sent as a notification (without an ID), the same `core.call` starts the
process and forgets it.

### 3.5 Notifications to the caller

A process can tell its caller about progress. It sends the notification
`core.notify {"name": "text", "params": {...}}`. The core gives it to the
caller as `core.notify {"call": <ID of the caller's request>, "name": "text",
"params": {...}}`. After the call has ended, the core drops such
notifications.

So a process never needs to know who called it.

### 3.6 Lifetime

A process lives until `main` returns. A process that uses the SDK lives
while one of these runs:

- a handler of a call,
- a background task,
- an endpoint that listens.

A process has three states:

1. **running**
2. **stopping**: the core has closed stdin. EOF on stdin means "stop now".
   - New calls of the process fail at once, and its pending calls fail.
   - A stop hook can still write files and send notifications. It cannot
     make calls, because no response can arrive.
   - After 5 seconds, the core kills the process.
3. **ended**: the core releases the locks of the process (section 6) and
   removes its folder in `/proc`.

Rules:

- `core.stop {pid}` starts the stopping state. `core.kill {pid}` kills the
  process at once.
- When a process ends, the core stops each of its attached children. All
  children stop in parallel, so a whole tree takes 5 seconds at most.
- A detached process has no parent. Only `core.stop`, `core.kill`, an
  upgrade (section 3.7), and the end of the host stop it.
- At host shutdown, the core starts no new processes, and it stops every
  process in parallel.

Note: on Unix, children outlive their parent. Here, an attached child ends
with its parent. Only `detach` makes a process independent.

### 3.7 Upgrades

A process is an instance of one plugin version. When a new version becomes
active (section 10.3):

1. New calls get the new version at once.
2. Each process of an older version gets a grace period to finish its work.
   The default is 60 seconds. A plugin can declare its own value in
   `rpc.discover` (`x-sausage-grace`). Then the core stops the process.
3. When the old process of a service has ended, the core starts the service
   again with the new version.
4. When the last process of an older version has ended, the core starts a
   process of the new version and calls `rpc.retired {"previous":
   "<sha256 of the older file>"}`. A plugin can use this hook to migrate its
   state, or to write a marker that its other new processes wait for. The
   result goes to the journal.

During the grace period, an old process can call a method that a new
provider no longer has.

If the host stops before the last old process has ended, `rpc.retired` does
not run for that version. So a new version must also work with state that
was not migrated. Check for a migration marker, and migrate when it is
missing.

### 3.8 Environment and network

- `SAUSAGE_PID`: the process ID.
- `SAUSAGE_PLUGIN`: the plugin name.
- `SAUSAGE_PARENT`: the parent: `console`, `core`, or a process ID.

Every process gets the network: outgoing HTTP and HTTPS through `wasi:http`,
and TCP, UDP, and DNS through `wasi:sockets`.

A process that computes for a long time does not block other processes. The
core interrupts it every 10 milliseconds.

### 3.9 Errors

The standard JSON-RPC codes apply. The core adds these codes:

| Code | Meaning |
| --- | --- |
| -32001 | There is no such plugin. |
| -32002 | The process ended without an answer. `data` has the exit status and the last journal lines. |
| -32003 | The activation was rejected. `data` has the reasons. |
| -32004 | `cargo.build` failed. `data` has the `output` of the failed step and the `tests`. |

## 4. The filesystem

Each process gets one preopen, `/`. The core resolves each path through a
mount table:

| Path | What it is | Access |
| --- | --- | --- |
| `/` | the world folder | read-write |
| `/proc` | generated by the core | read-only |
| `/self` | the folder `/proc/<own pid>/`, in memory | read-write |
| `/run` | in memory, shared, empty at each host start | read-write |

A listing of `/` shows `proc`, `run`, and `self`. A rename across two of
these areas fails.

```text
/proc/
  core/
    PROTOCOL.md          this file
    graph.json           plugins, imports, and the result of each check
    graph.dot            the same graph for Graphviz
    locks.json           held locks and waiting processes
    plugins/<name>/
      discover.json      the rpc.discover document of the current plugin.wasm
      client.rs          the Rust code that import!("<name>") generates
  <pid>/
    info.json            plugin, version, method, params, parent, detached, start time, state
    log                  the journal lines of this process
    <files>              the files that the process writes to /self
```

Rules:

- A process writes its own folder only through `/self`. Writes through
  `/proc` always fail.
- In a process folder, the names `info.json`, `log`, and `fs` belong to the
  core. A write to `/self/info.json` fails.
- A read of a generated file returns a snapshot that ends. For example,
  `/proc/<pid>/log` returns the lines up to the time of the open.
- The folder of a process goes away when the process ends. Its files are in
  memory. Large or persistent data belongs in `/state`.
- Do not send a `/self` path to another process. There it means a different
  folder, like `localhost` on another machine.

Principle: **inspect through files, act through calls.**

## 5. Endpoints

An endpoint is a JSON-RPC interface that a running process serves over TCP,
for example the control interface of a web server.

- The process binds a free port on 127.0.0.1. When the listener is ready, it
  writes an endpoint file in one atomic step (write a temporary file, then
  rename it). The file has one line: the address, the owner process, and the
  interface. Example: `tcp:127.0.0.1:41234 www-0197… control`.
- Usually the file is in `/self`. A service that others must find uses a
  fixed path in `/run`, for example `/run/secrets`.
- A client reads the file and connects. Its first request is
  `rpc.endpoint`, which returns `{"owner", "plugin", "interface"}`. If they
  do not match the file, the client disconnects. So a stale file or a reused
  port cannot connect a client to the wrong process.
- JSON-RPC runs on the connection, one message per line, in both directions.
  Notifications of a handler (`core.notify`) go over the connection.
- A plugin declares its endpoint interfaces in `rpc.discover`
  (`x-sausage-endpoints`).

## 6. Locks

- `core.lock {"path": "/state/notes/notes.json"}` returns `{"held": true}`
  when the caller holds the lock.
- Each acquisition gets its own place in a FIFO queue. The same process
  cannot take one lock twice: its second acquisition waits behind its first.
- With `"wait": false`, the call returns at once with `{"held": true}` or
  `{"held": false}`.
- `core.unlock {path}` (a notification) releases the lock.
- When a process ends, the core releases all of its locks, also after a
  crash or a kill. It waits until the last file operation of the process has
  returned. So a write of a killed process cannot land after the next
  process got the lock.
- Locks are advisory. The core does not block writes to the file: the path
  is only a name.
- SQLite has no file locks in WASI. Hold the lock while the database is open.

## 7. The journal

- An SQLite file at `/state/core/journal.sqlite`. Only the core writes it.
- Each entry has a UUIDv7 (it sorts by time), the process ID, the plugin, the
  stream (`stderr` or `core`), and the line.
- The core writes its own events too: start (with method and params), exit
  status, stop, kill, upgrades, and the warning about discarded stdout.
- `core.log {plugin?, pid?, since?, until?, grep?, limit?}` queries the
  journal. `/proc/<pid>/log` shows the lines of one running process.

## 8. Boot and services

At start, the core starts each service in `/config/boot.json`, in order:

```json
[
  {"method": "secrets.serve", "params": {}},
  {"method": "www.start", "params": {"port": 8080}}
]
```

- A service is a detached call with the parent `core`. The core waits for
  its result before it starts the next one. The result shows that the
  service is ready.
- If a service fails, the core writes the error to the journal and continues.
- After an upgrade of the plugin of a service, the core starts the service
  again with the new version (section 3.7).
- The core does not restart other detached processes.
- A singleton is a service with a fixed endpoint in `/run`. Only the core
  starts it, so two instances cannot start at the same time.

## 9. The core API

| Method | Params | Result |
| --- | --- | --- |
| `core.call` | `method`, `params`, `detach`? | the result of the method |
| `core.notify` | `name`, `params` | none (a notification) |
| `core.stop` | `pid` | `{}` when the process has ended |
| `core.kill` | `pid` | `{}` |
| `core.lock` | `path`, `wait`? | `{"held": bool}` |
| `core.unlock` | `path` | none (a notification) |
| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents |
| `core.revert` | `plugin`, `force`? | like `core.activate` |
| `core.log` | filters (section 7) | journal entries |
| `core.exit` | none | `{}`. Then the host stops with exit code 75. |

Everything else is a file.

## 10. Interfaces

### 10.1 `rpc.discover`

Each plugin answers `rpc.discover` with an OpenRPC 1.2 document. The schemas
are JSON Schema Draft 7, as OpenRPC requires. Shared schemas are in
`components.schemas`, and `$ref`s point to `#/components/schemas/<Name>`.
Params are by name. The document can have these extension fields:

- `x-sausage-params` for each method with params: a `$ref` to the schema of
  the params object.
- `x-sausage-endpoints`: the endpoint interfaces by name, each with its
  `methods`.
- `x-sausage-imports`: for each provider, the part of its document that the
  plugin uses.
- `x-sausage-grace`: the grace period in seconds.

The schema of params describes what the provider can read. The schema of a
result describes what the provider writes. Descriptions are part of the
interface: other agents read them to use it.

The core starts a process only to call `rpc.discover`, and it stops that
process after the answer. Do not do expensive work at start. If no answer
comes within 10 seconds, the discovery fails.

### 10.2 Compatibility

A provider can add methods, optional params, and result fields. For each
imported method:

1. The provider has the method.
2. Each params value that the consumer can send is valid for the provider.
3. Each result that the provider can return is valid for the consumer.

- The same rules apply to endpoint interfaces.
- There are no version numbers. This check replaces them.
- The check covers a fixed subset of JSON Schema: objects with `properties`
  and `required`, the primitive types, arrays, optional and nullable values,
  string enums, integer ranges, and `$ref`. A construct outside the subset
  fails the check.
- The check covers calls, not state. Old and new processes share the files in
  `/state`, and `core.revert` restores only code. A plugin must keep its
  state format compatible, or migrate it in `rpc.retired`.

### 10.3 Activation

`core.activate {"plugin": "notes", "wasm": "/state/cargo/builds/7/notes.wasm"}`:

1. The core copies the file. All further steps use this copy, even if the
   file changes.
2. It starts the copy and calls `rpc.discover`.
3. It gets the current documents of all plugins (section 10.4). A plugin
   whose discovery fails is "unavailable". Its imports cannot be checked,
   but it does not block the activation of other plugins.
4. It checks the imports of the new version against the current providers.
5. It checks each current consumer of the plugin against the new version.
6. If all checks pass, `plugin.wasm` becomes `plugin.prev.wasm`, and the copy
   becomes `plugin.wasm`. Then the upgrade starts (section 3.7). If the copy
   has the same hash as the current version, nothing changes.

If a check fails, nothing changes, and the error -32003 lists each problem,
with a counterexample where possible.

- The core runs one activation at a time.
- `plugins: [...]` activates several plugins together. The checks then use
  the new set.
- `force: true` skips steps 4 and 5.
- `core.revert {plugin}` activates `plugin.prev.wasm` in the same way, and
  the current version becomes `plugin.prev.wasm`.
- A person can also copy a file to `plugin.wasm` by hand. That skips the
  checks.

### 10.4 The graph

- The core keeps the `rpc.discover` documents of all plugins, by file hash.
  At start, it gets them in the background.
- A file watcher on `/plugins/*/plugin.wasm` updates them, for example after
  a copy by hand. `/proc/core/graph.json` then shows each broken import, and
  the journal records it.
- An import of a plugin that does not exist is "missing". An import of a
  plugin whose discovery failed is "unavailable".

## 11. Building and tests

The core has one native plugin: `cargo`. `cargo.build {"plugin": "notes"}`
builds `/plugins/notes/source/`:

1. It copies the source once. The tests and the build both use this copy.
2. It runs the tests under WASI.
3. It builds the release `.wasm` for `wasm32-wasip2`.
4. It returns `{"wasm": "/state/cargo/builds/<n>/notes.wasm", "tests":
   {"passed", "failed", "output"}, "warnings"}`. If a step fails, it returns
   an error with the compiler or test output.

- A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal
  entries, like other calls.
- The dependency `sausage-plugin = "0.1"` points to the SDK copy at
  `/plugins/cargo/sdk/`.
- Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2`
  point to a WASI clang.
- The build does not activate the result. Call `core.activate` next.

Tests:

- Each test runs in its own instance, so a panic ends only its own test.
- The normal Rust test setup works: `#[test]`, `tests/*.rs`, name filters,
  and `--ignored`. `#[should_panic]` cannot work, because WASI cannot catch
  a panic.
- Each test gets an empty temporary world as `/`, the package folder at its
  host path, and its own `/proc`, `/self`, and `/run`.
- The current folder of a test is `/`, not the package folder. Read fixture
  files through `env!("CARGO_MANIFEST_DIR")`.
- The test runner answers `core.lock`, `core.unlock`, `core.notify`, and
  `core.log`. A call to a plugin fails: a test has no other plugins.