sausage/PROTOCOL.md

# The Sausage protocol

This is the contract between the host and the actors. An actor in any
language that follows it works in Sausage. The host publishes this file at
`/actor/broker/public/PROTOCOL.md`, so every actor can read it.

## Actors

An actor is a WASI 0.2 command (`wasm32-wasip2`): a program with `main`. The
host runs one instance per actor. An instance can run forever.

The instance gets:

- **stdin:** one message per line, from the broker
- **stdout:** one message per line, to the broker
- **stderr:** free text, appended to `/actor/<id>/stderr.log`
- **environment:** `SAUSAGE_ACTOR` (its ID), `SAUSAGE_VERSION` (its version),
  and `SAUSAGE_CANDIDATE=1` during a test run
- **the network:** outgoing HTTP and HTTPS through `wasi:http` (the host does
  TLS), and TCP, UDP, and DNS through `wasi:sockets`, including listening on
  a port of the host
- **the filesystem** described below

## Messages

A line on **stdin**:

```json
{"id": "m-17", "from": "console", "reply_to": "agent/a5", "body": {"type": "text", "text": "hi"}}
```

A line on **stdout**:

```json
{"to": "console", "id": "a6", "reply_to": "m-17", "body": {"type": "text", "text": "hello"}}
```

- `body` is required. By convention it is an object with a `"type"`.
  Text for people is `{"type": "text", "text": "..."}`.
- `to` is the recipient. Without `to`, the broker publishes the message to
  every subscriber of the sender.
- `id` is an optional string. If the actor sets it, the message ID is `"<sender>/<id>"`.
  Otherwise, the broker assigns `m-<n>`. The broker always sets `from`.
- `reply_to` marks a reply: its value is the ID of the message it answers.
  There is no request/response. A reply is a new message.
- **Never answer a reply,** or two actors can answer each other forever.
  `broker` and `cargo` ignore messages that have a `reply_to`.
- Messages are fire and forget: delivery is at most once. Each actor has a
  mailbox of 1,000 messages, which keeps messages while the actor restarts
  or is upgraded. If the broker cannot deliver a message, the sender gets
  `{"type": "undeliverable", "reason": "...", "to": "..."}` from `broker`.
- An actor must keep reading stdin. If the broker must stop or upgrade an
  actor while a message waits for it to read stdin, it kills the actor.

Before the broker stops an instance, it sends
`{"id": "broker/shutdown", "from": "broker", "body": {"type": "shutdown"}}`
and then closes stdin. The broker still routes what the actor writes until
it exits. After 5 seconds, it kills the instance.

## Filesystem

Each instance sees four folders:

| Path | Access | Contents |
| --- | --- | --- |
| `/data` | read-write by its actor; readable by every actor | the actor's state |
| `/public` | read-write | the files that the actor publishes |
| `/actor` | read-only | the `actor/` folder of the world |
| `/config` | read-write by every actor | the world's shared `config/` folder; persists across restarts |

`/config` is the same world folder for every instance, including candidates;
writers coordinate by convention, not permissions. Do not put secrets there.

An actor can write only its own `/data`, but can read every actor's `/data` through `/actor/<id>/data/`. `/data` is not private: any unprotected secret file there is readable by every actor.

`/actor/<id>/` holds everything about an actor: `actor.wasm`, `actor.json`
(installed version, running, subscriptions, tapped, watches), `source/`, `public/`, `data/`,
`stderr.log`, and `versions/<n>/`. `ls /actor` lists all actors.

Files are plain POSIX files: a write is visible at once. To publish a file
so that readers never see half of it, write a temporary file and `rename` it
(the Rust SDK has `write_atomic`).

Send large or binary data as a path under `/actor/<id>/public/`, not inside a
message.

### Actor-served files: `/actor/<id>/fs/`

Files under `/actor/<id>/fs/` are not on disk: the actor `<id>` serves them.
Every actor can read and write them. The host turns each file operation into a
message from `fs` to `<id>` and waits up to 10 seconds for the reply. `path`
is relative to `fs/` and starts with `/` (the folder itself is `/`).

To make this virtual directory discoverable, an actor that serves it declares
`Actor-served files` and `fs_read` in its leading `//!` documentation in
`source/src/main.rs`. The host uses that declaration when listing
`/actor/<id>/`; it does not probe every actor (which could stall on actors that
do not handle filesystem messages). This is discovery metadata only: opening
and reading the path still requires a valid `fs_read` reply.

| Request | Reply |
| --- | --- |
| `{"type": "fs_read", "path"}` | `{"type": "fs_file", "text"}` (or `"base64"` instead of `"text"`), or `{"type": "fs_dir", "entries": [{"name", "type": "file" \| "dir"}]}` |
| `{"type": "fs_write", "path", "text"}` (or `"base64"` for non-UTF-8 data) | `{"type": "fs_done"}` |
| `{"type": "fs_mkdir", "path"}`, `{"type": "fs_remove", "path"}` | `{"type": "fs_done"}` |
| `{"type": "fs_rename", "path", "to"}` | `{"type": "fs_done"}` |

Any request can be answered with `{"type": "fs_error", "errno": "no-entry"}`.
`errno` is a `wasi:filesystem` error code: `no-entry`, `exist`,
`not-directory`, `is-directory`, `not-empty`, `access`, `not-permitted`,
`invalid`, `unsupported`, `busy`, or `io` (also for anything else).

- Opening a file sends `fs_read` once. Reads come from that copy.
- After a change, closing (or syncing) the file sends the whole file in one
  `fs_write`.
- No reply within 10 seconds, or an actor that does not exist, makes the
  operation fail (`io`, or `no-entry`). An actor that opens its own `fs/`
  waits for itself and fails after the timeout.
- `fs` is reserved: no actor can have that ID. It also sends `fs_changed`
  to watchers (see Watches).

## The broker

Requests to `broker`. The answer is a message whose `reply_to` is the ID of
the request: the request's `type` in past tense (`spawned`, `upgraded`,
`rolled_back`, `stopped`, `started`, `removed`, `subscribed`, `unsubscribed`,
`tapping`, `untapped`, `watching`, `unwatched`, `exiting`), or `{"type": "error", "reason": "..."}`.

| `type` | Fields | Effect |
| --- | --- | --- |
| `spawn` | `actor`, `wasm`, `source`?, `built_by`? | test, then start a new actor |
| `upgrade` | `actor`, `wasm`, `source`?, `built_by`? | test, then swap the version: a running actor is stopped (normal shutdown) and starts the new version; a stopped actor gets the new version and stays stopped, with its mailbox untouched, until `start`. The answer has `running` |
| `rollback` | `actor`, `version` | go back to an older version; like `upgrade`, a stopped actor stays stopped. The answer has `running` |
| `stop`, `start` | `actor` | stop or start an actor |
| `remove` | `actor` | stop the actor (normal shutdown), then delete its whole folder `/actor/<id>/` (all versions, `data/`, `public/`, logs); answer `{"type": "removed", "actor"}`. Messages to it are then undeliverable, and the ID can be spawned again. An unknown actor or a native one (`broker`, `console`, `cargo`, `fs`) is an `error` |
| `subscribe`, `unsubscribe` | `actor` | receive (or not) what `actor` publishes |
| `tap`, `untap` | | receive (or not) a copy of every message the broker routes |
| `watch`, `unwatch` | `path` | receive (or not) `fs_changed` for every change of `path` or anything under it; answer `{"type": "watching" \| "unwatched", "path"}` with the path as an `/actor/...` path |
| `exit` | | answer `exiting`, stop every actor, and end the `sausage` process with exit code 75 (a wrapper like `nix run .#loop` starts it again, updated) |

**Headless.** `sausage run --headless [dir]` (or `sausage --headless`, or `SAUSAGE_HEADLESS=1`) turns the console input off: the host does not read stdin, so end of input (`/dev/null`) does not stop the world. `nix run .#loop -- --headless <world>` passes it through the loop. It runs until SIGTERM or SIGINT (stops every actor, exit code 0) or a broker `exit` request (exit code 75, as above). Without the flag, EOF on stdin or `/quit` ends the host.

Paths are `/actor/...`, `/config/...`, or the sender's `/public/...` or `/data/...`.

**Taps.** After `tap`, the sender gets a copy of every message that the
broker routes (direct messages, replies, and publishes, between any actors,
including its own), for debugging, logging, or a live view of the world:

```json
{"type": "tapped", "message": {"id": "m-17", "from": "console", "to": "agent", "reply_to": null, "body": {...}}, "ts": 1792132579067}
```

- `to` is `null` for a publish: one copy per publish, not per subscriber.
- `ts` is the time of routing, in milliseconds since the Unix epoch.
- Copies come from `broker`. They are not routed themselves, so they are
  never tapped, and each message is tapped once per tapper.
- A copy that does not fit into the tapper's mailbox is dropped; routing
  never waits for a tapper, and nobody gets `undeliverable` for a copy.
- Never answer a copy: the answer would be tapped again, forever.
- Like subscriptions, a tap is saved in `actor.json` (`"tapped": true`) and
  survives restarts and upgrades. `untap` ends it.

**Watches.** The host watches the whole world folder with the OS file
watcher: every change inside it counts, whoever makes it (actors, a 9P mount,
a shell or `git` on the host). Actor-served files (`/actor/<id>/fs/`) are not
on disk and do not count. After `watch {path}`, the sender gets for each
change of `path` or a path under it (`/actor/a` covers `/actor/a/x`, not
`/actor/ab`):

```json
{"type": "fs_changed", "path": "/actor/notes/public/list.md", "op": "write"}
```

- `op` is `write`, `create`, `remove`, or `rename` (then `to` is the new
  path; a watcher of either path gets it).
- Changes of one path within about 100 ms become one event (a `write` after
  a `create` stays a `create`).
- The host's temporary files (names starting with `.sausage-tmp-`) are
  skipped; renaming one onto a file is a `write` of that file.
- Events come from `fs`. Like tap copies, they are not routed, a full
  mailbox drops them, and they never block the writer. Never answer them.
- If the OS drops events, every watcher gets one
  `{"type": "fs_changed", "path": "/actor", "op": "rescan"}`: anything may
  have changed.
- A watch is saved in `actor.json` (`"watches": [...]`) and survives
  restarts and upgrades. `unwatch` with the same path ends it.

**Versions.** A new version (from `spawn`, `upgrade`, or a changed
`actor.wasm` on disk) is tested first, while the old version keeps running.
Only when its tests pass does the broker copy it to `versions/<n>/`
(`actor.wasm`, `source/`, `meta.json`) and switch to it. So `versions/` holds
exactly the versions that were installed, and instances always run from
there. If the tests fail, nothing changes, and the sender gets an `error`
with the reason. All versions share the actor's `data/` and `public/`: a
new version must read the data of the old one.

**Tests.** The broker runs the candidate on a copy of the actor's `data/`
and `public/`, without network. The tests are JSON files in
`<source>/tests/*.json`, one scenario per file:

```json
{
  "description": "add, then list",
  "steps": [
    {"send": {"type": "add", "text": "milk"}, "expect": {"type": "added"}},
    {"send": {"type": "list"}, "expect": [{"type": "notes"}]}
  ]
}
```

Each step sends `send` (from `from`, default `test`), then reads the next
line that the candidate writes for each `expect` entry, in order, within 10
seconds. An entry matches when it is a subset of the message body. With
`to`, the message must also go to that recipient. Each file runs in a fresh
instance. A version without test files passes.

**Swap.** The broker holds the mailbox, stops the old instance, makes the
top-level `actor.wasm` and `source/` a copy of the new version, and starts
it. A stopped actor is not started: it keeps its mailbox and runs the new
version at the next `start`. A rollback does the same with the older version.

**Crashes.** The broker restarts a crashed actor after 0.5 s, then doubles
the delay up to 30 s. A new version that crashes 3 times within 30 seconds
after an upgrade goes back to the previous version. The actor that asked for
the upgrade gets `{"type": "rolled_back", "from", "to", "reason", "last_exit"}`.

**The folder is the truth.** At start, the broker boots every
`actor/<id>/actor.wasm`. Afterwards, the file watcher reports a new or
changed `actor/<id>/actor.wasm` (or a new folder `actor/<id>/`) once it is
quiet for about 100 ms: a new file becomes a new actor, a changed one an
upgrade, tested first with the tests in `actor/<id>/source/`. A file equal
to the installed version changes nothing. If its tests fail, the file stays
where it is, the reason goes to `stderr.log`, and the next change of the
file tries again. When the OS drops events, the broker checks every actor
folder again, as at start.

## The `cargo` actor

```json
{"type": "build", "source": "/actor/agent/public/drafts/notes"}
```

The build request may include `"skip_tests": true` to skip Rust unit tests.
By default, `cargo` first runs `cargo test` for the host target (not WASI),
so pure-logic unit tests in a `wasm32-wasip2` actor crate can run without a
WASI test runner. If tests fail or cannot compile, the build fails before the
WASI build. The reply carries `tests: {"passed": <count>, "failed": <count>,
"output": "<cargo test stdout and stderr>"}`; counts sum Cargo's test-result
summaries, and a test command/compilation failure without a summary counts as
one failure. With `skip_tests: true`, counts are zero and output says tests
were skipped.

The reply is `{"type": "built", "wasm": "/actor/cargo/public/builds/<n>/<name>.wasm", "built_by": "cargo", "tests": {...}, ...}`
or `{"type": "build_failed", "errors": "...", "tests": {...}}`. Pass
`wasm`, `source`, and `built_by` on to `spawn` or `upgrade`.

A Rust actor depends on the SDK with `sausage-actor = "0.1"`. `cargo` points
that dependency at the SDK of the host, which it publishes at
`/actor/cargo/public/sdk/`.