Supervise crashed boot service lifetimes with a latched crash limit

Keep one registration and restart owner across process cleanup and upgrades.
Restart on failed process exit before or after readiness; give up after more
than five crashes in a sliding ten-second window. Respect explicit stop and
shutdown, and suppress boot-service fallbacks from retired hooks.

Cover deterministic lifetime, boundary, give-up and admission races, plus
real WASI failures on both sides of readiness. Document the host contract.

Card: SAU-358
This commit is contained in:
SAU-358 implementer
2026-10-30 19:49:06 +00:00
parent c04baee67c
commit a953d80ce6
9 changed files with 646 additions and 67 deletions
+24 -5
View File
@@ -19,7 +19,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
| service | A call in `/config/boot.yaml`. The core starts it at boot and again after an upgrade of its plugin. |
| service | A call in `/config/boot.yaml`. The core starts it at boot, restarts it after a crash, and starts it again after an upgrade of its plugin. |
| import | An interface of another plugin that a plugin uses. |
| provider, consumer | The plugin that has an interface, and a plugin that imports it. |
| journal | The central log of the world. |
@@ -322,10 +322,29 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
- A service is a detached call with the parent `core`. The core waits for
its result before it starts the next one. The result shows that the
service is ready.
- If a service fails, the core writes the error to the journal and continues.
- After an upgrade of the plugin of a service, the core starts the service
again with the new version (section 3.7).
- The core does not restart other detached processes.
- If a service fails before answering, the core writes the error to the
journal and continues booting. A failure to admit a process (for example,
a missing plugin or invalid parameters) is logged, not retried in a loop.
- The core supervises the process lifetime, not its readiness response. A
nonzero exit, trap or failed host task restarts the same service with its
original method and parameters, including a crash before readiness. A
readiness response alone never triggers a restart; a clean exit does not
count as a crash or trigger an immediate restart.
- More than 5 crashes within 10 seconds gives up on that service and writes
`service gave up` to the journal. Each service has an independent sliding
window; crashes exactly 10 seconds old expire. There is no retry delay or
periodic liveness check. Giving up is latched for this host lifetime:
neither elapsed time nor an upgrade re-arms it.
- After an upgrade of the plugin of a service, the same supervisor starts
it again with the current version after its old process ends (section
3.7). Superseded exits are retirement, not crashes. Admission and cleanup
reserve a single owner/attempt, so an upgrade racing a crash cannot start
two owners. A matching boot-service start from `rpc.retired` is ignored
(returns `null`), even while restart is pending or after giving up; older
plugins' fallback starts do not bypass supervision.
- An explicit `core.stop` or `core.kill` disables automatic attempts for
that service, including during an upgrade. Host shutdown starts no new
processes. The core does not restart other detached processes.
- A singleton is a service with a fixed endpoint in `/run`. Only the core
starts it, so two instances cannot start at the same time.