Merge main service supervision with uniform upgrade grace

Keep SAU-358 supervision and SAU-357's five-second host policy together; adapt supervisor tests to the metadata-independent snapshot API.

Card: SAU-357
This commit is contained in:
2026-10-30 23:25:19 +00:00
10 changed files with 643 additions and 68 deletions
+24 -5
View File
@@ -19,7 +19,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
| service | A call in `/config/boot.yaml`. The core starts it at boot and again after an upgrade of its plugin. |
| service | A call in `/config/boot.yaml`. The core starts it at boot, restarts it after a crash, and starts it again after an upgrade of its plugin. |
| import | An interface of another plugin that a plugin uses. |
| provider, consumer | The plugin that has an interface, and a plugin that imports it. |
| journal | The central log of the world. |
@@ -325,10 +325,29 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
- A service is a detached call with the parent `core`. The core waits for
its result before it starts the next one. The result shows that the
service is ready.
- If a service fails, the core writes the error to the journal and continues.
- After an upgrade of the plugin of a service, the core starts the service
again with the new version (section 3.7).
- The core does not restart other detached processes.
- If a service fails before answering, the core writes the error to the
journal and continues booting. A failure to admit a process (for example,
a missing plugin or invalid parameters) is logged, not retried in a loop.
- The core supervises the process lifetime, not its readiness response. A
nonzero exit, trap or failed host task restarts the same service with its
original method and parameters, including a crash before readiness. A
readiness response alone never triggers a restart; a clean exit does not
count as a crash or trigger an immediate restart.
- More than 5 crashes within 10 seconds gives up on that service and writes
`service gave up` to the journal. Each service has an independent sliding
window; crashes exactly 10 seconds old expire. There is no retry delay or
periodic liveness check. Giving up is latched for this host lifetime:
neither elapsed time nor an upgrade re-arms it.
- After an upgrade of the plugin of a service, the same supervisor starts
it again with the current version after its old process ends (section
3.7). Superseded exits are retirement, not crashes. Admission and cleanup
reserve a single owner/attempt, so an upgrade racing a crash cannot start
two owners. A matching boot-service start from `rpc.retired` is ignored
(returns `null`), even while restart is pending or after giving up; older
plugins' fallback starts do not bypass supervision.
- An explicit `core.stop` or `core.kill` disables automatic attempts for
that service, including during an upgrade. Host shutdown starts no new
processes. The core does not restart other detached processes.
- A singleton is a service with a fixed endpoint in `/run`. Only the core
starts it, so two instances cannot start at the same time.