Supervise boot services with uniform five-second upgrade grace

This commit is contained in:
Sausage
2026-11-01 20:51:08 +00:00
parent 4da8a28392
commit 959ddf46ff
15 changed files with 943 additions and 99 deletions
+31 -8
View File
@@ -19,7 +19,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
| service | A call in `/config/boot.yaml`. The core starts it at boot and again after an upgrade of its plugin. |
| service | A call in `/config/boot.yaml`. The core starts it at boot, restarts it after a crash, and starts it again after an upgrade of its plugin. |
| import | An interface of another plugin that a plugin uses. |
| provider, consumer | The plugin that has an interface, and a plugin that imports it. |
| journal | The central log of the world. |
@@ -159,8 +159,11 @@ active (section 10.3):
continue to work. It can stop accepting work, finish its in-flight calls,
and exit early. Ignoring the notice changes nothing. The grace period is
the upper bound before the core stops the process (section 3.6); its
default is 60 seconds. A plugin can declare its own value in its old
`rpc.discover` document (`x-sausage-grace`).
duration is 5 seconds for every plugin. Discovery metadata, including an
explicit `x-sausage-grace`, does not change it. After grace, stdin closes
and the separate 5-second stop budget applies (section 3.6): at most two
5-second windows. Discovery-only processes do not receive this notice or
enter upgrade grace; they follow the discovery lifetime (section 10.1).
3. When the old process of a service has ended, the core starts the service
again with the new version.
4. When the last process of an older version has ended, the core starts a
@@ -322,10 +325,29 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
- A service is a detached call with the parent `core`. The core waits for
its result before it starts the next one. The result shows that the
service is ready.
- If a service fails, the core writes the error to the journal and continues.
- After an upgrade of the plugin of a service, the core starts the service
again with the new version (section 3.7).
- The core does not restart other detached processes.
- If a service fails before answering, the core writes the error to the
journal and continues booting. A failure to admit a process (for example,
a missing plugin or invalid parameters) is logged, not retried in a loop.
- The core supervises the process lifetime, not its readiness response. A
nonzero exit, trap or failed host task restarts the same service with its
original method and parameters, including a crash before readiness. A
readiness response alone never triggers a restart; a clean exit does not
count as a crash or trigger an immediate restart.
- More than 5 crashes within 10 seconds gives up on that service and writes
`service gave up` to the journal. Each service has an independent sliding
window; crashes exactly 10 seconds old expire. There is no retry delay or
periodic liveness check. Giving up is latched for this host lifetime:
neither elapsed time nor an upgrade re-arms it.
- After an upgrade of the plugin of a service, the same supervisor starts
it again with the current version after its old process ends (section
3.7). Superseded exits are retirement, not crashes. Admission and cleanup
reserve a single owner/attempt, so an upgrade racing a crash cannot start
two owners. A matching boot-service start from `rpc.retired` is ignored
(returns `null`), even while restart is pending or after giving up; older
plugins' fallback starts do not bypass supervision.
- An explicit `core.stop` or `core.kill` disables automatic attempts for
that service, including during an upgrade. Host shutdown starts no new
processes. The core does not restart other detached processes.
- A singleton is a service with a fixed endpoint in `/run`. Only the core
starts it, so two instances cannot start at the same time.
@@ -391,7 +413,8 @@ Params are by name. The document can have these extension fields:
`methods`.
- `x-sausage-imports`: for each provider, the part of its document that the
plugin uses.
- `x-sausage-grace`: the grace period in seconds.
- `x-sausage-grace`: legacy grace declaration in seconds. The core ignores
this field; upgrade grace is always 5 seconds (section 3.7).
The schema of params describes what the provider can read. The schema of a
result describes what the provider writes. Descriptions are part of the