Miguel Palhas
ae3091e2a2
The respawn runs with the session map unlocked, so two takes of the lock straddle it. A kill landing in that window deleted the sidecar before resume wrote it back, leaving a file for a session that was gone — the next recovery registered it as a dormant ghost. Resume now deletes the sidecar in the branch where it finds its own record gone. Two resumes both passed the `Running` check for the same reason. The record is marked in-flight under the first lock, so the second request is refused with `already_running` instead of losing a race for the FIFO and reporting a tmux failure. Every path out of resume clears the marker. Both races are driven deterministically in the tests: the manager's tmux spawn is parked mid-resume while the racing kill or resume runs. Closes #175
169 lines
9.5 KiB
Markdown
169 lines
9.5 KiB
Markdown
# Restart and upgrade contract
|
|
|
|
Maestro is upgraded in place: the operator reinstalls the binary and systemd
|
|
restarts the daemon under every browser tab and TUI that is attached to it.
|
|
tmux keeps the sessions alive across that (see `docs/INSTALL.md`), but clients
|
|
lose their sockets. This document is the contract for what a client does about
|
|
it — what the daemon publishes, how a client tells a restart from a blip, and
|
|
what the user sees while it happens.
|
|
|
|
## What the daemon publishes
|
|
|
|
`GET /api/info` carries two fields beyond identity and presets:
|
|
|
|
| field | meaning |
|
|
| ---------- | -------------------------------------------------------------- |
|
|
| `build` | `{ commit, commit_date, dirty }` — the commit the binary was compiled from |
|
|
| `instance` | uuid regenerated on every daemon start |
|
|
|
|
`build` is stamped at compile time by `crates/maestro-daemon/build.rs`, which
|
|
runs `git` in the checkout it is compiled in. The Nix build has no `.git` in the
|
|
crane-filtered source tree, so `flake.nix` passes the flake's own revision as
|
|
`MAESTRO_BUILD_COMMIT` / `MAESTRO_BUILD_DATE` / `MAESTRO_BUILD_DIRTY`. A build
|
|
with neither source reports empty strings — clients render that as
|
|
"unknown release" and fall back to `instance` alone.
|
|
|
|
`instance` is the one that never goes missing. It says *this process*, so two
|
|
replies that differ on it are two runs of the daemon, even when the commit did
|
|
not change (a plain `systemctl restart`, or a crash).
|
|
|
|
## How a client reads them
|
|
|
|
Three verdicts, from comparing the reply before an outage with the one after
|
|
(`compareIdentity` in `web/src/daemonLink.ts`, mirrored by
|
|
`crates/maestro-tui/src/daemon_link.rs`):
|
|
|
|
- **upgraded** — both commits are known and differ. New server code.
|
|
- **restarted** — commits match (or are unknown) and `instance` differs.
|
|
- **same** — neither changed; the socket blipped, the daemon never moved.
|
|
|
|
Only *upgraded* is special. A browser tab is holding a bundle that the upgraded
|
|
daemon no longer serves, so it reloads the page rather than driving new server
|
|
code from an old bundle. The TUI is its own binary — a reload would not replace
|
|
it — so it stays up, reconnects, and says which release it is now talking to;
|
|
the user restarts `maestro tui` when they want the new client.
|
|
|
|
## What the user sees
|
|
|
|
The **status indicator** lives in the sidebar footer, left of the spawn button,
|
|
in both clients. Its colour is the link state, and it names the running release
|
|
— commit hash and commit date — on hover (web) or in the status bar (TUI).
|
|
|
|
| state | colour | meaning |
|
|
| -------------- | ------ | ---------------------------------------------------- |
|
|
| `connecting` | amber | first probe hasn't landed |
|
|
| `online` | green | daemon answered; same process, or a resolved new one |
|
|
| `reconnecting` | amber | socket dropped, still inside the grace window |
|
|
| `down` | red | unreachable past the grace window — the UI blocks |
|
|
| `updating` | red | answered on a new commit; the page is reloading |
|
|
|
|
`updating` is web-only, because reloading is web-only. The TUI goes straight
|
|
back to `online` after an upgrade and puts `updated: <hash> (restart tui)` on
|
|
its status bar instead, until the user restarts it.
|
|
|
|
The **restart overlay** is up for exactly `down` and `updating`. It is not
|
|
dismissable and it swallows the app's keyboard shortcuts: everything behind it
|
|
would act on a daemon that is not there. In the TUI the quit binding still
|
|
resolves, so a blocked client is never a trapped one. It clears itself the
|
|
moment a probe succeeds — there is no "retry" button, because the client is
|
|
already retrying.
|
|
|
|
## What survives, and what comes back dormant
|
|
|
|
A daemon restart leaves tmux alone, so recovery re-adopts every
|
|
`maestro-<uuid>` session tmux still holds and the operator sees no change
|
|
beyond the reconnect above.
|
|
|
|
A **host reboot** is different: it kills the tmux server, so there is nothing
|
|
to adopt while every session's sidecar is still on disk. Recovery always asks
|
|
tmux — `tmux list-sessions` is the single source of truth for liveness, and the
|
|
kernel's `boot_id` only tells a reboot from a restart in the log (#164, #174).
|
|
A boot id changes without the tmux server dying whenever the state directory is
|
|
restored, a VM or container snapshot is rolled back, or the `boot_id` file is
|
|
truncated; a session tmux still holds is adopted `Running` in every one of
|
|
those cases. Recovery makes a
|
|
second pass over those sidecars and registers each as a `Dormant` session
|
|
(#162) — id, name, cwd, command, worktree and `created_at` intact. A dormant
|
|
session has no pane and no PTY bridge: it cannot be attached to, typed into or
|
|
resized, and neither watcher pass touches it.
|
|
|
|
A dormant session is brought back with `POST /api/sessions/:id/resume` (#166),
|
|
which respawns tmux under the **same** uuid and sidecar — `metadata.name`,
|
|
`created_at`, `env` and worktree ownership all survive, and only the tmux
|
|
session and the PTY bridge are new. The command is the matching spawn preset's
|
|
`resume_command` with `{agent_session}` filled in from the session's own
|
|
`agent_session_id`, so an agent comes back inside its old conversation; a
|
|
session with no matching preset, no `resume_command`, or no reported
|
|
conversation id is respawned on the command it was created with. `Session::command`
|
|
always keeps that original command, so a second resume resolves the same preset.
|
|
Resume is refused with `409` for a session that is already running
|
|
(`already_running`) and for one whose `cwd` has disappeared (`cwd_missing`) —
|
|
neither is something a respawn could fix. A session with a resume already in
|
|
flight is refused as `already_running` too (#175): the respawn happens with
|
|
the session map unlocked, so the record is marked in-flight before it starts
|
|
and the second request is answered on that state rather than on its tmux spawn
|
|
colliding with the first one's. A resume that loses a race with a kill takes
|
|
its own sidecar back off disk, so the killed session does not return as a
|
|
dormant ghost on the next recovery. The other thing to do with a dormant
|
|
session is dismiss it, which removes the sidecar so the next recovery does not
|
|
bring it back.
|
|
|
|
## What recovery does with them: `resume_on_boot`
|
|
|
|
Whether a recovered-but-dead session is left dormant at all is the operator's
|
|
call, per host, via `[daemon].resume_on_boot` (#167):
|
|
|
|
| value | what recovery does |
|
|
| ----------- | --------------------------------------------------------------------- |
|
|
| `"dormant"` | **Default.** Register each one dormant and wait for the operator. |
|
|
| `"auto"` | Resume each one during recovery, itself. |
|
|
| `"never"` | Prune the sidecars instead — a reboot is a clean slate. |
|
|
|
|
`dormant` is the default because resuming agents unattended is a foot-gun:
|
|
they come back on whatever flags they were spawned with, seconds after a
|
|
reboot, with nobody watching. `auto` exists for hosts where that is known to
|
|
be safe, and it is opted into rather than inherited.
|
|
|
|
`auto` calls the same `resume` the client's `POST /api/sessions/:id/resume`
|
|
does — same uuid, same sidecar, the preset's `resume_command` — so there is
|
|
one resume path and not two. A session that fails to resume is logged and
|
|
stays dormant; the rest still come back.
|
|
|
|
`never` prunes the same way #163 does for a missing `cwd`: the sidecar is
|
|
deleted and nothing is registered, so the session is not listed and the next
|
|
recovery does not find it. Whatever worktree it owned is left on disk.
|
|
|
|
The policy only decides the fate of sessions with no live pane. A session
|
|
tmux still holds is adopted `Running` by the first pass under every policy,
|
|
including `never`.
|
|
|
|
Recovery logs the policy that ran and the number of sessions it acted on, so
|
|
a host that resumes agents unattended says so in its own journal.
|
|
|
|
Attaching to a session that has no bridge answers `409 not_attachable`, not
|
|
`404 unknown_session`: the session exists, and a client that cannot tell "gone"
|
|
from "not running" cannot offer the resume.
|
|
|
|
`SessionStatus` is a wire type, so from this build forward it deserializes
|
|
leniently on both clients: a spelling the client does not know reads as `Dead`
|
|
rather than failing the whole session. That leniency lives in the client's own
|
|
binary, so it covers spellings added *after* this build, not this one — a CLI
|
|
or TUI still running an older build fails to parse a `Dormant` session, and
|
|
with it the whole session list, until it is upgraded. Clients render a dormant session as known-but-not-running —
|
|
see `docs/SIDEBAR.md` for the card and `docs/COMMAND-BAR.md` for how it ranks.
|
|
|
|
## Timing
|
|
|
|
- **Grace window** — 1200 ms between the socket dropping and the overlay
|
|
appearing. A wifi hiccup or a laptop waking up should not throw a modal.
|
|
- **Poll cadence** — `GET /api/info` every 700 ms while unreachable, flat, not
|
|
backed off: a restart takes a second or two and the point is to unlock as soon
|
|
as it is over. The events socket keeps its own exponential backoff for the
|
|
session stream; the two are independent, and the probe is what decides state.
|
|
|
|
The events socket dropping is the trigger, not a timer: the daemon cannot warn
|
|
anyone it is about to die, so its socket closing is the earliest signal there
|
|
is. A reconnect of that socket does **not** by itself clear the state — only a
|
|
probe does, because the socket carries no identity and reconnecting to an
|
|
upgraded daemon must reload rather than resume.
|