maestro/docs/RESTART.md

Restart and upgrade contract

Maestro is upgraded in place: the operator reinstalls the binary and systemd restarts the daemon under every browser tab and TUI that is attached to it. tmux keeps the sessions alive across that (see docs/INSTALL.md), but clients lose their sockets. This document is the contract for what a client does about it — what the daemon publishes, how a client tells a restart from a blip, and what the user sees while it happens.

What the daemon publishes

GET /api/info carries two fields beyond identity and presets:

field	meaning
build	{ commit, commit_date, dirty } — the commit the binary was compiled from
instance	uuid regenerated on every daemon start

build is stamped at compile time by crates/maestro-daemon/build.rs, which runs git in the checkout it is compiled in. The Nix build has no .git in the crane-filtered source tree, so flake.nix passes the flake's own revision as MAESTRO_BUILD_COMMIT / MAESTRO_BUILD_DATE / MAESTRO_BUILD_DIRTY. A build with neither source reports empty strings — clients render that as "unknown release" and fall back to instance alone.

instance is the one that never goes missing. It says this process, so two replies that differ on it are two runs of the daemon, even when the commit did not change (a plain systemctl restart, or a crash).

How a client reads them

Three verdicts, from comparing the reply before an outage with the one after (compareIdentity in web/src/daemonLink.ts, mirrored by crates/maestro-tui/src/daemon_link.rs):

• upgraded — both commits are known and differ. New server code.
• restarted — commits match (or are unknown) and instance differs.
• same — neither changed; the socket blipped, the daemon never moved.

Only upgraded is special. A browser tab is holding a bundle that the upgraded daemon no longer serves, so it reloads the page rather than driving new server code from an old bundle. The TUI is its own binary — a reload would not replace it — so it stays up, reconnects, and says which release it is now talking to; the user restarts maestro tui when they want the new client.

What the user sees

The status indicator lives in the sidebar footer, left of the spawn button, in both clients. Its colour is the link state, and it names the running release — commit hash and commit date — on hover (web) or in the status bar (TUI).

state	colour	meaning
connecting	amber	first probe hasn't landed
online	green	daemon answered; same process, or a resolved new one
reconnecting	amber	socket dropped, still inside the grace window
down	red	unreachable past the grace window — the UI blocks
updating	red	answered on a new commit; the page is reloading

updating is web-only, because reloading is web-only. The TUI goes straight back to online after an upgrade and puts updated: <hash> (restart tui) on its status bar instead, until the user restarts it.

The restart overlay is up for exactly down and updating. It is not dismissable and it swallows the app's keyboard shortcuts: everything behind it would act on a daemon that is not there. In the TUI the quit binding still resolves, so a blocked client is never a trapped one. It clears itself the moment a probe succeeds — there is no "retry" button, because the client is already retrying.

What survives, and what comes back dormant

A daemon restart leaves tmux alone, so recovery re-adopts every maestro-<uuid> session tmux still holds and the operator sees no change beyond the reconnect above.

A host reboot is different: it kills the tmux server, so there is nothing to adopt while every session's sidecar is still on disk. Recovery always asks tmux — tmux list-sessions is the single source of truth for liveness, and the kernel's boot_id only tells a reboot from a restart in the log (#164, #174). A boot id changes without the tmux server dying whenever the state directory is restored, a VM or container snapshot is rolled back, or the boot_id file is truncated; a session tmux still holds is adopted Running in every one of those cases. Recovery makes a second pass over those sidecars and registers each as a Dormant session (#162) — id, name, cwd, command, worktree and created_at intact. A dormant session has no pane and no PTY bridge: it cannot be attached to, typed into or resized, and neither watcher pass touches it.

A dormant session is brought back with POST /api/sessions/:id/resume (#166), which respawns tmux under the same uuid and sidecar — metadata.name, created_at, env and worktree ownership all survive, and only the tmux session and the PTY bridge are new. The command is the matching spawn preset's resume_command with {agent_session} filled in from the session's own agent_session_id, so an agent comes back inside its old conversation; a session with no matching preset, no resume_command, or no reported conversation id is respawned on the command it was created with. Session::command always keeps that original command, so a second resume resolves the same preset. Resume is refused with 409 for a session that is already running (already_running) and for one whose cwd has disappeared (cwd_missing) — neither is something a respawn could fix. The other thing to do with a dormant session is dismiss it, which removes the sidecar so the next recovery does not bring it back.

What recovery does with them: resume_on_boot

Whether a recovered-but-dead session is left dormant at all is the operator's call, per host, via [daemon].resume_on_boot (#167):

value	what recovery does
"dormant"	Default. Register each one dormant and wait for the operator.
"auto"	Resume each one during recovery, itself.
"never"	Prune the sidecars instead — a reboot is a clean slate.

dormant is the default because resuming agents unattended is a foot-gun: they come back on whatever flags they were spawned with, seconds after a reboot, with nobody watching. auto exists for hosts where that is known to be safe, and it is opted into rather than inherited.

auto calls the same resume the client's POST /api/sessions/:id/resume does — same uuid, same sidecar, the preset's resume_command — so there is one resume path and not two. A session that fails to resume is logged and stays dormant; the rest still come back.

never prunes the same way #163 does for a missing cwd: the sidecar is deleted and nothing is registered, so the session is not listed and the next recovery does not find it. Whatever worktree it owned is left on disk.

The policy only decides the fate of sessions with no live pane. A session tmux still holds is adopted Running by the first pass under every policy, including never.

Recovery logs the policy that ran and the number of sessions it acted on, so a host that resumes agents unattended says so in its own journal.

Attaching to a session that has no bridge answers 409 not_attachable, not 404 unknown_session: the session exists, and a client that cannot tell "gone" from "not running" cannot offer the resume.

SessionStatus is a wire type, so from this build forward it deserializes leniently on both clients: a spelling the client does not know reads as Dead rather than failing the whole session. That leniency lives in the client's own binary, so it covers spellings added after this build, not this one — a CLI or TUI still running an older build fails to parse a Dormant session, and with it the whole session list, until it is upgraded. Clients render a dormant session as known-but-not-running — see docs/SIDEBAR.md for the card and docs/COMMAND-BAR.md for how it ranks.

Timing

• Grace window — 1200 ms between the socket dropping and the overlay appearing. A wifi hiccup or a laptop waking up should not throw a modal.
• Poll cadence — GET /api/info every 700 ms while unreachable, flat, not backed off: a restart takes a second or two and the point is to unlock as soon as it is over. The events socket keeps its own exponential backoff for the session stream; the two are independent, and the probe is what decides state.

The events socket dropping is the trigger, not a timer: the daemon cannot warn anyone it is about to die, so its socket closing is the earliest signal there is. A reconnect of that socket does not by itself clear the state — only a probe does, because the socket carries no identity and reconnecting to an upgraded daemon must reload rather than resume.