Compare commits

...
Author SHA1 Message Date
Miguel Palhas d050df7d3d fix(maestro): always probe tmux during recovery
#164 skipped the tmux probe whenever the stored boot id differed from the
current one, assuming a changed boot id proves the tmux server is gone. It
does not: a restored state directory, a VM or container snapshot rollback,
or a truncated boot_id file all change it while tmux keeps running with
live sessions. Every one of those sessions was then registered Dormant and
stranded — resume respawns `maestro-<id>`, which collides with the name
tmux still holds, so it failed 502 forever.

Recovery now always runs `tmux list-sessions`, the single source of truth
for liveness the rest of the daemon assumes. The boot id keeps its other
job: telling a reboot from a restart in recovery's log.

The #164 test asserted the old behaviour with tmux still alive, so it is
replaced by one asserting the session is adopted Running, plus one covering
a real reboot — boot id changed and the tmux server dead — still going
dormant.

Refs #174
2026-09-04 09:13:15 +01:00
Miguel Palhas e51a7007a6 merge: resume_on_boot policy (#167)
Closes #167
2026-09-04 09:10:28 +01:00
Miguel PalhasandClaude Opus 5 02bf077c03 feat(maestro): add [daemon].resume_on_boot policy
Recovery now has a stated policy for sessions whose tmux pane is gone but
whose sidecar survived: dormant (default), auto, or never. auto goes down
the same SessionManager::resume the client's POST takes, so there is one
resume path. never prunes the sidecars the way #163 does for a missing cwd.

The default is dormant because resuming agents unattended seconds after a
reboot is a foot-gun; auto is opted into per host. Recovery logs which
policy ran and how many sessions it acted on.

Refs #167

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 09:06:51 +01:00
9 changed files with 493 additions and 27 deletions
+97
View File
@@ -129,6 +129,80 @@ pub struct DaemonSection {
/// `Some(0)` disables history entirely (no log written, no recents read).
#[serde(default)]
pub spawn_history_days: Option<u32>,
/// What recovery does with sessions that outlived their tmux server —
/// after a host reboot, or any restart that finds a sidecar with no pane
/// (#167). Default [`ResumeOnBoot::Dormant`].
#[serde(default, deserialize_with = "deserialize_resume_on_boot")]
pub resume_on_boot: Option<ResumeOnBoot>,
}
impl DaemonSection {
/// The configured `[daemon].resume_on_boot`, or the safe default:
/// recovered-but-dead sessions wait for the operator.
pub fn resume_on_boot(&self) -> ResumeOnBoot {
self.resume_on_boot.unwrap_or_default()
}
}
/// `[daemon].resume_on_boot` — what recovery does with a session whose tmux
/// pane is gone but whose sidecar is still on disk (#167).
///
/// Resuming agents unattended seconds after a reboot is a foot-gun: they run
/// with whatever flags they were spawned with, with nobody watching. So the
/// default waits for a human, and `auto` is opted into per host.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum ResumeOnBoot {
/// Register each one `Dormant` and wait for the operator to resume it.
#[default]
Dormant,
/// Resume each one during recovery, on the same path a client's
/// `POST /api/sessions/:id/resume` takes.
Auto,
/// Prune the sidecars instead: a reboot is a clean slate.
Never,
}
/// Every `[daemon].resume_on_boot` name and its variant, in documentation
/// order.
pub const RESUME_ON_BOOT_POLICIES: &[(&str, ResumeOnBoot)] = &[
("dormant", ResumeOnBoot::Dormant),
("auto", ResumeOnBoot::Auto),
("never", ResumeOnBoot::Never),
];
impl ResumeOnBoot {
/// The name this policy is spelled with in the config file and in logs.
pub fn as_str(&self) -> &'static str {
match self {
ResumeOnBoot::Dormant => "dormant",
ResumeOnBoot::Auto => "auto",
ResumeOnBoot::Never => "never",
}
}
}
fn deserialize_resume_on_boot<'de, D>(deserializer: D) -> Result<Option<ResumeOnBoot>, D::Error>
where
D: Deserializer<'de>,
{
let raw = Option::<String>::deserialize(deserializer)?;
raw.map(|name| {
RESUME_ON_BOOT_POLICIES
.iter()
.find(|(n, _)| *n == name)
.map(|(_, policy)| *policy)
.ok_or_else(|| {
let known = RESUME_ON_BOOT_POLICIES
.iter()
.map(|(n, _)| format!("{n:?}"))
.collect::<Vec<_>>()
.join(", ");
D::Error::custom(format!(
"invalid [daemon].resume_on_boot {name:?}; expected one of {known}"
))
})
})
.transpose()
}
/// `[tui]` — `maestro tui` settings (NAP-275 epic, phase 2 NAP-277).
@@ -775,4 +849,27 @@ bind_addr = "127.0.0.1:9090"
);
assert_eq!(cfg.resolve_daemon_url(None), "http://explicit:1");
}
#[test]
fn resume_on_boot_defaults_to_dormant_and_parses_every_policy() {
assert_eq!(
MaestroConfigFile::default().daemon.resume_on_boot(),
ResumeOnBoot::Dormant,
"an absent policy is the safe one"
);
for (name, expected) in RESUME_ON_BOOT_POLICIES {
let cfg = file_with(&format!("[daemon]\nresume_on_boot = \"{name}\"\n"));
assert_eq!(cfg.daemon.resume_on_boot(), *expected, "{name}");
assert_eq!(expected.as_str(), *name);
}
}
#[test]
fn resume_on_boot_rejects_an_unknown_policy() {
let err = toml::from_str::<MaestroConfigFile>("[daemon]\nresume_on_boot = \"maybe\"\n")
.expect_err("unknown policy must not silently fall back");
let msg = err.to_string();
assert!(msg.contains("resume_on_boot"), "{msg}");
assert!(msg.contains("\"auto\""), "{msg}");
}
}
+7 -4
View File
@@ -2,10 +2,13 @@
//!
//! `/proc/sys/kernel/random/boot_id` is fixed for the life of the kernel and
//! changes on every boot. [`SessionManager::recover`] compares it against the
//! value it stored on the previous run: a difference means every session from
//! before is provably dead, so the tmux probe can be skipped and those
//! sessions registered dormant directly instead of relying on tmux reporting
//! an empty list.
//! value it stored on the previous run and logs which of the two it was.
//!
//! It does not decide liveness (#174). A boot id changes without the tmux
//! server dying whenever the state directory is restored, a VM or container
//! snapshot is rolled back, or the `boot_id` file is truncated — so recovery
//! always probes `tmux list-sessions` and the boot id only says whether a
//! server that did not answer is coming back.
use std::path::Path;
+4 -1
View File
@@ -87,7 +87,10 @@ async fn run(config_override: Option<PathBuf>) -> Result<(), Box<dyn Error>> {
.with_boot_id(
config.boot_id_path(),
std::sync::Arc::new(crate::boot_id::ProcBootId),
);
)
// #167: recovery needs the presets to resolve a resumed session's command,
// so `auto` runs the same resume a client's POST would.
.with_resume_policy(config.resume_on_boot, config.spawn_presets.clone());
// Spawned sessions learn where to report back (#103). Derived from the
// configured bind address; set before any spawn can happen.
manager.set_daemon_url(format!("http://{}", config.bind_addr));
+35
View File
@@ -97,6 +97,10 @@ pub struct DaemonConfig {
/// never binds it itself; it serves it so the web client and the TUI read
/// one knob.
pub leader: String,
/// What recovery does with sessions whose tmux pane is gone but whose
/// sidecar survives (#167): `[daemon].resume_on_boot`, default
/// [`maestro_config::ResumeOnBoot::Dormant`].
pub resume_on_boot: maestro_config::ResumeOnBoot,
}
impl DaemonConfig {
@@ -149,6 +153,8 @@ pub fn load_config(path: &Path, env: EnvOverrides) -> Result<DaemonConfig, Confi
// validation (#22/#26).
let spawn_presets = file.spawn.resolve()?;
let resume_on_boot = daemon.resume_on_boot();
let hostname = daemon.hostname.unwrap_or_else(system_hostname);
let home_dir = dirs::home_dir()
.map(|p| p.to_string_lossy().into_owned())
@@ -209,6 +215,7 @@ pub fn load_config(path: &Path, env: EnvOverrides) -> Result<DaemonConfig, Confi
project_scan_depth,
hooks_auto_install: file.hooks.auto_install(),
leader: file.tui.leader().to_string(),
resume_on_boot,
})
}
@@ -289,6 +296,34 @@ auto_install = false
assert!(!cfg.hooks_auto_install);
}
#[test]
fn resume_on_boot_defaults_to_dormant_and_reads_the_file() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.toml");
std::fs::write(&path, "").unwrap();
assert_eq!(
load_config(&path, EnvOverrides::default())
.unwrap()
.resume_on_boot,
maestro_config::ResumeOnBoot::Dormant
);
std::fs::write(
&path,
r#"
[daemon]
resume_on_boot = "auto"
"#,
)
.unwrap();
assert_eq!(
load_config(&path, EnvOverrides::default())
.unwrap()
.resume_on_boot,
maestro_config::ResumeOnBoot::Auto
);
}
#[test]
fn empty_file_defaults_everything() {
let dir = tempfile::tempdir().unwrap();
+88 -13
View File
@@ -12,6 +12,7 @@ use std::sync::Arc;
use std::time::Duration;
use chrono::Utc;
use maestro_config::ResumeOnBoot;
use maestro_protocol::{
ActivityState, CatalogEvent, CreateSessionRequest, ErrorInfo, NameSource, Session,
SessionStatus, WorktreeSpec, NAME_KEY, NAME_SOURCE_KEY,
@@ -195,6 +196,11 @@ struct Inner {
/// reboot detection is unwired, which behaves exactly like a boot id
/// that could never be read: always fall back to probing tmux.
boot_id: Option<BootIdConfig>,
/// What recovery does with a session whose pane is gone (#167), and the
/// presets an `auto` resume resolves its command from. Defaults to
/// [`ResumeOnBoot::Dormant`] with no presets: register and wait.
resume_policy: ResumeOnBoot,
resume_presets: Vec<maestro_protocol::SpawnPreset>,
}
/// Where recovery persists the boot id it last saw, and how it reads the
@@ -253,6 +259,8 @@ impl SessionManager {
viewports: Mutex::new(Viewports::new()),
next_viewport: AtomicU64::new(0),
boot_id: None,
resume_policy: ResumeOnBoot::default(),
resume_presets: Vec::new(),
}),
}
}
@@ -297,6 +305,26 @@ impl SessionManager {
self
}
/// Wire the `[daemon].resume_on_boot` policy (#167) and the spawn presets
/// an `auto` resume resolves each session's command from. Builder, called
/// once right after construction like the other `with_*` methods. Without
/// it, recovery behaves as [`ResumeOnBoot::Dormant`].
///
/// # Panics
///
/// If the handle has already been cloned.
pub fn with_resume_policy(
mut self,
policy: ResumeOnBoot,
presets: Vec<maestro_protocol::SpawnPreset>,
) -> Self {
let inner = Arc::get_mut(&mut self.inner)
.expect("with_resume_policy must be called before the SessionManager is cloned");
inner.resume_policy = policy;
inner.resume_presets = presets;
self
}
/// Set the daemon's own HTTP base URL (#103), exported to every session
/// spawned afterwards as `MAESTRO_URL`. Unlike the `with_*` builders this
/// is a plain setter: the daemon runner learns its bound address after
@@ -819,20 +847,20 @@ impl SessionManager {
/// that no longer exists are still adopted — the spec does not require
/// us to repair them.
///
/// #164: when the kernel's boot id has changed since the last run, the
/// tmux probe below is skipped entirely — a reboot kills the tmux server,
/// so every previous session is already dead and there is nothing tmux
/// could truthfully adopt. [`Self::register_dormant`] then registers all
/// of them directly.
/// #164/#174: the kernel's boot id says whether the host rebooted since
/// the last run, but it never decides which sessions are alive —
/// `tmux list-sessions` does, always, which is what the rest of the
/// daemon assumes. A boot id can change without the tmux server dying
/// (a restored state directory, a VM or container snapshot restore, a
/// truncated `boot_id` file); skipping the probe then registered every
/// live session `Dormant`, and resume could never bring one back because
/// the tmux name was still taken. The boot id is now only a signal that a
/// tmux server which did not answer is not coming back, and it names
/// itself in recovery's log.
pub async fn recover(&self) -> Result<usize, SessionError> {
tokio::fs::create_dir_all(&self.inner.fifo_dir).await?;
let rebooted = self.check_boot_id().await;
let live = if rebooted {
// #164: every session from the previous boot is provably dead —
// no tmux server survives a reboot — so there is nothing to gain
// from asking it and no session it could honestly report back.
Vec::new()
} else {
let live = {
let tmux = self.inner.tmux.clone();
tmux_blocking(move || tmux.list_sessions()).await?
};
@@ -912,7 +940,9 @@ impl SessionManager {
///
/// A missing stored value (first run) or an unreadable current one
/// (non-Linux, restricted `/proc`, unwired in this build) both read as
/// "no reboot", which is today's behaviour: fall back to probing tmux.
/// "no reboot". Either way the tmux probe runs (#174): the answer only
/// tells recovery whether a tmux server that did not answer is coming
/// back, never which sessions are alive.
async fn check_boot_id(&self) -> bool {
let Some(cfg) = &self.inner.boot_id else {
return false;
@@ -959,7 +989,8 @@ impl SessionManager {
/// `rebooted` (#164) only changes what gets logged: a detected reboot
/// names the count even when it is zero, so the operator sees that
/// recovery told a reboot from a restart instead of a log that stays
/// silent either way.
/// silent either way. It never decides membership — everything here got
/// here by tmux not holding it (#174).
async fn register_dormant(&self, rebooted: bool) {
let stored = match self.inner.sidecar.load_all() {
Ok(stored) => stored,
@@ -970,6 +1001,7 @@ impl SessionManager {
};
let mut registered = Vec::new();
let mut pruned = 0usize;
let mut discarded = 0usize;
{
let mut sessions = self.inner.sessions.lock().await;
for mut session in stored {
@@ -984,6 +1016,16 @@ impl SessionManager {
pruned += 1;
continue;
}
if self.inner.resume_policy == ResumeOnBoot::Never {
// #167: this host wants a reboot to be a clean slate, so
// the sidecar goes the way #163's does — nothing is
// registered and the next recovery finds nothing.
if let Err(e) = self.inner.sidecar.delete(session.id) {
tracing::warn!(error = ?e, id = %session.id, "recover: failed to prune sidecar under resume_on_boot=never");
}
discarded += 1;
continue;
}
let id = session.id;
session.status = SessionStatus::Dormant;
session.foreground = None;
@@ -1025,12 +1067,45 @@ impl SessionManager {
if pruned > 0 {
tracing::info!(pruned, "pruned sidecars whose cwd no longer exists");
}
let resumable: Vec<Uuid> = registered.iter().map(|s| s.id).collect();
for session in registered {
let _ = self
.inner
.events
.send(CatalogEvent::SessionAdded { session });
}
// #167: say which policy ran and how many sessions it acted on, so a
// host that resumes agents unattended says so in its own log. The
// match is exhaustive on purpose: a fourth policy must decide what it
// does here rather than inherit whichever arm happened to be last.
match self.inner.resume_policy {
ResumeOnBoot::Dormant => tracing::info!(
policy = ResumeOnBoot::Dormant.as_str(),
sessions = resumable.len(),
"resume_on_boot: sessions left dormant for the operator"
),
ResumeOnBoot::Never => tracing::info!(
policy = ResumeOnBoot::Never.as_str(),
sessions = discarded,
"resume_on_boot: pruned the sidecars of sessions with no live pane"
),
ResumeOnBoot::Auto => {
tracing::info!(
policy = ResumeOnBoot::Auto.as_str(),
sessions = resumable.len(),
"resume_on_boot: resuming sessions with no live pane"
);
let presets = self.inner.resume_presets.clone();
for id in resumable {
// The same call the client's POST /api/sessions/:id/resume
// makes — one resume path, not two.
if let Err(e) = self.resume(id, &presets).await {
tracing::warn!(error = %e, id = %id, "resume_on_boot=auto: resume failed; session stays dormant");
}
}
}
}
}
/// Forward keystrokes to a session's pane. A dead or dormant session is
@@ -0,0 +1,161 @@
//! `[daemon].resume_on_boot` (#167): what recovery does with sessions whose
//! tmux pane is gone but whose sidecar survived, against real tmux.
//!
//! The default is deliberately the timid one. Resuming agents unattended
//! seconds after a reboot is a foot-gun, so `auto` is opted into per host and
//! `never` throws the sidecars away instead.
mod common;
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
use common::{wait_for_async, DaemonFixture};
use maestro_config::ResumeOnBoot;
use maestro_daemon::SessionManager;
use maestro_protocol::{CreateSessionRequest, Session, SessionStatus, SpawnPreset};
use tempfile::TempDir;
/// A fresh manager over the same tmux server, sidecars and fifos — what the
/// daemon builds after a restart — carrying `policy`.
fn revive(fx: &DaemonFixture, policy: ResumeOnBoot, presets: Vec<SpawnPreset>) -> SessionManager {
SessionManager::new(
Arc::new(fx.tmux.tmux.clone()),
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir()),
maestro_daemon::worktree::WorktreeManager::new(None),
fx.fifo_dir(),
)
.with_resume_policy(policy, presets)
}
/// A preset whose `resume_command` touches a marker file, so a test can see
/// that the resumed session ran the resume command rather than the original.
fn marker_preset(command: &str, marker_dir: &TempDir) -> Vec<SpawnPreset> {
vec![SpawnPreset {
id: "agent".into(),
label: "Agent".into(),
command: command.into(),
resume_command: Some(format!(
"sh -c 'touch {}/resumed; sleep 30'",
marker_dir.path().display()
)),
}]
}
async fn create(fx: &DaemonFixture, cwd: &TempDir, command: &str) -> Session {
fx.manager
.create(CreateSessionRequest {
cwd: cwd.path().to_path_buf(),
command: command.into(),
env: HashMap::new(),
metadata: HashMap::new(),
worktree: None,
})
.await
.expect("create")
}
/// The default: a reboot leaves the session listed but not running, waiting
/// for the operator.
#[tokio::test]
async fn dormant_is_the_default_and_leaves_the_session_waiting() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let session = create(&fx, &cwd, "sleep 30").await;
fx.tmux.tmux.kill_server().expect("kill tmux server");
let revived = revive(&fx, ResumeOnBoot::Dormant, Vec::new());
revived.recover().await.expect("recover");
assert_eq!(
revived.status(session.id).await,
Some(SessionStatus::Dormant)
);
assert!(fx
.sidecar_dir()
.join(format!("{}.json", session.id))
.exists());
assert!(
fx.tmux.tmux.list_sessions().expect("list").is_empty(),
"nothing was respawned"
);
}
/// `auto` brings each one back itself, down the same path a client's resume
/// takes: same uuid, same sidecar, the preset's `resume_command`.
#[tokio::test]
async fn auto_resumes_every_dormant_session_during_recovery() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let markers = TempDir::new().unwrap();
let session = create(&fx, &cwd, "sleep 30").await;
fx.tmux.tmux.kill_server().expect("kill tmux server");
let revived = revive(&fx, ResumeOnBoot::Auto, marker_preset("sleep 30", &markers));
revived.recover().await.expect("recover");
assert_eq!(
revived.status(session.id).await,
Some(SessionStatus::Running),
"auto resumed it"
);
assert_eq!(
fx.tmux.tmux.list_sessions().expect("list"),
vec![session.id],
"back under the same uuid"
);
let stored = maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir())
.read(session.id)
.expect("read sidecar")
.expect("same sidecar");
assert_eq!(stored.created_at, session.created_at);
assert_eq!(stored.command, "sleep 30", "the original command is kept");
// It really ran the preset's resume command, not the spawn command.
let marker = markers.path().join("resumed");
wait_for_async(Duration::from_secs(5), || {
let marker = marker.clone();
async move { marker.exists() }
})
.await;
assert!(marker.exists(), "resume_command ran");
}
/// `never` makes a reboot a clean slate: the sidecars go, and the session is
/// not listed at all.
#[tokio::test]
async fn never_prunes_the_sidecars_instead() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let session = create(&fx, &cwd, "sleep 30").await;
fx.tmux.tmux.kill_server().expect("kill tmux server");
let revived = revive(&fx, ResumeOnBoot::Never, Vec::new());
revived.recover().await.expect("recover");
assert!(!revived.has(session.id).await, "not listed");
assert!(
!fx.sidecar_dir()
.join(format!("{}.json", session.id))
.exists(),
"sidecar pruned"
);
assert!(fx.tmux.tmux.list_sessions().expect("list").is_empty());
}
/// The policy only decides what happens to sessions that lost their pane. A
/// session tmux still holds is adopted running by the first pass under every
/// policy — including `never`, which must not delete a live session's sidecar.
#[tokio::test]
async fn the_policy_leaves_sessions_tmux_still_holds_alone() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let live = create(&fx, &cwd, "sleep 30").await;
let revived = revive(&fx, ResumeOnBoot::Never, Vec::new());
assert_eq!(revived.recover().await.expect("recover"), 1);
assert_eq!(revived.status(live.id).await, Some(SessionStatus::Running));
assert!(fx.sidecar_dir().join(format!("{}.json", live.id)).exists());
}
+42 -8
View File
@@ -326,12 +326,14 @@ fn revive_with_boot_id(
)
}
/// #164: a changed boot id means every session from the previous boot is
/// already dead, so recovery must not trust tmux to say so — it skips the
/// probe outright. Proven by leaving the tmux session alive: if the probe
/// ran, it would come back `Running`, not `Dormant`.
/// #174: a boot id can change without the tmux server dying — a restored
/// state directory, a VM or container snapshot restore, a truncated
/// `boot_id` file. `tmux list-sessions` stays the single source of truth for
/// liveness, so a session tmux still holds is adopted `Running` no matter
/// what the boot id says. Registering it dormant instead stranded it: resume
/// would collide with the tmux name still in use and fail forever.
#[tokio::test]
async fn recover_skips_tmux_probe_when_boot_id_changed() {
async fn recover_adopts_live_session_when_boot_id_changed() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let s = fx
@@ -349,15 +351,15 @@ async fn recover_skips_tmux_probe_when_boot_id_changed() {
let boot_id_path = fx.state_dir.path().join("boot_id");
std::fs::write(&boot_id_path, "previous-boot").unwrap();
// tmux session is still alive — a probe would find it if one ran.
// tmux session is still alive: the probe runs and finds it.
let revived = revive_with_boot_id(&fx, boot_id_path.clone(), Some("current-boot"));
let adopted = revived.recover().await.expect("recover");
assert_eq!(adopted, 0, "reboot detected: nothing adopted live");
assert_eq!(adopted, 1, "tmux still holds the session; adopt it");
let snap = revived.snapshot().await;
assert_eq!(snap.len(), 1);
assert_eq!(snap[0].id, s.id);
assert_eq!(snap[0].status, SessionStatus::Dormant);
assert_eq!(snap[0].status, SessionStatus::Running);
// The current boot id is now the stored one, for the next comparison.
assert_eq!(
@@ -366,6 +368,38 @@ async fn recover_skips_tmux_probe_when_boot_id_changed() {
);
}
/// #164, still true after #174: a real reboot takes the tmux server with it,
/// so the probe comes back empty and every sidecar is registered dormant.
#[tokio::test]
async fn recover_registers_dormant_when_boot_id_changed_and_tmux_gone() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let s = fx
.manager
.create(CreateSessionRequest {
cwd: cwd.path().to_path_buf(),
command: "sleep 30".into(),
env: HashMap::new(),
metadata: HashMap::new(),
worktree: None,
})
.await
.expect("create");
let boot_id_path = fx.state_dir.path().join("boot_id");
std::fs::write(&boot_id_path, "previous-boot").unwrap();
fx.tmux.tmux.kill_server().expect("kill tmux server");
let revived = revive_with_boot_id(&fx, boot_id_path, Some("current-boot"));
let adopted = revived.recover().await.expect("recover");
assert_eq!(adopted, 0, "no tmux server survived the reboot");
let snap = revived.snapshot().await;
assert_eq!(snap.len(), 1);
assert_eq!(snap[0].id, s.id);
assert_eq!(snap[0].status, SessionStatus::Dormant);
}
/// #164: an unchanged boot id is a plain restart — recovery probes tmux as
/// before and adopts the still-live session `Running`.
#[tokio::test]
+20
View File
@@ -243,6 +243,26 @@ Pick `/tmp` if the daemon must outlive your login session and you can't run
tmux sockets with it). The env var `MAESTRO_DAEMON_TMUX_TMPDIR` takes
precedence over the file.
## What happens to sessions after a reboot
A host reboot kills the tmux server, so every session the daemon knew is dead
while its record on disk survives. What recovery does with those is
`[daemon].resume_on_boot`:
```toml
[daemon]
resume_on_boot = "dormant" # "dormant" (default) | "auto" | "never"
```
- `dormant` — the sessions are listed but not running, and you resume the ones
you want from the web UI or the TUI.
- `auto` — the daemon resumes them all itself during startup. Only pick this
on a host where restarting every agent unattended is safe: they come back on
the flags they were spawned with, with nobody watching.
- `never` — the records are deleted and a reboot is a clean slate.
See `docs/RESTART.md` for the full contract.
## What the systemd unit looks like
The wizard writes a *user* unit (`systemctl --user`), since `maestro daemon`
+39 -1
View File
@@ -75,7 +75,13 @@ A daemon restart leaves tmux alone, so recovery re-adopts every
beyond the reconnect above.
A **host reboot** is different: it kills the tmux server, so there is nothing
to adopt while every session's sidecar is still on disk. Recovery makes a
to adopt while every session's sidecar is still on disk. Recovery always asks
tmux — `tmux list-sessions` is the single source of truth for liveness, and the
kernel's `boot_id` only tells a reboot from a restart in the log (#164, #174).
A boot id changes without the tmux server dying whenever the state directory is
restored, a VM or container snapshot is rolled back, or the `boot_id` file is
truncated; a session tmux still holds is adopted `Running` in every one of
those cases. Recovery makes a
second pass over those sidecars and registers each as a `Dormant` session
(#162) — id, name, cwd, command, worktree and `created_at` intact. A dormant
session has no pane and no PTY bridge: it cannot be attached to, typed into or
@@ -96,6 +102,38 @@ neither is something a respawn could fix. The other thing to do with a dormant
session is dismiss it, which removes the sidecar so the next recovery does not
bring it back.
## What recovery does with them: `resume_on_boot`
Whether a recovered-but-dead session is left dormant at all is the operator's
call, per host, via `[daemon].resume_on_boot` (#167):
| value | what recovery does |
| ----------- | --------------------------------------------------------------------- |
| `"dormant"` | **Default.** Register each one dormant and wait for the operator. |
| `"auto"` | Resume each one during recovery, itself. |
| `"never"` | Prune the sidecars instead — a reboot is a clean slate. |
`dormant` is the default because resuming agents unattended is a foot-gun:
they come back on whatever flags they were spawned with, seconds after a
reboot, with nobody watching. `auto` exists for hosts where that is known to
be safe, and it is opted into rather than inherited.
`auto` calls the same `resume` the client's `POST /api/sessions/:id/resume`
does — same uuid, same sidecar, the preset's `resume_command` — so there is
one resume path and not two. A session that fails to resume is logged and
stays dormant; the rest still come back.
`never` prunes the same way #163 does for a missing `cwd`: the sidecar is
deleted and nothing is registered, so the session is not listed and the next
recovery does not find it. Whatever worktree it owned is left on disk.
The policy only decides the fate of sessions with no live pane. A session
tmux still holds is adopted `Running` by the first pass under every policy,
including `never`.
Recovery logs the policy that ran and the number of sessions it acted on, so
a host that resumes agents unattended says so in its own journal.
Attaching to a session that has no bridge answers `409 not_attachable`, not
`404 unknown_session`: the session exists, and a client that cannot tell "gone"
from "not running" cannot offer the resume.