fix(maestro): always probe tmux during recovery

#164 skipped the tmux probe whenever the stored boot id differed from the
current one, assuming a changed boot id proves the tmux server is gone. It
does not: a restored state directory, a VM or container snapshot rollback,
or a truncated boot_id file all change it while tmux keeps running with
live sessions. Every one of those sessions was then registered Dormant and
stranded — resume respawns `maestro-<id>`, which collides with the name
tmux still holds, so it failed 502 forever.

Recovery now always runs `tmux list-sessions`, the single source of truth
for liveness the rest of the daemon assumes. The boot id keeps its other
job: telling a reboot from a restart in recovery's log.

The #164 test asserted the old behaviour with tmux still alive, so it is
replaced by one asserting the session is adopted Running, plus one covering
a real reboot — boot id changed and the tmux server dead — still going
dormant.

Refs #174
This commit is contained in:
Miguel Palhas
2026-09-04 09:13:15 +01:00
parent e51a7007a6
commit d050df7d3d
4 changed files with 72 additions and 26 deletions
+7 -4
View File
@@ -2,10 +2,13 @@
//!
//! `/proc/sys/kernel/random/boot_id` is fixed for the life of the kernel and
//! changes on every boot. [`SessionManager::recover`] compares it against the
//! value it stored on the previous run: a difference means every session from
//! before is provably dead, so the tmux probe can be skipped and those
//! sessions registered dormant directly instead of relying on tmux reporting
//! an empty list.
//! value it stored on the previous run and logs which of the two it was.
//!
//! It does not decide liveness (#174). A boot id changes without the tmux
//! server dying whenever the state directory is restored, a VM or container
//! snapshot is rolled back, or the `boot_id` file is truncated — so recovery
//! always probes `tmux list-sessions` and the boot id only says whether a
//! server that did not answer is coming back.
use std::path::Path;
+16 -13
View File
@@ -847,20 +847,20 @@ impl SessionManager {
/// that no longer exists are still adopted — the spec does not require
/// us to repair them.
///
/// #164: when the kernel's boot id has changed since the last run, the
/// tmux probe below is skipped entirely — a reboot kills the tmux server,
/// so every previous session is already dead and there is nothing tmux
/// could truthfully adopt. [`Self::register_dormant`] then registers all
/// of them directly.
/// #164/#174: the kernel's boot id says whether the host rebooted since
/// the last run, but it never decides which sessions are alive —
/// `tmux list-sessions` does, always, which is what the rest of the
/// daemon assumes. A boot id can change without the tmux server dying
/// (a restored state directory, a VM or container snapshot restore, a
/// truncated `boot_id` file); skipping the probe then registered every
/// live session `Dormant`, and resume could never bring one back because
/// the tmux name was still taken. The boot id is now only a signal that a
/// tmux server which did not answer is not coming back, and it names
/// itself in recovery's log.
pub async fn recover(&self) -> Result<usize, SessionError> {
tokio::fs::create_dir_all(&self.inner.fifo_dir).await?;
let rebooted = self.check_boot_id().await;
let live = if rebooted {
// #164: every session from the previous boot is provably dead —
// no tmux server survives a reboot — so there is nothing to gain
// from asking it and no session it could honestly report back.
Vec::new()
} else {
let live = {
let tmux = self.inner.tmux.clone();
tmux_blocking(move || tmux.list_sessions()).await?
};
@@ -940,7 +940,9 @@ impl SessionManager {
///
/// A missing stored value (first run) or an unreadable current one
/// (non-Linux, restricted `/proc`, unwired in this build) both read as
/// "no reboot", which is today's behaviour: fall back to probing tmux.
/// "no reboot". Either way the tmux probe runs (#174): the answer only
/// tells recovery whether a tmux server that did not answer is coming
/// back, never which sessions are alive.
async fn check_boot_id(&self) -> bool {
let Some(cfg) = &self.inner.boot_id else {
return false;
@@ -987,7 +989,8 @@ impl SessionManager {
/// `rebooted` (#164) only changes what gets logged: a detected reboot
/// names the count even when it is zero, so the operator sees that
/// recovery told a reboot from a restart instead of a log that stays
/// silent either way.
/// silent either way. It never decides membership — everything here got
/// here by tmux not holding it (#174).
async fn register_dormant(&self, rebooted: bool) {
let stored = match self.inner.sidecar.load_all() {
Ok(stored) => stored,