Compare commits

..
Author SHA1 Message Date
Miguel PalhasandClaude Opus 5 5a47f6fbf6 feat(maestro): persist PTY scrollback across a reboot
A resumed session opened on a blank pane: the agent's conversation came
back, the terminal history the operator was reading did not. Each PTY
bridge's replay ring is now mirrored to `<state_dir>/scrollback/<id>.bin`
and handed to the new bridge on resume, and on the recovery pass that
re-adopts a session tmux still holds — that one lost its ring with the
previous daemon too. Restored bytes go out ahead of the live ones, and
are dropped once the ring holds a full budget of new output of its own.

Flushed every 10 seconds by the liveness watcher, and once more when a
bridge is sealed on exit. The timer is the part that matters: a host
reboot runs no exit path at all, so a scrollback written only on a clean
shutdown is missing in exactly the case this exists for. Ten seconds
bounds the loss to the last few lines — this is visual continuity, not
state — and costs at most one file write per active session per tick,
since a session whose bridge has read nothing since the last flush is
skipped.

The file is capped at the existing `pty_replay_bytes` budget rather than
a second one, is deleted on kill and when recovery prunes a sidecar, and
a missing, unreadable or oversized one degrades to an empty pane instead
of failing the resume.

Closes #171

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 09:37:53 +01:00
Miguel Palhas e7830d15b0 merge: replace agent_session_id on a new report (#176)
Closes #176
2026-09-04 09:29:49 +01:00
Miguel Palhas 87f1e73dec fix(maestro): replace agent_session_id when a report names a new one
A resume issues a new conversation id, so write-once left the stored id
pointing at the pre-resume conversation and every later resume reattached
it. Rewrite the sidecar when the reported id differs from the stored one;
a repeated report of the same id still writes nothing.

Closes #176
2026-09-04 09:26:16 +01:00
Miguel Palhas 0fd107be77 merge: dormant cards and resume in the web sidebar (#168)
Closes #168
2026-09-04 09:26:02 +01:00
Miguel Palhas aebd10c2da merge: guard resume against kill and second resume (#175)
Closes #175
2026-09-04 09:24:35 +01:00
Miguel Palhas ae3091e2a2 fix(maestro): guard resume against kill and second resume
The respawn runs with the session map unlocked, so two takes of the
lock straddle it. A kill landing in that window deleted the sidecar
before resume wrote it back, leaving a file for a session that was
gone — the next recovery registered it as a dormant ghost. Resume now
deletes the sidecar in the branch where it finds its own record gone.

Two resumes both passed the `Running` check for the same reason. The
record is marked in-flight under the first lock, so the second request
is refused with `already_running` instead of losing a race for the
FIFO and reporting a tmux failure. Every path out of resume clears the
marker.

Both races are driven deterministically in the tests: the manager's
tmux spawn is parked mid-resume while the racing kill or resume runs.

Closes #175
2026-09-04 09:19:59 +01:00
Miguel Palhas 1366c2d430 merge: always probe tmux during recovery (#174)
Closes #174
2026-09-04 09:15:45 +01:00
Miguel Palhas d050df7d3d fix(maestro): always probe tmux during recovery
#164 skipped the tmux probe whenever the stored boot id differed from the
current one, assuming a changed boot id proves the tmux server is gone. It
does not: a restored state directory, a VM or container snapshot rollback,
or a truncated boot_id file all change it while tmux keeps running with
live sessions. Every one of those sessions was then registered Dormant and
stranded — resume respawns `maestro-<id>`, which collides with the name
tmux still holds, so it failed 502 forever.

Recovery now always runs `tmux list-sessions`, the single source of truth
for liveness the rest of the daemon assumes. The boot id keeps its other
job: telling a reboot from a restart in recovery's log.

The #164 test asserted the old behaviour with tmux still alive, so it is
replaced by one asserting the session is adopted Running, plus one covering
a real reboot — boot id changed and the tmux server dead — still going
dormant.

Refs #174
2026-09-04 09:13:15 +01:00
Miguel Palhas e51a7007a6 merge: resume_on_boot policy (#167)
Closes #167
2026-09-04 09:10:28 +01:00
Miguel PalhasandClaude Opus 5 02bf077c03 feat(maestro): add [daemon].resume_on_boot policy
Recovery now has a stated policy for sessions whose tmux pane is gone but
whose sidecar survived: dormant (default), auto, or never. auto goes down
the same SessionManager::resume the client's POST takes, so there is one
resume path. never prunes the sidecars the way #163 does for a missing cwd.

The default is dormant because resuming agents unattended seconds after a
reboot is a foot-gun; auto is opted into per host. Recovery logs which
policy ran and how many sessions it acted on.

Refs #167

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 09:06:51 +01:00
16 changed files with 1533 additions and 53 deletions
+97
View File
@@ -129,6 +129,80 @@ pub struct DaemonSection {
/// `Some(0)` disables history entirely (no log written, no recents read).
#[serde(default)]
pub spawn_history_days: Option<u32>,
/// What recovery does with sessions that outlived their tmux server —
/// after a host reboot, or any restart that finds a sidecar with no pane
/// (#167). Default [`ResumeOnBoot::Dormant`].
#[serde(default, deserialize_with = "deserialize_resume_on_boot")]
pub resume_on_boot: Option<ResumeOnBoot>,
}
impl DaemonSection {
/// The configured `[daemon].resume_on_boot`, or the safe default:
/// recovered-but-dead sessions wait for the operator.
pub fn resume_on_boot(&self) -> ResumeOnBoot {
self.resume_on_boot.unwrap_or_default()
}
}
/// `[daemon].resume_on_boot` — what recovery does with a session whose tmux
/// pane is gone but whose sidecar is still on disk (#167).
///
/// Resuming agents unattended seconds after a reboot is a foot-gun: they run
/// with whatever flags they were spawned with, with nobody watching. So the
/// default waits for a human, and `auto` is opted into per host.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum ResumeOnBoot {
/// Register each one `Dormant` and wait for the operator to resume it.
#[default]
Dormant,
/// Resume each one during recovery, on the same path a client's
/// `POST /api/sessions/:id/resume` takes.
Auto,
/// Prune the sidecars instead: a reboot is a clean slate.
Never,
}
/// Every `[daemon].resume_on_boot` name and its variant, in documentation
/// order.
pub const RESUME_ON_BOOT_POLICIES: &[(&str, ResumeOnBoot)] = &[
("dormant", ResumeOnBoot::Dormant),
("auto", ResumeOnBoot::Auto),
("never", ResumeOnBoot::Never),
];
impl ResumeOnBoot {
/// The name this policy is spelled with in the config file and in logs.
pub fn as_str(&self) -> &'static str {
match self {
ResumeOnBoot::Dormant => "dormant",
ResumeOnBoot::Auto => "auto",
ResumeOnBoot::Never => "never",
}
}
}
fn deserialize_resume_on_boot<'de, D>(deserializer: D) -> Result<Option<ResumeOnBoot>, D::Error>
where
D: Deserializer<'de>,
{
let raw = Option::<String>::deserialize(deserializer)?;
raw.map(|name| {
RESUME_ON_BOOT_POLICIES
.iter()
.find(|(n, _)| *n == name)
.map(|(_, policy)| *policy)
.ok_or_else(|| {
let known = RESUME_ON_BOOT_POLICIES
.iter()
.map(|(n, _)| format!("{n:?}"))
.collect::<Vec<_>>()
.join(", ");
D::Error::custom(format!(
"invalid [daemon].resume_on_boot {name:?}; expected one of {known}"
))
})
})
.transpose()
}
/// `[tui]` — `maestro tui` settings (NAP-275 epic, phase 2 NAP-277).
@@ -775,4 +849,27 @@ bind_addr = "127.0.0.1:9090"
);
assert_eq!(cfg.resolve_daemon_url(None), "http://explicit:1");
}
#[test]
fn resume_on_boot_defaults_to_dormant_and_parses_every_policy() {
assert_eq!(
MaestroConfigFile::default().daemon.resume_on_boot(),
ResumeOnBoot::Dormant,
"an absent policy is the safe one"
);
for (name, expected) in RESUME_ON_BOOT_POLICIES {
let cfg = file_with(&format!("[daemon]\nresume_on_boot = \"{name}\"\n"));
assert_eq!(cfg.daemon.resume_on_boot(), *expected, "{name}");
assert_eq!(expected.as_str(), *name);
}
}
#[test]
fn resume_on_boot_rejects_an_unknown_policy() {
let err = toml::from_str::<MaestroConfigFile>("[daemon]\nresume_on_boot = \"maybe\"\n")
.expect_err("unknown policy must not silently fall back");
let msg = err.to_string();
assert!(msg.contains("resume_on_boot"), "{msg}");
assert!(msg.contains("\"auto\""), "{msg}");
}
}
+7 -4
View File
@@ -2,10 +2,13 @@
//!
//! `/proc/sys/kernel/random/boot_id` is fixed for the life of the kernel and
//! changes on every boot. [`SessionManager::recover`] compares it against the
//! value it stored on the previous run: a difference means every session from
//! before is provably dead, so the tmux probe can be skipped and those
//! sessions registered dormant directly instead of relying on tmux reporting
//! an empty list.
//! value it stored on the previous run and logs which of the two it was.
//!
//! It does not decide liveness (#174). A boot id changes without the tmux
//! server dying whenever the state directory is restored, a VM or container
//! snapshot is rolled back, or the `boot_id` file is truncated — so recovery
//! always probes `tmux list-sessions` and the boot id only says whether a
//! server that did not answer is coming back.
use std::path::Path;
+7 -1
View File
@@ -84,10 +84,16 @@ async fn run(config_override: Option<PathBuf>) -> Result<(), Box<dyn Error>> {
config.spawn_history_path(),
config.spawn_history_days,
))
// #171: mirror PTY scrollback so a resumed session opens on what it was
// showing, instead of a blank pane.
.with_scrollback(config.scrollback_dir())
.with_boot_id(
config.boot_id_path(),
std::sync::Arc::new(crate::boot_id::ProcBootId),
);
)
// #167: recovery needs the presets to resolve a resumed session's command,
// so `auto` runs the same resume a client's POST would.
.with_resume_policy(config.resume_on_boot, config.spawn_presets.clone());
// Spawned sessions learn where to report back (#103). Derived from the
// configured bind address; set before any spawn can happen.
manager.set_daemon_url(format!("http://{}", config.bind_addr));
+41
View File
@@ -97,6 +97,10 @@ pub struct DaemonConfig {
/// never binds it itself; it serves it so the web client and the TUI read
/// one knob.
pub leader: String,
/// What recovery does with sessions whose tmux pane is gone but whose
/// sidecar survives (#167): `[daemon].resume_on_boot`, default
/// [`maestro_config::ResumeOnBoot::Dormant`].
pub resume_on_boot: maestro_config::ResumeOnBoot,
}
impl DaemonConfig {
@@ -113,6 +117,12 @@ impl DaemonConfig {
self.state_dir.join("spawn-history.jsonl")
}
/// Where PTY scrollback is mirrored so a resumed session opens on what
/// it printed before (#171): `<state_dir>/scrollback`.
pub fn scrollback_dir(&self) -> PathBuf {
self.state_dir.join("scrollback")
}
/// Where recovery records the kernel boot id it saw last (#164):
/// `<state_dir>/boot_id`.
pub fn boot_id_path(&self) -> PathBuf {
@@ -149,6 +159,8 @@ pub fn load_config(path: &Path, env: EnvOverrides) -> Result<DaemonConfig, Confi
// validation (#22/#26).
let spawn_presets = file.spawn.resolve()?;
let resume_on_boot = daemon.resume_on_boot();
let hostname = daemon.hostname.unwrap_or_else(system_hostname);
let home_dir = dirs::home_dir()
.map(|p| p.to_string_lossy().into_owned())
@@ -209,6 +221,7 @@ pub fn load_config(path: &Path, env: EnvOverrides) -> Result<DaemonConfig, Confi
project_scan_depth,
hooks_auto_install: file.hooks.auto_install(),
leader: file.tui.leader().to_string(),
resume_on_boot,
})
}
@@ -289,6 +302,34 @@ auto_install = false
assert!(!cfg.hooks_auto_install);
}
#[test]
fn resume_on_boot_defaults_to_dormant_and_reads_the_file() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.toml");
std::fs::write(&path, "").unwrap();
assert_eq!(
load_config(&path, EnvOverrides::default())
.unwrap()
.resume_on_boot,
maestro_config::ResumeOnBoot::Dormant
);
std::fs::write(
&path,
r#"
[daemon]
resume_on_boot = "auto"
"#,
)
.unwrap();
assert_eq!(
load_config(&path, EnvOverrides::default())
.unwrap()
.resume_on_boot,
maestro_config::ResumeOnBoot::Auto
);
}
#[test]
fn empty_file_defaults_everything() {
let dir = tempfile::tempdir().unwrap();
+1
View File
@@ -26,6 +26,7 @@ pub mod projects;
pub mod pty;
pub mod release;
pub mod resume;
pub mod scrollback;
pub mod session;
pub mod sidecar;
pub mod spawn_history;
+143 -1
View File
@@ -21,6 +21,11 @@
//! one-shot prefix before joining the live broadcast, so attaching to an
//! idle session immediately shows the last screenful instead of a blank
//! terminal.
//! - #171: a bridge can be started with scrollback restored from disk — what
//! the session printed before the daemon (or the host) went away. Those
//! bytes are handed to subscribers ahead of the live ones until the ring
//! has a full budget of new output of its own, at which point the restored
//! history is older than anything the budget covers and is dropped.
use std::collections::VecDeque;
use std::ffi::CString;
@@ -58,6 +63,10 @@ const READ_BUF_BYTES: usize = 4096;
#[derive(Debug)]
pub struct PtyBridge {
fifo_path: PathBuf,
/// Scrollback from a previous life of this session (#171), replayed
/// ahead of the ring. Empty for a fresh spawn. Never appended to, so it
/// can never duplicate a byte the ring also holds.
restored: Vec<u8>,
/// `None` once [`Self::seal`] has run: live subscribers see the channel
/// close, but the replay buffer stays readable.
sender: Option<broadcast::Sender<Vec<u8>>>,
@@ -79,6 +88,21 @@ impl PtyBridge {
/// Variant that lets the caller pick a custom replay-buffer cap (bytes).
/// Pass `0` to disable replay (subscribers see only live bytes).
pub async fn start_with_replay(fifo_path: PathBuf, replay_bytes: usize) -> io::Result<Self> {
Self::start_restoring(fifo_path, replay_bytes, Vec::new()).await
}
/// As [`Self::start_with_replay`], seeded with `restored` scrollback read
/// back from disk (#171). The seed is truncated to the same
/// `replay_bytes` budget the ring uses — there is one budget here, not
/// two — and is dropped entirely when replay is disabled.
pub async fn start_restoring(
fifo_path: PathBuf,
replay_bytes: usize,
mut restored: Vec<u8>,
) -> io::Result<Self> {
if restored.len() > replay_bytes {
restored.drain(..restored.len() - replay_bytes);
}
if let Some(parent) = fifo_path.parent() {
tokio::fs::create_dir_all(parent).await?;
}
@@ -102,6 +126,7 @@ impl PtyBridge {
});
Ok(Self {
fifo_path,
restored,
sender: Some(sender),
replay,
last_activity,
@@ -155,7 +180,8 @@ impl PtyBridge {
F: FnOnce() -> Option<Vec<u8>>,
{
let guard = self.replay.lock_for_snapshot();
let snapshot = capture().unwrap_or_else(|| guard.snapshot());
let live = capture().unwrap_or_else(|| guard.snapshot());
let snapshot = guard.with_restored_prefix(&self.restored, live);
let rx = match &self.sender {
Some(tx) => tx.subscribe(),
// Sealed: hand back the scrollback plus a receiver that is already
@@ -171,6 +197,25 @@ impl PtyBridge {
(snapshot, rx)
}
/// Everything worth persisting for a later life of this session (#171):
/// the restored scrollback and the ring, trimmed together to the replay
/// budget so the file on disk is never bigger than the ring it mirrors.
pub fn scrollback(&self) -> Vec<u8> {
let guard = self.replay.lock_for_snapshot();
guard.with_restored_prefix(&self.restored, guard.snapshot())
}
/// Total bytes this bridge has appended to its ring. The scrollback flush
/// pass compares it against what it last wrote, so an idle session costs
/// no disk writes at all.
pub fn bytes_seen(&self) -> u64 {
self.replay
.inner
.lock()
.expect("ReplayBuffer poisoned")
.seen
}
/// Stop reading and close the live stream, keeping the replay buffer.
///
/// Attached clients see the channel close — without that a `maestro attach`
@@ -201,6 +246,8 @@ struct ReplayBuffer {
struct ReplayInner {
buf: VecDeque<u8>,
cap: usize,
/// Bytes ever appended, not bytes held — the ring evicts, this does not.
seen: u64,
}
impl ReplayBuffer {
@@ -209,6 +256,7 @@ impl ReplayBuffer {
inner: Arc::new(Mutex::new(ReplayInner {
buf: VecDeque::with_capacity(cap.min(READ_BUF_BYTES * 2)),
cap,
seen: 0,
})),
}
}
@@ -225,6 +273,7 @@ impl ReplayBuffer {
// If the incoming chunk alone exceeds the cap, take only its tail —
// anything in the buffer would be evicted before this finishes anyway.
let take_from = bytes.len().saturating_sub(s.cap);
s.seen += bytes.len() as u64;
let bytes = &bytes[take_from..];
s.buf.extend(bytes);
while s.buf.len() > s.cap {
@@ -247,6 +296,24 @@ struct ReplayGuard<'a> {
}
impl ReplayGuard<'_> {
/// `live` with the restored scrollback (#171) in front of it, while the
/// ring still holds less than a full budget of its own output. Once it
/// holds a full budget, everything restored is older than what the
/// budget covers, so it is dropped rather than kept forever.
fn with_restored_prefix(&self, restored: &[u8], live: Vec<u8>) -> Vec<u8> {
let cap = self.inner.cap;
let held = self.inner.buf.len();
if restored.is_empty() || cap == 0 || held >= cap {
return live;
}
let keep = cap - held;
let start = restored.len().saturating_sub(keep);
let mut out = Vec::with_capacity((restored.len() - start) + live.len());
out.extend_from_slice(&restored[start..]);
out.extend_from_slice(&live);
out
}
fn snapshot(&self) -> Vec<u8> {
let (a, b) = self.inner.buf.as_slices();
let mut out = Vec::with_capacity(a.len() + b.len());
@@ -292,6 +359,7 @@ async fn run_reader(
// never both, never neither.
let mut s = replay.inner.lock().expect("ReplayBuffer poisoned");
if s.cap > 0 {
s.seen += n as u64;
let take_from = n.saturating_sub(s.cap);
s.buf.extend(&buf[take_from..n]);
while s.buf.len() > s.cap {
@@ -345,6 +413,80 @@ fn shell_quote(s: &str) -> String {
mod tests {
use super::*;
async fn bridge(cap: usize, restored: &[u8]) -> (tempfile::TempDir, PtyBridge) {
let dir = tempfile::TempDir::new().expect("tempdir");
let bridge =
PtyBridge::start_restoring(dir.path().join("pty.fifo"), cap, restored.to_vec())
.await
.expect("start bridge");
(dir, bridge)
}
/// Feed the bridge through its own FIFO and wait for the reader to have
/// taken the bytes, so the ring state is deterministic.
async fn feed(bridge: &PtyBridge, bytes: &[u8]) {
use std::io::Write;
let before = bridge.bytes_seen();
let mut f = std::fs::OpenOptions::new()
.write(true)
.open(bridge.fifo_path())
.expect("open fifo for write");
f.write_all(bytes).expect("write fifo");
drop(f);
for _ in 0..400 {
if bridge.bytes_seen() >= before + bytes.len() as u64 {
return;
}
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
panic!("bridge never read the bytes written to its fifo");
}
#[tokio::test]
async fn restored_scrollback_is_replayed_ahead_of_live_bytes() {
let (_dir, bridge) = bridge(64, b"before the reboot\n").await;
feed(&bridge, b"after\n").await;
let (replay, _rx) = bridge.subscribe_with_replay();
assert_eq!(replay, b"before the reboot\nafter\n");
}
#[tokio::test]
async fn restored_scrollback_is_trimmed_to_the_replay_budget() {
let (_dir, bridge) = bridge(4, b"abcdefghij").await;
let (replay, _rx) = bridge.subscribe_with_replay();
assert_eq!(replay, b"ghij");
}
#[tokio::test]
async fn restored_scrollback_yields_once_the_ring_holds_a_full_budget() {
let (_dir, bridge) = bridge(4, b"old!").await;
feed(&bridge, b"ab").await;
let (replay, _rx) = bridge.subscribe_with_replay();
assert_eq!(replay, b"d!ab", "prefix plus ring stays inside the budget");
feed(&bridge, b"cd").await;
let (replay, _rx) = bridge.subscribe_with_replay();
assert_eq!(
replay, b"abcd",
"restored history is now older than the budget"
);
}
#[tokio::test]
async fn scrollback_carries_the_restored_prefix_and_the_ring() {
let (_dir, bridge) = bridge(8, b"old").await;
feed(&bridge, b"new").await;
assert_eq!(bridge.scrollback(), b"oldnew");
assert_eq!(bridge.bytes_seen(), 3);
}
#[tokio::test]
async fn no_replay_budget_means_no_restored_scrollback() {
let (_dir, bridge) = bridge(0, b"old").await;
let (replay, _rx) = bridge.subscribe_with_replay();
assert!(replay.is_empty());
assert!(bridge.scrollback().is_empty());
}
#[test]
fn replay_buffer_keeps_last_cap_bytes() {
let r = ReplayBuffer::new(4);
+148
View File
@@ -0,0 +1,148 @@
//! Persisted PTY scrollback (#171).
//!
//! The PTY replay ring lives in memory, so everything a session printed is
//! gone the moment the daemon process is. A resumed session then opens on a
//! blank pane: the agent's conversation is back, but the terminal history the
//! operator was reading is not.
//!
//! This store mirrors each bridge's ring to `<state_dir>/scrollback/<id>.bin`,
//! and hands it back when a session gets a new bridge — on resume, and on the
//! recovery pass that re-adopts a session tmux still holds. The file is
//! capped at the same `replay_bytes` budget the in-memory ring uses, so disk
//! cost is bounded by the same number the operator already configures.
//!
//! Nothing here is allowed to fail a resume. The payload is raw PTY bytes
//! with no framing to be corrupt, and a file that is missing, unreadable or
//! oversized degrades to an empty (or truncated) pane rather than an error.
use std::io;
use std::path::{Path, PathBuf};
use uuid::Uuid;
/// Filesystem-backed scrollback store, rooted at a directory.
#[derive(Debug, Clone)]
pub struct ScrollbackStore {
dir: PathBuf,
}
impl ScrollbackStore {
pub fn new(dir: impl Into<PathBuf>) -> Self {
Self { dir: dir.into() }
}
pub fn dir(&self) -> &Path {
&self.dir
}
pub fn path_for(&self, id: Uuid) -> PathBuf {
self.dir.join(format!("{id}.bin"))
}
/// Atomically write `bytes` to `<dir>/<id>.bin`, creating the directory if
/// needed. Written through a temp file and renamed, so a reboot in the
/// middle of a flush leaves the previous scrollback intact rather than a
/// half-file.
pub fn write(&self, id: Uuid, bytes: &[u8]) -> io::Result<()> {
std::fs::create_dir_all(&self.dir)?;
let tmp = self.dir.join(format!("{id}.bin.tmp"));
std::fs::write(&tmp, bytes)?;
std::fs::rename(&tmp, self.path_for(id))
}
/// The stored scrollback for `id`, capped to the last `budget` bytes.
///
/// Never fails: a missing file, an unreadable one, or a `budget` of zero
/// all read as "no scrollback", which is exactly the blank pane the
/// operator had before this existed.
pub fn read(&self, id: Uuid, budget: usize) -> Vec<u8> {
if budget == 0 {
return Vec::new();
}
let path = self.path_for(id);
let mut bytes = match std::fs::read(&path) {
Ok(b) => b,
Err(e) if e.kind() == io::ErrorKind::NotFound => return Vec::new(),
Err(e) => {
tracing::warn!(error = ?e, %id, path = %path.display(), "scrollback: unreadable, resuming with a blank pane");
return Vec::new();
}
};
// A file written under a larger budget than this daemon now runs
// with keeps its tail, the same end of the stream the ring would.
if bytes.len() > budget {
bytes.drain(..bytes.len() - budget);
}
bytes
}
/// Delete the scrollback for `id`. No-op if it is not there.
pub fn delete(&self, id: Uuid) -> io::Result<()> {
match std::fs::remove_file(self.path_for(id)) {
Ok(()) => Ok(()),
Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()),
Err(e) => Err(e),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn store() -> (tempfile::TempDir, ScrollbackStore) {
let dir = tempfile::TempDir::new().expect("tempdir");
let store = ScrollbackStore::new(dir.path().join("scrollback"));
(dir, store)
}
#[test]
fn round_trips_bytes() {
let (_dir, store) = store();
let id = Uuid::new_v4();
store.write(id, b"hello world").expect("write");
assert_eq!(store.read(id, 64), b"hello world");
}
#[test]
fn a_missing_file_reads_as_empty() {
let (_dir, store) = store();
assert!(store.read(Uuid::new_v4(), 64).is_empty());
}
#[test]
fn a_file_over_budget_keeps_its_tail() {
let (_dir, store) = store();
let id = Uuid::new_v4();
store.write(id, b"abcdefghij").expect("write");
assert_eq!(store.read(id, 4), b"ghij");
}
#[test]
fn a_zero_budget_reads_as_empty() {
let (_dir, store) = store();
let id = Uuid::new_v4();
store.write(id, b"abcdefghij").expect("write");
assert!(store.read(id, 0).is_empty());
}
#[test]
fn delete_removes_the_file_and_tolerates_a_missing_one() {
let (_dir, store) = store();
let id = Uuid::new_v4();
store.write(id, b"gone soon").expect("write");
store.delete(id).expect("delete");
assert!(!store.path_for(id).exists());
store.delete(id).expect("second delete is a no-op");
}
#[test]
fn a_second_write_replaces_the_first() {
let (_dir, store) = store();
let id = Uuid::new_v4();
store.write(id, b"first").expect("write");
store.write(id, b"second").expect("rewrite");
assert_eq!(store.read(id, 64), b"second");
assert!(!store.dir().join(format!("{id}.bin.tmp")).exists());
}
}
+289 -29
View File
@@ -12,6 +12,7 @@ use std::sync::Arc;
use std::time::Duration;
use chrono::Utc;
use maestro_config::ResumeOnBoot;
use maestro_protocol::{
ActivityState, CatalogEvent, CreateSessionRequest, ErrorInfo, NameSource, Session,
SessionStatus, WorktreeSpec, NAME_KEY, NAME_SOURCE_KEY,
@@ -22,6 +23,7 @@ use uuid::Uuid;
use crate::boot_id::BootIdSource;
use crate::panes::Panes;
use crate::pty::{pipe_pane_command, PtyBridge};
use crate::scrollback::ScrollbackStore;
use crate::sidecar::{SidecarError, SidecarStore};
use crate::spawn_history::{RecentDir, SpawnHistory, SpawnHistoryRecord};
use crate::tmux::TmuxError;
@@ -43,6 +45,16 @@ pub const DEFAULT_LIVENESS_INTERVAL: Duration = Duration::from_secs(2);
/// attempt, which is all #83 needs.
pub const DEFAULT_BLOCKED_WORK_INTERVAL: Duration = Duration::from_secs(15);
/// How often the liveness watcher mirrors PTY scrollback to disk (#171).
///
/// A host reboot gives no warning and runs no exit path, so this interval is
/// the whole guarantee: whatever a session printed in the last few seconds
/// before the power went is what a resumed pane is missing. Ten seconds keeps
/// that loss cosmetic — scrollback is visual continuity, not state — while
/// costing at most one file write per *active* session per tick, since a
/// session whose bridge read nothing is skipped entirely.
pub const DEFAULT_SCROLLBACK_INTERVAL: Duration = Duration::from_secs(10);
/// Gap between the text of a `send` and the Enter that submits it.
///
/// Agents that run a full-screen prompt (Claude Code, opencode) coalesce a
@@ -86,11 +98,20 @@ pub struct SessionRecord {
/// (`POST /.../focus`), `None` while unfocused. Daemon-local; never rides
/// the wire.
pub focused_since: Option<std::time::Instant>,
/// Set for as long as a [`SessionManager::resume`] is between its two
/// takes of the map lock (#175). The spawn happens with the lock released,
/// so the `Running` check alone lets a second request through; this is the
/// state the second one is refused on. Daemon-local; never rides the wire.
pub resuming: bool,
/// Sticky acknowledgement applied by [`crate::activity::apply_focus_suppression`]:
/// once the pane has been focused long enough to clear an `AwaitingInput`
/// glow, this holds through focus loss until the session does new work.
/// Daemon-local; never rides the wire.
pub awaiting_suppressed: bool,
/// [`PtyBridge::bytes_seen`] as of this session's last scrollback flush
/// (#171). The flush pass skips a session whose bridge has read nothing
/// since, so an idle session costs no disk writes.
pub flushed_bytes: u64,
}
#[derive(Debug, thiserror::Error)]
@@ -195,6 +216,15 @@ struct Inner {
/// reboot detection is unwired, which behaves exactly like a boot id
/// that could never be read: always fall back to probing tmux.
boot_id: Option<BootIdConfig>,
/// What recovery does with a session whose pane is gone (#167), and the
/// presets an `auto` resume resolves its command from. Defaults to
/// [`ResumeOnBoot::Dormant`] with no presets: register and wait.
resume_policy: ResumeOnBoot,
resume_presets: Vec<maestro_protocol::SpawnPreset>,
/// Where PTY scrollback is mirrored so it survives the daemon, and the
/// host (#171). `None` — the default — means scrollback is not persisted
/// at all: a resumed session opens on a blank pane, exactly as before.
scrollback: Option<ScrollbackStore>,
}
/// Where recovery persists the boot id it last saw, and how it reads the
@@ -253,6 +283,9 @@ impl SessionManager {
viewports: Mutex::new(Viewports::new()),
next_viewport: AtomicU64::new(0),
boot_id: None,
resume_policy: ResumeOnBoot::default(),
resume_presets: Vec::new(),
scrollback: None,
}),
}
}
@@ -290,6 +323,21 @@ impl SessionManager {
/// # Panics
///
/// If the handle has already been cloned.
/// Persist PTY scrollback under `dir` (#171), so a resumed session comes
/// back showing what it printed before the daemon — or the host — went
/// away. Builder, called once right after construction like the other
/// `with_*` methods. Without it nothing is written, read or deleted.
///
/// # Panics
///
/// If the handle has already been cloned; see [`Self::with_spawn_history`].
pub fn with_scrollback(mut self, dir: PathBuf) -> Self {
let inner = Arc::get_mut(&mut self.inner)
.expect("with_scrollback must be called before the SessionManager is cloned");
inner.scrollback = Some(ScrollbackStore::new(dir));
self
}
pub fn with_boot_id(mut self, path: PathBuf, source: Arc<dyn BootIdSource>) -> Self {
let inner = Arc::get_mut(&mut self.inner)
.expect("with_boot_id must be called before the SessionManager is cloned");
@@ -297,6 +345,26 @@ impl SessionManager {
self
}
/// Wire the `[daemon].resume_on_boot` policy (#167) and the spawn presets
/// an `auto` resume resolves each session's command from. Builder, called
/// once right after construction like the other `with_*` methods. Without
/// it, recovery behaves as [`ResumeOnBoot::Dormant`].
///
/// # Panics
///
/// If the handle has already been cloned.
pub fn with_resume_policy(
mut self,
policy: ResumeOnBoot,
presets: Vec<maestro_protocol::SpawnPreset>,
) -> Self {
let inner = Arc::get_mut(&mut self.inner)
.expect("with_resume_policy must be called before the SessionManager is cloned");
inner.resume_policy = policy;
inner.resume_presets = presets;
self
}
/// Set the daemon's own HTTP base URL (#103), exported to every session
/// spawned afterwards as `MAESTRO_URL`. Unlike the `with_*` builders this
/// is a plain setter: the daemon runner learns its bound address after
@@ -567,6 +635,8 @@ impl SessionManager {
naming: false,
focused_since: None,
awaiting_suppressed: false,
resuming: false,
flushed_bytes: 0,
},
);
}
@@ -618,15 +688,23 @@ impl SessionManager {
id: Uuid,
presets: &[maestro_protocol::SpawnPreset],
) -> Result<Session, SessionError> {
// The spawn below runs with the map lock released, so the `Running`
// check alone is not enough to keep two resumes apart (#175): both
// would pass it, both would open a `PtyBridge` on the same FIFO, and
// the loser would report a tmux failure instead of the conflict it
// is. The record is marked in-flight under this first lock, and every
// path out of here clears it again.
let mut session = {
let sessions = self.inner.sessions.lock().await;
let rec = sessions.get(&id).ok_or(SessionError::NotFound(id))?;
if rec.session.status == SessionStatus::Running {
let mut sessions = self.inner.sessions.lock().await;
let rec = sessions.get_mut(&id).ok_or(SessionError::NotFound(id))?;
if rec.session.status == SessionStatus::Running || rec.resuming {
return Err(SessionError::AlreadyRunning(id));
}
rec.resuming = true;
rec.session.clone()
};
if !session.cwd.exists() {
self.clear_resuming(id).await;
return Err(SessionError::CwdMissing(session.cwd.clone()));
}
let command = crate::resume::resume_command(&session, presets);
@@ -634,8 +712,14 @@ impl SessionManager {
// Same fifo path as the original spawn: it is derived from the id, and
// `mkfifo` tolerates the stale node the previous run left behind.
let fifo_path = self.inner.fifo_dir.join(format!("{id}.fifo"));
let bridge =
PtyBridge::start_with_replay(fifo_path.clone(), self.inner.replay_bytes).await?;
// #171: the pane is new, but the scrollback the operator was reading
// before the reboot is on disk. It goes in ahead of the live bytes.
let bridge = PtyBridge::start_restoring(
fifo_path.clone(),
self.inner.replay_bytes,
self.restored_scrollback(id),
)
.await?;
let mut session_env = sanitize_env(&session.env)?;
session_env.insert("MAESTRO_SESSION_ID".into(), id.to_string());
if let Some(url) = self
@@ -658,6 +742,7 @@ impl SessionManager {
if let Err(e) = spawned {
// `bridge` drops here, closing the FIFO reader. Nothing else was
// touched, so the session stays exactly as dormant as it was.
self.clear_resuming(id).await;
return Err(SessionError::Tmux(e));
}
@@ -677,6 +762,7 @@ impl SessionManager {
// sidecar does not describe.
let tmux = self.inner.tmux.clone();
let _ = tmux_blocking(move || tmux.kill_session(id)).await;
self.clear_resuming(id).await;
return Err(SessionError::Sidecar(e));
}
@@ -684,9 +770,17 @@ impl SessionManager {
let mut sessions = self.inner.sessions.lock().await;
let Some(rec) = sessions.get_mut(&id) else {
// Killed while we were spawning. Undo the pane instead of
// leaving one behind with no record of it.
// leaving one behind with no record of it — and delete the
// sidecar the write above just put back (#175). The kill
// deleted it before that write landed, so leaving it would
// have the next recovery register a dormant ghost for a
// session that is gone and whose worktree is already removed.
drop(sessions);
let tmux = self.inner.tmux.clone();
let _ = tmux_blocking(move || tmux.kill_session(id)).await;
if let Err(e) = self.inner.sidecar.delete(id) {
tracing::warn!(error = ?e, %id, "resume: failed to delete the sidecar of a session killed mid-resume");
}
return Err(SessionError::NotFound(id));
};
rec.session = session.clone();
@@ -696,6 +790,7 @@ impl SessionManager {
rec.settled = false;
rec.focused_since = None;
rec.awaiting_suppressed = false;
rec.resuming = false;
}
let _ = self.inner.events.send(CatalogEvent::SessionUpdated {
session: session.clone(),
@@ -703,6 +798,53 @@ impl SessionManager {
Ok(session)
}
/// Delete the scrollback of a sidecar recovery threw away (#171): the
/// session will never be listed again, so nothing would ever read it.
fn prune_scrollback(&self, id: Uuid) {
if let Some(store) = &self.inner.scrollback {
if let Err(e) = store.delete(id) {
tracing::warn!(error = ?e, %id, "recover: failed to prune scrollback");
}
}
}
/// Scrollback persisted for `id` by an earlier life of this session
/// (#171), capped to the replay budget. Empty when scrollback is not
/// wired, and empty — never an error — when the file is missing or
/// unreadable: a blank pane is the honest fallback, and it is what the
/// operator had before any of this existed.
fn restored_scrollback(&self, id: Uuid) -> Vec<u8> {
self.inner
.scrollback
.as_ref()
.map(|store| store.read(id, self.inner.replay_bytes))
.unwrap_or_default()
}
/// Mirror every live bridge's scrollback to disk (#171).
///
/// Called on a timer by the liveness watcher rather than only on a clean
/// exit, because the case this exists for — a host reboot — runs no exit
/// path at all: anything flushed only on shutdown is missing exactly when
/// it is needed. Sessions whose bridge has read nothing since the last
/// flush are skipped, so an idle host writes nothing.
///
/// Best-effort throughout: a failed write is warned and the session keeps
/// running.
pub async fn flush_scrollback(&self) {
run_scrollback_pass(&self.inner).await;
}
/// Drop the in-flight marker [`Self::resume`] set under its first lock
/// (#175), so a refused or failed resume can be retried. A no-op when the
/// record is already gone: a kill that landed mid-resume took the marker
/// with it.
async fn clear_resuming(&self, id: Uuid) {
if let Some(rec) = self.inner.sessions.lock().await.get_mut(&id) {
rec.resuming = false;
}
}
/// Best-effort removal of a worktree created earlier in `create` when a
/// later step fails. No-op when the request had no worktree.
fn rollback_worktree(&self, base: &Option<PathBuf>, path: &Option<PathBuf>) {
@@ -793,6 +935,10 @@ impl SessionManager {
// Start "due" so the first live tick probes: a recovered daemon
// should not wait a full interval to show what blocks.
let mut since_probe = blocked_work_interval;
// Scrollback starts *not* due: a daemon that just recovered has
// nothing new to mirror, and the sessions it adopted were primed
// from the files this would overwrite (#171).
let mut since_flush = Duration::ZERO;
loop {
ticker.tick().await;
let Some(inner) = weak.upgrade() else {
@@ -807,6 +953,11 @@ impl SessionManager {
since_probe = Duration::ZERO;
run_blocked_work_pass(&inner).await;
}
since_flush += interval;
if since_flush >= DEFAULT_SCROLLBACK_INTERVAL {
since_flush = Duration::ZERO;
run_scrollback_pass(&inner).await;
}
}
});
}
@@ -819,20 +970,20 @@ impl SessionManager {
/// that no longer exists are still adopted — the spec does not require
/// us to repair them.
///
/// #164: when the kernel's boot id has changed since the last run, the
/// tmux probe below is skipped entirely — a reboot kills the tmux server,
/// so every previous session is already dead and there is nothing tmux
/// could truthfully adopt. [`Self::register_dormant`] then registers all
/// of them directly.
/// #164/#174: the kernel's boot id says whether the host rebooted since
/// the last run, but it never decides which sessions are alive —
/// `tmux list-sessions` does, always, which is what the rest of the
/// daemon assumes. A boot id can change without the tmux server dying
/// (a restored state directory, a VM or container snapshot restore, a
/// truncated `boot_id` file); skipping the probe then registered every
/// live session `Dormant`, and resume could never bring one back because
/// the tmux name was still taken. The boot id is now only a signal that a
/// tmux server which did not answer is not coming back, and it names
/// itself in recovery's log.
pub async fn recover(&self) -> Result<usize, SessionError> {
tokio::fs::create_dir_all(&self.inner.fifo_dir).await?;
let rebooted = self.check_boot_id().await;
let live = if rebooted {
// #164: every session from the previous boot is provably dead —
// no tmux server survives a reboot — so there is nothing to gain
// from asking it and no session it could honestly report back.
Vec::new()
} else {
let live = {
let tmux = self.inner.tmux.clone();
tmux_blocking(move || tmux.list_sessions()).await?
};
@@ -866,8 +1017,15 @@ impl SessionManager {
},
};
let fifo_path = self.inner.fifo_dir.join(format!("{id}.fifo"));
let bridge =
PtyBridge::start_with_replay(fifo_path.clone(), self.inner.replay_bytes).await?;
// The ring the previous daemon held died with it, so an adopted
// session would otherwise replay nothing until tmux's own screen
// capture — which is one screenful, not scrollback (#171).
let bridge = PtyBridge::start_restoring(
fifo_path.clone(),
self.inner.replay_bytes,
self.restored_scrollback(id),
)
.await?;
// Re-pipe so output starts flowing again. `Tmux::pipe_pane` runs
// without tmux's `-o` flag, so any stale pipe the previous daemon
// left running to a now-unread FIFO is replaced (NAP-228).
@@ -894,6 +1052,8 @@ impl SessionManager {
naming: false,
focused_since: None,
awaiting_suppressed: false,
resuming: false,
flushed_bytes: 0,
},
);
let _ = self
@@ -912,7 +1072,9 @@ impl SessionManager {
///
/// A missing stored value (first run) or an unreadable current one
/// (non-Linux, restricted `/proc`, unwired in this build) both read as
/// "no reboot", which is today's behaviour: fall back to probing tmux.
/// "no reboot". Either way the tmux probe runs (#174): the answer only
/// tells recovery whether a tmux server that did not answer is coming
/// back, never which sessions are alive.
async fn check_boot_id(&self) -> bool {
let Some(cfg) = &self.inner.boot_id else {
return false;
@@ -959,7 +1121,8 @@ impl SessionManager {
/// `rebooted` (#164) only changes what gets logged: a detected reboot
/// names the count even when it is zero, so the operator sees that
/// recovery told a reboot from a restart instead of a log that stays
/// silent either way.
/// silent either way. It never decides membership — everything here got
/// here by tmux not holding it (#174).
async fn register_dormant(&self, rebooted: bool) {
let stored = match self.inner.sidecar.load_all() {
Ok(stored) => stored,
@@ -970,6 +1133,7 @@ impl SessionManager {
};
let mut registered = Vec::new();
let mut pruned = 0usize;
let mut discarded = 0usize;
{
let mut sessions = self.inner.sessions.lock().await;
for mut session in stored {
@@ -981,9 +1145,21 @@ impl SessionManager {
if let Err(e) = self.inner.sidecar.delete(session.id) {
tracing::warn!(error = ?e, id = %session.id, "recover: failed to prune sidecar with missing cwd");
}
self.prune_scrollback(session.id);
pruned += 1;
continue;
}
if self.inner.resume_policy == ResumeOnBoot::Never {
// #167: this host wants a reboot to be a clean slate, so
// the sidecar goes the way #163's does — nothing is
// registered and the next recovery finds nothing.
if let Err(e) = self.inner.sidecar.delete(session.id) {
tracing::warn!(error = ?e, id = %session.id, "recover: failed to prune sidecar under resume_on_boot=never");
}
self.prune_scrollback(session.id);
discarded += 1;
continue;
}
let id = session.id;
session.status = SessionStatus::Dormant;
session.foreground = None;
@@ -1006,6 +1182,8 @@ impl SessionManager {
naming: false,
focused_since: None,
awaiting_suppressed: false,
resuming: false,
flushed_bytes: 0,
},
);
registered.push(session);
@@ -1025,12 +1203,45 @@ impl SessionManager {
if pruned > 0 {
tracing::info!(pruned, "pruned sidecars whose cwd no longer exists");
}
let resumable: Vec<Uuid> = registered.iter().map(|s| s.id).collect();
for session in registered {
let _ = self
.inner
.events
.send(CatalogEvent::SessionAdded { session });
}
// #167: say which policy ran and how many sessions it acted on, so a
// host that resumes agents unattended says so in its own log. The
// match is exhaustive on purpose: a fourth policy must decide what it
// does here rather than inherit whichever arm happened to be last.
match self.inner.resume_policy {
ResumeOnBoot::Dormant => tracing::info!(
policy = ResumeOnBoot::Dormant.as_str(),
sessions = resumable.len(),
"resume_on_boot: sessions left dormant for the operator"
),
ResumeOnBoot::Never => tracing::info!(
policy = ResumeOnBoot::Never.as_str(),
sessions = discarded,
"resume_on_boot: pruned the sidecars of sessions with no live pane"
),
ResumeOnBoot::Auto => {
tracing::info!(
policy = ResumeOnBoot::Auto.as_str(),
sessions = resumable.len(),
"resume_on_boot: resuming sessions with no live pane"
);
let presets = self.inner.resume_presets.clone();
for id in resumable {
// The same call the client's POST /api/sessions/:id/resume
// makes — one resume path, not two.
if let Err(e) = self.resume(id, &presets).await {
tracing::warn!(error = %e, id = %id, "resume_on_boot=auto: resume failed; session stays dormant");
}
}
}
}
}
/// Forward keystrokes to a session's pane. A dead or dormant session is
@@ -1193,10 +1404,12 @@ impl SessionManager {
/// and stays silent on the event stream at steady state.
///
/// `agent_session_id` (#161) is the agent's own conversation id, parsed
/// by the caller from the hook payload's `session_id` field. It is
/// written once — the first report that carries one wins — and the
/// sidecar is only rewritten on that transition, never on every report,
/// since an agent's session id never changes mid-session.
/// by the caller from the hook payload's `session_id` field. The
/// sidecar is rewritten only when the reported id differs from the
/// stored one — a repeated report of the same id is pure churn from the
/// hook's `PreToolUse` cadence and writes nothing, but a resume (#176)
/// issues a new conversation id and must replace the stale one, or
/// every later resume reattaches the pre-resume conversation.
pub async fn report_activity(
&self,
id: Uuid,
@@ -1218,16 +1431,16 @@ impl SessionManager {
state,
self.inner.report_ttl,
));
let newly_identified =
rec.session.agent_session_id.is_none() && agent_session_id.is_some();
if newly_identified {
let id_changed =
agent_session_id.is_some() && rec.session.agent_session_id != agent_session_id;
if id_changed {
rec.session.agent_session_id = agent_session_id;
self.inner
.sidecar
.write(&rec.session)
.map_err(SessionError::Sidecar)?;
}
(changed || newly_identified).then(|| rec.session.clone())
(changed || id_changed).then(|| rec.session.clone())
};
if let Some(session) = updated {
let _ = self
@@ -1610,6 +1823,36 @@ async fn run_blocked_work_pass(inner: &Arc<Inner>) {
}
}
/// Mirror every live bridge's scrollback to disk. See
/// [`SessionManager::flush_scrollback`].
async fn run_scrollback_pass(inner: &Arc<Inner>) {
let Some(store) = inner.scrollback.clone() else {
return;
};
// Snapshot under the lock, write with it released: the writes are
// filesystem work and every attach needs this map.
let pending: Vec<(Uuid, Vec<u8>)> = {
let mut sessions = inner.sessions.lock().await;
sessions
.iter_mut()
.filter_map(|(id, rec)| {
let bridge = rec.bridge.as_ref()?;
let seen = bridge.bytes_seen();
if seen == rec.flushed_bytes {
return None;
}
rec.flushed_bytes = seen;
Some((*id, bridge.scrollback()))
})
.collect()
};
for (id, bytes) in pending {
if let Err(e) = store.write(id, &bytes) {
tracing::warn!(error = ?e, %id, "scrollback: flush failed");
}
}
}
/// Mark a session whose tmux session has disappeared as [`SessionStatus::Dead`]
/// and settle its worktree per the natural-exit rule (#82).
///
@@ -1636,6 +1879,15 @@ async fn mark_session_dead(inner: &Arc<Inner>, id: Uuid) {
// attach.
if let Some(bridge) = record.bridge.as_mut() {
bridge.seal();
// The clean-exit half of #171: the last thing the session
// printed is what a later resume should open on, and it
// will produce nothing more for the flush pass to notice.
if let Some(store) = &inner.scrollback {
if let Err(e) = store.write(id, &bridge.scrollback()) {
tracing::warn!(error = ?e, %id, "scrollback: final flush failed");
}
record.flushed_bytes = bridge.bytes_seen();
}
}
let session = record.session.clone();
let worktree = record
@@ -1757,6 +2009,14 @@ async fn cleanup_record(
if let Err(e) = inner.sidecar.delete(id) {
tracing::warn!(error = ?e, %id, "cleanup: failed to delete sidecar");
}
// #171: the session is gone, so its scrollback goes with it — nothing
// will ever read the file again, and leaving it would keep a killed
// session's output on disk indefinitely.
if let Some(store) = &inner.scrollback {
if let Err(e) = store.delete(id) {
tracing::warn!(error = ?e, %id, "cleanup: failed to delete scrollback");
}
}
if let (Some(base), Some(path)) = (&record.worktree_base_repo, &record.worktree_path) {
if keep_worktree {
tracing::info!(
+31 -4
View File
@@ -187,10 +187,30 @@ async fn a_reported_session_id_is_recorded_on_the_session() {
assert_eq!(s.agent_session_id.as_deref(), Some("conv-abc123"));
}
/// #161: a session id does not change mid-session, so once set it is never
/// overwritten by a later report — even one that names a different id.
/// #161: repeated reports of the same id are pure churn from the hook's
/// `PreToolUse` cadence — the sidecar must not be rewritten for them.
#[tokio::test]
async fn a_reported_session_id_is_written_once_and_never_replaced() {
async fn a_repeated_report_of_the_same_session_id_writes_nothing() {
let fx = fixture(Duration::from_secs(300)).await;
let id = spawn_sleeping(&fx.manager).await;
fx.manager
.report_activity(id, ActivityState::Active, None, Some("conv-first".into()))
.await
.unwrap();
fx.manager
.report_activity(id, ActivityState::Active, None, Some("conv-first".into()))
.await
.unwrap();
let s = observed(&fx.manager, id).await;
assert_eq!(s.agent_session_id.as_deref(), Some("conv-first"));
}
/// #176: a resume issues a new conversation id. The stored id must move to
/// it, or every later resume reattaches the stale pre-resume conversation.
#[tokio::test]
async fn a_reported_session_id_is_replaced_when_it_differs() {
let fx = fixture(Duration::from_secs(300)).await;
let id = spawn_sleeping(&fx.manager).await;
@@ -204,7 +224,14 @@ async fn a_reported_session_id_is_written_once_and_never_replaced() {
.unwrap();
let s = observed(&fx.manager, id).await;
assert_eq!(s.agent_session_id.as_deref(), Some("conv-first"));
assert_eq!(s.agent_session_id.as_deref(), Some("conv-second"));
let stored = fx
.sidecar
.read(id)
.expect("sidecar read")
.expect("sidecar present");
assert_eq!(stored.agent_session_id.as_deref(), Some("conv-second"));
}
#[tokio::test]
+21 -4
View File
@@ -151,27 +151,40 @@ pub struct DaemonFixture {
impl DaemonFixture {
pub fn new() -> Self {
Self::build(None)
Self::build(None, maestro_daemon::pty::DEFAULT_REPLAY_BYTES)
}
/// Variant whose manager runs a custom PTY replay budget (NAP-229), which
/// also caps the scrollback it persists (#171).
pub fn new_with_replay(replay_bytes: usize) -> Self {
Self::build(None, replay_bytes)
}
/// Variant whose manager has a durable spawn history (#24) attached, with
/// `retention_days` recency cap, logging to `<state_dir>/spawn-history.jsonl`.
pub fn new_with_history(retention_days: u32) -> Self {
Self::build(Some(retention_days))
Self::build(
Some(retention_days),
maestro_daemon::pty::DEFAULT_REPLAY_BYTES,
)
}
fn build(history_days: Option<u32>) -> Self {
fn build(history_days: Option<u32>, replay_bytes: usize) -> Self {
let tmux = TmuxFixture::new();
let state_dir = TempDir::new().unwrap();
let sidecar = SidecarStore::new(state_dir.path().join("sessions"));
// Default placement: worktrees land under their base repo (#56).
let worktree_mgr = WorktreeManager::new(None);
let mut manager = SessionManager::new(
let mut manager = SessionManager::new_with_replay(
Arc::new(tmux.tmux.clone()),
sidecar,
worktree_mgr,
state_dir.path().join("fifos"),
replay_bytes,
);
// #171: production wires this, so the fixtures do too — every test
// then exercises the same flush/restore/delete paths the daemon runs.
manager = manager.with_scrollback(state_dir.path().join("scrollback"));
if let Some(days) = history_days {
manager =
manager.with_spawn_history(SpawnHistory::new(Self::history_path(&state_dir), days));
@@ -198,4 +211,8 @@ impl DaemonFixture {
pub fn fifo_dir(&self) -> PathBuf {
self.state_dir.path().join("fifos")
}
pub fn scrollback_dir(&self) -> PathBuf {
self.state_dir.path().join("scrollback")
}
}
+234
View File
@@ -357,3 +357,237 @@ async fn post_resume_on_an_unknown_session_is_404() {
.unwrap();
assert_eq!(resp.status(), reqwest::StatusCode::NOT_FOUND);
}
// ---------------------------------------------------------------------------
// Races (#175)
// ---------------------------------------------------------------------------
/// Real tmux, with the *first* `new_session_piped` held inside the call until
/// the test releases it. Later calls pass straight through, so a second resume
/// racing the first never blocks on a gate the first one owns.
///
/// This is what makes the two races below deterministic rather than timing
/// dependent: the resume under test is parked in exactly the window the bug
/// lives in, and the racing operation runs to completion before it moves.
#[derive(Debug)]
struct GatedSpawn {
inner: maestro_daemon::tmux::Tmux,
entered: tokio::sync::mpsc::UnboundedSender<()>,
release: std::sync::Mutex<std::sync::mpsc::Receiver<()>>,
gated: std::sync::atomic::AtomicBool,
}
/// The test's end of a [`GatedSpawn`].
struct Gate {
entered: tokio::sync::mpsc::UnboundedReceiver<()>,
release: std::sync::mpsc::SyncSender<()>,
}
impl Gate {
/// Wait until the gated spawn is parked mid-resume.
async fn wait_entered(&mut self) {
tokio::time::timeout(Duration::from_secs(10), self.entered.recv())
.await
.expect("resume reached the tmux spawn")
.expect("gate alive");
}
fn release(&self) {
self.release.send(()).expect("gate receiver alive");
}
}
fn gated_panes(inner: maestro_daemon::tmux::Tmux) -> (Arc<GatedSpawn>, Gate) {
let (entered_tx, entered_rx) = tokio::sync::mpsc::unbounded_channel();
let (release_tx, release_rx) = std::sync::mpsc::sync_channel(1);
(
Arc::new(GatedSpawn {
inner,
entered: entered_tx,
release: std::sync::Mutex::new(release_rx),
gated: std::sync::atomic::AtomicBool::new(true),
}),
Gate {
entered: entered_rx,
release: release_tx,
},
)
}
impl maestro_daemon::panes::Panes for GatedSpawn {
fn new_session_piped(
&self,
id: Uuid,
cwd: &std::path::Path,
command: &str,
session_env: &std::collections::BTreeMap<String, String>,
sink_command: &str,
) -> Result<(), maestro_daemon::tmux::TmuxError> {
if self.gated.swap(false, std::sync::atomic::Ordering::SeqCst) {
let _ = self.entered.send(());
let _ = self.release.lock().expect("gate mutex").recv();
}
self.inner
.new_session_piped(id, cwd, command, session_env, sink_command)
}
fn kill_session(&self, id: Uuid) -> Result<(), maestro_daemon::tmux::TmuxError> {
self.inner.kill_session(id)
}
fn list_sessions(&self) -> Result<Vec<Uuid>, maestro_daemon::tmux::TmuxError> {
self.inner.list_sessions()
}
fn pane_info(
&self,
id: Uuid,
) -> Result<Option<maestro_daemon::tmux::PaneInfo>, maestro_daemon::tmux::TmuxError> {
self.inner.pane_info(id)
}
fn send_keys(&self, id: Uuid, bytes: &[u8]) -> Result<(), maestro_daemon::tmux::TmuxError> {
self.inner.send_keys(id, bytes)
}
fn pipe_pane(&self, id: Uuid, sink: &str) -> Result<(), maestro_daemon::tmux::TmuxError> {
self.inner.pipe_pane(id, sink)
}
fn resize_window(
&self,
id: Uuid,
cols: u16,
rows: u16,
) -> Result<(), maestro_daemon::tmux::TmuxError> {
self.inner.resize_window(id, cols, rows)
}
fn capture_pane(&self, id: Uuid) -> Result<Option<Vec<u8>>, maestro_daemon::tmux::TmuxError> {
self.inner.capture_pane(id)
}
}
/// A dormant session whose manager's tmux spawn can be parked by the test.
async fn dormant_gated(fx: &DaemonFixture, cwd: &TempDir) -> (SessionManager, Session, Gate) {
let session = fx
.manager
.create(CreateSessionRequest {
cwd: cwd.path().to_path_buf(),
command: "sleep 30".into(),
env: HashMap::new(),
metadata: HashMap::new(),
worktree: None,
})
.await
.expect("create");
fx.tmux.tmux.kill_server().expect("kill tmux server");
let (panes, gate) = gated_panes(fx.tmux.tmux.clone());
let manager = SessionManager::new(
panes,
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir()),
maestro_daemon::worktree::WorktreeManager::new(None),
fx.fifo_dir(),
);
manager.recover().await.expect("recover");
assert_eq!(
manager.status(session.id).await,
Some(SessionStatus::Dormant)
);
(manager, session, gate)
}
/// A kill that lands while a resume is inside the tmux spawn must not leave
/// the sidecar behind. The kill deletes it; the resume writes it back a moment
/// later and then discovers its own record is gone. The sidecar it resurrected
/// would otherwise make the next recovery register a ghost dormant session for
/// a session that was killed.
#[tokio::test]
async fn a_kill_during_resume_leaves_no_sidecar_behind() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let (manager, session, mut gate) = dormant_gated(&fx, &cwd).await;
let id = session.id;
let resuming = tokio::spawn({
let manager = manager.clone();
async move { manager.resume(id, &[]).await }
});
gate.wait_entered().await;
manager.kill(id, false).await.expect("kill mid-resume");
gate.release();
let err = resuming.await.unwrap().expect_err("record is gone");
assert!(matches!(err, SessionError::NotFound(got) if got == id));
assert!(
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir())
.read(id)
.expect("read sidecar")
.is_none(),
"the killed session's sidecar must not be resurrected by the losing resume"
);
// And so the next recovery registers no ghost.
let after = revive(&fx);
after.recover().await.expect("recover");
assert!(
after.snapshot().await.is_empty(),
"a killed session must not come back dormant"
);
}
/// Two resumes of the same session must not both proceed. The second is
/// refused on the state — `already_running` — not on its tmux spawn colliding
/// with the first one's.
#[tokio::test]
async fn a_second_resume_is_refused_while_the_first_is_in_flight() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let (manager, session, mut gate) = dormant_gated(&fx, &cwd).await;
let id = session.id;
let resuming = tokio::spawn({
let manager = manager.clone();
async move { manager.resume(id, &[]).await }
});
gate.wait_entered().await;
let err = manager
.resume(id, &[])
.await
.expect_err("a resume is already in flight");
assert!(matches!(err, SessionError::AlreadyRunning(got) if got == id));
assert_eq!(err.to_error_info().code, "already_running");
gate.release();
let first = resuming.await.unwrap().expect("the first resume wins");
assert_eq!(first.status, SessionStatus::Running);
assert_eq!(
fx.tmux.tmux.list_sessions().expect("list"),
vec![id],
"exactly one pane for the session"
);
}
/// A refused resume must not leave the session marked in-flight: once the
/// reason for the refusal is gone, resuming works.
#[tokio::test]
async fn a_refused_resume_can_be_retried() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let (manager, session) = dormant_session(&fx, &cwd, "sleep 30", "retried").await;
std::fs::remove_dir_all(cwd.path()).unwrap();
let err = manager
.resume(session.id, &[])
.await
.expect_err("cwd is gone");
assert_eq!(err.to_error_info().code, "cwd_missing");
std::fs::create_dir_all(cwd.path()).unwrap();
let resumed = manager.resume(session.id, &[]).await.expect("resume");
assert_eq!(resumed.status, SessionStatus::Running);
}
@@ -0,0 +1,161 @@
//! `[daemon].resume_on_boot` (#167): what recovery does with sessions whose
//! tmux pane is gone but whose sidecar survived, against real tmux.
//!
//! The default is deliberately the timid one. Resuming agents unattended
//! seconds after a reboot is a foot-gun, so `auto` is opted into per host and
//! `never` throws the sidecars away instead.
mod common;
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
use common::{wait_for_async, DaemonFixture};
use maestro_config::ResumeOnBoot;
use maestro_daemon::SessionManager;
use maestro_protocol::{CreateSessionRequest, Session, SessionStatus, SpawnPreset};
use tempfile::TempDir;
/// A fresh manager over the same tmux server, sidecars and fifos — what the
/// daemon builds after a restart — carrying `policy`.
fn revive(fx: &DaemonFixture, policy: ResumeOnBoot, presets: Vec<SpawnPreset>) -> SessionManager {
SessionManager::new(
Arc::new(fx.tmux.tmux.clone()),
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir()),
maestro_daemon::worktree::WorktreeManager::new(None),
fx.fifo_dir(),
)
.with_resume_policy(policy, presets)
}
/// A preset whose `resume_command` touches a marker file, so a test can see
/// that the resumed session ran the resume command rather than the original.
fn marker_preset(command: &str, marker_dir: &TempDir) -> Vec<SpawnPreset> {
vec![SpawnPreset {
id: "agent".into(),
label: "Agent".into(),
command: command.into(),
resume_command: Some(format!(
"sh -c 'touch {}/resumed; sleep 30'",
marker_dir.path().display()
)),
}]
}
async fn create(fx: &DaemonFixture, cwd: &TempDir, command: &str) -> Session {
fx.manager
.create(CreateSessionRequest {
cwd: cwd.path().to_path_buf(),
command: command.into(),
env: HashMap::new(),
metadata: HashMap::new(),
worktree: None,
})
.await
.expect("create")
}
/// The default: a reboot leaves the session listed but not running, waiting
/// for the operator.
#[tokio::test]
async fn dormant_is_the_default_and_leaves_the_session_waiting() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let session = create(&fx, &cwd, "sleep 30").await;
fx.tmux.tmux.kill_server().expect("kill tmux server");
let revived = revive(&fx, ResumeOnBoot::Dormant, Vec::new());
revived.recover().await.expect("recover");
assert_eq!(
revived.status(session.id).await,
Some(SessionStatus::Dormant)
);
assert!(fx
.sidecar_dir()
.join(format!("{}.json", session.id))
.exists());
assert!(
fx.tmux.tmux.list_sessions().expect("list").is_empty(),
"nothing was respawned"
);
}
/// `auto` brings each one back itself, down the same path a client's resume
/// takes: same uuid, same sidecar, the preset's `resume_command`.
#[tokio::test]
async fn auto_resumes_every_dormant_session_during_recovery() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let markers = TempDir::new().unwrap();
let session = create(&fx, &cwd, "sleep 30").await;
fx.tmux.tmux.kill_server().expect("kill tmux server");
let revived = revive(&fx, ResumeOnBoot::Auto, marker_preset("sleep 30", &markers));
revived.recover().await.expect("recover");
assert_eq!(
revived.status(session.id).await,
Some(SessionStatus::Running),
"auto resumed it"
);
assert_eq!(
fx.tmux.tmux.list_sessions().expect("list"),
vec![session.id],
"back under the same uuid"
);
let stored = maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir())
.read(session.id)
.expect("read sidecar")
.expect("same sidecar");
assert_eq!(stored.created_at, session.created_at);
assert_eq!(stored.command, "sleep 30", "the original command is kept");
// It really ran the preset's resume command, not the spawn command.
let marker = markers.path().join("resumed");
wait_for_async(Duration::from_secs(5), || {
let marker = marker.clone();
async move { marker.exists() }
})
.await;
assert!(marker.exists(), "resume_command ran");
}
/// `never` makes a reboot a clean slate: the sidecars go, and the session is
/// not listed at all.
#[tokio::test]
async fn never_prunes_the_sidecars_instead() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let session = create(&fx, &cwd, "sleep 30").await;
fx.tmux.tmux.kill_server().expect("kill tmux server");
let revived = revive(&fx, ResumeOnBoot::Never, Vec::new());
revived.recover().await.expect("recover");
assert!(!revived.has(session.id).await, "not listed");
assert!(
!fx.sidecar_dir()
.join(format!("{}.json", session.id))
.exists(),
"sidecar pruned"
);
assert!(fx.tmux.tmux.list_sessions().expect("list").is_empty());
}
/// The policy only decides what happens to sessions that lost their pane. A
/// session tmux still holds is adopted running by the first pass under every
/// policy — including `never`, which must not delete a live session's sidecar.
#[tokio::test]
async fn the_policy_leaves_sessions_tmux_still_holds_alone() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let live = create(&fx, &cwd, "sleep 30").await;
let revived = revive(&fx, ResumeOnBoot::Never, Vec::new());
assert_eq!(revived.recover().await.expect("recover"), 1);
assert_eq!(revived.status(live.id).await, Some(SessionStatus::Running));
assert!(fx.sidecar_dir().join(format!("{}.json", live.id)).exists());
}
+219
View File
@@ -0,0 +1,219 @@
//! Persisted PTY scrollback across a reboot (#171), against real tmux.
//!
//! Everything else in the reboot milestone restores what a session *is*; this
//! restores what it *looked like*. The cases that matter are the ones a clean
//! shutdown never gets to run: the host went down, and the only scrollback
//! that exists is whatever the last periodic flush wrote.
mod common;
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
use common::{wait_for_async, DaemonFixture};
use maestro_daemon::scrollback::ScrollbackStore;
use maestro_daemon::SessionManager;
use maestro_protocol::{CreateSessionRequest, Session, SessionStatus};
use tempfile::TempDir;
use uuid::Uuid;
/// A fresh manager over the same tmux server, sidecars, fifos and scrollback —
/// what the daemon builds after a restart.
fn revive(fx: &DaemonFixture) -> SessionManager {
SessionManager::new(
Arc::new(fx.tmux.tmux.clone()),
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir()),
maestro_daemon::worktree::WorktreeManager::new(None),
fx.fifo_dir(),
)
.with_scrollback(fx.scrollback_dir())
}
async fn spawn(fx: &DaemonFixture, cwd: &TempDir, command: &str) -> Session {
fx.manager
.create(CreateSessionRequest {
cwd: cwd.path().to_path_buf(),
command: command.into(),
env: HashMap::new(),
metadata: HashMap::new(),
worktree: None,
})
.await
.expect("create")
}
/// Wait until the session has printed `needle`, flushing as we go — the same
/// thing the liveness watcher's flush pass does on its timer.
async fn wait_for_flushed(
manager: &SessionManager,
store: &ScrollbackStore,
id: Uuid,
needle: &str,
) {
wait_for_async(Duration::from_secs(10), || async {
manager.flush_scrollback().await;
String::from_utf8_lossy(&store.read(id, 64 * 1024)).contains(needle)
})
.await;
}
fn replay_text(replay: &[u8]) -> String {
String::from_utf8_lossy(replay).to_string()
}
/// The keystone: the host reboots with no exit path run at all, and the
/// resumed session opens showing what it printed before.
#[tokio::test]
async fn a_resumed_session_replays_the_scrollback_from_before_the_reboot() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let store = ScrollbackStore::new(fx.scrollback_dir());
let session = spawn(&fx, &cwd, "sh -c 'echo before-the-reboot; sleep 30'").await;
wait_for_flushed(&fx.manager, &store, session.id, "before-the-reboot").await;
// The reboot: tmux dies where it stands. No kill, no seal, no shutdown
// hook — only whatever the periodic flush had already written.
fx.tmux.tmux.kill_server().expect("kill tmux server");
let manager = revive(&fx);
manager.recover().await.expect("recover");
assert_eq!(
manager.status(session.id).await,
Some(SessionStatus::Dormant)
);
manager.resume(session.id, &[]).await.expect("resume");
let (replay, _rx) = manager
.subscribe_pty_with_replay(session.id)
.await
.expect("attachable again");
assert!(
replay_text(&replay).contains("before-the-reboot"),
"resumed pane should open on the old scrollback, got {:?}",
replay_text(&replay)
);
}
/// A daemon restart leaves tmux alone, so the session is adopted rather than
/// resumed — and it too lost its in-memory ring with the old process.
#[tokio::test]
async fn an_adopted_session_replays_the_scrollback_of_the_previous_daemon() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let store = ScrollbackStore::new(fx.scrollback_dir());
let session = spawn(&fx, &cwd, "sh -c 'echo printed-earlier; sleep 30'").await;
wait_for_flushed(&fx.manager, &store, session.id, "printed-earlier").await;
// Daemon restart: the manager goes, tmux stays.
let manager = revive(&fx);
manager.recover().await.expect("recover");
assert_eq!(
manager.status(session.id).await,
Some(SessionStatus::Running)
);
let (replay, _rx) = manager
.subscribe_pty_with_replay(session.id)
.await
.expect("attachable");
assert!(
replay_text(&replay).contains("printed-earlier"),
"adopted pane should carry the old scrollback ahead of tmux's screen, got {:?}",
replay_text(&replay)
);
}
/// Killing a session takes its scrollback with it: nothing will ever read the
/// file again, and the operator's output should not linger on disk.
#[tokio::test]
async fn killing_a_session_deletes_its_scrollback() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let store = ScrollbackStore::new(fx.scrollback_dir());
let session = spawn(&fx, &cwd, "sh -c 'echo doomed; sleep 30'").await;
wait_for_flushed(&fx.manager, &store, session.id, "doomed").await;
assert!(store.path_for(session.id).exists());
fx.manager.kill(session.id, false).await.expect("kill");
assert!(
!store.path_for(session.id).exists(),
"scrollback should not outlive the session"
);
}
/// A session that exits on its own does get an exit path, and takes it: the
/// last thing it printed is flushed when the bridge is sealed, with no wait
/// for the next tick.
#[tokio::test]
async fn a_session_that_exits_flushes_its_last_output() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let store = ScrollbackStore::new(fx.scrollback_dir());
let session = spawn(&fx, &cwd, "sh -c 'echo last-words'").await;
fx.manager
.start_liveness_watcher(Duration::from_millis(100));
wait_for_async(Duration::from_secs(10), || async {
fx.manager.status(session.id).await == Some(SessionStatus::Dead)
})
.await;
assert!(
String::from_utf8_lossy(&store.read(session.id, 64 * 1024)).contains("last-words"),
"a sealed bridge should have flushed what it printed"
);
}
/// Nothing here may fail a resume. A scrollback file that is gone, or one full
/// of bytes no terminal would emit, degrades to a pane with no history — not
/// to an error.
#[tokio::test]
async fn a_missing_or_corrupt_scrollback_still_resumes() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let store = ScrollbackStore::new(fx.scrollback_dir());
let gone = spawn(&fx, &cwd, "sleep 30").await;
let garbled = spawn(&fx, &cwd, "sleep 30").await;
fx.manager.flush_scrollback().await;
fx.tmux.tmux.kill_server().expect("kill tmux server");
std::fs::create_dir_all(store.dir()).expect("scrollback dir");
std::fs::remove_file(store.path_for(gone.id)).ok();
std::fs::write(store.path_for(garbled.id), [0xff, 0xfe, 0x00, 0x80]).expect("write garbage");
let manager = revive(&fx);
manager.recover().await.expect("recover");
for id in [gone.id, garbled.id] {
manager.resume(id, &[]).await.expect("resume");
assert_eq!(manager.status(id).await, Some(SessionStatus::Running));
assert!(manager.subscribe_pty_with_replay(id).await.is_some());
}
}
/// The disk cost is the replay budget, not a second one of its own.
#[tokio::test]
async fn the_flushed_file_stays_inside_the_replay_budget() {
let fx = DaemonFixture::new_with_replay(256);
let cwd = TempDir::new().unwrap();
let store = ScrollbackStore::new(fx.scrollback_dir());
let session = spawn(
&fx,
&cwd,
"sh -c 'for i in $(seq 1 400); do echo line-$i; done; sleep 30'",
)
.await;
wait_for_flushed(&fx.manager, &store, session.id, "line-400").await;
let size = std::fs::metadata(store.path_for(session.id))
.expect("stat scrollback")
.len();
assert!(
size <= 256,
"scrollback file was {size} bytes, budget is 256"
);
}
+42 -8
View File
@@ -326,12 +326,14 @@ fn revive_with_boot_id(
)
}
/// #164: a changed boot id means every session from the previous boot is
/// already dead, so recovery must not trust tmux to say so — it skips the
/// probe outright. Proven by leaving the tmux session alive: if the probe
/// ran, it would come back `Running`, not `Dormant`.
/// #174: a boot id can change without the tmux server dying — a restored
/// state directory, a VM or container snapshot restore, a truncated
/// `boot_id` file. `tmux list-sessions` stays the single source of truth for
/// liveness, so a session tmux still holds is adopted `Running` no matter
/// what the boot id says. Registering it dormant instead stranded it: resume
/// would collide with the tmux name still in use and fail forever.
#[tokio::test]
async fn recover_skips_tmux_probe_when_boot_id_changed() {
async fn recover_adopts_live_session_when_boot_id_changed() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let s = fx
@@ -349,15 +351,15 @@ async fn recover_skips_tmux_probe_when_boot_id_changed() {
let boot_id_path = fx.state_dir.path().join("boot_id");
std::fs::write(&boot_id_path, "previous-boot").unwrap();
// tmux session is still alive — a probe would find it if one ran.
// tmux session is still alive: the probe runs and finds it.
let revived = revive_with_boot_id(&fx, boot_id_path.clone(), Some("current-boot"));
let adopted = revived.recover().await.expect("recover");
assert_eq!(adopted, 0, "reboot detected: nothing adopted live");
assert_eq!(adopted, 1, "tmux still holds the session; adopt it");
let snap = revived.snapshot().await;
assert_eq!(snap.len(), 1);
assert_eq!(snap[0].id, s.id);
assert_eq!(snap[0].status, SessionStatus::Dormant);
assert_eq!(snap[0].status, SessionStatus::Running);
// The current boot id is now the stored one, for the next comparison.
assert_eq!(
@@ -366,6 +368,38 @@ async fn recover_skips_tmux_probe_when_boot_id_changed() {
);
}
/// #164, still true after #174: a real reboot takes the tmux server with it,
/// so the probe comes back empty and every sidecar is registered dormant.
#[tokio::test]
async fn recover_registers_dormant_when_boot_id_changed_and_tmux_gone() {
let fx = DaemonFixture::new();
let cwd = TempDir::new().unwrap();
let s = fx
.manager
.create(CreateSessionRequest {
cwd: cwd.path().to_path_buf(),
command: "sleep 30".into(),
env: HashMap::new(),
metadata: HashMap::new(),
worktree: None,
})
.await
.expect("create");
let boot_id_path = fx.state_dir.path().join("boot_id");
std::fs::write(&boot_id_path, "previous-boot").unwrap();
fx.tmux.tmux.kill_server().expect("kill tmux server");
let revived = revive_with_boot_id(&fx, boot_id_path, Some("current-boot"));
let adopted = revived.recover().await.expect("recover");
assert_eq!(adopted, 0, "no tmux server survived the reboot");
let snap = revived.snapshot().await;
assert_eq!(snap.len(), 1);
assert_eq!(snap[0].id, s.id);
assert_eq!(snap[0].status, SessionStatus::Dormant);
}
/// #164: an unchanged boot id is a plain restart — recovery probes tmux as
/// before and adopts the still-live session `Running`.
#[tokio::test]
+20
View File
@@ -243,6 +243,26 @@ Pick `/tmp` if the daemon must outlive your login session and you can't run
tmux sockets with it). The env var `MAESTRO_DAEMON_TMUX_TMPDIR` takes
precedence over the file.
## What happens to sessions after a reboot
A host reboot kills the tmux server, so every session the daemon knew is dead
while its record on disk survives. What recovery does with those is
`[daemon].resume_on_boot`:
```toml
[daemon]
resume_on_boot = "dormant" # "dormant" (default) | "auto" | "never"
```
- `dormant` — the sessions are listed but not running, and you resume the ones
you want from the web UI or the TUI.
- `auto` — the daemon resumes them all itself during startup. Only pick this
on a host where restarting every agent unattended is safe: they come back on
the flags they were spawned with, with nobody watching.
- `never` — the records are deleted and a reboot is a clean slate.
See `docs/RESTART.md` for the full contract.
## What the systemd unit looks like
The wizard writes a *user* unit (`systemctl --user`), since `maestro daemon`
+72 -2
View File
@@ -75,7 +75,13 @@ A daemon restart leaves tmux alone, so recovery re-adopts every
beyond the reconnect above.
A **host reboot** is different: it kills the tmux server, so there is nothing
to adopt while every session's sidecar is still on disk. Recovery makes a
to adopt while every session's sidecar is still on disk. Recovery always asks
tmux — `tmux list-sessions` is the single source of truth for liveness, and the
kernel's `boot_id` only tells a reboot from a restart in the log (#164, #174).
A boot id changes without the tmux server dying whenever the state directory is
restored, a VM or container snapshot is rolled back, or the `boot_id` file is
truncated; a session tmux still holds is adopted `Running` in every one of
those cases. Recovery makes a
second pass over those sidecars and registers each as a `Dormant` session
(#162) — id, name, cwd, command, worktree and `created_at` intact. A dormant
session has no pane and no PTY bridge: it cannot be attached to, typed into or
@@ -92,10 +98,74 @@ conversation id is respawned on the command it was created with. `Session::comma
always keeps that original command, so a second resume resolves the same preset.
Resume is refused with `409` for a session that is already running
(`already_running`) and for one whose `cwd` has disappeared (`cwd_missing`) —
neither is something a respawn could fix. The other thing to do with a dormant
neither is something a respawn could fix. A session with a resume already in
flight is refused as `already_running` too (#175): the respawn happens with
the session map unlocked, so the record is marked in-flight before it starts
and the second request is answered on that state rather than on its tmux spawn
colliding with the first one's. A resume that loses a race with a kill takes
its own sidecar back off disk, so the killed session does not return as a
dormant ghost on the next recovery. The other thing to do with a dormant
session is dismiss it, which removes the sidecar so the next recovery does not
bring it back.
## What the pane looks like when it comes back
The PTY replay ring lives in memory, so a resumed session would otherwise open
on a blank pane: the conversation is back, the terminal history the operator
was reading is not. Each bridge's ring is mirrored to
`<state_dir>/scrollback/<id>.bin` (#171) and handed to the new bridge on
resume, and on the recovery pass that re-adopts a session tmux still holds —
that one lost its ring with the previous daemon too. The restored bytes go out
ahead of the live ones, and are dropped once the ring holds a full budget of
new output of its own, which is the point where everything restored is older
than the budget covers.
The file is capped at the same `[daemon] pty_replay_bytes` budget the ring
uses — one budget, not two — and it is deleted when the session is killed, and
when recovery prunes its sidecar. Nothing here can fail a resume: a missing,
unreadable or oversized file degrades to an empty pane, which is exactly what
the operator had before it existed.
It is flushed every 10 seconds by the liveness watcher, and once more when a
session's bridge is sealed on exit. The timer is the part that matters: a host
reboot runs no exit path at all, so scrollback written only on a clean shutdown
would be missing in the one case this exists for. Ten seconds bounds the loss
to the last few lines — scrollback is visual continuity, not state — and costs
at most one file write per *active* session per tick, since a session whose
bridge has read nothing since the last flush is skipped.
## What recovery does with them: `resume_on_boot`
Whether a recovered-but-dead session is left dormant at all is the operator's
call, per host, via `[daemon].resume_on_boot` (#167):
| value | what recovery does |
| ----------- | --------------------------------------------------------------------- |
| `"dormant"` | **Default.** Register each one dormant and wait for the operator. |
| `"auto"` | Resume each one during recovery, itself. |
| `"never"` | Prune the sidecars instead — a reboot is a clean slate. |
`dormant` is the default because resuming agents unattended is a foot-gun:
they come back on whatever flags they were spawned with, seconds after a
reboot, with nobody watching. `auto` exists for hosts where that is known to
be safe, and it is opted into rather than inherited.
`auto` calls the same `resume` the client's `POST /api/sessions/:id/resume`
does — same uuid, same sidecar, the preset's `resume_command` — so there is
one resume path and not two. A session that fails to resume is logged and
stays dormant; the rest still come back.
`never` prunes the same way #163 does for a missing `cwd`: the sidecar is
deleted and nothing is registered, so the session is not listed and the next
recovery does not find it. Whatever worktree it owned is left on disk.
The policy only decides the fate of sessions with no live pane. A session
tmux still holds is adopted `Running` by the first pass under every policy,
including `never`.
Recovery logs the policy that ran and the number of sessions it acted on, so
a host that resumes agents unattended says so in its own journal.
Attaching to a session that has no bridge answers `409 not_attachable`, not
`404 unknown_session`: the session exists, and a client that cannot tell "gone"
from "not running" cannot offer the resume.