Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5a47f6fbf6 | ||
|
|
e7830d15b0 | ||
|
|
87f1e73dec | ||
|
|
0fd107be77 | ||
|
|
aebd10c2da | ||
|
|
ae3091e2a2 | ||
|
|
1366c2d430 | ||
|
|
d050df7d3d | ||
|
|
e51a7007a6 | ||
|
|
02bf077c03 |
@@ -129,6 +129,80 @@ pub struct DaemonSection {
|
||||
/// `Some(0)` disables history entirely (no log written, no recents read).
|
||||
#[serde(default)]
|
||||
pub spawn_history_days: Option<u32>,
|
||||
/// What recovery does with sessions that outlived their tmux server —
|
||||
/// after a host reboot, or any restart that finds a sidecar with no pane
|
||||
/// (#167). Default [`ResumeOnBoot::Dormant`].
|
||||
#[serde(default, deserialize_with = "deserialize_resume_on_boot")]
|
||||
pub resume_on_boot: Option<ResumeOnBoot>,
|
||||
}
|
||||
|
||||
impl DaemonSection {
|
||||
/// The configured `[daemon].resume_on_boot`, or the safe default:
|
||||
/// recovered-but-dead sessions wait for the operator.
|
||||
pub fn resume_on_boot(&self) -> ResumeOnBoot {
|
||||
self.resume_on_boot.unwrap_or_default()
|
||||
}
|
||||
}
|
||||
|
||||
/// `[daemon].resume_on_boot` — what recovery does with a session whose tmux
|
||||
/// pane is gone but whose sidecar is still on disk (#167).
|
||||
///
|
||||
/// Resuming agents unattended seconds after a reboot is a foot-gun: they run
|
||||
/// with whatever flags they were spawned with, with nobody watching. So the
|
||||
/// default waits for a human, and `auto` is opted into per host.
|
||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
||||
pub enum ResumeOnBoot {
|
||||
/// Register each one `Dormant` and wait for the operator to resume it.
|
||||
#[default]
|
||||
Dormant,
|
||||
/// Resume each one during recovery, on the same path a client's
|
||||
/// `POST /api/sessions/:id/resume` takes.
|
||||
Auto,
|
||||
/// Prune the sidecars instead: a reboot is a clean slate.
|
||||
Never,
|
||||
}
|
||||
|
||||
/// Every `[daemon].resume_on_boot` name and its variant, in documentation
|
||||
/// order.
|
||||
pub const RESUME_ON_BOOT_POLICIES: &[(&str, ResumeOnBoot)] = &[
|
||||
("dormant", ResumeOnBoot::Dormant),
|
||||
("auto", ResumeOnBoot::Auto),
|
||||
("never", ResumeOnBoot::Never),
|
||||
];
|
||||
|
||||
impl ResumeOnBoot {
|
||||
/// The name this policy is spelled with in the config file and in logs.
|
||||
pub fn as_str(&self) -> &'static str {
|
||||
match self {
|
||||
ResumeOnBoot::Dormant => "dormant",
|
||||
ResumeOnBoot::Auto => "auto",
|
||||
ResumeOnBoot::Never => "never",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn deserialize_resume_on_boot<'de, D>(deserializer: D) -> Result<Option<ResumeOnBoot>, D::Error>
|
||||
where
|
||||
D: Deserializer<'de>,
|
||||
{
|
||||
let raw = Option::<String>::deserialize(deserializer)?;
|
||||
raw.map(|name| {
|
||||
RESUME_ON_BOOT_POLICIES
|
||||
.iter()
|
||||
.find(|(n, _)| *n == name)
|
||||
.map(|(_, policy)| *policy)
|
||||
.ok_or_else(|| {
|
||||
let known = RESUME_ON_BOOT_POLICIES
|
||||
.iter()
|
||||
.map(|(n, _)| format!("{n:?}"))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ");
|
||||
D::Error::custom(format!(
|
||||
"invalid [daemon].resume_on_boot {name:?}; expected one of {known}"
|
||||
))
|
||||
})
|
||||
})
|
||||
.transpose()
|
||||
}
|
||||
|
||||
/// `[tui]` — `maestro tui` settings (NAP-275 epic, phase 2 NAP-277).
|
||||
@@ -775,4 +849,27 @@ bind_addr = "127.0.0.1:9090"
|
||||
);
|
||||
assert_eq!(cfg.resolve_daemon_url(None), "http://explicit:1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resume_on_boot_defaults_to_dormant_and_parses_every_policy() {
|
||||
assert_eq!(
|
||||
MaestroConfigFile::default().daemon.resume_on_boot(),
|
||||
ResumeOnBoot::Dormant,
|
||||
"an absent policy is the safe one"
|
||||
);
|
||||
for (name, expected) in RESUME_ON_BOOT_POLICIES {
|
||||
let cfg = file_with(&format!("[daemon]\nresume_on_boot = \"{name}\"\n"));
|
||||
assert_eq!(cfg.daemon.resume_on_boot(), *expected, "{name}");
|
||||
assert_eq!(expected.as_str(), *name);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resume_on_boot_rejects_an_unknown_policy() {
|
||||
let err = toml::from_str::<MaestroConfigFile>("[daemon]\nresume_on_boot = \"maybe\"\n")
|
||||
.expect_err("unknown policy must not silently fall back");
|
||||
let msg = err.to_string();
|
||||
assert!(msg.contains("resume_on_boot"), "{msg}");
|
||||
assert!(msg.contains("\"auto\""), "{msg}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,10 +2,13 @@
|
||||
//!
|
||||
//! `/proc/sys/kernel/random/boot_id` is fixed for the life of the kernel and
|
||||
//! changes on every boot. [`SessionManager::recover`] compares it against the
|
||||
//! value it stored on the previous run: a difference means every session from
|
||||
//! before is provably dead, so the tmux probe can be skipped and those
|
||||
//! sessions registered dormant directly instead of relying on tmux reporting
|
||||
//! an empty list.
|
||||
//! value it stored on the previous run and logs which of the two it was.
|
||||
//!
|
||||
//! It does not decide liveness (#174). A boot id changes without the tmux
|
||||
//! server dying whenever the state directory is restored, a VM or container
|
||||
//! snapshot is rolled back, or the `boot_id` file is truncated — so recovery
|
||||
//! always probes `tmux list-sessions` and the boot id only says whether a
|
||||
//! server that did not answer is coming back.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
|
||||
@@ -84,10 +84,16 @@ async fn run(config_override: Option<PathBuf>) -> Result<(), Box<dyn Error>> {
|
||||
config.spawn_history_path(),
|
||||
config.spawn_history_days,
|
||||
))
|
||||
// #171: mirror PTY scrollback so a resumed session opens on what it was
|
||||
// showing, instead of a blank pane.
|
||||
.with_scrollback(config.scrollback_dir())
|
||||
.with_boot_id(
|
||||
config.boot_id_path(),
|
||||
std::sync::Arc::new(crate::boot_id::ProcBootId),
|
||||
);
|
||||
)
|
||||
// #167: recovery needs the presets to resolve a resumed session's command,
|
||||
// so `auto` runs the same resume a client's POST would.
|
||||
.with_resume_policy(config.resume_on_boot, config.spawn_presets.clone());
|
||||
// Spawned sessions learn where to report back (#103). Derived from the
|
||||
// configured bind address; set before any spawn can happen.
|
||||
manager.set_daemon_url(format!("http://{}", config.bind_addr));
|
||||
|
||||
@@ -97,6 +97,10 @@ pub struct DaemonConfig {
|
||||
/// never binds it itself; it serves it so the web client and the TUI read
|
||||
/// one knob.
|
||||
pub leader: String,
|
||||
/// What recovery does with sessions whose tmux pane is gone but whose
|
||||
/// sidecar survives (#167): `[daemon].resume_on_boot`, default
|
||||
/// [`maestro_config::ResumeOnBoot::Dormant`].
|
||||
pub resume_on_boot: maestro_config::ResumeOnBoot,
|
||||
}
|
||||
|
||||
impl DaemonConfig {
|
||||
@@ -113,6 +117,12 @@ impl DaemonConfig {
|
||||
self.state_dir.join("spawn-history.jsonl")
|
||||
}
|
||||
|
||||
/// Where PTY scrollback is mirrored so a resumed session opens on what
|
||||
/// it printed before (#171): `<state_dir>/scrollback`.
|
||||
pub fn scrollback_dir(&self) -> PathBuf {
|
||||
self.state_dir.join("scrollback")
|
||||
}
|
||||
|
||||
/// Where recovery records the kernel boot id it saw last (#164):
|
||||
/// `<state_dir>/boot_id`.
|
||||
pub fn boot_id_path(&self) -> PathBuf {
|
||||
@@ -149,6 +159,8 @@ pub fn load_config(path: &Path, env: EnvOverrides) -> Result<DaemonConfig, Confi
|
||||
// validation (#22/#26).
|
||||
let spawn_presets = file.spawn.resolve()?;
|
||||
|
||||
let resume_on_boot = daemon.resume_on_boot();
|
||||
|
||||
let hostname = daemon.hostname.unwrap_or_else(system_hostname);
|
||||
let home_dir = dirs::home_dir()
|
||||
.map(|p| p.to_string_lossy().into_owned())
|
||||
@@ -209,6 +221,7 @@ pub fn load_config(path: &Path, env: EnvOverrides) -> Result<DaemonConfig, Confi
|
||||
project_scan_depth,
|
||||
hooks_auto_install: file.hooks.auto_install(),
|
||||
leader: file.tui.leader().to_string(),
|
||||
resume_on_boot,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -289,6 +302,34 @@ auto_install = false
|
||||
assert!(!cfg.hooks_auto_install);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resume_on_boot_defaults_to_dormant_and_reads_the_file() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("config.toml");
|
||||
std::fs::write(&path, "").unwrap();
|
||||
assert_eq!(
|
||||
load_config(&path, EnvOverrides::default())
|
||||
.unwrap()
|
||||
.resume_on_boot,
|
||||
maestro_config::ResumeOnBoot::Dormant
|
||||
);
|
||||
|
||||
std::fs::write(
|
||||
&path,
|
||||
r#"
|
||||
[daemon]
|
||||
resume_on_boot = "auto"
|
||||
"#,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
load_config(&path, EnvOverrides::default())
|
||||
.unwrap()
|
||||
.resume_on_boot,
|
||||
maestro_config::ResumeOnBoot::Auto
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_file_defaults_everything() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
|
||||
@@ -26,6 +26,7 @@ pub mod projects;
|
||||
pub mod pty;
|
||||
pub mod release;
|
||||
pub mod resume;
|
||||
pub mod scrollback;
|
||||
pub mod session;
|
||||
pub mod sidecar;
|
||||
pub mod spawn_history;
|
||||
|
||||
@@ -21,6 +21,11 @@
|
||||
//! one-shot prefix before joining the live broadcast, so attaching to an
|
||||
//! idle session immediately shows the last screenful instead of a blank
|
||||
//! terminal.
|
||||
//! - #171: a bridge can be started with scrollback restored from disk — what
|
||||
//! the session printed before the daemon (or the host) went away. Those
|
||||
//! bytes are handed to subscribers ahead of the live ones until the ring
|
||||
//! has a full budget of new output of its own, at which point the restored
|
||||
//! history is older than anything the budget covers and is dropped.
|
||||
|
||||
use std::collections::VecDeque;
|
||||
use std::ffi::CString;
|
||||
@@ -58,6 +63,10 @@ const READ_BUF_BYTES: usize = 4096;
|
||||
#[derive(Debug)]
|
||||
pub struct PtyBridge {
|
||||
fifo_path: PathBuf,
|
||||
/// Scrollback from a previous life of this session (#171), replayed
|
||||
/// ahead of the ring. Empty for a fresh spawn. Never appended to, so it
|
||||
/// can never duplicate a byte the ring also holds.
|
||||
restored: Vec<u8>,
|
||||
/// `None` once [`Self::seal`] has run: live subscribers see the channel
|
||||
/// close, but the replay buffer stays readable.
|
||||
sender: Option<broadcast::Sender<Vec<u8>>>,
|
||||
@@ -79,6 +88,21 @@ impl PtyBridge {
|
||||
/// Variant that lets the caller pick a custom replay-buffer cap (bytes).
|
||||
/// Pass `0` to disable replay (subscribers see only live bytes).
|
||||
pub async fn start_with_replay(fifo_path: PathBuf, replay_bytes: usize) -> io::Result<Self> {
|
||||
Self::start_restoring(fifo_path, replay_bytes, Vec::new()).await
|
||||
}
|
||||
|
||||
/// As [`Self::start_with_replay`], seeded with `restored` scrollback read
|
||||
/// back from disk (#171). The seed is truncated to the same
|
||||
/// `replay_bytes` budget the ring uses — there is one budget here, not
|
||||
/// two — and is dropped entirely when replay is disabled.
|
||||
pub async fn start_restoring(
|
||||
fifo_path: PathBuf,
|
||||
replay_bytes: usize,
|
||||
mut restored: Vec<u8>,
|
||||
) -> io::Result<Self> {
|
||||
if restored.len() > replay_bytes {
|
||||
restored.drain(..restored.len() - replay_bytes);
|
||||
}
|
||||
if let Some(parent) = fifo_path.parent() {
|
||||
tokio::fs::create_dir_all(parent).await?;
|
||||
}
|
||||
@@ -102,6 +126,7 @@ impl PtyBridge {
|
||||
});
|
||||
Ok(Self {
|
||||
fifo_path,
|
||||
restored,
|
||||
sender: Some(sender),
|
||||
replay,
|
||||
last_activity,
|
||||
@@ -155,7 +180,8 @@ impl PtyBridge {
|
||||
F: FnOnce() -> Option<Vec<u8>>,
|
||||
{
|
||||
let guard = self.replay.lock_for_snapshot();
|
||||
let snapshot = capture().unwrap_or_else(|| guard.snapshot());
|
||||
let live = capture().unwrap_or_else(|| guard.snapshot());
|
||||
let snapshot = guard.with_restored_prefix(&self.restored, live);
|
||||
let rx = match &self.sender {
|
||||
Some(tx) => tx.subscribe(),
|
||||
// Sealed: hand back the scrollback plus a receiver that is already
|
||||
@@ -171,6 +197,25 @@ impl PtyBridge {
|
||||
(snapshot, rx)
|
||||
}
|
||||
|
||||
/// Everything worth persisting for a later life of this session (#171):
|
||||
/// the restored scrollback and the ring, trimmed together to the replay
|
||||
/// budget so the file on disk is never bigger than the ring it mirrors.
|
||||
pub fn scrollback(&self) -> Vec<u8> {
|
||||
let guard = self.replay.lock_for_snapshot();
|
||||
guard.with_restored_prefix(&self.restored, guard.snapshot())
|
||||
}
|
||||
|
||||
/// Total bytes this bridge has appended to its ring. The scrollback flush
|
||||
/// pass compares it against what it last wrote, so an idle session costs
|
||||
/// no disk writes at all.
|
||||
pub fn bytes_seen(&self) -> u64 {
|
||||
self.replay
|
||||
.inner
|
||||
.lock()
|
||||
.expect("ReplayBuffer poisoned")
|
||||
.seen
|
||||
}
|
||||
|
||||
/// Stop reading and close the live stream, keeping the replay buffer.
|
||||
///
|
||||
/// Attached clients see the channel close — without that a `maestro attach`
|
||||
@@ -201,6 +246,8 @@ struct ReplayBuffer {
|
||||
struct ReplayInner {
|
||||
buf: VecDeque<u8>,
|
||||
cap: usize,
|
||||
/// Bytes ever appended, not bytes held — the ring evicts, this does not.
|
||||
seen: u64,
|
||||
}
|
||||
|
||||
impl ReplayBuffer {
|
||||
@@ -209,6 +256,7 @@ impl ReplayBuffer {
|
||||
inner: Arc::new(Mutex::new(ReplayInner {
|
||||
buf: VecDeque::with_capacity(cap.min(READ_BUF_BYTES * 2)),
|
||||
cap,
|
||||
seen: 0,
|
||||
})),
|
||||
}
|
||||
}
|
||||
@@ -225,6 +273,7 @@ impl ReplayBuffer {
|
||||
// If the incoming chunk alone exceeds the cap, take only its tail —
|
||||
// anything in the buffer would be evicted before this finishes anyway.
|
||||
let take_from = bytes.len().saturating_sub(s.cap);
|
||||
s.seen += bytes.len() as u64;
|
||||
let bytes = &bytes[take_from..];
|
||||
s.buf.extend(bytes);
|
||||
while s.buf.len() > s.cap {
|
||||
@@ -247,6 +296,24 @@ struct ReplayGuard<'a> {
|
||||
}
|
||||
|
||||
impl ReplayGuard<'_> {
|
||||
/// `live` with the restored scrollback (#171) in front of it, while the
|
||||
/// ring still holds less than a full budget of its own output. Once it
|
||||
/// holds a full budget, everything restored is older than what the
|
||||
/// budget covers, so it is dropped rather than kept forever.
|
||||
fn with_restored_prefix(&self, restored: &[u8], live: Vec<u8>) -> Vec<u8> {
|
||||
let cap = self.inner.cap;
|
||||
let held = self.inner.buf.len();
|
||||
if restored.is_empty() || cap == 0 || held >= cap {
|
||||
return live;
|
||||
}
|
||||
let keep = cap - held;
|
||||
let start = restored.len().saturating_sub(keep);
|
||||
let mut out = Vec::with_capacity((restored.len() - start) + live.len());
|
||||
out.extend_from_slice(&restored[start..]);
|
||||
out.extend_from_slice(&live);
|
||||
out
|
||||
}
|
||||
|
||||
fn snapshot(&self) -> Vec<u8> {
|
||||
let (a, b) = self.inner.buf.as_slices();
|
||||
let mut out = Vec::with_capacity(a.len() + b.len());
|
||||
@@ -292,6 +359,7 @@ async fn run_reader(
|
||||
// never both, never neither.
|
||||
let mut s = replay.inner.lock().expect("ReplayBuffer poisoned");
|
||||
if s.cap > 0 {
|
||||
s.seen += n as u64;
|
||||
let take_from = n.saturating_sub(s.cap);
|
||||
s.buf.extend(&buf[take_from..n]);
|
||||
while s.buf.len() > s.cap {
|
||||
@@ -345,6 +413,80 @@ fn shell_quote(s: &str) -> String {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
async fn bridge(cap: usize, restored: &[u8]) -> (tempfile::TempDir, PtyBridge) {
|
||||
let dir = tempfile::TempDir::new().expect("tempdir");
|
||||
let bridge =
|
||||
PtyBridge::start_restoring(dir.path().join("pty.fifo"), cap, restored.to_vec())
|
||||
.await
|
||||
.expect("start bridge");
|
||||
(dir, bridge)
|
||||
}
|
||||
|
||||
/// Feed the bridge through its own FIFO and wait for the reader to have
|
||||
/// taken the bytes, so the ring state is deterministic.
|
||||
async fn feed(bridge: &PtyBridge, bytes: &[u8]) {
|
||||
use std::io::Write;
|
||||
let before = bridge.bytes_seen();
|
||||
let mut f = std::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(bridge.fifo_path())
|
||||
.expect("open fifo for write");
|
||||
f.write_all(bytes).expect("write fifo");
|
||||
drop(f);
|
||||
for _ in 0..400 {
|
||||
if bridge.bytes_seen() >= before + bytes.len() as u64 {
|
||||
return;
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
|
||||
}
|
||||
panic!("bridge never read the bytes written to its fifo");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn restored_scrollback_is_replayed_ahead_of_live_bytes() {
|
||||
let (_dir, bridge) = bridge(64, b"before the reboot\n").await;
|
||||
feed(&bridge, b"after\n").await;
|
||||
let (replay, _rx) = bridge.subscribe_with_replay();
|
||||
assert_eq!(replay, b"before the reboot\nafter\n");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn restored_scrollback_is_trimmed_to_the_replay_budget() {
|
||||
let (_dir, bridge) = bridge(4, b"abcdefghij").await;
|
||||
let (replay, _rx) = bridge.subscribe_with_replay();
|
||||
assert_eq!(replay, b"ghij");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn restored_scrollback_yields_once_the_ring_holds_a_full_budget() {
|
||||
let (_dir, bridge) = bridge(4, b"old!").await;
|
||||
feed(&bridge, b"ab").await;
|
||||
let (replay, _rx) = bridge.subscribe_with_replay();
|
||||
assert_eq!(replay, b"d!ab", "prefix plus ring stays inside the budget");
|
||||
feed(&bridge, b"cd").await;
|
||||
let (replay, _rx) = bridge.subscribe_with_replay();
|
||||
assert_eq!(
|
||||
replay, b"abcd",
|
||||
"restored history is now older than the budget"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scrollback_carries_the_restored_prefix_and_the_ring() {
|
||||
let (_dir, bridge) = bridge(8, b"old").await;
|
||||
feed(&bridge, b"new").await;
|
||||
assert_eq!(bridge.scrollback(), b"oldnew");
|
||||
assert_eq!(bridge.bytes_seen(), 3);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_replay_budget_means_no_restored_scrollback() {
|
||||
let (_dir, bridge) = bridge(0, b"old").await;
|
||||
let (replay, _rx) = bridge.subscribe_with_replay();
|
||||
assert!(replay.is_empty());
|
||||
assert!(bridge.scrollback().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_buffer_keeps_last_cap_bytes() {
|
||||
let r = ReplayBuffer::new(4);
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
//! Persisted PTY scrollback (#171).
|
||||
//!
|
||||
//! The PTY replay ring lives in memory, so everything a session printed is
|
||||
//! gone the moment the daemon process is. A resumed session then opens on a
|
||||
//! blank pane: the agent's conversation is back, but the terminal history the
|
||||
//! operator was reading is not.
|
||||
//!
|
||||
//! This store mirrors each bridge's ring to `<state_dir>/scrollback/<id>.bin`,
|
||||
//! and hands it back when a session gets a new bridge — on resume, and on the
|
||||
//! recovery pass that re-adopts a session tmux still holds. The file is
|
||||
//! capped at the same `replay_bytes` budget the in-memory ring uses, so disk
|
||||
//! cost is bounded by the same number the operator already configures.
|
||||
//!
|
||||
//! Nothing here is allowed to fail a resume. The payload is raw PTY bytes
|
||||
//! with no framing to be corrupt, and a file that is missing, unreadable or
|
||||
//! oversized degrades to an empty (or truncated) pane rather than an error.
|
||||
|
||||
use std::io;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use uuid::Uuid;
|
||||
|
||||
/// Filesystem-backed scrollback store, rooted at a directory.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ScrollbackStore {
|
||||
dir: PathBuf,
|
||||
}
|
||||
|
||||
impl ScrollbackStore {
|
||||
pub fn new(dir: impl Into<PathBuf>) -> Self {
|
||||
Self { dir: dir.into() }
|
||||
}
|
||||
|
||||
pub fn dir(&self) -> &Path {
|
||||
&self.dir
|
||||
}
|
||||
|
||||
pub fn path_for(&self, id: Uuid) -> PathBuf {
|
||||
self.dir.join(format!("{id}.bin"))
|
||||
}
|
||||
|
||||
/// Atomically write `bytes` to `<dir>/<id>.bin`, creating the directory if
|
||||
/// needed. Written through a temp file and renamed, so a reboot in the
|
||||
/// middle of a flush leaves the previous scrollback intact rather than a
|
||||
/// half-file.
|
||||
pub fn write(&self, id: Uuid, bytes: &[u8]) -> io::Result<()> {
|
||||
std::fs::create_dir_all(&self.dir)?;
|
||||
let tmp = self.dir.join(format!("{id}.bin.tmp"));
|
||||
std::fs::write(&tmp, bytes)?;
|
||||
std::fs::rename(&tmp, self.path_for(id))
|
||||
}
|
||||
|
||||
/// The stored scrollback for `id`, capped to the last `budget` bytes.
|
||||
///
|
||||
/// Never fails: a missing file, an unreadable one, or a `budget` of zero
|
||||
/// all read as "no scrollback", which is exactly the blank pane the
|
||||
/// operator had before this existed.
|
||||
pub fn read(&self, id: Uuid, budget: usize) -> Vec<u8> {
|
||||
if budget == 0 {
|
||||
return Vec::new();
|
||||
}
|
||||
let path = self.path_for(id);
|
||||
let mut bytes = match std::fs::read(&path) {
|
||||
Ok(b) => b,
|
||||
Err(e) if e.kind() == io::ErrorKind::NotFound => return Vec::new(),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, %id, path = %path.display(), "scrollback: unreadable, resuming with a blank pane");
|
||||
return Vec::new();
|
||||
}
|
||||
};
|
||||
// A file written under a larger budget than this daemon now runs
|
||||
// with keeps its tail, the same end of the stream the ring would.
|
||||
if bytes.len() > budget {
|
||||
bytes.drain(..bytes.len() - budget);
|
||||
}
|
||||
bytes
|
||||
}
|
||||
|
||||
/// Delete the scrollback for `id`. No-op if it is not there.
|
||||
pub fn delete(&self, id: Uuid) -> io::Result<()> {
|
||||
match std::fs::remove_file(self.path_for(id)) {
|
||||
Ok(()) => Ok(()),
|
||||
Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()),
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn store() -> (tempfile::TempDir, ScrollbackStore) {
|
||||
let dir = tempfile::TempDir::new().expect("tempdir");
|
||||
let store = ScrollbackStore::new(dir.path().join("scrollback"));
|
||||
(dir, store)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn round_trips_bytes() {
|
||||
let (_dir, store) = store();
|
||||
let id = Uuid::new_v4();
|
||||
store.write(id, b"hello world").expect("write");
|
||||
assert_eq!(store.read(id, 64), b"hello world");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_file_reads_as_empty() {
|
||||
let (_dir, store) = store();
|
||||
assert!(store.read(Uuid::new_v4(), 64).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_file_over_budget_keeps_its_tail() {
|
||||
let (_dir, store) = store();
|
||||
let id = Uuid::new_v4();
|
||||
store.write(id, b"abcdefghij").expect("write");
|
||||
assert_eq!(store.read(id, 4), b"ghij");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_zero_budget_reads_as_empty() {
|
||||
let (_dir, store) = store();
|
||||
let id = Uuid::new_v4();
|
||||
store.write(id, b"abcdefghij").expect("write");
|
||||
assert!(store.read(id, 0).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delete_removes_the_file_and_tolerates_a_missing_one() {
|
||||
let (_dir, store) = store();
|
||||
let id = Uuid::new_v4();
|
||||
store.write(id, b"gone soon").expect("write");
|
||||
store.delete(id).expect("delete");
|
||||
assert!(!store.path_for(id).exists());
|
||||
store.delete(id).expect("second delete is a no-op");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_write_replaces_the_first() {
|
||||
let (_dir, store) = store();
|
||||
let id = Uuid::new_v4();
|
||||
store.write(id, b"first").expect("write");
|
||||
store.write(id, b"second").expect("rewrite");
|
||||
assert_eq!(store.read(id, 64), b"second");
|
||||
assert!(!store.dir().join(format!("{id}.bin.tmp")).exists());
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,7 @@ use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::Utc;
|
||||
use maestro_config::ResumeOnBoot;
|
||||
use maestro_protocol::{
|
||||
ActivityState, CatalogEvent, CreateSessionRequest, ErrorInfo, NameSource, Session,
|
||||
SessionStatus, WorktreeSpec, NAME_KEY, NAME_SOURCE_KEY,
|
||||
@@ -22,6 +23,7 @@ use uuid::Uuid;
|
||||
use crate::boot_id::BootIdSource;
|
||||
use crate::panes::Panes;
|
||||
use crate::pty::{pipe_pane_command, PtyBridge};
|
||||
use crate::scrollback::ScrollbackStore;
|
||||
use crate::sidecar::{SidecarError, SidecarStore};
|
||||
use crate::spawn_history::{RecentDir, SpawnHistory, SpawnHistoryRecord};
|
||||
use crate::tmux::TmuxError;
|
||||
@@ -43,6 +45,16 @@ pub const DEFAULT_LIVENESS_INTERVAL: Duration = Duration::from_secs(2);
|
||||
/// attempt, which is all #83 needs.
|
||||
pub const DEFAULT_BLOCKED_WORK_INTERVAL: Duration = Duration::from_secs(15);
|
||||
|
||||
/// How often the liveness watcher mirrors PTY scrollback to disk (#171).
|
||||
///
|
||||
/// A host reboot gives no warning and runs no exit path, so this interval is
|
||||
/// the whole guarantee: whatever a session printed in the last few seconds
|
||||
/// before the power went is what a resumed pane is missing. Ten seconds keeps
|
||||
/// that loss cosmetic — scrollback is visual continuity, not state — while
|
||||
/// costing at most one file write per *active* session per tick, since a
|
||||
/// session whose bridge read nothing is skipped entirely.
|
||||
pub const DEFAULT_SCROLLBACK_INTERVAL: Duration = Duration::from_secs(10);
|
||||
|
||||
/// Gap between the text of a `send` and the Enter that submits it.
|
||||
///
|
||||
/// Agents that run a full-screen prompt (Claude Code, opencode) coalesce a
|
||||
@@ -86,11 +98,20 @@ pub struct SessionRecord {
|
||||
/// (`POST /.../focus`), `None` while unfocused. Daemon-local; never rides
|
||||
/// the wire.
|
||||
pub focused_since: Option<std::time::Instant>,
|
||||
/// Set for as long as a [`SessionManager::resume`] is between its two
|
||||
/// takes of the map lock (#175). The spawn happens with the lock released,
|
||||
/// so the `Running` check alone lets a second request through; this is the
|
||||
/// state the second one is refused on. Daemon-local; never rides the wire.
|
||||
pub resuming: bool,
|
||||
/// Sticky acknowledgement applied by [`crate::activity::apply_focus_suppression`]:
|
||||
/// once the pane has been focused long enough to clear an `AwaitingInput`
|
||||
/// glow, this holds through focus loss until the session does new work.
|
||||
/// Daemon-local; never rides the wire.
|
||||
pub awaiting_suppressed: bool,
|
||||
/// [`PtyBridge::bytes_seen`] as of this session's last scrollback flush
|
||||
/// (#171). The flush pass skips a session whose bridge has read nothing
|
||||
/// since, so an idle session costs no disk writes.
|
||||
pub flushed_bytes: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
@@ -195,6 +216,15 @@ struct Inner {
|
||||
/// reboot detection is unwired, which behaves exactly like a boot id
|
||||
/// that could never be read: always fall back to probing tmux.
|
||||
boot_id: Option<BootIdConfig>,
|
||||
/// What recovery does with a session whose pane is gone (#167), and the
|
||||
/// presets an `auto` resume resolves its command from. Defaults to
|
||||
/// [`ResumeOnBoot::Dormant`] with no presets: register and wait.
|
||||
resume_policy: ResumeOnBoot,
|
||||
resume_presets: Vec<maestro_protocol::SpawnPreset>,
|
||||
/// Where PTY scrollback is mirrored so it survives the daemon, and the
|
||||
/// host (#171). `None` — the default — means scrollback is not persisted
|
||||
/// at all: a resumed session opens on a blank pane, exactly as before.
|
||||
scrollback: Option<ScrollbackStore>,
|
||||
}
|
||||
|
||||
/// Where recovery persists the boot id it last saw, and how it reads the
|
||||
@@ -253,6 +283,9 @@ impl SessionManager {
|
||||
viewports: Mutex::new(Viewports::new()),
|
||||
next_viewport: AtomicU64::new(0),
|
||||
boot_id: None,
|
||||
resume_policy: ResumeOnBoot::default(),
|
||||
resume_presets: Vec::new(),
|
||||
scrollback: None,
|
||||
}),
|
||||
}
|
||||
}
|
||||
@@ -290,6 +323,21 @@ impl SessionManager {
|
||||
/// # Panics
|
||||
///
|
||||
/// If the handle has already been cloned.
|
||||
/// Persist PTY scrollback under `dir` (#171), so a resumed session comes
|
||||
/// back showing what it printed before the daemon — or the host — went
|
||||
/// away. Builder, called once right after construction like the other
|
||||
/// `with_*` methods. Without it nothing is written, read or deleted.
|
||||
///
|
||||
/// # Panics
|
||||
///
|
||||
/// If the handle has already been cloned; see [`Self::with_spawn_history`].
|
||||
pub fn with_scrollback(mut self, dir: PathBuf) -> Self {
|
||||
let inner = Arc::get_mut(&mut self.inner)
|
||||
.expect("with_scrollback must be called before the SessionManager is cloned");
|
||||
inner.scrollback = Some(ScrollbackStore::new(dir));
|
||||
self
|
||||
}
|
||||
|
||||
pub fn with_boot_id(mut self, path: PathBuf, source: Arc<dyn BootIdSource>) -> Self {
|
||||
let inner = Arc::get_mut(&mut self.inner)
|
||||
.expect("with_boot_id must be called before the SessionManager is cloned");
|
||||
@@ -297,6 +345,26 @@ impl SessionManager {
|
||||
self
|
||||
}
|
||||
|
||||
/// Wire the `[daemon].resume_on_boot` policy (#167) and the spawn presets
|
||||
/// an `auto` resume resolves each session's command from. Builder, called
|
||||
/// once right after construction like the other `with_*` methods. Without
|
||||
/// it, recovery behaves as [`ResumeOnBoot::Dormant`].
|
||||
///
|
||||
/// # Panics
|
||||
///
|
||||
/// If the handle has already been cloned.
|
||||
pub fn with_resume_policy(
|
||||
mut self,
|
||||
policy: ResumeOnBoot,
|
||||
presets: Vec<maestro_protocol::SpawnPreset>,
|
||||
) -> Self {
|
||||
let inner = Arc::get_mut(&mut self.inner)
|
||||
.expect("with_resume_policy must be called before the SessionManager is cloned");
|
||||
inner.resume_policy = policy;
|
||||
inner.resume_presets = presets;
|
||||
self
|
||||
}
|
||||
|
||||
/// Set the daemon's own HTTP base URL (#103), exported to every session
|
||||
/// spawned afterwards as `MAESTRO_URL`. Unlike the `with_*` builders this
|
||||
/// is a plain setter: the daemon runner learns its bound address after
|
||||
@@ -567,6 +635,8 @@ impl SessionManager {
|
||||
naming: false,
|
||||
focused_since: None,
|
||||
awaiting_suppressed: false,
|
||||
resuming: false,
|
||||
flushed_bytes: 0,
|
||||
},
|
||||
);
|
||||
}
|
||||
@@ -618,15 +688,23 @@ impl SessionManager {
|
||||
id: Uuid,
|
||||
presets: &[maestro_protocol::SpawnPreset],
|
||||
) -> Result<Session, SessionError> {
|
||||
// The spawn below runs with the map lock released, so the `Running`
|
||||
// check alone is not enough to keep two resumes apart (#175): both
|
||||
// would pass it, both would open a `PtyBridge` on the same FIFO, and
|
||||
// the loser would report a tmux failure instead of the conflict it
|
||||
// is. The record is marked in-flight under this first lock, and every
|
||||
// path out of here clears it again.
|
||||
let mut session = {
|
||||
let sessions = self.inner.sessions.lock().await;
|
||||
let rec = sessions.get(&id).ok_or(SessionError::NotFound(id))?;
|
||||
if rec.session.status == SessionStatus::Running {
|
||||
let mut sessions = self.inner.sessions.lock().await;
|
||||
let rec = sessions.get_mut(&id).ok_or(SessionError::NotFound(id))?;
|
||||
if rec.session.status == SessionStatus::Running || rec.resuming {
|
||||
return Err(SessionError::AlreadyRunning(id));
|
||||
}
|
||||
rec.resuming = true;
|
||||
rec.session.clone()
|
||||
};
|
||||
if !session.cwd.exists() {
|
||||
self.clear_resuming(id).await;
|
||||
return Err(SessionError::CwdMissing(session.cwd.clone()));
|
||||
}
|
||||
let command = crate::resume::resume_command(&session, presets);
|
||||
@@ -634,8 +712,14 @@ impl SessionManager {
|
||||
// Same fifo path as the original spawn: it is derived from the id, and
|
||||
// `mkfifo` tolerates the stale node the previous run left behind.
|
||||
let fifo_path = self.inner.fifo_dir.join(format!("{id}.fifo"));
|
||||
let bridge =
|
||||
PtyBridge::start_with_replay(fifo_path.clone(), self.inner.replay_bytes).await?;
|
||||
// #171: the pane is new, but the scrollback the operator was reading
|
||||
// before the reboot is on disk. It goes in ahead of the live bytes.
|
||||
let bridge = PtyBridge::start_restoring(
|
||||
fifo_path.clone(),
|
||||
self.inner.replay_bytes,
|
||||
self.restored_scrollback(id),
|
||||
)
|
||||
.await?;
|
||||
let mut session_env = sanitize_env(&session.env)?;
|
||||
session_env.insert("MAESTRO_SESSION_ID".into(), id.to_string());
|
||||
if let Some(url) = self
|
||||
@@ -658,6 +742,7 @@ impl SessionManager {
|
||||
if let Err(e) = spawned {
|
||||
// `bridge` drops here, closing the FIFO reader. Nothing else was
|
||||
// touched, so the session stays exactly as dormant as it was.
|
||||
self.clear_resuming(id).await;
|
||||
return Err(SessionError::Tmux(e));
|
||||
}
|
||||
|
||||
@@ -677,6 +762,7 @@ impl SessionManager {
|
||||
// sidecar does not describe.
|
||||
let tmux = self.inner.tmux.clone();
|
||||
let _ = tmux_blocking(move || tmux.kill_session(id)).await;
|
||||
self.clear_resuming(id).await;
|
||||
return Err(SessionError::Sidecar(e));
|
||||
}
|
||||
|
||||
@@ -684,9 +770,17 @@ impl SessionManager {
|
||||
let mut sessions = self.inner.sessions.lock().await;
|
||||
let Some(rec) = sessions.get_mut(&id) else {
|
||||
// Killed while we were spawning. Undo the pane instead of
|
||||
// leaving one behind with no record of it.
|
||||
// leaving one behind with no record of it — and delete the
|
||||
// sidecar the write above just put back (#175). The kill
|
||||
// deleted it before that write landed, so leaving it would
|
||||
// have the next recovery register a dormant ghost for a
|
||||
// session that is gone and whose worktree is already removed.
|
||||
drop(sessions);
|
||||
let tmux = self.inner.tmux.clone();
|
||||
let _ = tmux_blocking(move || tmux.kill_session(id)).await;
|
||||
if let Err(e) = self.inner.sidecar.delete(id) {
|
||||
tracing::warn!(error = ?e, %id, "resume: failed to delete the sidecar of a session killed mid-resume");
|
||||
}
|
||||
return Err(SessionError::NotFound(id));
|
||||
};
|
||||
rec.session = session.clone();
|
||||
@@ -696,6 +790,7 @@ impl SessionManager {
|
||||
rec.settled = false;
|
||||
rec.focused_since = None;
|
||||
rec.awaiting_suppressed = false;
|
||||
rec.resuming = false;
|
||||
}
|
||||
let _ = self.inner.events.send(CatalogEvent::SessionUpdated {
|
||||
session: session.clone(),
|
||||
@@ -703,6 +798,53 @@ impl SessionManager {
|
||||
Ok(session)
|
||||
}
|
||||
|
||||
/// Delete the scrollback of a sidecar recovery threw away (#171): the
|
||||
/// session will never be listed again, so nothing would ever read it.
|
||||
fn prune_scrollback(&self, id: Uuid) {
|
||||
if let Some(store) = &self.inner.scrollback {
|
||||
if let Err(e) = store.delete(id) {
|
||||
tracing::warn!(error = ?e, %id, "recover: failed to prune scrollback");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Scrollback persisted for `id` by an earlier life of this session
|
||||
/// (#171), capped to the replay budget. Empty when scrollback is not
|
||||
/// wired, and empty — never an error — when the file is missing or
|
||||
/// unreadable: a blank pane is the honest fallback, and it is what the
|
||||
/// operator had before any of this existed.
|
||||
fn restored_scrollback(&self, id: Uuid) -> Vec<u8> {
|
||||
self.inner
|
||||
.scrollback
|
||||
.as_ref()
|
||||
.map(|store| store.read(id, self.inner.replay_bytes))
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Mirror every live bridge's scrollback to disk (#171).
|
||||
///
|
||||
/// Called on a timer by the liveness watcher rather than only on a clean
|
||||
/// exit, because the case this exists for — a host reboot — runs no exit
|
||||
/// path at all: anything flushed only on shutdown is missing exactly when
|
||||
/// it is needed. Sessions whose bridge has read nothing since the last
|
||||
/// flush are skipped, so an idle host writes nothing.
|
||||
///
|
||||
/// Best-effort throughout: a failed write is warned and the session keeps
|
||||
/// running.
|
||||
pub async fn flush_scrollback(&self) {
|
||||
run_scrollback_pass(&self.inner).await;
|
||||
}
|
||||
|
||||
/// Drop the in-flight marker [`Self::resume`] set under its first lock
|
||||
/// (#175), so a refused or failed resume can be retried. A no-op when the
|
||||
/// record is already gone: a kill that landed mid-resume took the marker
|
||||
/// with it.
|
||||
async fn clear_resuming(&self, id: Uuid) {
|
||||
if let Some(rec) = self.inner.sessions.lock().await.get_mut(&id) {
|
||||
rec.resuming = false;
|
||||
}
|
||||
}
|
||||
|
||||
/// Best-effort removal of a worktree created earlier in `create` when a
|
||||
/// later step fails. No-op when the request had no worktree.
|
||||
fn rollback_worktree(&self, base: &Option<PathBuf>, path: &Option<PathBuf>) {
|
||||
@@ -793,6 +935,10 @@ impl SessionManager {
|
||||
// Start "due" so the first live tick probes: a recovered daemon
|
||||
// should not wait a full interval to show what blocks.
|
||||
let mut since_probe = blocked_work_interval;
|
||||
// Scrollback starts *not* due: a daemon that just recovered has
|
||||
// nothing new to mirror, and the sessions it adopted were primed
|
||||
// from the files this would overwrite (#171).
|
||||
let mut since_flush = Duration::ZERO;
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
let Some(inner) = weak.upgrade() else {
|
||||
@@ -807,6 +953,11 @@ impl SessionManager {
|
||||
since_probe = Duration::ZERO;
|
||||
run_blocked_work_pass(&inner).await;
|
||||
}
|
||||
since_flush += interval;
|
||||
if since_flush >= DEFAULT_SCROLLBACK_INTERVAL {
|
||||
since_flush = Duration::ZERO;
|
||||
run_scrollback_pass(&inner).await;
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -819,20 +970,20 @@ impl SessionManager {
|
||||
/// that no longer exists are still adopted — the spec does not require
|
||||
/// us to repair them.
|
||||
///
|
||||
/// #164: when the kernel's boot id has changed since the last run, the
|
||||
/// tmux probe below is skipped entirely — a reboot kills the tmux server,
|
||||
/// so every previous session is already dead and there is nothing tmux
|
||||
/// could truthfully adopt. [`Self::register_dormant`] then registers all
|
||||
/// of them directly.
|
||||
/// #164/#174: the kernel's boot id says whether the host rebooted since
|
||||
/// the last run, but it never decides which sessions are alive —
|
||||
/// `tmux list-sessions` does, always, which is what the rest of the
|
||||
/// daemon assumes. A boot id can change without the tmux server dying
|
||||
/// (a restored state directory, a VM or container snapshot restore, a
|
||||
/// truncated `boot_id` file); skipping the probe then registered every
|
||||
/// live session `Dormant`, and resume could never bring one back because
|
||||
/// the tmux name was still taken. The boot id is now only a signal that a
|
||||
/// tmux server which did not answer is not coming back, and it names
|
||||
/// itself in recovery's log.
|
||||
pub async fn recover(&self) -> Result<usize, SessionError> {
|
||||
tokio::fs::create_dir_all(&self.inner.fifo_dir).await?;
|
||||
let rebooted = self.check_boot_id().await;
|
||||
let live = if rebooted {
|
||||
// #164: every session from the previous boot is provably dead —
|
||||
// no tmux server survives a reboot — so there is nothing to gain
|
||||
// from asking it and no session it could honestly report back.
|
||||
Vec::new()
|
||||
} else {
|
||||
let live = {
|
||||
let tmux = self.inner.tmux.clone();
|
||||
tmux_blocking(move || tmux.list_sessions()).await?
|
||||
};
|
||||
@@ -866,8 +1017,15 @@ impl SessionManager {
|
||||
},
|
||||
};
|
||||
let fifo_path = self.inner.fifo_dir.join(format!("{id}.fifo"));
|
||||
let bridge =
|
||||
PtyBridge::start_with_replay(fifo_path.clone(), self.inner.replay_bytes).await?;
|
||||
// The ring the previous daemon held died with it, so an adopted
|
||||
// session would otherwise replay nothing until tmux's own screen
|
||||
// capture — which is one screenful, not scrollback (#171).
|
||||
let bridge = PtyBridge::start_restoring(
|
||||
fifo_path.clone(),
|
||||
self.inner.replay_bytes,
|
||||
self.restored_scrollback(id),
|
||||
)
|
||||
.await?;
|
||||
// Re-pipe so output starts flowing again. `Tmux::pipe_pane` runs
|
||||
// without tmux's `-o` flag, so any stale pipe the previous daemon
|
||||
// left running to a now-unread FIFO is replaced (NAP-228).
|
||||
@@ -894,6 +1052,8 @@ impl SessionManager {
|
||||
naming: false,
|
||||
focused_since: None,
|
||||
awaiting_suppressed: false,
|
||||
resuming: false,
|
||||
flushed_bytes: 0,
|
||||
},
|
||||
);
|
||||
let _ = self
|
||||
@@ -912,7 +1072,9 @@ impl SessionManager {
|
||||
///
|
||||
/// A missing stored value (first run) or an unreadable current one
|
||||
/// (non-Linux, restricted `/proc`, unwired in this build) both read as
|
||||
/// "no reboot", which is today's behaviour: fall back to probing tmux.
|
||||
/// "no reboot". Either way the tmux probe runs (#174): the answer only
|
||||
/// tells recovery whether a tmux server that did not answer is coming
|
||||
/// back, never which sessions are alive.
|
||||
async fn check_boot_id(&self) -> bool {
|
||||
let Some(cfg) = &self.inner.boot_id else {
|
||||
return false;
|
||||
@@ -959,7 +1121,8 @@ impl SessionManager {
|
||||
/// `rebooted` (#164) only changes what gets logged: a detected reboot
|
||||
/// names the count even when it is zero, so the operator sees that
|
||||
/// recovery told a reboot from a restart instead of a log that stays
|
||||
/// silent either way.
|
||||
/// silent either way. It never decides membership — everything here got
|
||||
/// here by tmux not holding it (#174).
|
||||
async fn register_dormant(&self, rebooted: bool) {
|
||||
let stored = match self.inner.sidecar.load_all() {
|
||||
Ok(stored) => stored,
|
||||
@@ -970,6 +1133,7 @@ impl SessionManager {
|
||||
};
|
||||
let mut registered = Vec::new();
|
||||
let mut pruned = 0usize;
|
||||
let mut discarded = 0usize;
|
||||
{
|
||||
let mut sessions = self.inner.sessions.lock().await;
|
||||
for mut session in stored {
|
||||
@@ -981,9 +1145,21 @@ impl SessionManager {
|
||||
if let Err(e) = self.inner.sidecar.delete(session.id) {
|
||||
tracing::warn!(error = ?e, id = %session.id, "recover: failed to prune sidecar with missing cwd");
|
||||
}
|
||||
self.prune_scrollback(session.id);
|
||||
pruned += 1;
|
||||
continue;
|
||||
}
|
||||
if self.inner.resume_policy == ResumeOnBoot::Never {
|
||||
// #167: this host wants a reboot to be a clean slate, so
|
||||
// the sidecar goes the way #163's does — nothing is
|
||||
// registered and the next recovery finds nothing.
|
||||
if let Err(e) = self.inner.sidecar.delete(session.id) {
|
||||
tracing::warn!(error = ?e, id = %session.id, "recover: failed to prune sidecar under resume_on_boot=never");
|
||||
}
|
||||
self.prune_scrollback(session.id);
|
||||
discarded += 1;
|
||||
continue;
|
||||
}
|
||||
let id = session.id;
|
||||
session.status = SessionStatus::Dormant;
|
||||
session.foreground = None;
|
||||
@@ -1006,6 +1182,8 @@ impl SessionManager {
|
||||
naming: false,
|
||||
focused_since: None,
|
||||
awaiting_suppressed: false,
|
||||
resuming: false,
|
||||
flushed_bytes: 0,
|
||||
},
|
||||
);
|
||||
registered.push(session);
|
||||
@@ -1025,12 +1203,45 @@ impl SessionManager {
|
||||
if pruned > 0 {
|
||||
tracing::info!(pruned, "pruned sidecars whose cwd no longer exists");
|
||||
}
|
||||
let resumable: Vec<Uuid> = registered.iter().map(|s| s.id).collect();
|
||||
for session in registered {
|
||||
let _ = self
|
||||
.inner
|
||||
.events
|
||||
.send(CatalogEvent::SessionAdded { session });
|
||||
}
|
||||
|
||||
// #167: say which policy ran and how many sessions it acted on, so a
|
||||
// host that resumes agents unattended says so in its own log. The
|
||||
// match is exhaustive on purpose: a fourth policy must decide what it
|
||||
// does here rather than inherit whichever arm happened to be last.
|
||||
match self.inner.resume_policy {
|
||||
ResumeOnBoot::Dormant => tracing::info!(
|
||||
policy = ResumeOnBoot::Dormant.as_str(),
|
||||
sessions = resumable.len(),
|
||||
"resume_on_boot: sessions left dormant for the operator"
|
||||
),
|
||||
ResumeOnBoot::Never => tracing::info!(
|
||||
policy = ResumeOnBoot::Never.as_str(),
|
||||
sessions = discarded,
|
||||
"resume_on_boot: pruned the sidecars of sessions with no live pane"
|
||||
),
|
||||
ResumeOnBoot::Auto => {
|
||||
tracing::info!(
|
||||
policy = ResumeOnBoot::Auto.as_str(),
|
||||
sessions = resumable.len(),
|
||||
"resume_on_boot: resuming sessions with no live pane"
|
||||
);
|
||||
let presets = self.inner.resume_presets.clone();
|
||||
for id in resumable {
|
||||
// The same call the client's POST /api/sessions/:id/resume
|
||||
// makes — one resume path, not two.
|
||||
if let Err(e) = self.resume(id, &presets).await {
|
||||
tracing::warn!(error = %e, id = %id, "resume_on_boot=auto: resume failed; session stays dormant");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Forward keystrokes to a session's pane. A dead or dormant session is
|
||||
@@ -1193,10 +1404,12 @@ impl SessionManager {
|
||||
/// and stays silent on the event stream at steady state.
|
||||
///
|
||||
/// `agent_session_id` (#161) is the agent's own conversation id, parsed
|
||||
/// by the caller from the hook payload's `session_id` field. It is
|
||||
/// written once — the first report that carries one wins — and the
|
||||
/// sidecar is only rewritten on that transition, never on every report,
|
||||
/// since an agent's session id never changes mid-session.
|
||||
/// by the caller from the hook payload's `session_id` field. The
|
||||
/// sidecar is rewritten only when the reported id differs from the
|
||||
/// stored one — a repeated report of the same id is pure churn from the
|
||||
/// hook's `PreToolUse` cadence and writes nothing, but a resume (#176)
|
||||
/// issues a new conversation id and must replace the stale one, or
|
||||
/// every later resume reattaches the pre-resume conversation.
|
||||
pub async fn report_activity(
|
||||
&self,
|
||||
id: Uuid,
|
||||
@@ -1218,16 +1431,16 @@ impl SessionManager {
|
||||
state,
|
||||
self.inner.report_ttl,
|
||||
));
|
||||
let newly_identified =
|
||||
rec.session.agent_session_id.is_none() && agent_session_id.is_some();
|
||||
if newly_identified {
|
||||
let id_changed =
|
||||
agent_session_id.is_some() && rec.session.agent_session_id != agent_session_id;
|
||||
if id_changed {
|
||||
rec.session.agent_session_id = agent_session_id;
|
||||
self.inner
|
||||
.sidecar
|
||||
.write(&rec.session)
|
||||
.map_err(SessionError::Sidecar)?;
|
||||
}
|
||||
(changed || newly_identified).then(|| rec.session.clone())
|
||||
(changed || id_changed).then(|| rec.session.clone())
|
||||
};
|
||||
if let Some(session) = updated {
|
||||
let _ = self
|
||||
@@ -1610,6 +1823,36 @@ async fn run_blocked_work_pass(inner: &Arc<Inner>) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Mirror every live bridge's scrollback to disk. See
|
||||
/// [`SessionManager::flush_scrollback`].
|
||||
async fn run_scrollback_pass(inner: &Arc<Inner>) {
|
||||
let Some(store) = inner.scrollback.clone() else {
|
||||
return;
|
||||
};
|
||||
// Snapshot under the lock, write with it released: the writes are
|
||||
// filesystem work and every attach needs this map.
|
||||
let pending: Vec<(Uuid, Vec<u8>)> = {
|
||||
let mut sessions = inner.sessions.lock().await;
|
||||
sessions
|
||||
.iter_mut()
|
||||
.filter_map(|(id, rec)| {
|
||||
let bridge = rec.bridge.as_ref()?;
|
||||
let seen = bridge.bytes_seen();
|
||||
if seen == rec.flushed_bytes {
|
||||
return None;
|
||||
}
|
||||
rec.flushed_bytes = seen;
|
||||
Some((*id, bridge.scrollback()))
|
||||
})
|
||||
.collect()
|
||||
};
|
||||
for (id, bytes) in pending {
|
||||
if let Err(e) = store.write(id, &bytes) {
|
||||
tracing::warn!(error = ?e, %id, "scrollback: flush failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Mark a session whose tmux session has disappeared as [`SessionStatus::Dead`]
|
||||
/// and settle its worktree per the natural-exit rule (#82).
|
||||
///
|
||||
@@ -1636,6 +1879,15 @@ async fn mark_session_dead(inner: &Arc<Inner>, id: Uuid) {
|
||||
// attach.
|
||||
if let Some(bridge) = record.bridge.as_mut() {
|
||||
bridge.seal();
|
||||
// The clean-exit half of #171: the last thing the session
|
||||
// printed is what a later resume should open on, and it
|
||||
// will produce nothing more for the flush pass to notice.
|
||||
if let Some(store) = &inner.scrollback {
|
||||
if let Err(e) = store.write(id, &bridge.scrollback()) {
|
||||
tracing::warn!(error = ?e, %id, "scrollback: final flush failed");
|
||||
}
|
||||
record.flushed_bytes = bridge.bytes_seen();
|
||||
}
|
||||
}
|
||||
let session = record.session.clone();
|
||||
let worktree = record
|
||||
@@ -1757,6 +2009,14 @@ async fn cleanup_record(
|
||||
if let Err(e) = inner.sidecar.delete(id) {
|
||||
tracing::warn!(error = ?e, %id, "cleanup: failed to delete sidecar");
|
||||
}
|
||||
// #171: the session is gone, so its scrollback goes with it — nothing
|
||||
// will ever read the file again, and leaving it would keep a killed
|
||||
// session's output on disk indefinitely.
|
||||
if let Some(store) = &inner.scrollback {
|
||||
if let Err(e) = store.delete(id) {
|
||||
tracing::warn!(error = ?e, %id, "cleanup: failed to delete scrollback");
|
||||
}
|
||||
}
|
||||
if let (Some(base), Some(path)) = (&record.worktree_base_repo, &record.worktree_path) {
|
||||
if keep_worktree {
|
||||
tracing::info!(
|
||||
|
||||
@@ -187,10 +187,30 @@ async fn a_reported_session_id_is_recorded_on_the_session() {
|
||||
assert_eq!(s.agent_session_id.as_deref(), Some("conv-abc123"));
|
||||
}
|
||||
|
||||
/// #161: a session id does not change mid-session, so once set it is never
|
||||
/// overwritten by a later report — even one that names a different id.
|
||||
/// #161: repeated reports of the same id are pure churn from the hook's
|
||||
/// `PreToolUse` cadence — the sidecar must not be rewritten for them.
|
||||
#[tokio::test]
|
||||
async fn a_reported_session_id_is_written_once_and_never_replaced() {
|
||||
async fn a_repeated_report_of_the_same_session_id_writes_nothing() {
|
||||
let fx = fixture(Duration::from_secs(300)).await;
|
||||
let id = spawn_sleeping(&fx.manager).await;
|
||||
|
||||
fx.manager
|
||||
.report_activity(id, ActivityState::Active, None, Some("conv-first".into()))
|
||||
.await
|
||||
.unwrap();
|
||||
fx.manager
|
||||
.report_activity(id, ActivityState::Active, None, Some("conv-first".into()))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let s = observed(&fx.manager, id).await;
|
||||
assert_eq!(s.agent_session_id.as_deref(), Some("conv-first"));
|
||||
}
|
||||
|
||||
/// #176: a resume issues a new conversation id. The stored id must move to
|
||||
/// it, or every later resume reattaches the stale pre-resume conversation.
|
||||
#[tokio::test]
|
||||
async fn a_reported_session_id_is_replaced_when_it_differs() {
|
||||
let fx = fixture(Duration::from_secs(300)).await;
|
||||
let id = spawn_sleeping(&fx.manager).await;
|
||||
|
||||
@@ -204,7 +224,14 @@ async fn a_reported_session_id_is_written_once_and_never_replaced() {
|
||||
.unwrap();
|
||||
|
||||
let s = observed(&fx.manager, id).await;
|
||||
assert_eq!(s.agent_session_id.as_deref(), Some("conv-first"));
|
||||
assert_eq!(s.agent_session_id.as_deref(), Some("conv-second"));
|
||||
|
||||
let stored = fx
|
||||
.sidecar
|
||||
.read(id)
|
||||
.expect("sidecar read")
|
||||
.expect("sidecar present");
|
||||
assert_eq!(stored.agent_session_id.as_deref(), Some("conv-second"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -151,27 +151,40 @@ pub struct DaemonFixture {
|
||||
|
||||
impl DaemonFixture {
|
||||
pub fn new() -> Self {
|
||||
Self::build(None)
|
||||
Self::build(None, maestro_daemon::pty::DEFAULT_REPLAY_BYTES)
|
||||
}
|
||||
|
||||
/// Variant whose manager runs a custom PTY replay budget (NAP-229), which
|
||||
/// also caps the scrollback it persists (#171).
|
||||
pub fn new_with_replay(replay_bytes: usize) -> Self {
|
||||
Self::build(None, replay_bytes)
|
||||
}
|
||||
|
||||
/// Variant whose manager has a durable spawn history (#24) attached, with
|
||||
/// `retention_days` recency cap, logging to `<state_dir>/spawn-history.jsonl`.
|
||||
pub fn new_with_history(retention_days: u32) -> Self {
|
||||
Self::build(Some(retention_days))
|
||||
Self::build(
|
||||
Some(retention_days),
|
||||
maestro_daemon::pty::DEFAULT_REPLAY_BYTES,
|
||||
)
|
||||
}
|
||||
|
||||
fn build(history_days: Option<u32>) -> Self {
|
||||
fn build(history_days: Option<u32>, replay_bytes: usize) -> Self {
|
||||
let tmux = TmuxFixture::new();
|
||||
let state_dir = TempDir::new().unwrap();
|
||||
let sidecar = SidecarStore::new(state_dir.path().join("sessions"));
|
||||
// Default placement: worktrees land under their base repo (#56).
|
||||
let worktree_mgr = WorktreeManager::new(None);
|
||||
let mut manager = SessionManager::new(
|
||||
let mut manager = SessionManager::new_with_replay(
|
||||
Arc::new(tmux.tmux.clone()),
|
||||
sidecar,
|
||||
worktree_mgr,
|
||||
state_dir.path().join("fifos"),
|
||||
replay_bytes,
|
||||
);
|
||||
// #171: production wires this, so the fixtures do too — every test
|
||||
// then exercises the same flush/restore/delete paths the daemon runs.
|
||||
manager = manager.with_scrollback(state_dir.path().join("scrollback"));
|
||||
if let Some(days) = history_days {
|
||||
manager =
|
||||
manager.with_spawn_history(SpawnHistory::new(Self::history_path(&state_dir), days));
|
||||
@@ -198,4 +211,8 @@ impl DaemonFixture {
|
||||
pub fn fifo_dir(&self) -> PathBuf {
|
||||
self.state_dir.path().join("fifos")
|
||||
}
|
||||
|
||||
pub fn scrollback_dir(&self) -> PathBuf {
|
||||
self.state_dir.path().join("scrollback")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -357,3 +357,237 @@ async fn post_resume_on_an_unknown_session_is_404() {
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), reqwest::StatusCode::NOT_FOUND);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Races (#175)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Real tmux, with the *first* `new_session_piped` held inside the call until
|
||||
/// the test releases it. Later calls pass straight through, so a second resume
|
||||
/// racing the first never blocks on a gate the first one owns.
|
||||
///
|
||||
/// This is what makes the two races below deterministic rather than timing
|
||||
/// dependent: the resume under test is parked in exactly the window the bug
|
||||
/// lives in, and the racing operation runs to completion before it moves.
|
||||
#[derive(Debug)]
|
||||
struct GatedSpawn {
|
||||
inner: maestro_daemon::tmux::Tmux,
|
||||
entered: tokio::sync::mpsc::UnboundedSender<()>,
|
||||
release: std::sync::Mutex<std::sync::mpsc::Receiver<()>>,
|
||||
gated: std::sync::atomic::AtomicBool,
|
||||
}
|
||||
|
||||
/// The test's end of a [`GatedSpawn`].
|
||||
struct Gate {
|
||||
entered: tokio::sync::mpsc::UnboundedReceiver<()>,
|
||||
release: std::sync::mpsc::SyncSender<()>,
|
||||
}
|
||||
|
||||
impl Gate {
|
||||
/// Wait until the gated spawn is parked mid-resume.
|
||||
async fn wait_entered(&mut self) {
|
||||
tokio::time::timeout(Duration::from_secs(10), self.entered.recv())
|
||||
.await
|
||||
.expect("resume reached the tmux spawn")
|
||||
.expect("gate alive");
|
||||
}
|
||||
|
||||
fn release(&self) {
|
||||
self.release.send(()).expect("gate receiver alive");
|
||||
}
|
||||
}
|
||||
|
||||
fn gated_panes(inner: maestro_daemon::tmux::Tmux) -> (Arc<GatedSpawn>, Gate) {
|
||||
let (entered_tx, entered_rx) = tokio::sync::mpsc::unbounded_channel();
|
||||
let (release_tx, release_rx) = std::sync::mpsc::sync_channel(1);
|
||||
(
|
||||
Arc::new(GatedSpawn {
|
||||
inner,
|
||||
entered: entered_tx,
|
||||
release: std::sync::Mutex::new(release_rx),
|
||||
gated: std::sync::atomic::AtomicBool::new(true),
|
||||
}),
|
||||
Gate {
|
||||
entered: entered_rx,
|
||||
release: release_tx,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
impl maestro_daemon::panes::Panes for GatedSpawn {
|
||||
fn new_session_piped(
|
||||
&self,
|
||||
id: Uuid,
|
||||
cwd: &std::path::Path,
|
||||
command: &str,
|
||||
session_env: &std::collections::BTreeMap<String, String>,
|
||||
sink_command: &str,
|
||||
) -> Result<(), maestro_daemon::tmux::TmuxError> {
|
||||
if self.gated.swap(false, std::sync::atomic::Ordering::SeqCst) {
|
||||
let _ = self.entered.send(());
|
||||
let _ = self.release.lock().expect("gate mutex").recv();
|
||||
}
|
||||
self.inner
|
||||
.new_session_piped(id, cwd, command, session_env, sink_command)
|
||||
}
|
||||
|
||||
fn kill_session(&self, id: Uuid) -> Result<(), maestro_daemon::tmux::TmuxError> {
|
||||
self.inner.kill_session(id)
|
||||
}
|
||||
|
||||
fn list_sessions(&self) -> Result<Vec<Uuid>, maestro_daemon::tmux::TmuxError> {
|
||||
self.inner.list_sessions()
|
||||
}
|
||||
|
||||
fn pane_info(
|
||||
&self,
|
||||
id: Uuid,
|
||||
) -> Result<Option<maestro_daemon::tmux::PaneInfo>, maestro_daemon::tmux::TmuxError> {
|
||||
self.inner.pane_info(id)
|
||||
}
|
||||
|
||||
fn send_keys(&self, id: Uuid, bytes: &[u8]) -> Result<(), maestro_daemon::tmux::TmuxError> {
|
||||
self.inner.send_keys(id, bytes)
|
||||
}
|
||||
|
||||
fn pipe_pane(&self, id: Uuid, sink: &str) -> Result<(), maestro_daemon::tmux::TmuxError> {
|
||||
self.inner.pipe_pane(id, sink)
|
||||
}
|
||||
|
||||
fn resize_window(
|
||||
&self,
|
||||
id: Uuid,
|
||||
cols: u16,
|
||||
rows: u16,
|
||||
) -> Result<(), maestro_daemon::tmux::TmuxError> {
|
||||
self.inner.resize_window(id, cols, rows)
|
||||
}
|
||||
|
||||
fn capture_pane(&self, id: Uuid) -> Result<Option<Vec<u8>>, maestro_daemon::tmux::TmuxError> {
|
||||
self.inner.capture_pane(id)
|
||||
}
|
||||
}
|
||||
|
||||
/// A dormant session whose manager's tmux spawn can be parked by the test.
|
||||
async fn dormant_gated(fx: &DaemonFixture, cwd: &TempDir) -> (SessionManager, Session, Gate) {
|
||||
let session = fx
|
||||
.manager
|
||||
.create(CreateSessionRequest {
|
||||
cwd: cwd.path().to_path_buf(),
|
||||
command: "sleep 30".into(),
|
||||
env: HashMap::new(),
|
||||
metadata: HashMap::new(),
|
||||
worktree: None,
|
||||
})
|
||||
.await
|
||||
.expect("create");
|
||||
fx.tmux.tmux.kill_server().expect("kill tmux server");
|
||||
|
||||
let (panes, gate) = gated_panes(fx.tmux.tmux.clone());
|
||||
let manager = SessionManager::new(
|
||||
panes,
|
||||
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir()),
|
||||
maestro_daemon::worktree::WorktreeManager::new(None),
|
||||
fx.fifo_dir(),
|
||||
);
|
||||
manager.recover().await.expect("recover");
|
||||
assert_eq!(
|
||||
manager.status(session.id).await,
|
||||
Some(SessionStatus::Dormant)
|
||||
);
|
||||
(manager, session, gate)
|
||||
}
|
||||
|
||||
/// A kill that lands while a resume is inside the tmux spawn must not leave
|
||||
/// the sidecar behind. The kill deletes it; the resume writes it back a moment
|
||||
/// later and then discovers its own record is gone. The sidecar it resurrected
|
||||
/// would otherwise make the next recovery register a ghost dormant session for
|
||||
/// a session that was killed.
|
||||
#[tokio::test]
|
||||
async fn a_kill_during_resume_leaves_no_sidecar_behind() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let (manager, session, mut gate) = dormant_gated(&fx, &cwd).await;
|
||||
let id = session.id;
|
||||
|
||||
let resuming = tokio::spawn({
|
||||
let manager = manager.clone();
|
||||
async move { manager.resume(id, &[]).await }
|
||||
});
|
||||
gate.wait_entered().await;
|
||||
|
||||
manager.kill(id, false).await.expect("kill mid-resume");
|
||||
gate.release();
|
||||
|
||||
let err = resuming.await.unwrap().expect_err("record is gone");
|
||||
assert!(matches!(err, SessionError::NotFound(got) if got == id));
|
||||
|
||||
assert!(
|
||||
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir())
|
||||
.read(id)
|
||||
.expect("read sidecar")
|
||||
.is_none(),
|
||||
"the killed session's sidecar must not be resurrected by the losing resume"
|
||||
);
|
||||
|
||||
// And so the next recovery registers no ghost.
|
||||
let after = revive(&fx);
|
||||
after.recover().await.expect("recover");
|
||||
assert!(
|
||||
after.snapshot().await.is_empty(),
|
||||
"a killed session must not come back dormant"
|
||||
);
|
||||
}
|
||||
|
||||
/// Two resumes of the same session must not both proceed. The second is
|
||||
/// refused on the state — `already_running` — not on its tmux spawn colliding
|
||||
/// with the first one's.
|
||||
#[tokio::test]
|
||||
async fn a_second_resume_is_refused_while_the_first_is_in_flight() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let (manager, session, mut gate) = dormant_gated(&fx, &cwd).await;
|
||||
let id = session.id;
|
||||
|
||||
let resuming = tokio::spawn({
|
||||
let manager = manager.clone();
|
||||
async move { manager.resume(id, &[]).await }
|
||||
});
|
||||
gate.wait_entered().await;
|
||||
|
||||
let err = manager
|
||||
.resume(id, &[])
|
||||
.await
|
||||
.expect_err("a resume is already in flight");
|
||||
assert!(matches!(err, SessionError::AlreadyRunning(got) if got == id));
|
||||
assert_eq!(err.to_error_info().code, "already_running");
|
||||
|
||||
gate.release();
|
||||
let first = resuming.await.unwrap().expect("the first resume wins");
|
||||
assert_eq!(first.status, SessionStatus::Running);
|
||||
assert_eq!(
|
||||
fx.tmux.tmux.list_sessions().expect("list"),
|
||||
vec![id],
|
||||
"exactly one pane for the session"
|
||||
);
|
||||
}
|
||||
|
||||
/// A refused resume must not leave the session marked in-flight: once the
|
||||
/// reason for the refusal is gone, resuming works.
|
||||
#[tokio::test]
|
||||
async fn a_refused_resume_can_be_retried() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let (manager, session) = dormant_session(&fx, &cwd, "sleep 30", "retried").await;
|
||||
|
||||
std::fs::remove_dir_all(cwd.path()).unwrap();
|
||||
let err = manager
|
||||
.resume(session.id, &[])
|
||||
.await
|
||||
.expect_err("cwd is gone");
|
||||
assert_eq!(err.to_error_info().code, "cwd_missing");
|
||||
|
||||
std::fs::create_dir_all(cwd.path()).unwrap();
|
||||
let resumed = manager.resume(session.id, &[]).await.expect("resume");
|
||||
assert_eq!(resumed.status, SessionStatus::Running);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
//! `[daemon].resume_on_boot` (#167): what recovery does with sessions whose
|
||||
//! tmux pane is gone but whose sidecar survived, against real tmux.
|
||||
//!
|
||||
//! The default is deliberately the timid one. Resuming agents unattended
|
||||
//! seconds after a reboot is a foot-gun, so `auto` is opted into per host and
|
||||
//! `never` throws the sidecars away instead.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use common::{wait_for_async, DaemonFixture};
|
||||
use maestro_config::ResumeOnBoot;
|
||||
use maestro_daemon::SessionManager;
|
||||
use maestro_protocol::{CreateSessionRequest, Session, SessionStatus, SpawnPreset};
|
||||
use tempfile::TempDir;
|
||||
|
||||
/// A fresh manager over the same tmux server, sidecars and fifos — what the
|
||||
/// daemon builds after a restart — carrying `policy`.
|
||||
fn revive(fx: &DaemonFixture, policy: ResumeOnBoot, presets: Vec<SpawnPreset>) -> SessionManager {
|
||||
SessionManager::new(
|
||||
Arc::new(fx.tmux.tmux.clone()),
|
||||
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir()),
|
||||
maestro_daemon::worktree::WorktreeManager::new(None),
|
||||
fx.fifo_dir(),
|
||||
)
|
||||
.with_resume_policy(policy, presets)
|
||||
}
|
||||
|
||||
/// A preset whose `resume_command` touches a marker file, so a test can see
|
||||
/// that the resumed session ran the resume command rather than the original.
|
||||
fn marker_preset(command: &str, marker_dir: &TempDir) -> Vec<SpawnPreset> {
|
||||
vec![SpawnPreset {
|
||||
id: "agent".into(),
|
||||
label: "Agent".into(),
|
||||
command: command.into(),
|
||||
resume_command: Some(format!(
|
||||
"sh -c 'touch {}/resumed; sleep 30'",
|
||||
marker_dir.path().display()
|
||||
)),
|
||||
}]
|
||||
}
|
||||
|
||||
async fn create(fx: &DaemonFixture, cwd: &TempDir, command: &str) -> Session {
|
||||
fx.manager
|
||||
.create(CreateSessionRequest {
|
||||
cwd: cwd.path().to_path_buf(),
|
||||
command: command.into(),
|
||||
env: HashMap::new(),
|
||||
metadata: HashMap::new(),
|
||||
worktree: None,
|
||||
})
|
||||
.await
|
||||
.expect("create")
|
||||
}
|
||||
|
||||
/// The default: a reboot leaves the session listed but not running, waiting
|
||||
/// for the operator.
|
||||
#[tokio::test]
|
||||
async fn dormant_is_the_default_and_leaves_the_session_waiting() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let session = create(&fx, &cwd, "sleep 30").await;
|
||||
fx.tmux.tmux.kill_server().expect("kill tmux server");
|
||||
|
||||
let revived = revive(&fx, ResumeOnBoot::Dormant, Vec::new());
|
||||
revived.recover().await.expect("recover");
|
||||
|
||||
assert_eq!(
|
||||
revived.status(session.id).await,
|
||||
Some(SessionStatus::Dormant)
|
||||
);
|
||||
assert!(fx
|
||||
.sidecar_dir()
|
||||
.join(format!("{}.json", session.id))
|
||||
.exists());
|
||||
assert!(
|
||||
fx.tmux.tmux.list_sessions().expect("list").is_empty(),
|
||||
"nothing was respawned"
|
||||
);
|
||||
}
|
||||
|
||||
/// `auto` brings each one back itself, down the same path a client's resume
|
||||
/// takes: same uuid, same sidecar, the preset's `resume_command`.
|
||||
#[tokio::test]
|
||||
async fn auto_resumes_every_dormant_session_during_recovery() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let markers = TempDir::new().unwrap();
|
||||
let session = create(&fx, &cwd, "sleep 30").await;
|
||||
fx.tmux.tmux.kill_server().expect("kill tmux server");
|
||||
|
||||
let revived = revive(&fx, ResumeOnBoot::Auto, marker_preset("sleep 30", &markers));
|
||||
revived.recover().await.expect("recover");
|
||||
|
||||
assert_eq!(
|
||||
revived.status(session.id).await,
|
||||
Some(SessionStatus::Running),
|
||||
"auto resumed it"
|
||||
);
|
||||
assert_eq!(
|
||||
fx.tmux.tmux.list_sessions().expect("list"),
|
||||
vec![session.id],
|
||||
"back under the same uuid"
|
||||
);
|
||||
let stored = maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir())
|
||||
.read(session.id)
|
||||
.expect("read sidecar")
|
||||
.expect("same sidecar");
|
||||
assert_eq!(stored.created_at, session.created_at);
|
||||
assert_eq!(stored.command, "sleep 30", "the original command is kept");
|
||||
|
||||
// It really ran the preset's resume command, not the spawn command.
|
||||
let marker = markers.path().join("resumed");
|
||||
wait_for_async(Duration::from_secs(5), || {
|
||||
let marker = marker.clone();
|
||||
async move { marker.exists() }
|
||||
})
|
||||
.await;
|
||||
assert!(marker.exists(), "resume_command ran");
|
||||
}
|
||||
|
||||
/// `never` makes a reboot a clean slate: the sidecars go, and the session is
|
||||
/// not listed at all.
|
||||
#[tokio::test]
|
||||
async fn never_prunes_the_sidecars_instead() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let session = create(&fx, &cwd, "sleep 30").await;
|
||||
fx.tmux.tmux.kill_server().expect("kill tmux server");
|
||||
|
||||
let revived = revive(&fx, ResumeOnBoot::Never, Vec::new());
|
||||
revived.recover().await.expect("recover");
|
||||
|
||||
assert!(!revived.has(session.id).await, "not listed");
|
||||
assert!(
|
||||
!fx.sidecar_dir()
|
||||
.join(format!("{}.json", session.id))
|
||||
.exists(),
|
||||
"sidecar pruned"
|
||||
);
|
||||
assert!(fx.tmux.tmux.list_sessions().expect("list").is_empty());
|
||||
}
|
||||
|
||||
/// The policy only decides what happens to sessions that lost their pane. A
|
||||
/// session tmux still holds is adopted running by the first pass under every
|
||||
/// policy — including `never`, which must not delete a live session's sidecar.
|
||||
#[tokio::test]
|
||||
async fn the_policy_leaves_sessions_tmux_still_holds_alone() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let live = create(&fx, &cwd, "sleep 30").await;
|
||||
|
||||
let revived = revive(&fx, ResumeOnBoot::Never, Vec::new());
|
||||
assert_eq!(revived.recover().await.expect("recover"), 1);
|
||||
|
||||
assert_eq!(revived.status(live.id).await, Some(SessionStatus::Running));
|
||||
assert!(fx.sidecar_dir().join(format!("{}.json", live.id)).exists());
|
||||
}
|
||||
@@ -0,0 +1,219 @@
|
||||
//! Persisted PTY scrollback across a reboot (#171), against real tmux.
|
||||
//!
|
||||
//! Everything else in the reboot milestone restores what a session *is*; this
|
||||
//! restores what it *looked like*. The cases that matter are the ones a clean
|
||||
//! shutdown never gets to run: the host went down, and the only scrollback
|
||||
//! that exists is whatever the last periodic flush wrote.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use common::{wait_for_async, DaemonFixture};
|
||||
use maestro_daemon::scrollback::ScrollbackStore;
|
||||
use maestro_daemon::SessionManager;
|
||||
use maestro_protocol::{CreateSessionRequest, Session, SessionStatus};
|
||||
use tempfile::TempDir;
|
||||
use uuid::Uuid;
|
||||
|
||||
/// A fresh manager over the same tmux server, sidecars, fifos and scrollback —
|
||||
/// what the daemon builds after a restart.
|
||||
fn revive(fx: &DaemonFixture) -> SessionManager {
|
||||
SessionManager::new(
|
||||
Arc::new(fx.tmux.tmux.clone()),
|
||||
maestro_daemon::sidecar::SidecarStore::new(fx.sidecar_dir()),
|
||||
maestro_daemon::worktree::WorktreeManager::new(None),
|
||||
fx.fifo_dir(),
|
||||
)
|
||||
.with_scrollback(fx.scrollback_dir())
|
||||
}
|
||||
|
||||
async fn spawn(fx: &DaemonFixture, cwd: &TempDir, command: &str) -> Session {
|
||||
fx.manager
|
||||
.create(CreateSessionRequest {
|
||||
cwd: cwd.path().to_path_buf(),
|
||||
command: command.into(),
|
||||
env: HashMap::new(),
|
||||
metadata: HashMap::new(),
|
||||
worktree: None,
|
||||
})
|
||||
.await
|
||||
.expect("create")
|
||||
}
|
||||
|
||||
/// Wait until the session has printed `needle`, flushing as we go — the same
|
||||
/// thing the liveness watcher's flush pass does on its timer.
|
||||
async fn wait_for_flushed(
|
||||
manager: &SessionManager,
|
||||
store: &ScrollbackStore,
|
||||
id: Uuid,
|
||||
needle: &str,
|
||||
) {
|
||||
wait_for_async(Duration::from_secs(10), || async {
|
||||
manager.flush_scrollback().await;
|
||||
String::from_utf8_lossy(&store.read(id, 64 * 1024)).contains(needle)
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
fn replay_text(replay: &[u8]) -> String {
|
||||
String::from_utf8_lossy(replay).to_string()
|
||||
}
|
||||
|
||||
/// The keystone: the host reboots with no exit path run at all, and the
|
||||
/// resumed session opens showing what it printed before.
|
||||
#[tokio::test]
|
||||
async fn a_resumed_session_replays_the_scrollback_from_before_the_reboot() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let store = ScrollbackStore::new(fx.scrollback_dir());
|
||||
|
||||
let session = spawn(&fx, &cwd, "sh -c 'echo before-the-reboot; sleep 30'").await;
|
||||
wait_for_flushed(&fx.manager, &store, session.id, "before-the-reboot").await;
|
||||
|
||||
// The reboot: tmux dies where it stands. No kill, no seal, no shutdown
|
||||
// hook — only whatever the periodic flush had already written.
|
||||
fx.tmux.tmux.kill_server().expect("kill tmux server");
|
||||
let manager = revive(&fx);
|
||||
manager.recover().await.expect("recover");
|
||||
assert_eq!(
|
||||
manager.status(session.id).await,
|
||||
Some(SessionStatus::Dormant)
|
||||
);
|
||||
|
||||
manager.resume(session.id, &[]).await.expect("resume");
|
||||
let (replay, _rx) = manager
|
||||
.subscribe_pty_with_replay(session.id)
|
||||
.await
|
||||
.expect("attachable again");
|
||||
assert!(
|
||||
replay_text(&replay).contains("before-the-reboot"),
|
||||
"resumed pane should open on the old scrollback, got {:?}",
|
||||
replay_text(&replay)
|
||||
);
|
||||
}
|
||||
|
||||
/// A daemon restart leaves tmux alone, so the session is adopted rather than
|
||||
/// resumed — and it too lost its in-memory ring with the old process.
|
||||
#[tokio::test]
|
||||
async fn an_adopted_session_replays_the_scrollback_of_the_previous_daemon() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let store = ScrollbackStore::new(fx.scrollback_dir());
|
||||
|
||||
let session = spawn(&fx, &cwd, "sh -c 'echo printed-earlier; sleep 30'").await;
|
||||
wait_for_flushed(&fx.manager, &store, session.id, "printed-earlier").await;
|
||||
|
||||
// Daemon restart: the manager goes, tmux stays.
|
||||
let manager = revive(&fx);
|
||||
manager.recover().await.expect("recover");
|
||||
assert_eq!(
|
||||
manager.status(session.id).await,
|
||||
Some(SessionStatus::Running)
|
||||
);
|
||||
|
||||
let (replay, _rx) = manager
|
||||
.subscribe_pty_with_replay(session.id)
|
||||
.await
|
||||
.expect("attachable");
|
||||
assert!(
|
||||
replay_text(&replay).contains("printed-earlier"),
|
||||
"adopted pane should carry the old scrollback ahead of tmux's screen, got {:?}",
|
||||
replay_text(&replay)
|
||||
);
|
||||
}
|
||||
|
||||
/// Killing a session takes its scrollback with it: nothing will ever read the
|
||||
/// file again, and the operator's output should not linger on disk.
|
||||
#[tokio::test]
|
||||
async fn killing_a_session_deletes_its_scrollback() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let store = ScrollbackStore::new(fx.scrollback_dir());
|
||||
|
||||
let session = spawn(&fx, &cwd, "sh -c 'echo doomed; sleep 30'").await;
|
||||
wait_for_flushed(&fx.manager, &store, session.id, "doomed").await;
|
||||
assert!(store.path_for(session.id).exists());
|
||||
|
||||
fx.manager.kill(session.id, false).await.expect("kill");
|
||||
assert!(
|
||||
!store.path_for(session.id).exists(),
|
||||
"scrollback should not outlive the session"
|
||||
);
|
||||
}
|
||||
|
||||
/// A session that exits on its own does get an exit path, and takes it: the
|
||||
/// last thing it printed is flushed when the bridge is sealed, with no wait
|
||||
/// for the next tick.
|
||||
#[tokio::test]
|
||||
async fn a_session_that_exits_flushes_its_last_output() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let store = ScrollbackStore::new(fx.scrollback_dir());
|
||||
|
||||
let session = spawn(&fx, &cwd, "sh -c 'echo last-words'").await;
|
||||
fx.manager
|
||||
.start_liveness_watcher(Duration::from_millis(100));
|
||||
wait_for_async(Duration::from_secs(10), || async {
|
||||
fx.manager.status(session.id).await == Some(SessionStatus::Dead)
|
||||
})
|
||||
.await;
|
||||
|
||||
assert!(
|
||||
String::from_utf8_lossy(&store.read(session.id, 64 * 1024)).contains("last-words"),
|
||||
"a sealed bridge should have flushed what it printed"
|
||||
);
|
||||
}
|
||||
|
||||
/// Nothing here may fail a resume. A scrollback file that is gone, or one full
|
||||
/// of bytes no terminal would emit, degrades to a pane with no history — not
|
||||
/// to an error.
|
||||
#[tokio::test]
|
||||
async fn a_missing_or_corrupt_scrollback_still_resumes() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let store = ScrollbackStore::new(fx.scrollback_dir());
|
||||
|
||||
let gone = spawn(&fx, &cwd, "sleep 30").await;
|
||||
let garbled = spawn(&fx, &cwd, "sleep 30").await;
|
||||
fx.manager.flush_scrollback().await;
|
||||
|
||||
fx.tmux.tmux.kill_server().expect("kill tmux server");
|
||||
std::fs::create_dir_all(store.dir()).expect("scrollback dir");
|
||||
std::fs::remove_file(store.path_for(gone.id)).ok();
|
||||
std::fs::write(store.path_for(garbled.id), [0xff, 0xfe, 0x00, 0x80]).expect("write garbage");
|
||||
|
||||
let manager = revive(&fx);
|
||||
manager.recover().await.expect("recover");
|
||||
for id in [gone.id, garbled.id] {
|
||||
manager.resume(id, &[]).await.expect("resume");
|
||||
assert_eq!(manager.status(id).await, Some(SessionStatus::Running));
|
||||
assert!(manager.subscribe_pty_with_replay(id).await.is_some());
|
||||
}
|
||||
}
|
||||
|
||||
/// The disk cost is the replay budget, not a second one of its own.
|
||||
#[tokio::test]
|
||||
async fn the_flushed_file_stays_inside_the_replay_budget() {
|
||||
let fx = DaemonFixture::new_with_replay(256);
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let store = ScrollbackStore::new(fx.scrollback_dir());
|
||||
|
||||
let session = spawn(
|
||||
&fx,
|
||||
&cwd,
|
||||
"sh -c 'for i in $(seq 1 400); do echo line-$i; done; sleep 30'",
|
||||
)
|
||||
.await;
|
||||
wait_for_flushed(&fx.manager, &store, session.id, "line-400").await;
|
||||
|
||||
let size = std::fs::metadata(store.path_for(session.id))
|
||||
.expect("stat scrollback")
|
||||
.len();
|
||||
assert!(
|
||||
size <= 256,
|
||||
"scrollback file was {size} bytes, budget is 256"
|
||||
);
|
||||
}
|
||||
@@ -326,12 +326,14 @@ fn revive_with_boot_id(
|
||||
)
|
||||
}
|
||||
|
||||
/// #164: a changed boot id means every session from the previous boot is
|
||||
/// already dead, so recovery must not trust tmux to say so — it skips the
|
||||
/// probe outright. Proven by leaving the tmux session alive: if the probe
|
||||
/// ran, it would come back `Running`, not `Dormant`.
|
||||
/// #174: a boot id can change without the tmux server dying — a restored
|
||||
/// state directory, a VM or container snapshot restore, a truncated
|
||||
/// `boot_id` file. `tmux list-sessions` stays the single source of truth for
|
||||
/// liveness, so a session tmux still holds is adopted `Running` no matter
|
||||
/// what the boot id says. Registering it dormant instead stranded it: resume
|
||||
/// would collide with the tmux name still in use and fail forever.
|
||||
#[tokio::test]
|
||||
async fn recover_skips_tmux_probe_when_boot_id_changed() {
|
||||
async fn recover_adopts_live_session_when_boot_id_changed() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let s = fx
|
||||
@@ -349,15 +351,15 @@ async fn recover_skips_tmux_probe_when_boot_id_changed() {
|
||||
let boot_id_path = fx.state_dir.path().join("boot_id");
|
||||
std::fs::write(&boot_id_path, "previous-boot").unwrap();
|
||||
|
||||
// tmux session is still alive — a probe would find it if one ran.
|
||||
// tmux session is still alive: the probe runs and finds it.
|
||||
let revived = revive_with_boot_id(&fx, boot_id_path.clone(), Some("current-boot"));
|
||||
let adopted = revived.recover().await.expect("recover");
|
||||
assert_eq!(adopted, 0, "reboot detected: nothing adopted live");
|
||||
assert_eq!(adopted, 1, "tmux still holds the session; adopt it");
|
||||
|
||||
let snap = revived.snapshot().await;
|
||||
assert_eq!(snap.len(), 1);
|
||||
assert_eq!(snap[0].id, s.id);
|
||||
assert_eq!(snap[0].status, SessionStatus::Dormant);
|
||||
assert_eq!(snap[0].status, SessionStatus::Running);
|
||||
|
||||
// The current boot id is now the stored one, for the next comparison.
|
||||
assert_eq!(
|
||||
@@ -366,6 +368,38 @@ async fn recover_skips_tmux_probe_when_boot_id_changed() {
|
||||
);
|
||||
}
|
||||
|
||||
/// #164, still true after #174: a real reboot takes the tmux server with it,
|
||||
/// so the probe comes back empty and every sidecar is registered dormant.
|
||||
#[tokio::test]
|
||||
async fn recover_registers_dormant_when_boot_id_changed_and_tmux_gone() {
|
||||
let fx = DaemonFixture::new();
|
||||
let cwd = TempDir::new().unwrap();
|
||||
let s = fx
|
||||
.manager
|
||||
.create(CreateSessionRequest {
|
||||
cwd: cwd.path().to_path_buf(),
|
||||
command: "sleep 30".into(),
|
||||
env: HashMap::new(),
|
||||
metadata: HashMap::new(),
|
||||
worktree: None,
|
||||
})
|
||||
.await
|
||||
.expect("create");
|
||||
|
||||
let boot_id_path = fx.state_dir.path().join("boot_id");
|
||||
std::fs::write(&boot_id_path, "previous-boot").unwrap();
|
||||
fx.tmux.tmux.kill_server().expect("kill tmux server");
|
||||
|
||||
let revived = revive_with_boot_id(&fx, boot_id_path, Some("current-boot"));
|
||||
let adopted = revived.recover().await.expect("recover");
|
||||
assert_eq!(adopted, 0, "no tmux server survived the reboot");
|
||||
|
||||
let snap = revived.snapshot().await;
|
||||
assert_eq!(snap.len(), 1);
|
||||
assert_eq!(snap[0].id, s.id);
|
||||
assert_eq!(snap[0].status, SessionStatus::Dormant);
|
||||
}
|
||||
|
||||
/// #164: an unchanged boot id is a plain restart — recovery probes tmux as
|
||||
/// before and adopts the still-live session `Running`.
|
||||
#[tokio::test]
|
||||
|
||||
@@ -243,6 +243,26 @@ Pick `/tmp` if the daemon must outlive your login session and you can't run
|
||||
tmux sockets with it). The env var `MAESTRO_DAEMON_TMUX_TMPDIR` takes
|
||||
precedence over the file.
|
||||
|
||||
## What happens to sessions after a reboot
|
||||
|
||||
A host reboot kills the tmux server, so every session the daemon knew is dead
|
||||
while its record on disk survives. What recovery does with those is
|
||||
`[daemon].resume_on_boot`:
|
||||
|
||||
```toml
|
||||
[daemon]
|
||||
resume_on_boot = "dormant" # "dormant" (default) | "auto" | "never"
|
||||
```
|
||||
|
||||
- `dormant` — the sessions are listed but not running, and you resume the ones
|
||||
you want from the web UI or the TUI.
|
||||
- `auto` — the daemon resumes them all itself during startup. Only pick this
|
||||
on a host where restarting every agent unattended is safe: they come back on
|
||||
the flags they were spawned with, with nobody watching.
|
||||
- `never` — the records are deleted and a reboot is a clean slate.
|
||||
|
||||
See `docs/RESTART.md` for the full contract.
|
||||
|
||||
## What the systemd unit looks like
|
||||
|
||||
The wizard writes a *user* unit (`systemctl --user`), since `maestro daemon`
|
||||
|
||||
+72
-2
@@ -75,7 +75,13 @@ A daemon restart leaves tmux alone, so recovery re-adopts every
|
||||
beyond the reconnect above.
|
||||
|
||||
A **host reboot** is different: it kills the tmux server, so there is nothing
|
||||
to adopt while every session's sidecar is still on disk. Recovery makes a
|
||||
to adopt while every session's sidecar is still on disk. Recovery always asks
|
||||
tmux — `tmux list-sessions` is the single source of truth for liveness, and the
|
||||
kernel's `boot_id` only tells a reboot from a restart in the log (#164, #174).
|
||||
A boot id changes without the tmux server dying whenever the state directory is
|
||||
restored, a VM or container snapshot is rolled back, or the `boot_id` file is
|
||||
truncated; a session tmux still holds is adopted `Running` in every one of
|
||||
those cases. Recovery makes a
|
||||
second pass over those sidecars and registers each as a `Dormant` session
|
||||
(#162) — id, name, cwd, command, worktree and `created_at` intact. A dormant
|
||||
session has no pane and no PTY bridge: it cannot be attached to, typed into or
|
||||
@@ -92,10 +98,74 @@ conversation id is respawned on the command it was created with. `Session::comma
|
||||
always keeps that original command, so a second resume resolves the same preset.
|
||||
Resume is refused with `409` for a session that is already running
|
||||
(`already_running`) and for one whose `cwd` has disappeared (`cwd_missing`) —
|
||||
neither is something a respawn could fix. The other thing to do with a dormant
|
||||
neither is something a respawn could fix. A session with a resume already in
|
||||
flight is refused as `already_running` too (#175): the respawn happens with
|
||||
the session map unlocked, so the record is marked in-flight before it starts
|
||||
and the second request is answered on that state rather than on its tmux spawn
|
||||
colliding with the first one's. A resume that loses a race with a kill takes
|
||||
its own sidecar back off disk, so the killed session does not return as a
|
||||
dormant ghost on the next recovery. The other thing to do with a dormant
|
||||
session is dismiss it, which removes the sidecar so the next recovery does not
|
||||
bring it back.
|
||||
|
||||
## What the pane looks like when it comes back
|
||||
|
||||
The PTY replay ring lives in memory, so a resumed session would otherwise open
|
||||
on a blank pane: the conversation is back, the terminal history the operator
|
||||
was reading is not. Each bridge's ring is mirrored to
|
||||
`<state_dir>/scrollback/<id>.bin` (#171) and handed to the new bridge on
|
||||
resume, and on the recovery pass that re-adopts a session tmux still holds —
|
||||
that one lost its ring with the previous daemon too. The restored bytes go out
|
||||
ahead of the live ones, and are dropped once the ring holds a full budget of
|
||||
new output of its own, which is the point where everything restored is older
|
||||
than the budget covers.
|
||||
|
||||
The file is capped at the same `[daemon] pty_replay_bytes` budget the ring
|
||||
uses — one budget, not two — and it is deleted when the session is killed, and
|
||||
when recovery prunes its sidecar. Nothing here can fail a resume: a missing,
|
||||
unreadable or oversized file degrades to an empty pane, which is exactly what
|
||||
the operator had before it existed.
|
||||
|
||||
It is flushed every 10 seconds by the liveness watcher, and once more when a
|
||||
session's bridge is sealed on exit. The timer is the part that matters: a host
|
||||
reboot runs no exit path at all, so scrollback written only on a clean shutdown
|
||||
would be missing in the one case this exists for. Ten seconds bounds the loss
|
||||
to the last few lines — scrollback is visual continuity, not state — and costs
|
||||
at most one file write per *active* session per tick, since a session whose
|
||||
bridge has read nothing since the last flush is skipped.
|
||||
|
||||
## What recovery does with them: `resume_on_boot`
|
||||
|
||||
Whether a recovered-but-dead session is left dormant at all is the operator's
|
||||
call, per host, via `[daemon].resume_on_boot` (#167):
|
||||
|
||||
| value | what recovery does |
|
||||
| ----------- | --------------------------------------------------------------------- |
|
||||
| `"dormant"` | **Default.** Register each one dormant and wait for the operator. |
|
||||
| `"auto"` | Resume each one during recovery, itself. |
|
||||
| `"never"` | Prune the sidecars instead — a reboot is a clean slate. |
|
||||
|
||||
`dormant` is the default because resuming agents unattended is a foot-gun:
|
||||
they come back on whatever flags they were spawned with, seconds after a
|
||||
reboot, with nobody watching. `auto` exists for hosts where that is known to
|
||||
be safe, and it is opted into rather than inherited.
|
||||
|
||||
`auto` calls the same `resume` the client's `POST /api/sessions/:id/resume`
|
||||
does — same uuid, same sidecar, the preset's `resume_command` — so there is
|
||||
one resume path and not two. A session that fails to resume is logged and
|
||||
stays dormant; the rest still come back.
|
||||
|
||||
`never` prunes the same way #163 does for a missing `cwd`: the sidecar is
|
||||
deleted and nothing is registered, so the session is not listed and the next
|
||||
recovery does not find it. Whatever worktree it owned is left on disk.
|
||||
|
||||
The policy only decides the fate of sessions with no live pane. A session
|
||||
tmux still holds is adopted `Running` by the first pass under every policy,
|
||||
including `never`.
|
||||
|
||||
Recovery logs the policy that ran and the number of sessions it acted on, so
|
||||
a host that resumes agents unattended says so in its own journal.
|
||||
|
||||
Attaching to a session that has no bridge answers `409 not_attachable`, not
|
||||
`404 unknown_session`: the session exists, and a client that cannot tell "gone"
|
||||
from "not running" cannot offer the resume.
|
||||
|
||||
Reference in New Issue
Block a user