feat(nightshift): calibrated quota verdict + scoped milestone checks
usage-window.py stores a calibrated window quota and emits an explicit verdict, so a run can no longer park on an uncalibrated raw token count. Milestone assess is scoped; full suite moves to the gated push points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YUXS63P1WCdC6bEKWcnAE
This commit is contained in:
@@ -41,7 +41,7 @@ Then fan out. Agents implementing against frozen signatures can run in parallel
|
|||||||
Repeat until done or stopped:
|
Repeat until done or stopped:
|
||||||
|
|
||||||
1. **Check the limit** (see below). Park if close.
|
1. **Check the limit** (see below). Park if close.
|
||||||
2. **Assess** — run the project's gate (typecheck + lint + tests). Read what landed since last time.
|
2. **Assess** — typecheck + lint, plus tests **for what changed this milestone**. Read what landed since last time. The full suite is a push-time check, not a milestone habit — see "Local verification budget" in COMMON.md; you are sharing this machine with other autonomous sessions and running everything every milestone is what OOMs it.
|
||||||
3. **Decide** — one architectural decision, written down. If nothing needs deciding, you are done; say so.
|
3. **Decide** — one architectural decision, written down. If nothing needs deciding, you are done; say so.
|
||||||
4. **Decompose** into subtasks with **disjoint file ownership**.
|
4. **Decompose** into subtasks with **disjoint file ownership**.
|
||||||
5. **Delegate** — subagents, in parallel where files don't collide.
|
5. **Delegate** — subagents, in parallel where files don't collide.
|
||||||
@@ -59,16 +59,43 @@ python3 <skills-root>/nightshift/usage-window.py --json # machine
|
|||||||
|
|
||||||
It sums billable tokens across **every Claude Code session on this machine** — all projects, not just yours, since parallel sessions in other repos share the same window.
|
It sums billable tokens across **every Claude Code session on this machine** — all projects, not just yours, since parallel sessions in other repos share the same window.
|
||||||
|
|
||||||
**It is a floor, never a ceiling.** Nothing local states the real quota, and the script cannot see Claude Code on another machine, claude.ai web usage, or direct API calls. Real usage is whatever it reports *plus* however much the account is being used elsewhere. If the user works across several machines, treat a comfortable reading with suspicion and take a wider margin.
|
### Act on the percentage. Never on the raw number.
|
||||||
|
|
||||||
Because of that gap, the reactive backstop matters as much as the estimate: **an agent that dies on a limit error is telling you the truth this script only guesses at.** Believe it immediately, and park — do not retry into the wall.
|
**A token count is a numerator.** On its own it means nothing. "4.8 million" sounds enormous and can be 30% of the window.
|
||||||
|
|
||||||
How to act on it:
|
The script prints an explicit `VERDICT`. Use it:
|
||||||
|
|
||||||
- **Window looks heavy** (well into millions, and climbing fast across recent milestones): finish what's running, commit, push, and **park**. Do not dispatch new agents.
|
| Verdict | Do |
|
||||||
- **Parking** = `ScheduleWakeup` for the time `oldestAgesOutInSeconds` reports, plus a margin. Sleeping until the window loosens is strictly better than having three agents killed halfway through their tasks.
|
|---|---|
|
||||||
- **Before parking, always**: commit, push (parking is one of the deliberate push points, see §8), and write the current state and the next intended step into the log. The run must be resumable by a different session that has none of your context.
|
| `clear` (<60%) | Dispatch freely. The window refills — unused capacity is wasted, not saved. |
|
||||||
|
| `wrap-up` (60–80%) | Keep working. Prefer shorter tasks over long fan-outs, so nothing large is in flight if it tightens. |
|
||||||
|
| `park` (≥80%) | Finish what's running, commit, push, write the next step down, park. |
|
||||||
|
| `unknown` | **No quota is calibrated.** Do NOT infer one from magnitude. See below. |
|
||||||
|
|
||||||
|
**If the verdict is `unknown`, the correct response is to ask, not to guess.** Ask the user what percentage Claude Code reports (`/status`), then:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 <skills-root>/nightshift/usage-window.py --calibrate 30
|
||||||
|
```
|
||||||
|
|
||||||
|
That stores the implied quota in `~/.claude/nightshift-quota.json` and every later run gets a real percentage. It takes one question. Until then you are flying blind and must not park on a hunch.
|
||||||
|
|
||||||
|
> **This section exists because it went wrong.** A run parked itself at what turned out to be 30% of the window, having reasoned that ~4.8M tokens was "well into millions and climbing" — the exact phrasing this skill used to contain. Hours of remaining capacity went unused and the user had to intervene. An unanchored magnitude threshold is not conservatism, it is a made-up number. Parking early is not free: it wastes the window the run was given.
|
||||||
|
|
||||||
|
### Both directions are failures
|
||||||
|
|
||||||
|
Backing off too late kills agents mid-task. Backing off too early wastes the run. Neither is the safe default — the whole point of measuring is to avoid *guessing* in either direction.
|
||||||
|
|
||||||
|
**It is a floor, never a ceiling.** The script cannot see Claude Code on another machine, claude.ai web usage, or direct API calls. Real usage is whatever it reports *plus* however much the account is being used elsewhere. If the user works across several machines, treat a comfortable reading with suspicion — but treat it with suspicion by *asking* or by widening the margin a little, not by inventing a threshold.
|
||||||
|
|
||||||
|
Because of that gap, the reactive backstop matters as much as the estimate: **an agent that dies on a limit error is telling you the truth this script only estimates.** Believe it immediately, and park — do not retry into the wall.
|
||||||
|
|
||||||
|
Other rules:
|
||||||
|
|
||||||
|
- **Parking** = `ScheduleWakeup` for the time `oldestAgesOutInSeconds` reports, plus a margin. Sleeping until the window loosens beats having three agents killed halfway through their tasks.
|
||||||
|
- **Before parking, always**: commit, push (parking is one of the deliberate push points, see §8), and write the current state and the next intended step into the log. The run must be resumable by a different session with none of your context.
|
||||||
- **If an agent dies on a limit error anyway**: do not immediately retry. Check the tree still passes the gate, commit whatever is green with a message stating plainly that it is **unverified** and what was left half-done, then park.
|
- **If an agent dies on a limit error anyway**: do not immediately retry. Check the tree still passes the gate, commit whatever is green with a message stating plainly that it is **unverified** and what was left half-done, then park.
|
||||||
|
- **If the user is present, say the number and let them decide.** They can see the real figure; you cannot. A one-line "window at 31%, continuing" costs nothing and catches a miscalibration immediately.
|
||||||
- Never *silently* burn the window to zero. If the user is asleep, they will wake to a stalled run and no explanation.
|
- Never *silently* burn the window to zero. If the user is asleep, they will wake to a stalled run and no explanation.
|
||||||
|
|
||||||
Scale the check to the work: a run doing small mechanical tasks needs it rarely; a run fanning out three heavy agents per milestone needs it every time.
|
Scale the check to the work: a run doing small mechanical tasks needs it rarely; a run fanning out three heavy agents per milestone needs it every time.
|
||||||
@@ -83,6 +110,8 @@ What actually works, learned the hard way:
|
|||||||
- **Demand verification the task can actually support.** "Tests pass" is not enough for anything a human will look at or listen to. Require a screenshot, a measured number, a browser run. Say plainly when something can only be verified by a human.
|
- **Demand verification the task can actually support.** "Tests pass" is not enough for anything a human will look at or listen to. Require a screenshot, a measured number, a browser run. Say plainly when something can only be verified by a human.
|
||||||
- **Model choice**: strongest model for design-heavy or feel-critical work; a cheaper one is fine for mechanical, well-specified changes.
|
- **Model choice**: strongest model for design-heavy or feel-critical work; a cheaper one is fine for mechanical, well-specified changes.
|
||||||
- Instruct them to **commit their own work locally** when it's coherent, so a killed agent loses less — and explicitly **not to push**. A dozen subagent pushes is a dozen CI runs on half-finished work.
|
- Instruct them to **commit their own work locally** when it's coherent, so a killed agent loses less — and explicitly **not to push**. A dozen subagent pushes is a dozen CI runs on half-finished work.
|
||||||
|
- **Tell them not to run the full suite.** Scoped checks on the files they own, nothing more. Five agents each running every test is five copies of the same work and enough memory pressure to kill the run. You run the full suite once, at push time, through `gate.sh`.
|
||||||
|
- **Cap the fan-out at 3 concurrent subagents, 2 if their tasks compile or test.** More agents is not more throughput on a box this size — it is swap. `<skills-root>/linear-common/scripts/gate.sh --status` shows how much of the machine other sessions are already using; dispatch fewer when it is contended, and remember other `/yolo` and `/nightshift` runs are competing for the same RAM.
|
||||||
|
|
||||||
## 6. Reviewing what lands
|
## 6. Reviewing what lands
|
||||||
|
|
||||||
@@ -110,6 +139,7 @@ This is what makes an overnight run reviewable by a human who slept through it.
|
|||||||
- All work goes on **one branch** in the worktree. Subtasks commit to it **locally**.
|
- All work goes on **one branch** in the worktree. Subtasks commit to it **locally**.
|
||||||
- **Push is a deliberate act, not a milestone habit.** Every push runs CI, and a night of milestone pushes is a night of CI runs on work that was half-finished at the time — noisy, expensive, and it trains the user to ignore the build.
|
- **Push is a deliberate act, not a milestone habit.** Every push runs CI, and a night of milestone pushes is a night of CI runs on work that was half-finished at the time — noisy, expensive, and it trains the user to ignore the build.
|
||||||
- **Push when:** the run finishes, you park on a limit, or the user asks. That's it.
|
- **Push when:** the run finishes, you park on a limit, or the user asks. That's it.
|
||||||
|
- **Before each of those pushes**, run the full suite once through the gate: `<skills-root>/linear-common/scripts/gate.sh -- <buildCommand>`. Exit 75 means the machine was busy and it never ran — push and say so plainly in the PR body under what is unverified. Exit 137 is the memory cap, not a failing test.
|
||||||
- **Then open the PR** describing what landed, what is unverified, what you decided and why, and what needs a human. The build log (§7) is most of that text already.
|
- **Then open the PR** describing what landed, what is unverified, what you decided and why, and what needs a human. The build log (§7) is most of that text already.
|
||||||
|
|
||||||
Forge-agnostic:
|
Forge-agnostic:
|
||||||
|
|||||||
@@ -7,6 +7,11 @@ block, in ~/.claude/projects/**/*.jsonl. Summing those over a rolling window
|
|||||||
gives a usable proxy: not the quota, but a consistent measure of how hard the
|
gives a usable proxy: not the quota, but a consistent measure of how hard the
|
||||||
window has been worked, plus when the oldest tokens in it age out.
|
window has been worked, plus when the oldest tokens in it age out.
|
||||||
|
|
||||||
|
A RAW TOKEN COUNT IS NOT A SIGNAL. It is a numerator. "4.8 million" sounds
|
||||||
|
alarming and means nothing without the denominator — on a large plan that can
|
||||||
|
be 30% of the window. Decisions about backing off MUST be made on percentage,
|
||||||
|
which means the quota has to be calibrated at least once. See --calibrate.
|
||||||
|
|
||||||
SCOPE — read this before trusting the number:
|
SCOPE — read this before trusting the number:
|
||||||
|
|
||||||
Counted: every Claude Code session on THIS machine, all projects, not
|
Counted: every Claude Code session on THIS machine, all projects, not
|
||||||
@@ -29,6 +34,14 @@ Usage:
|
|||||||
usage-window.py # human summary
|
usage-window.py # human summary
|
||||||
usage-window.py --json # machine readable
|
usage-window.py --json # machine readable
|
||||||
usage-window.py --hours 5 # window size (default 5)
|
usage-window.py --hours 5 # window size (default 5)
|
||||||
|
usage-window.py --quota 16e6 # one-off quota override
|
||||||
|
usage-window.py --calibrate 30 # "Claude Code says I'm at 30%" -> store implied quota
|
||||||
|
|
||||||
|
Calibration:
|
||||||
|
Ask the user what percentage their /status (or the Claude Code UI) reports,
|
||||||
|
then run --calibrate with it. The implied quota is written to
|
||||||
|
~/.claude/nightshift-quota.json and used by every later run. Re-calibrate
|
||||||
|
if the plan changes. One data point beats zero; two beats one.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -40,6 +53,14 @@ from datetime import datetime, timedelta, timezone
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
PROJECTS = Path.home() / ".claude" / "projects"
|
PROJECTS = Path.home() / ".claude" / "projects"
|
||||||
|
QUOTA_FILE = Path.home() / ".claude" / "nightshift-quota.json"
|
||||||
|
|
||||||
|
# Percentage bands. These are about RISK TO IN-FLIGHT WORK, not thrift — the
|
||||||
|
# window refills continuously, and an unused window is wasted capacity, not
|
||||||
|
# saved money. The only thing being avoided is agents dying mid-task.
|
||||||
|
BAND_CLEAR = 60.0 # below this: dispatch freely
|
||||||
|
BAND_WRAP = 80.0 # below this: keep going, prefer shorter tasks
|
||||||
|
# at or above BAND_WRAP: finish what's running, commit, push, park
|
||||||
|
|
||||||
|
|
||||||
def parse_ts(value: str | None) -> datetime | None:
|
def parse_ts(value: str | None) -> datetime | None:
|
||||||
@@ -65,6 +86,52 @@ def billable(usage: dict) -> int:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_quota() -> tuple[int | None, str | None]:
|
||||||
|
"""Calibrated window quota, if the user has ever supplied one.
|
||||||
|
|
||||||
|
Precedence: --quota (handled by caller) > env > stored file > unknown.
|
||||||
|
Returns (quota, source) so the output can say where the number came from —
|
||||||
|
an uncalibrated guess must never masquerade as a measurement.
|
||||||
|
"""
|
||||||
|
env = os.environ.get("NIGHTSHIFT_WINDOW_QUOTA")
|
||||||
|
if env:
|
||||||
|
try:
|
||||||
|
return int(float(env)), "env NIGHTSHIFT_WINDOW_QUOTA"
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
data = json.loads(QUOTA_FILE.read_text(encoding="utf-8"))
|
||||||
|
quota = int(data["windowTokenQuota"])
|
||||||
|
at = data.get("calibratedAt", "unknown date")
|
||||||
|
return quota, f"calibrated {at[:10]}"
|
||||||
|
except (OSError, KeyError, ValueError, TypeError):
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
def save_quota(quota: int, observed_tokens: int, observed_pct: float) -> None:
|
||||||
|
QUOTA_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
QUOTA_FILE.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"windowTokenQuota": quota,
|
||||||
|
"calibratedAt": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"observedTokens": observed_tokens,
|
||||||
|
"observedPercent": observed_pct,
|
||||||
|
"note": (
|
||||||
|
"Implied quota = observedTokens / (observedPercent/100). "
|
||||||
|
"Derived from what Claude Code reported at one moment, so it "
|
||||||
|
"inherits this script's blind spots (other machines, web, API). "
|
||||||
|
"Re-run --calibrate after a plan change or if parking decisions "
|
||||||
|
"start feeling wrong."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
+ "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def collect(hours: float) -> dict:
|
def collect(hours: float) -> dict:
|
||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
cutoff = now - timedelta(hours=hours)
|
cutoff = now - timedelta(hours=hours)
|
||||||
@@ -140,25 +207,86 @@ def collect(hours: float) -> dict:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def add_verdict(report: dict, quota: int | None, source: str | None) -> dict:
|
||||||
|
"""Attach percent-of-quota and an explicit act-on-this verdict.
|
||||||
|
|
||||||
|
Without a calibrated quota there is NO verdict — deliberately. An earlier
|
||||||
|
version of this tool emitted only a raw count, and the reading agent
|
||||||
|
invented a magnitude threshold to compare it against, then parked a run at
|
||||||
|
what turned out to be 30% of the window. Saying "unknown" is strictly better
|
||||||
|
than handing back a number that invites a made-up denominator.
|
||||||
|
"""
|
||||||
|
report["quota"] = quota
|
||||||
|
report["quotaSource"] = source
|
||||||
|
if not quota or quota <= 0:
|
||||||
|
report["percentUsed"] = None
|
||||||
|
report["verdict"] = "unknown"
|
||||||
|
report["verdictReason"] = (
|
||||||
|
"No calibrated quota. Percentage is unknown, so the raw token count says "
|
||||||
|
"NOTHING about how close the window is. Do not park on magnitude. "
|
||||||
|
"Ask the user what percent Claude Code reports, then run --calibrate."
|
||||||
|
)
|
||||||
|
return report
|
||||||
|
|
||||||
|
pct = 100.0 * report["tokens"] / quota
|
||||||
|
report["percentUsed"] = round(pct, 1)
|
||||||
|
if pct < BAND_CLEAR:
|
||||||
|
report["verdict"] = "clear"
|
||||||
|
report["verdictReason"] = "Dispatch freely. The window refills; unused capacity is wasted."
|
||||||
|
elif pct < BAND_WRAP:
|
||||||
|
report["verdict"] = "wrap-up"
|
||||||
|
report["verdictReason"] = (
|
||||||
|
"Keep working, but prefer shorter tasks over long fan-outs so nothing "
|
||||||
|
"large is in flight if the window tightens."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
report["verdict"] = "park"
|
||||||
|
report["verdictReason"] = (
|
||||||
|
"Finish what is running, commit, push, write the next step down, and park."
|
||||||
|
)
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
def human(report: dict) -> str:
|
def human(report: dict) -> str:
|
||||||
if "error" in report:
|
if "error" in report:
|
||||||
return f"usage: unavailable ({report['error']})"
|
return f"usage: unavailable ({report['error']})"
|
||||||
|
|
||||||
lines = [
|
lines = [
|
||||||
f"rolling {report['windowHours']}h window — all local Claude Code sessions",
|
f"rolling {report['windowHours']}h window — all local Claude Code sessions",
|
||||||
f" tokens {report['tokens']:,} ({report['outputTokens']:,} output) "
|
|
||||||
f"over {report['messages']:,} messages",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
|
quota = report.get("quota")
|
||||||
|
pct = report.get("percentUsed")
|
||||||
|
if quota and pct is not None:
|
||||||
|
lines.append(
|
||||||
|
f" usage {report['tokens']:,} / {quota:,} tokens = {pct:.1f}%"
|
||||||
|
f" [{report['quotaSource']}]"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
lines.append(f" tokens {report['tokens']:,} (no calibrated quota — percentage UNKNOWN)")
|
||||||
|
|
||||||
|
lines.append(
|
||||||
|
f" detail {report['outputTokens']:,} output over {report['messages']:,} messages"
|
||||||
|
)
|
||||||
|
|
||||||
if report["oldestAgesOutInSeconds"] is not None:
|
if report["oldestAgesOutInSeconds"] is not None:
|
||||||
secs = report["oldestAgesOutInSeconds"]
|
secs = report["oldestAgesOutInSeconds"]
|
||||||
lines.append(f" oldest entry ages out in {secs // 60}m{secs % 60:02d}s")
|
lines.append(f" oldest entry ages out in {secs // 60}m{secs % 60:02d}s")
|
||||||
|
|
||||||
top = list(report["projects"].items())[:3]
|
top = list(report["projects"].items())[:3]
|
||||||
if top:
|
if top:
|
||||||
lines.append(" busiest: " + ", ".join(f"{k} {v:,}" for k, v in top))
|
lines.append(" busiest: " + ", ".join(f"{k} {v:,}" for k, v in top))
|
||||||
lines.append(f" sessions: {len(report['projects'])} project(s) contributing")
|
lines.append(f" sessions: {len(report['projects'])} project(s) contributing")
|
||||||
|
|
||||||
|
lines.append(f" VERDICT: {report['verdict']} — {report['verdictReason']}")
|
||||||
lines.append(
|
lines.append(
|
||||||
" FLOOR, not a quota: misses other machines, claude.ai and direct API use"
|
" FLOOR, not a ceiling: misses other machines, claude.ai and direct API use"
|
||||||
)
|
)
|
||||||
|
if not quota:
|
||||||
|
lines.append(
|
||||||
|
" calibrate: ask the user their /status percentage, then "
|
||||||
|
"usage-window.py --calibrate <pct>"
|
||||||
|
)
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
@@ -166,9 +294,48 @@ def main() -> None:
|
|||||||
parser = argparse.ArgumentParser(description=__doc__)
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
parser.add_argument("--hours", type=float, default=5.0)
|
parser.add_argument("--hours", type=float, default=5.0)
|
||||||
parser.add_argument("--json", action="store_true")
|
parser.add_argument("--json", action="store_true")
|
||||||
|
parser.add_argument(
|
||||||
|
"--quota",
|
||||||
|
type=float,
|
||||||
|
default=None,
|
||||||
|
help="Window token quota for this run only (accepts 16e6). Overrides stored calibration.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--calibrate",
|
||||||
|
type=float,
|
||||||
|
default=None,
|
||||||
|
metavar="PCT",
|
||||||
|
help="Percent-used that Claude Code currently reports. Stores the implied quota.",
|
||||||
|
)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
report = collect(args.hours)
|
report = collect(args.hours)
|
||||||
|
|
||||||
|
if args.calibrate is not None:
|
||||||
|
if "error" in report:
|
||||||
|
print(human(report))
|
||||||
|
raise SystemExit(1)
|
||||||
|
if not 0 < args.calibrate <= 100:
|
||||||
|
print("calibrate: percentage must be in (0, 100]")
|
||||||
|
raise SystemExit(2)
|
||||||
|
implied = int(report["tokens"] / (args.calibrate / 100.0))
|
||||||
|
save_quota(implied, report["tokens"], args.calibrate)
|
||||||
|
print(
|
||||||
|
f"calibrated: {report['tokens']:,} tokens reported as {args.calibrate}% "
|
||||||
|
f"=> window quota ~{implied:,}\n"
|
||||||
|
f"written to {QUOTA_FILE}\n"
|
||||||
|
"Note this inherits the script's blind spots (other machines, web, direct API),\n"
|
||||||
|
"so the true quota is this or LARGER. Re-calibrate after a plan change."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
if args.quota is not None:
|
||||||
|
quota, source = int(args.quota), "--quota flag"
|
||||||
|
else:
|
||||||
|
quota, source = load_quota()
|
||||||
|
|
||||||
|
report = add_verdict(report, quota, source)
|
||||||
|
|
||||||
if args.json:
|
if args.json:
|
||||||
print(json.dumps(report, indent=2))
|
print(json.dumps(report, indent=2))
|
||||||
else:
|
else:
|
||||||
|
|||||||
Reference in New Issue
Block a user