feat(skills): machine-wide gate for heavy test runs
Parallel yolo/nightshift/blitz sessions each ran the full suite and OOMed the box. gate.sh caps concurrency, memory and build parallelism; skills now run scoped checks in the loop and one gated full run per push. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YUXS63P1WCdC6bEKWcnAE
This commit is contained in:
naps62
@@ -37,6 +37,15 @@ imports = [ inputs.agent-skills.homeModules.default ];
|
||||
|
||||
`recursive = true` links files individually, so machine-local skills can coexist in the same dir. `nixos-rebuild switch` to apply/update.
|
||||
|
||||
## Shared machine, many sessions
|
||||
|
||||
Several autonomous runs share one box. `skills/linear-common/scripts/gate.sh` is a machine-wide semaphore for heavy commands (full test suites, whole-project builds): bounded slots, memory + CPU cap via a systemd user scope, pinned build/test parallelism. Skills run scoped checks in the inner loop and put only the once-per-push full suite through the gate; exit 75 means it never ran and CI takes over. Policy lives in `linear-common/COMMON.md` under "Local verification budget".
|
||||
|
||||
```sh
|
||||
~/.claude/skills/linear-common/scripts/gate.sh --status
|
||||
AGENT_GATE_SLOTS=3 AGENT_GATE_MEM_MAX=4G ~/.claude/skills/linear-common/scripts/gate.sh -- cargo test
|
||||
```
|
||||
|
||||
## Adding a skill
|
||||
|
||||
Drop a new `skills/<name>/SKILL.md` (+ optional `scripts/`, `references/`, `assets/`). Commit. Non-Nix: re-run `bin/link.sh`. Nix: rebuild.
|
||||
@@ -49,5 +58,5 @@ Drop a new `skills/<name>/SKILL.md` (+ optional `scripts/`, `references/`, `asse
|
||||
| `yolo` | quick ship; optional `land` handoff |
|
||||
| `land` | drive an open PR to green + merged (canonical CI/review loop) |
|
||||
| `blitz` | drive a whole milestone to done |
|
||||
| `linear-common` | shared config/setup/worktree conventions (dependency of work/yolo/blitz) |
|
||||
| `linear-common` | shared config/setup/worktree conventions + local verification budget (dependency of work/yolo/blitz/nightshift) |
|
||||
| `crit`, `humanizer`, `impeccable`, `improve-codebase-architecture` | misc |
|
||||
|
||||
@@ -52,11 +52,11 @@ This skill targets **`tracker: gitea`** (milestones live in the repo's Gitea tra
|
||||
Each pass:
|
||||
|
||||
1. Recompute the **ready set** (§2.4).
|
||||
2. **Fan out**: spawn one issue subagent per ready issue, **in parallel** (multiple `Agent` calls in a single message), `isolation: "worktree"`. Cap concurrency at a sane number (≈4–6). Each subagent prompt:
|
||||
- "Implement Gitea issue #N (`<title>`) in this repo following the `/yolo` flow and `COMMON.md`. You are on integration branch `blitz/<slug>`; create branch `<slug>/N-<issue-slug>` **off it**. Read the issue body + its linked spec/epic; that plus the repo is your full context. Implement, run `buildCommand`, commit in logical steps. **Do not merge to any shared branch and do not close the issue** — push your branch and return the result. If you discover a bug or missing work outside this issue's scope, do not fix it silently; report it in `newFindings`."
|
||||
- Force a structured return (schema): `{ issue, done, branch, summary, buildPassed, newFindings: [{title, body}] }`.
|
||||
2. **Fan out**: spawn one issue subagent per ready issue, **in parallel** (multiple `Agent` calls in a single message), `isolation: "worktree"`. **Cap concurrency at 3** — each worktree carries its own build artifacts and test run, and other autonomous sessions are on the same box. Drop to 2 when `<skills-root>/linear-common/scripts/gate.sh --status` shows the machine already contended. Each subagent prompt:
|
||||
- "Implement Gitea issue #N (`<title>`) in this repo following the `/yolo` flow and `COMMON.md`. You are on integration branch `blitz/<slug>`; create branch `<slug>/N-<issue-slug>` **off it**. Read the issue body + its linked spec/epic; that plus the repo is your full context. Implement and commit in logical steps. Check **only what you touched** as you go; run `buildCommand` at most once at the end, and run it as `<skills-root>/linear-common/scripts/gate.sh -- <buildCommand>` — exit 75 means the machine was busy and it did not run, so return `buildPassed: null` rather than retrying. **Do not merge to any shared branch and do not close the issue** — push your branch and return the result. If you discover a bug or missing work outside this issue's scope, do not fix it silently; report it in `newFindings`."
|
||||
- Force a structured return (schema): `{ issue, done, branch, summary, buildPassed, newFindings: [{title, body}] }`. `buildPassed: null` = the gate was busy, so the integration build is the first real check that branch gets.
|
||||
- **Strict rule**: never spawn a subagent for a blocked issue. Dependencies are load-bearing.
|
||||
3. **Integrate serially** (orchestrator, to avoid parallel-merge conflicts): for each finished subagent whose `done && buildPassed`, merge its branch into `blitz/<slug>`, resolve conflicts, run `buildCommand`. If the merge or build breaks, fix on the integration branch (or bounce the issue back for another pass).
|
||||
3. **Integrate serially** (orchestrator, to avoid parallel-merge conflicts): for each finished subagent whose `done` and whose `buildPassed` is not `false`, merge its branch into `blitz/<slug>` and resolve conflicts. Run `buildCommand` **once per wave, after the last merge** — not once per branch — and through the gate: `<skills-root>/linear-common/scripts/gate.sh -- <buildCommand>`. If the merge or build breaks, fix on the integration branch (or bounce the issue back for another pass); with several branches merged, `git log --oneline` on the failing area tells you which one to bounce.
|
||||
4. **Close** each successfully integrated issue on Gitea (`Closes #N` in the merge commit, or PATCH `state:closed`). Epics whose blockers are now all closed: close them too.
|
||||
5. **Integration review (cadence-gated) — do NOT skip.** After each wave (or every ~3 integrated issues, whichever comes first), audit the *accumulated* diff of `blitz/<slug>` vs `defaultBranch` — not each issue in isolation. Run `/code-review` on that diff, or spawn a reviewer subagent, hunting the cross-issue drift that blind parallel work causes: inconsistent data shapes / contracts between issues, divergent naming, duplicated or conflicting logic, dead code, regressions, misbehavior. **Findings are top priority**: fix them (inline, or file + wire as blocking issues) *before* spawning the next fan-out wave. This is the load-bearing coherence check — parallel subagents can't see each other's work, so this is the only place drift gets caught.
|
||||
6. **Fold in findings**: for each `newFindings` item and any bug you find, create a new Gitea issue in this milestone (`milestone: MS_ID`), wire dependencies if it blocks/relies on others, and let the next pass pick it up. Fix trivial bugs inline instead of filing.
|
||||
@@ -66,7 +66,7 @@ Each pass:
|
||||
|
||||
Declare the milestone **ready** only when ALL hold:
|
||||
- Every workable issue closed; every epic closed.
|
||||
- `buildCommand` green on `blitz/<slug>`.
|
||||
- `buildCommand` green on `blitz/<slug>` (through the gate; a run that never happened is not green — wait for a slot here, this is the one check that must actually execute).
|
||||
- A real smoke/verify of the app passes — drive the actual feature (use `/verify` or `/run`, or the project's verify skill), not just unit tests. This catches integration breakage the per-issue subagents couldn't see.
|
||||
|
||||
If the gate fails, file/fix the gap as a finding and run another pass.
|
||||
|
||||
@@ -238,6 +238,47 @@ If already in a worktree: stay here. Check out the issue branch if the current b
|
||||
5. Never amend commits — always create new ones.
|
||||
6. Keep the tracking issue updated (Linear, Gitea, or GitHub, per `tracker`) if scope changes significantly.
|
||||
|
||||
## Local verification budget (shared machine)
|
||||
|
||||
Several autonomous sessions (`/yolo`, `/nightshift`, `/work`, `/blitz`) run on one box at once, each with subagents. If every one runs the full suite whenever it feels like it, the machine OOMs and all of them die. If instead everything gets pushed for CI to check, CI is clogged and feedback is slow. So: **scoped checks locally, full suite once per push, heavy commands through the gate.**
|
||||
|
||||
### Tiers
|
||||
|
||||
| When | Run | How |
|
||||
|---|---|---|
|
||||
| Inner loop, after each change | typecheck + lint + tests **for the touched module only** | direct, no gate |
|
||||
| Once, right before push/PR | the project's `buildCommand` (full suite) | through `gate.sh` |
|
||||
| Anything else | nothing | let CI do it |
|
||||
|
||||
Never run the full suite twice for the same push. Never run it "to be sure" after a green scoped run of the same code.
|
||||
|
||||
### The gate
|
||||
|
||||
Any command that compiles the whole project or runs the whole suite goes through the machine-wide semaphore:
|
||||
|
||||
```bash
|
||||
<skills-root>/linear-common/scripts/gate.sh -- <buildCommand>
|
||||
```
|
||||
|
||||
It bounds concurrency machine-wide (default `nproc/4` slots), caps the command's memory and CPU via a systemd scope, and pins test/build parallelism env vars (`CARGO_BUILD_JOBS`, `RUST_TEST_THREADS`, `VITEST_MAX_*`, `MAKEFLAGS`, `GOMAXPROCS`, node heap) so the suite doesn't fan out to every core.
|
||||
|
||||
- **Exit 75** = no slot or too little free RAM within the wait budget. It did **not** run. That is a normal outcome, not an error: push and let CI cover it, and say so in the commit/PR body. Do not retry in a loop, do not bypass the gate by running the command directly.
|
||||
- **Exit 137** = killed by the memory cap, NOT a test failure. Do not go hunting for a bug that isn't there. Re-run once with `AGENT_GATE_MEM_MAX=6G` if the suite genuinely needs more.
|
||||
- `gate.sh --status` shows slots busy and free RAM. Cheap; check it before deciding to run anything heavy.
|
||||
- If the file isn't executable on this install, prefix with `bash`.
|
||||
- Knobs (env): `AGENT_GATE_SLOTS`, `AGENT_GATE_WAIT` (default 600s), `AGENT_GATE_MEM_MAX` (3G), `AGENT_GATE_MEM_FLOOR` (2000MB), `AGENT_GATE_JOBS`.
|
||||
|
||||
### Subagents
|
||||
|
||||
- Subagents **never run the full suite**, ever. They run scoped checks on what they touched. The session that dispatched them runs the full suite once, at the end.
|
||||
- Cap concurrent subagents at **3** per session, **2** if their tasks build or test. `gate.sh --status` showing no free slots is a signal to dispatch fewer, not to wait.
|
||||
|
||||
### Pushing
|
||||
|
||||
- One push per finished unit of work. Not per commit, not per milestone.
|
||||
- Don't push to trigger CI as a substitute for the local scoped checks — that is how CI gets clogged.
|
||||
- When the gate was skipped (exit 75), the push is doing real verification work: arm the one-shot CI watcher so a red build surfaces.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Never ask the user** during autonomous work unless you hit a genuine blocker (architectural contradiction, missing credentials, ambiguous requirements that could go very wrong).
|
||||
|
||||
Executable
+101
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env bash
|
||||
# Machine-wide semaphore for heavy verification (test suites, builds).
|
||||
# Many autonomous sessions share one box; without this they all run the full
|
||||
# suite at once and OOM the machine. Exit 75 means "did not run, hand to CI".
|
||||
set -uo pipefail
|
||||
|
||||
LOCKDIR="${AGENT_GATE_DIR:-${XDG_RUNTIME_DIR:-/tmp}/agent-gate}"
|
||||
SLOTS="${AGENT_GATE_SLOTS:-}"
|
||||
WAIT="${AGENT_GATE_WAIT:-600}"
|
||||
MEM_FLOOR_MB="${AGENT_GATE_MEM_FLOOR:-2000}"
|
||||
MEM_MAX="${AGENT_GATE_MEM_MAX:-3G}"
|
||||
CPU_QUOTA="${AGENT_GATE_CPU_QUOTA:-}"
|
||||
JOBS="${AGENT_GATE_JOBS:-}"
|
||||
|
||||
cores=$(nproc 2>/dev/null || echo 4)
|
||||
[ -n "$SLOTS" ] || SLOTS=$(( cores / 4 )); [ "$SLOTS" -lt 1 ] && SLOTS=1
|
||||
[ -n "$JOBS" ] || JOBS=$(( cores / SLOTS / 2 )); [ "$JOBS" -lt 1 ] && JOBS=1
|
||||
[ -n "$CPU_QUOTA" ] || CPU_QUOTA="$(( JOBS * 100 ))%"
|
||||
|
||||
avail_mb() { awk '/MemAvailable/ {print int($2/1024); exit}' /proc/meminfo 2>/dev/null || echo 99999; }
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
gate.sh — run a heavy command under a machine-wide concurrency + memory cap.
|
||||
|
||||
gate.sh [-w SECS] [-s SLOTS] [-m MEMMAX] -- <command...>
|
||||
gate.sh --status
|
||||
|
||||
Exit codes: command's own status, or 75 if no slot / not enough free RAM
|
||||
within the wait budget (caller should skip the check and let CI cover it).
|
||||
|
||||
Env: AGENT_GATE_SLOTS AGENT_GATE_WAIT AGENT_GATE_MEM_FLOOR
|
||||
AGENT_GATE_MEM_MAX AGENT_GATE_CPU_QUOTA AGENT_GATE_JOBS AGENT_GATE_DIR
|
||||
EOF
|
||||
}
|
||||
|
||||
status() {
|
||||
mkdir -p "$LOCKDIR"
|
||||
local free=0 busy=0 i
|
||||
for i in $(seq 1 "$SLOTS"); do
|
||||
if flock -n "$LOCKDIR/slot.$i" true 2>/dev/null; then free=$((free+1)); else busy=$((busy+1)); fi
|
||||
done
|
||||
echo "slots: $busy busy / $SLOTS total (${free} free)"
|
||||
echo "mem: $(avail_mb) MB available (floor ${MEM_FLOOR_MB} MB)"
|
||||
[ -s "$LOCKDIR/holders" ] && { echo "recent holders:"; tail -5 "$LOCKDIR/holders"; }
|
||||
return 0
|
||||
}
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-w|--wait) WAIT="$2"; shift 2 ;;
|
||||
-s|--slots) SLOTS="$2"; shift 2 ;;
|
||||
-m|--mem-max) MEM_MAX="$2"; shift 2 ;;
|
||||
--status) status; exit 0 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
--) shift; break ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
[ $# -gt 0 ] || { usage; exit 2; }
|
||||
|
||||
mkdir -p "$LOCKDIR"
|
||||
deadline=$(( $(date +%s) + WAIT ))
|
||||
fd=""
|
||||
|
||||
while :; do
|
||||
if [ "$(avail_mb)" -ge "$MEM_FLOOR_MB" ]; then
|
||||
for i in $(seq 1 "$SLOTS"); do
|
||||
exec {try}>"$LOCKDIR/slot.$i"
|
||||
if flock -n "$try"; then fd="$try"; break; fi
|
||||
exec {try}>&-
|
||||
done
|
||||
fi
|
||||
[ -n "$fd" ] && break
|
||||
left=$(( deadline - $(date +%s) ))
|
||||
[ "$left" -le 0 ] && {
|
||||
echo "gate: no slot within ${WAIT}s (avail $(avail_mb) MB) — skipping, CI covers this" >&2
|
||||
exit 75
|
||||
}
|
||||
sleep $(( left < 10 ? left : 10 ))
|
||||
done
|
||||
|
||||
echo "$(date -Iseconds) pid=$$ cwd=$PWD cmd=$*" >>"$LOCKDIR/holders"
|
||||
|
||||
export CARGO_BUILD_JOBS="$JOBS" RUST_TEST_THREADS="$JOBS" \
|
||||
VITEST_MAX_THREADS="$JOBS" VITEST_MAX_FORKS="$JOBS" \
|
||||
MAKEFLAGS="-j$JOBS" GOMAXPROCS="$JOBS" \
|
||||
NODE_OPTIONS="${NODE_OPTIONS:-} --max-old-space-size=2048"
|
||||
|
||||
if command -v systemd-run >/dev/null 2>&1 && \
|
||||
systemd-run --user --scope -q -p MemoryMax=64M true >/dev/null 2>&1; then
|
||||
systemd-run --user --scope -q --collect \
|
||||
-p "MemoryMax=$MEM_MAX" -p "MemorySwapMax=0" -p "CPUQuota=$CPU_QUOTA" \
|
||||
-- "$@"
|
||||
else
|
||||
"$@"
|
||||
fi
|
||||
rc=$?
|
||||
|
||||
exec {fd}>&-
|
||||
exit $rc
|
||||
@@ -34,12 +34,13 @@ Autonomous workflow: tracking issue -> worktree -> implementation -> PR -> revie
|
||||
Follow the implementation guidelines from COMMON.md.
|
||||
|
||||
After implementation is complete:
|
||||
1. Run the `buildCommand` from the config. All checks must pass before opening a PR.
|
||||
1. Run the `buildCommand` from the config through the gate — `<skills-root>/linear-common/scripts/gate.sh -- <buildCommand>` (see "Local verification budget" in COMMON.md). All checks must pass before opening a PR. Exit 75 = the machine was busy and it never ran: open the PR and let CI be the check, saying so in the PR body. Exit 137 = memory cap, not a failing test.
|
||||
2. If tests fail, fix them. Do not ship broken code.
|
||||
3. During implementation, check only the module you touched. This is the one full run.
|
||||
|
||||
### 3.5 Self-review gate (before opening the PR)
|
||||
|
||||
Spawn the **`cavecrew-reviewer`** subagent on your own diff (Agent tool, `subagent_type: caveman:cavecrew-reviewer`; point it at `git diff <defaultBranch>...HEAD`). It returns one line per finding, severity-tagged, no praise. Fix anything real it surfaces, then re-run the `buildCommand`. This catches obvious issues before `/land` waits on human/bot review — cheaper than a review round-trip.
|
||||
Spawn the **`cavecrew-reviewer`** subagent on your own diff (Agent tool, `subagent_type: caveman:cavecrew-reviewer`; point it at `git diff <defaultBranch>...HEAD`). It returns one line per finding, severity-tagged, no praise. Fix anything real it surfaces, then re-check **only the modules those fixes touched** — don't run the whole suite a second time; CI covers the rest. This catches obvious issues before `/land` waits on human/bot review — cheaper than a review round-trip.
|
||||
|
||||
Advisory, not a hard gate: for a trivial diff (typo, one-liner, config bump) skip it. Don't loop on it — one pass, address the genuine findings, move on.
|
||||
|
||||
|
||||
@@ -28,7 +28,12 @@ Follow the implementation guidelines from COMMON.md. Move fast — this is yolo
|
||||
|
||||
- Skip formal planning. Read the issue, understand it, start coding.
|
||||
- Still write tests if the project has them, but don't block on edge cases.
|
||||
- Run the `buildCommand` if configured. If it fails, fix it. If a failure is minor and unrelated to your change, warn the user but keep going.
|
||||
- While coding, check **only what you touched** — the module's own tests, typecheck, lint. Not the whole suite.
|
||||
- Once, before pushing, run the configured `buildCommand` through the gate (see "Local verification budget" in COMMON.md):
|
||||
```
|
||||
<skills-root>/linear-common/scripts/gate.sh -- <buildCommand>
|
||||
```
|
||||
If it fails, fix it. If a failure is minor and unrelated to your change, warn the user but keep going. **Exit 75** means the machine was busy and it never ran — push anyway, note it in the commit body, and arm the CI watcher below. **Exit 137** is the memory cap, not a bug.
|
||||
|
||||
### 3. Ship
|
||||
|
||||
@@ -46,7 +51,7 @@ Follow the implementation guidelines from COMMON.md. Move fast — this is yolo
|
||||
- **gitea**: close it — `curl -sS -X PATCH -H "Authorization: token $GITEA_TOKEN" -H "Content-Type: application/json" "$BASE/api/v1/repos/$REPO/issues/$N" -d '{"state":"closed"}'` (or put `Closes #N` in the final commit merged to the default branch).
|
||||
4. That's it. No PR, no review loop.
|
||||
|
||||
**Optional CI confirm (no loop).** If the repo runs CI on push/merge and you want to catch a red build without babysitting, arm ONE background watcher that wakes you only if CI fails, then stop. Don't sit polling.
|
||||
**CI confirm (no loop).** If the repo runs CI on push/merge, arm ONE background watcher that wakes you only if CI fails, then stop. Don't sit polling. **Mandatory when the local gate returned 75** — that push is the only verification the change has had.
|
||||
- GitHub: `gh pr checks <N> --watch --fail-fast` (if a PR exists) or `gh run watch <run-id> --exit-status` via Bash `run_in_background: true`. Non-zero exit → report the failing job to the user.
|
||||
- Gitea: one-shot Monitor on `"$BASE/api/v1/repos/$REPO/commits/<sha>/statuses"` that exits on `success|failure|error` (same snippet as `/land`'s Gitea CI watcher). Report only on `failure`/`error`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user