feat(week-review): scan pi/opencode, report model, effort, cost

Scanner covered Claude Code only. It now also reads pi jsonl sessions, the
opencode sqlite store, and Codex (rollout files plus the sqlite thread index
as fallback), and records per-session model, effort level, token counts, tool
errors and cost.

Cost is reported natively by pi and opencode; Claude Code and Codex are
estimated from pricing.json and marked as such, since a subscription seat is
not billed those numbers.

New models.md output ranks (tool, model, effort) by spend with cost per human
turn and a push-back count, and SKILL.md step 4 says how to read it without
turning a regex into a verdict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Miguel Palhas
2026-08-19 21:45:48 +01:00
parent de37048f12
commit 2022bbc881
3 changed files with 599 additions and 75 deletions
+59 -9
View File
@@ -1,6 +1,6 @@
---
name: week-review
description: Review the past week of Claude Code and Codex sessions, find recurring friction, and turn it into concrete config or tooling changes. Use when the user asks to review the week, review recent sessions, or asks what to improve about their setup. Also picks up carry-over items filed as issues on the agent-skills repo.
description: Review the past week of agent sessions across Claude Code, Codex, pi and opencode, find recurring friction, judge which models and effort levels earned their cost, and turn it into concrete config or tooling changes. Use when the user asks to review the week, review recent sessions, asks what to improve about their setup, or asks which model or tool is worth the spend. Also picks up carry-over items filed as issues on the agent-skills repo.
user-invocable: true
argument-hint: "[--days N | --since YYYY-MM-DD]"
allowed-tools:
@@ -46,10 +46,22 @@ deferred; a deferred item stays open and gets one line in the summary.
python3 <skill-dir>/scripts/scan-sessions.py --days 7 --out <scratch>
```
Writes `sessions.json` (one record per session) and `userturns.txt` (every
human turn, grouped). It drops subagent transcripts and flags swarm runs —
collapse those to a single line, since one `/code-review ultra` can be 500+
sessions and 40% of the week's bytes without being 40% of the week's work.
Covers Claude Code, Codex, pi and opencode in one pass — Claude Code is the
bulk of most weeks, but a friction pattern that only shows up in the other
three is exactly the one nobody has noticed yet. Narrow with
`--tools claude,pi` when a run only needs one of them.
Writes three files:
- `sessions.json` — one record per session: tool, cwd, models, effort levels,
token counts, cost, tool-error count.
- `userturns.txt` — every human turn, grouped, with the tool, model, effort and
cost in each session header.
- `models.md` — spend and friction per (tool, model, effort), for step 4.
It drops subagent transcripts and flags swarm runs — collapse those to a single
line, since one `/code-review ultra` can be 500+ sessions and 40% of the week's
bytes without being 40% of the week's work.
Read `userturns.txt` in full. It is the primary evidence and it is usually
40-100k tokens. Do not sample it.
@@ -73,7 +85,44 @@ instance felt. In order of signal strength:
Quote the user verbatim with the date and repo. A finding without a quote is a
guess, and the user can tell.
## 4. Check the docs before recommending
## 4. Judge the models and effort levels
`models.md` holds one row per (tool, model, effort) with sessions, human turns,
push-back count, tool errors, output tokens, cost, and cost per human turn.
Cost per turn is the honest headline, not total cost. A model that bills three
times as much but reaches the same place in a third of the turns is the cheaper
one, and the raw total will say the opposite.
Read the numbers with these limits in mind:
- **Claude Code and Codex costs are estimated**, from token counts and
`scripts/pricing.json`. A subscription seat is not billed this. Call it
API-equivalent spend every time you quote it. pi and opencode report their
own real cost — those are the only invoice-true numbers in the table.
- **Unpriced models count as zero.** If the unpriced list at the bottom of the
table is long, the ranking is wrong until the rates are added.
- **`push-back` is a regex**, not a verdict. It counts human turns that read
like a correction. Use it to pick which sessions to read, then quote what the
user actually said.
- A single session never establishes that a model is worse. Two rows are
comparable only when they did comparable work.
What the comparison is for:
- **Effort.** Find work that ran at high effort and did not need it — small
mechanical edits, single-file renames — and work that ran too low and came
back with push-back. The fix is a per-task-class default, not a global one.
- **Model and tool choice.** Where the same class of task ran under two models
or two tools in the same week, compare turns-to-done and push-back, not
tokens.
- **Cost concentrated in one repo or one skill.** A skill that reliably costs
ten times the median per turn is a skill to reread, not a model problem.
Recommendations from this step change a default in config; they never end at
"use the cheaper model".
## 5. Check the docs before recommending
Model behaviour changes and last year's advice rots. Before proposing a
prompt, skill, or config change, read the relevant page — do not answer from
@@ -91,7 +140,7 @@ output style, so a rule written in dense prose teaches dense prose.
Search for community practice too, and say which source a recommendation came
from.
## 5. Measure before trimming
## 6. Measure before trimming
Always-loaded and on-demand are different budgets, and conflating them
produces wrong advice.
@@ -107,7 +156,7 @@ Count lines, not words — Anthropic's target is under 200 lines per file.
Splitting one file into `@import`s saves nothing; only deleting content or
adding `paths:` scoping does.
## 6. Apply, then file the rest
## 7. Apply, then file the rest
Propose a ranked shortlist with an appetite for each. Apply what the user
agrees to, in this repo, and push. For anything deferred or too large, file a
@@ -134,6 +183,7 @@ landed.
## Scope
Config, skills, hooks, and prompts. Not a project status report — the user has
Config, skills, hooks, prompts, and which model, effort level and tool each
class of work should default to. Not a project status report — the user has
trackers for that. If a week's biggest problem is a product bug, say so in one
line and move on.