feat(week-review): scan pi/opencode, report model, effort, cost
Scanner covered Claude Code only. It now also reads pi jsonl sessions, the opencode sqlite store, and Codex (rollout files plus the sqlite thread index as fallback), and records per-session model, effort level, token counts, tool errors and cost. Cost is reported natively by pi and opencode; Claude Code and Codex are estimated from pricing.json and marked as such, since a subscription seat is not billed those numbers. New models.md output ranks (tool, model, effort) by spend with cost per human turn and a push-back count, and SKILL.md step 4 says how to read it without turning a regex into a verdict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: week-review
|
||||
description: Review the past week of Claude Code and Codex sessions, find recurring friction, and turn it into concrete config or tooling changes. Use when the user asks to review the week, review recent sessions, or asks what to improve about their setup. Also picks up carry-over items filed as issues on the agent-skills repo.
|
||||
description: Review the past week of agent sessions across Claude Code, Codex, pi and opencode, find recurring friction, judge which models and effort levels earned their cost, and turn it into concrete config or tooling changes. Use when the user asks to review the week, review recent sessions, asks what to improve about their setup, or asks which model or tool is worth the spend. Also picks up carry-over items filed as issues on the agent-skills repo.
|
||||
user-invocable: true
|
||||
argument-hint: "[--days N | --since YYYY-MM-DD]"
|
||||
allowed-tools:
|
||||
@@ -46,10 +46,22 @@ deferred; a deferred item stays open and gets one line in the summary.
|
||||
python3 <skill-dir>/scripts/scan-sessions.py --days 7 --out <scratch>
|
||||
```
|
||||
|
||||
Writes `sessions.json` (one record per session) and `userturns.txt` (every
|
||||
human turn, grouped). It drops subagent transcripts and flags swarm runs —
|
||||
collapse those to a single line, since one `/code-review ultra` can be 500+
|
||||
sessions and 40% of the week's bytes without being 40% of the week's work.
|
||||
Covers Claude Code, Codex, pi and opencode in one pass — Claude Code is the
|
||||
bulk of most weeks, but a friction pattern that only shows up in the other
|
||||
three is exactly the one nobody has noticed yet. Narrow with
|
||||
`--tools claude,pi` when a run only needs one of them.
|
||||
|
||||
Writes three files:
|
||||
|
||||
- `sessions.json` — one record per session: tool, cwd, models, effort levels,
|
||||
token counts, cost, tool-error count.
|
||||
- `userturns.txt` — every human turn, grouped, with the tool, model, effort and
|
||||
cost in each session header.
|
||||
- `models.md` — spend and friction per (tool, model, effort), for step 4.
|
||||
|
||||
It drops subagent transcripts and flags swarm runs — collapse those to a single
|
||||
line, since one `/code-review ultra` can be 500+ sessions and 40% of the week's
|
||||
bytes without being 40% of the week's work.
|
||||
|
||||
Read `userturns.txt` in full. It is the primary evidence and it is usually
|
||||
40-100k tokens. Do not sample it.
|
||||
@@ -73,7 +85,44 @@ instance felt. In order of signal strength:
|
||||
Quote the user verbatim with the date and repo. A finding without a quote is a
|
||||
guess, and the user can tell.
|
||||
|
||||
## 4. Check the docs before recommending
|
||||
## 4. Judge the models and effort levels
|
||||
|
||||
`models.md` holds one row per (tool, model, effort) with sessions, human turns,
|
||||
push-back count, tool errors, output tokens, cost, and cost per human turn.
|
||||
|
||||
Cost per turn is the honest headline, not total cost. A model that bills three
|
||||
times as much but reaches the same place in a third of the turns is the cheaper
|
||||
one, and the raw total will say the opposite.
|
||||
|
||||
Read the numbers with these limits in mind:
|
||||
|
||||
- **Claude Code and Codex costs are estimated**, from token counts and
|
||||
`scripts/pricing.json`. A subscription seat is not billed this. Call it
|
||||
API-equivalent spend every time you quote it. pi and opencode report their
|
||||
own real cost — those are the only invoice-true numbers in the table.
|
||||
- **Unpriced models count as zero.** If the unpriced list at the bottom of the
|
||||
table is long, the ranking is wrong until the rates are added.
|
||||
- **`push-back` is a regex**, not a verdict. It counts human turns that read
|
||||
like a correction. Use it to pick which sessions to read, then quote what the
|
||||
user actually said.
|
||||
- A single session never establishes that a model is worse. Two rows are
|
||||
comparable only when they did comparable work.
|
||||
|
||||
What the comparison is for:
|
||||
|
||||
- **Effort.** Find work that ran at high effort and did not need it — small
|
||||
mechanical edits, single-file renames — and work that ran too low and came
|
||||
back with push-back. The fix is a per-task-class default, not a global one.
|
||||
- **Model and tool choice.** Where the same class of task ran under two models
|
||||
or two tools in the same week, compare turns-to-done and push-back, not
|
||||
tokens.
|
||||
- **Cost concentrated in one repo or one skill.** A skill that reliably costs
|
||||
ten times the median per turn is a skill to reread, not a model problem.
|
||||
|
||||
Recommendations from this step change a default in config; they never end at
|
||||
"use the cheaper model".
|
||||
|
||||
## 5. Check the docs before recommending
|
||||
|
||||
Model behaviour changes and last year's advice rots. Before proposing a
|
||||
prompt, skill, or config change, read the relevant page — do not answer from
|
||||
@@ -91,7 +140,7 @@ output style, so a rule written in dense prose teaches dense prose.
|
||||
Search for community practice too, and say which source a recommendation came
|
||||
from.
|
||||
|
||||
## 5. Measure before trimming
|
||||
## 6. Measure before trimming
|
||||
|
||||
Always-loaded and on-demand are different budgets, and conflating them
|
||||
produces wrong advice.
|
||||
@@ -107,7 +156,7 @@ Count lines, not words — Anthropic's target is under 200 lines per file.
|
||||
Splitting one file into `@import`s saves nothing; only deleting content or
|
||||
adding `paths:` scoping does.
|
||||
|
||||
## 6. Apply, then file the rest
|
||||
## 7. Apply, then file the rest
|
||||
|
||||
Propose a ranked shortlist with an appetite for each. Apply what the user
|
||||
agrees to, in this repo, and push. For anything deferred or too large, file a
|
||||
@@ -134,6 +183,7 @@ landed.
|
||||
|
||||
## Scope
|
||||
|
||||
Config, skills, hooks, and prompts. Not a project status report — the user has
|
||||
Config, skills, hooks, prompts, and which model, effort level and tool each
|
||||
class of work should default to. Not a project status report — the user has
|
||||
trackers for that. If a week's biggest problem is a product bug, say so in one
|
||||
line and move on.
|
||||
|
||||
Reference in New Issue
Block a user