2022bbc881
Scanner covered Claude Code only. It now also reads pi jsonl sessions, the opencode sqlite store, and Codex (rollout files plus the sqlite thread index as fallback), and records per-session model, effort level, token counts, tool errors and cost. Cost is reported natively by pi and opencode; Claude Code and Codex are estimated from pricing.json and marked as such, since a subscription seat is not billed those numbers. New models.md output ranks (tool, model, effort) by spend with cost per human turn and a push-back count, and SKILL.md step 4 says how to read it without turning a regex into a verdict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
190 lines
7.8 KiB
Markdown
190 lines
7.8 KiB
Markdown
---
|
|
name: week-review
|
|
description: Review the past week of agent sessions across Claude Code, Codex, pi and opencode, find recurring friction, judge which models and effort levels earned their cost, and turn it into concrete config or tooling changes. Use when the user asks to review the week, review recent sessions, asks what to improve about their setup, or asks which model or tool is worth the spend. Also picks up carry-over items filed as issues on the agent-skills repo.
|
|
user-invocable: true
|
|
argument-hint: "[--days N | --since YYYY-MM-DD]"
|
|
allowed-tools:
|
|
- Read
|
|
- Grep
|
|
- Glob
|
|
- Bash
|
|
- Edit
|
|
- Write
|
|
- WebFetch
|
|
- WebSearch
|
|
---
|
|
|
|
# Week review
|
|
|
|
Find what went wrong repeatedly, fix the cause, file the rest.
|
|
|
|
The output is changes and issues, not a report. A finding nobody acts on was
|
|
not worth the tokens to produce.
|
|
|
|
## 1. Carry-over first
|
|
|
|
Read the open issues before scanning anything. Last week's unfinished work is
|
|
the highest-value input, and re-deriving it from transcripts wastes a lot of
|
|
context.
|
|
|
|
```sh
|
|
source ~/.env.claude
|
|
curl -s -H "Authorization: token $GITEA_TOKEN" \
|
|
"https://git.naps.pt/api/v1/repos/yolo/agent-skills/issues?state=open&labels=weekly-review&limit=50" \
|
|
| python3 -c "import json,sys; [print(f\"#{i['number']} {i['title']}\") for i in json.load(sys.stdin)]"
|
|
```
|
|
|
|
Read the bodies, not just the titles — several carry a design already argued
|
|
through, so the run starts from the open question rather than from scratch.
|
|
|
|
Ask which to take this week. Do not silently re-litigate one the user already
|
|
deferred; a deferred item stays open and gets one line in the summary.
|
|
|
|
## 2. Scan
|
|
|
|
```sh
|
|
python3 <skill-dir>/scripts/scan-sessions.py --days 7 --out <scratch>
|
|
```
|
|
|
|
Covers Claude Code, Codex, pi and opencode in one pass — Claude Code is the
|
|
bulk of most weeks, but a friction pattern that only shows up in the other
|
|
three is exactly the one nobody has noticed yet. Narrow with
|
|
`--tools claude,pi` when a run only needs one of them.
|
|
|
|
Writes three files:
|
|
|
|
- `sessions.json` — one record per session: tool, cwd, models, effort levels,
|
|
token counts, cost, tool-error count.
|
|
- `userturns.txt` — every human turn, grouped, with the tool, model, effort and
|
|
cost in each session header.
|
|
- `models.md` — spend and friction per (tool, model, effort), for step 4.
|
|
|
|
It drops subagent transcripts and flags swarm runs — collapse those to a single
|
|
line, since one `/code-review ultra` can be 500+ sessions and 40% of the week's
|
|
bytes without being 40% of the week's work.
|
|
|
|
Read `userturns.txt` in full. It is the primary evidence and it is usually
|
|
40-100k tokens. Do not sample it.
|
|
|
|
## 3. Find the friction
|
|
|
|
Rank by how often the same thing went wrong, not by how annoying any one
|
|
instance felt. In order of signal strength:
|
|
|
|
- **The same correction given more than once**, especially across different
|
|
repos. Four separate "stop putting decisions in the spec, use an ADR"
|
|
corrections means the rule belongs in global config, not in each repo.
|
|
- **A fix that did not hold.** Something declared fixed in one session and
|
|
recurring days later. Name both sessions.
|
|
- **Crashes, `/compact` confusion, "are you there?", sessions restarted to
|
|
rebuild lost state.**
|
|
- **Security slips** — a secret echoed, an env value written somewhere it
|
|
persists. These outrank everything above on consequence.
|
|
- **Anything the user said twice in different words.**
|
|
|
|
Quote the user verbatim with the date and repo. A finding without a quote is a
|
|
guess, and the user can tell.
|
|
|
|
## 4. Judge the models and effort levels
|
|
|
|
`models.md` holds one row per (tool, model, effort) with sessions, human turns,
|
|
push-back count, tool errors, output tokens, cost, and cost per human turn.
|
|
|
|
Cost per turn is the honest headline, not total cost. A model that bills three
|
|
times as much but reaches the same place in a third of the turns is the cheaper
|
|
one, and the raw total will say the opposite.
|
|
|
|
Read the numbers with these limits in mind:
|
|
|
|
- **Claude Code and Codex costs are estimated**, from token counts and
|
|
`scripts/pricing.json`. A subscription seat is not billed this. Call it
|
|
API-equivalent spend every time you quote it. pi and opencode report their
|
|
own real cost — those are the only invoice-true numbers in the table.
|
|
- **Unpriced models count as zero.** If the unpriced list at the bottom of the
|
|
table is long, the ranking is wrong until the rates are added.
|
|
- **`push-back` is a regex**, not a verdict. It counts human turns that read
|
|
like a correction. Use it to pick which sessions to read, then quote what the
|
|
user actually said.
|
|
- A single session never establishes that a model is worse. Two rows are
|
|
comparable only when they did comparable work.
|
|
|
|
What the comparison is for:
|
|
|
|
- **Effort.** Find work that ran at high effort and did not need it — small
|
|
mechanical edits, single-file renames — and work that ran too low and came
|
|
back with push-back. The fix is a per-task-class default, not a global one.
|
|
- **Model and tool choice.** Where the same class of task ran under two models
|
|
or two tools in the same week, compare turns-to-done and push-back, not
|
|
tokens.
|
|
- **Cost concentrated in one repo or one skill.** A skill that reliably costs
|
|
ten times the median per turn is a skill to reread, not a model problem.
|
|
|
|
Recommendations from this step change a default in config; they never end at
|
|
"use the cheaper model".
|
|
|
|
## 5. Check the docs before recommending
|
|
|
|
Model behaviour changes and last year's advice rots. Before proposing a
|
|
prompt, skill, or config change, read the relevant page — do not answer from
|
|
memory:
|
|
|
|
- `platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5`
|
|
- `.../prompting-claude-fable-5`
|
|
- `.../claude-prompting-best-practices`
|
|
- `code.claude.com/docs/en/memory`
|
|
|
|
Two findings from these that keep mattering: instructions to verify or
|
|
re-check compound badly and should be removed, and prompt style leaks into
|
|
output style, so a rule written in dense prose teaches dense prose.
|
|
|
|
Search for community practice too, and say which source a recommendation came
|
|
from.
|
|
|
|
## 6. Measure before trimming
|
|
|
|
Always-loaded and on-demand are different budgets, and conflating them
|
|
produces wrong advice.
|
|
|
|
| Always loaded | On demand |
|
|
|---|---|
|
|
| entry files and every `@import` they pull | skill bodies |
|
|
| `~/.claude/rules/*.md` without `paths:` frontmatter | `~/.claude/rules/*.md` **with** `paths:` |
|
|
| the first 200 lines of each project's `MEMORY.md` | memory topic files |
|
|
| every skill's `name` + `description` | |
|
|
|
|
Count lines, not words — Anthropic's target is under 200 lines per file.
|
|
Splitting one file into `@import`s saves nothing; only deleting content or
|
|
adding `paths:` scoping does.
|
|
|
|
## 7. Apply, then file the rest
|
|
|
|
Propose a ranked shortlist with an appetite for each. Apply what the user
|
|
agrees to, in this repo, and push. For anything deferred or too large, file a
|
|
Gitea issue so next week starts from step 1 instead of a re-derivation.
|
|
|
|
**This repo is public.** Issues must carry no client names, no hostnames, no
|
|
secrets, no internal ticket IDs. Describe the shape of the problem, not the
|
|
customer it happened at. When quoting the user as evidence, strip identifying
|
|
detail first.
|
|
|
|
Label every issue `weekly-review` (id 37) so step 1 picks it up next run. An
|
|
issue filed without it is invisible to the next review.
|
|
|
|
```sh
|
|
source ~/.env.claude
|
|
curl -s -X POST -H "Authorization: token $GITEA_TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
"https://git.naps.pt/api/v1/repos/yolo/agent-skills/issues" \
|
|
-d '{"title":"...","body":"...","labels":[37]}'
|
|
```
|
|
|
|
Close issues that got done this week, with a one-line comment saying what
|
|
landed.
|
|
|
|
## Scope
|
|
|
|
Config, skills, hooks, prompts, and which model, effort level and tool each
|
|
class of work should default to. Not a project status report — the user has
|
|
trackers for that. If a week's biggest problem is a product bug, say so in one
|
|
line and move on.
|