This is a deliberately small Promptfoo harness for comparing a Claude Code or Codex instruction change. It protects subscription capacity rather than trying to maximize throughput:

A run is disabled until EVAL_ENABLE_AGENT_RUNS=1 is set.
It permits six rollouts by default (EVAL_RUN_BUDGET=6).
It terminates a rollout after ten minutes by default.
It never runs providers in parallel.
It checks the local Claude and Codex quota signals before every rollout and parks when either provider reports pressure.
It uses shell verifiers only. There is no API-backed LLM judge or generated red-team data.

The checks are a conservative floor. Claude's local estimator cannot see other machines or claude.ai activity, so do not override a warning just because this directory says a window is clear.

First run

The included case makes no changes. It only proves that the selected CLI is available, follows an instruction, and leaves a fixture untouched.

cd evals/agent-harness
EVAL_ENABLE_AGENT_RUNS=1 EVAL_PROVIDER=codex npx promptfoo@latest eval --no-cache

Use EVAL_PROVIDER=claude for Claude Code. Run one provider at a time; this is intentional. The provider wrapper uses the subscription login already held by the CLI, not OPENAI_API_KEY or ANTHROPIC_API_KEY.

Each attempted rollout is recorded in .runs.json (ignored by git). The default budget is six. Increase it deliberately when a suite grows:

EVAL_ENABLE_AGENT_RUNS=1 EVAL_PROVIDER=codex EVAL_RUN_BUDGET=12 \
  npx promptfoo@latest eval --no-cache

Set EVAL_ROLLOUT_TIMEOUT_SECONDS only for a fixture that needs longer than the ten-minute default.

Start with this smoke test, then add one real regression at a time. Every fixture needs a verify.sh that performs the acceptance checks without a model. Keep fixtures small and independent; the provider copies one to a fresh temporary directory for each case.

Do not add llm-rubric, Promptfoo red-team generation, or API model providers to this suite without a separate spend decision.

Native Codex SDK rollout

promptfooconfig.pr-skills.codex-land.yaml uses Promptfoo's native openai:codex-sdk provider instead of nesting codex exec inside an existing Codex session. It reuses the local Codex login and does not require an API key. The native provider owns a fixed disposable workspace, so prepare it once, then run the eval and its deterministic verifier:

cd evals/agent-harness
bin/prepare-codex-fixture.sh land-ci
promptfoo eval -c promptfooconfig.pr-skills.codex-land.yaml --no-cache
bash .runtime/codex-land-ci/verify.sh

The fixture uses a local bare Git remote and a mocked gh; network and web search are disabled for the Codex rollout. The preparation command deliberately refuses to overwrite a previous runtime. Inspect or remove that single ignored runtime directory before preparing another fresh rollout. Its Git metadata is kept in .workgit (with GIT_DIR set for the agent) because the SDK sandbox correctly makes a literal .git directory read-only.

The same verifier is a Promptfoo JavaScript assertion, so the eval result fails when the agent's filesystem effects do not satisfy the acceptance checks; the final shell command is a readable independent confirmation.

promptfooconfig.pr-skills.codex-review.yaml uses the same native setup for the untrusted review-pr case. Substitute review-untrusted for land-ci in the preparation and verification commands, and use that config filename.