test(evals): cover cross-forge PR skill rollouts #25

Merged

naps62-yolo merged 19 commits from moderato into main 2026-09-06 14:01:36 +01:00

Conversation 0 Commits 19 Files Changed 104

+1420 -3

naps62-yolo commented 2026-09-04 16:27:19 +01:00

Owner

Copy Link

Copy Source

Summary

• Add real Promptfoo fixtures for land and review-pr across GitHub and Gitea.
• Cover trusted and untrusted review behavior on Claude and native Codex.
• Make native Codex fixture paths portable across checkouts.

Issue

• Evaluation harness improvement; no tracker issue.

Test plan

• Claude: trusted and untrusted review-pr flows on GitHub and Gitea.
• Native Codex: trusted and untrusted review-pr flows on GitHub and Gitea.
• Native Codex: land CI flows on GitHub and Gitea.

## Summary - Add real Promptfoo fixtures for land and review-pr across GitHub and Gitea. - Cover trusted and untrusted review behavior on Claude and native Codex. - Make native Codex fixture paths portable across checkouts. ## Issue - Evaluation harness improvement; no tracker issue. ## Test plan - Claude: trusted and untrusted review-pr flows on GitHub and Gitea. - Native Codex: trusted and untrusted review-pr flows on GitHub and Gitea. - Native Codex: land CI flows on GitHub and Gitea. <!-- agent-meta: {"model": "gpt-5", "harness": "codex"} -->

naps62-yolo added 17 commits 2026-09-04 16:27:19 +01:00

feat(evals): add quota-guarded agent harness 38657ab442

fix(evals): capture provider transcript d851c6215f

fix(evals): bound subscription rollout time 4850ecdb7a

feat(evals): add real harness tasks c486ecefdc

fix(evals): run real tasks by default 19a33208bf

fix(evals): permit Claude test commands e7f2cdd88d

feat(evals): add PR skill fixtures 6db55acbea

fix(evals): allow review diff reads 1764c1e954

test(evals): explain review verifier failures 037df8a041

feat(evals): add native codex skill rollouts 7101f774fe

test(evals): cover github and gitea skill flows 34050b0402

fix(evals): isolate native gitea environment 6ecf8c7ce9

fix(evals): isolate gitea credentials in native runs 3535ce068a

test(evals): add review-pr adversarial cases c9be845f32

fix(evals): isolate trusted review configuration d4e43264a8

test(evals): add portable trusted Codex review f835be5295

test(evals): cover trusted Gitea reviews d185d4f519

naps62-yolo added 1 commit 2026-09-04 19:42:32 +01:00

test(evals): cover land ready states 2098d399e0

naps62-yolo added 1 commit 2026-09-06 14:00:54 +01:00

fix(lint): satisfy eval fixture checks 2cfb6ef42a

naps62-yolo merged commit 74632e5929 into main 2026-09-06 14:01:36 +01:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

weekly-review

Carry-over for the next /week-review run

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

hermes (Hermes (agent)) naps62 (Miguel Palhas) naps62-yolo (naps62-yolo (agent))

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: yolo/agent-skills#25