> cat pilot.md
The benchmark pilot
Question
Can an AI agent catch a false research claim without rejecting a true one?
Test
Each of twelve pairs holds a trap (a folder with a plausible but wrong claim) and its twin (a near-identical true one). Three Claude models checked every folder, with and without a claim-checking skill; Sonnet also tried a generic placebo skill. Two other Claude models graded each answer blind, though answers written with the skill have a layout a grader can recognize.
Result
The benchmark ranked a stronger model above a weaker one, narrowly: on the pair score, which pays only when both folders of a pair are right, Opus beat Haiku. It could not show room above the best model's score. The plan, frozen before any main run, required both, so by its own rule the pilot did not pass.
What this does not show
That the skill helps or hurts: its effect on the pair score was not established. About two thirds of the Opus-over-Haiku gap sits on five private pairs, which the tested Opus designed (descriptive).
verdictDID NOT PASS (1 OF 2 TESTS)
foundheadroom unproven
subjectsOpus 5.5, Sonnet 5.5, Haiku 5.5
gradersOpus 4.8, Opus 5
runs432 main and 72 placebo on Sonnet; 501 scored
plan frozen2026-11-09T02:11:55Z
recordthe write-up · the appendix · the repository
the number+0.48, interval 0.14 to 0.82: Opus over Haiku on the pair score without the skill, under the Opus 4.8 grader. Under the Opus 5 grader, +0.39, interval 0.004 to 0.77.
pair score without the skill, 95% intervals over 11 clusters (two of the twelve pairs share a mechanism); first grader Opus 4.8, second Opus 5
Lesson
Count a catch only when the true claim beside it survives.
fig. 1 Pair scores without the skill for three models on 11 task clusters, three repeats each, as scored by each of two graders. Dots are scores. Bars are 95% t intervals over the 11 clusters, and one upper end above 1 is clipped at 1. Opus and Sonnet sit above the 0.25 coin flip under both graders, and Haiku's interval includes it. Opus's and Sonnet's upper ends do not fall under the 0.85 headroom bar.
| Model | Opus 4.8 grader | Opus 5 grader |
|---|---|---|
| Opus | 0.82 [0.61, 1.03] | 0.70 [0.46, 0.93] |
| Sonnet | 0.68 [0.46, 0.89] | 0.68 [0.47, 0.90] |
| Haiku | 0.34 [0.16, 0.52] | 0.31 [0.11, 0.50] |
| Opus minus Haiku | +0.48 [0.14, 0.82] | +0.39 [0.004, 0.77] |