> cat pilot.md

The benchmark pilot

Question

Can an AI agent catch a false research claim without rejecting a true one?

Test

Each of twelve pairs holds a trap (a folder with a plausible but wrong claim) and its twin (a near-identical true one). Three Claude models checked every folder, with and without a claim-checking skill; Sonnet also tried a generic placebo skill. Two other Claude models graded each answer blind, though answers written with the skill have a layout a grader can recognize.

Result

The benchmark ranked a stronger model above a weaker one, narrowly: on the pair score, which pays only when both folders of a pair are right, Opus beat Haiku. It could not show room above the best model's score. The plan, frozen before any main run, required both, so by its own rule the pilot did not pass.

What this does not show

That the skill helps or hurts: its effect on the pair score was not established. About two thirds of the Opus-over-Haiku gap sits on five private pairs, which the tested Opus designed (descriptive).

verdictDID NOT PASS (1 OF 2 TESTS)

foundheadroom unproven

subjectsOpus 5.5, Sonnet 5.5, Haiku 5.5

gradersOpus 4.8, Opus 5

runs432 main and 72 placebo on Sonnet; 501 scored

plan frozen2026-11-09T02:11:55Z

recordthe write-up · the appendix · the repository

the number+0.48, interval 0.14 to 0.82: Opus over Haiku on the pair score without the skill, under the Opus 4.8 grader. Under the Opus 5 grader, +0.39, interval 0.004 to 0.77.

pair score without the skill, 95% intervals over 11 clusters (two of the twelve pairs share a mechanism); first grader Opus 4.8, second Opus 5

Lesson

Count a catch only when the true claim beside it survives.

> cat reward-audit.md

Dot-and-whisker chart of the pair scores of Opus, Sonnet and Haiku under each of two graders, with a line at the 0.25 coin flip and a line at the 0.85 headroom bar. coin flip 0.25 headroom bar 0.85 0 0.5 1 Pair score, with 95% interval Opus 0.82 graded by Opus 4.8 0.70 by Opus 5 Sonnet 0.68 0.68 Haiku 0.34 0.31 Arrow: upper end 1.03, clipped at 1 Dot-and-whisker chart of the pair scores of Opus, Sonnet and Haiku under each of two graders, with a line at the 0.25 coin flip and a line at the 0.85 headroom bar. coin flip 0.25 headroom bar 0.85 0 0.5 1 Pair score, with 95% interval Opus 0.82 by Opus 4.8 0.70 by Opus 5 Sonnet 0.68 0.68 Haiku 0.34 0.31 Arrow: upper end 1.03, clipped at 1

fig. 1 Pair scores without the skill for three models on 11 task clusters, three repeats each, as scored by each of two graders. Dots are scores. Bars are 95% t intervals over the 11 clusters, and one upper end above 1 is clipped at 1. Opus and Sonnet sit above the 0.25 coin flip under both graders, and Haiku's interval includes it. Opus's and Sonnet's upper ends do not fall under the 0.85 headroom bar.

Pair scores without the skill, all 11 clusters, with 95% t intervals; a bound above 1 means 1. A coin flip scores 0.25; the headroom bar is 0.85.
ModelOpus 4.8 graderOpus 5 grader
Opus0.82 [0.61, 1.03]0.70 [0.46, 0.93]
Sonnet0.68 [0.46, 0.89]0.68 [0.47, 0.90]
Haiku0.34 [0.16, 0.52]0.31 [0.11, 0.50]
Opus minus Haiku+0.48 [0.14, 0.82]+0.39 [0.004, 0.77]