> cat open-models.md
Two open models
Question
What do two open models score on the seven public pairs?
Test
Each pair has a trap (a folder with a wrong claim) and its twin (a near-identical true one). Two open models ran each task three times, with and without the checklist (the skill's text as a system prompt).
Result
All four pair scores, which pay only when trap and twin are both right, were under 0.25, what a coin flip on the verdict earns if it always shows the key evidence. The per-answer reward ran 0.21 to 0.54. deepseek rejected nearly everything, which that reward pays and the pair score does not.
What this does not show
Not comparable with the pilot: the tools were read-only. One grader, which saw the answers unredacted and so may have told the settings apart. The tasks and keys are public and the checklist was tuned on these traps, so a model may have seen them. The checklist's effect was not established.
verdict3 OF 4 INTERVALS BELOW A COIN FLIP
foundpair score 0.00 to 0.11
subjectsdeepseek-v4.1-flash, qwen3.6-35b-a3b
graderOpus 4.8 only; a second failed its check
runs168
plan frozenbefore any paid call
date2026-11-11
recordthe write-up · the results · the raw traces
the numberThe four pair scores ran from 0.00 to 0.11; a coin flip scores 0.25. The upper end of the 95% interval was under 0.25 in three of the four rows and reached 0.292 in the fourth.
a coin flip scores 0.25; the upper end of the 95% interval was under 0.25 in three of the four rows and reached 0.292 in the fourth
Lesson
Score each answer alone and you pay an agent for saying no.
fig. 1 Pair scores of deepseek-v4.1-flash and qwen3.6-35b-a3b over six task clusters, each task run three times, without and with the checklist, as scored by one grader. Dots are scores. Bars are 95% t intervals over the six clusters, and a lower end below 0 is drawn at 0. All four dots sit under the 0.25 coin flip, and one bar reaches past it. The trap and twin counts are runs that passed the task (verdict and key evidence both marked PASS), descriptive only.
| Model and setting | Pair score | 95% t interval | Trap runs passed | Twin runs passed |
|---|---|---|---|---|
| deepseek-v4.1-flash, without the checklist | 0.056 | [-0.087, 0.198] | 19 of 21 | 1 of 21 |
| deepseek-v4.1-flash, with the checklist | 0.056 | [-0.087, 0.198] | 21 of 21 | 1 of 21 (an answer cut off at the output limit) |
| qwen3.6-35b-a3b, without the checklist | 0.000 | [0.000, 0.000] | 1 of 21 | 0 of 21 |
| qwen3.6-35b-a3b, with the checklist | 0.111 | [-0.070, 0.292] | 12 of 21 | 3 of 21 |