> cat open-models.md

Two open models

Question

What do two open models score on the seven public pairs?

Test

Each pair has a trap (a folder with a wrong claim) and its twin (a near-identical true one). Two open models ran each task three times, with and without the checklist (the skill's text as a system prompt).

Result

All four pair scores, which pay only when trap and twin are both right, were under 0.25, what a coin flip on the verdict earns if it always shows the key evidence. The per-answer reward ran 0.21 to 0.54. deepseek rejected nearly everything, which that reward pays and the pair score does not.

What this does not show

Not comparable with the pilot: the tools were read-only. One grader, which saw the answers unredacted and so may have told the settings apart. The tasks and keys are public and the checklist was tuned on these traps, so a model may have seen them. The checklist's effect was not established.

verdict3 OF 4 INTERVALS BELOW A COIN FLIP

foundpair score 0.00 to 0.11

subjectsdeepseek-v4.1-flash, qwen3.6-35b-a3b

graderOpus 4.8 only; a second failed its check

runs168

plan frozenbefore any paid call

date2026-11-11

recordthe write-up · the results · the raw traces

the numberThe four pair scores ran from 0.00 to 0.11; a coin flip scores 0.25. The upper end of the 95% interval was under 0.25 in three of the four rows and reached 0.292 in the fourth.

a coin flip scores 0.25; the upper end of the 95% interval was under 0.25 in three of the four rows and reached 0.292 in the fourth

Lesson

Score each answer alone and you pay an agent for saying no.

> cat errors

Two open models: pair scores with 95% intervals, without and with the checklist, against a coin flipPair score on six task clusters, three repeats each, one grader; a coin flip scores 0.25. deepseek-v4.1-flash without the checklist 0.06, interval -0.09 to 0.20, 19 of 21 trap runs and 1 of 21 twin runs passed; with it 0.06, interval -0.09 to 0.20, 21 of 21 and 1 of 21. qwen3.6-35b-a3b without the checklist 0.00, interval 0.00 to 0.00, 1 of 21 and 0 of 21; with it 0.11, interval -0.07 to 0.29, 12 of 21 and 3 of 21. A lower end below 0 is drawn at 0.coin flip 0.25deepseek-v4.1-flashwithout the checklist 0.06traps 19 of 21, twins 1 of 21with the checklist 0.06traps 21 of 21, twins 1 of 21qwen3.6-35b-a3bwithout the checklist 0.00traps 1 of 21, twins 0 of 21with the checklist 0.11traps 12 of 21, twins 3 of 2100.51Pair score, with 95% intervalTwo open models: pair scores with 95% intervals, without and with the checklist, against a coin flipPair score on six task clusters, three repeats each, one grader; a coin flip scores 0.25. deepseek-v4.1-flash without the checklist 0.06, interval -0.09 to 0.20, 19 of 21 trap runs and 1 of 21 twin runs passed; with it 0.06, interval -0.09 to 0.20, 21 of 21 and 1 of 21. qwen3.6-35b-a3b without the checklist 0.00, interval 0.00 to 0.00, 1 of 21 and 0 of 21; with it 0.11, interval -0.07 to 0.29, 12 of 21 and 3 of 21. A lower end below 0 is drawn at 0.coin flip 0.25deepseek-v4.1-flashwithout the checklist 0.06traps 19 of 21, twins 1 of 21with the checklist 0.06traps 21 of 21, twins 1 of 21qwen3.6-35b-a3bwithout the checklist 0.00traps 1 of 21, twins 0 of 21with the checklist 0.11traps 12 of 21, twins 3 of 2100.51Pair score, with 95% interval

fig. 1 Pair scores of deepseek-v4.1-flash and qwen3.6-35b-a3b over six task clusters, each task run three times, without and with the checklist, as scored by one grader. Dots are scores. Bars are 95% t intervals over the six clusters, and a lower end below 0 is drawn at 0. All four dots sit under the 0.25 coin flip, and one bar reaches past it. The trap and twin counts are runs that passed the task (verdict and key evidence both marked PASS), descriptive only.

deepseek-v4.1-flash and qwen3.6-35b-a3b: pair score with its 95% t interval over six clusters, and runs that passed the task; a coin flip scores 0.25. The interval is not clipped, so it can dip below 0 although the score cannot. The pass counts are descriptive, not a test.
Model and settingPair score95% t intervalTrap runs passedTwin runs passed
deepseek-v4.1-flash, without the checklist0.056[-0.087, 0.198]19 of 211 of 21
deepseek-v4.1-flash, with the checklist0.056[-0.087, 0.198]21 of 211 of 21 (an answer cut off at the output limit)
qwen3.6-35b-a3b, without the checklist0.000[0.000, 0.000]1 of 210 of 21
qwen3.6-35b-a3b, with the checklist0.111[-0.070, 0.292]12 of 213 of 21