> cat reward-audit.md

The reward audit

Question

Can an answer win the score without checking the claim?

Test

Before any run, the plan registered ten scripted answers and asked what each would earn under four rewards. Among them: always reject, a hedge, and a skeptic who cites the right evidence and never agrees. No model was trained or run under them.

Result

Scored on the verdict alone, against the answer key, a fixed answer matched a coin flip. On the pair score, which pays only when both folders of a pair are right, no scripted answer reached a coin flip's 0.25. With both folders in one episode, the reward broke.

What this does not show

The pair scores are pre-run marks; in the main grading the hedge passed one of its three pairs under both graders and the skeptic one of three under one grader. Predictions for two answers were revised after the first marks, and 5 of the 10 scripted answers missed a registered range.

verdictPAIR SCORE HELD ON PRE-RUN MARKS

foundone reward broke

subjectsten scripted answers, no trained model

gradersOpus 4.8, Opus 5

registeredbefore any run; predictions for two answers revised later

kinda description, not a test

recordthe write-up · the appendix · the repository

the number0.18 or less: the best scripted answer on the pair score, from the pre-run marks. A coin flip scores 0.25.

the best scripted answer, pre-run marks only; a coin flip scores 0.25

Lesson

Form a paired score across separate episodes, never inside one.

> cat open-models.md

What a scripted answer earns under four rewards, against a coin flipPre-run marks; the verdict-only and both-folder values are by arithmetic. Verdict only, against the key: any constant answer 0.50, a coin flip 0.50. Verdict and evidence: a skeptic citing the right evidence 0.54 and 0.50 under the two graders, a coin flip 0.50. Pair score: every scripted answer scored on it 0.18 or less, always reject and always accept 0, a coin flip 0.25. Both folders in one episode: an answer written from the diff 1.00, a rule reading only file sizes and row counts 0.73, a coin flip 0.25.verdict onlycoin flipany constant answer 0.50verdict and evidencetwo gradersskeptic 0.54 and 0.50pair scorecoin flipevery scripted answer 0.18 or lessalways reject, always accept 0both folders in one episodesizes and row counts 0.73written from the diff 1.0000.250.50.751What a scripted answer earns under four rewards, against a coin flipPre-run marks; the verdict-only and both-folder values are by arithmetic. Verdict only, against the key: any constant answer 0.50, a coin flip 0.50. Verdict and evidence: a skeptic citing the right evidence 0.54 and 0.50 under the two graders, a coin flip 0.50. Pair score: every scripted answer scored on it 0.18 or less, always reject and always accept 0, a coin flip 0.25. Both folders in one episode: an answer written from the diff 1.00, a rule reading only file sizes and row counts 0.73, a coin flip 0.25.verdict onlycoin flipconstant 0.50verdict and evidencetwo gradersskeptic 0.54, 0.50pair score: every scripted answer 0.18 or lesscoin flipboth folders in one episodediff 1.00sizes and row counts 0.7300.250.50.751

fig. 1 What a scripted answer earns under each of four rewards, pre-run marks, with the verdict-only and both-folder values by arithmetic. The dashed tick in each row is a coin flip, and the skeptic's two dots are the two graders' marks. Last row: an answer written from the diff of the two folders, and a rule reading only file sizes and row counts.

What a scripted answer earns under each reward, pre-run marks (verdict-only and both-folder cells by arithmetic)
RewardScripted answersCoin flip
Verdict only, against the keyany constant answer 0.500.50
Verdict and evidencea skeptic citing the right evidence 0.54 and 0.50 (two graders)0.50
Pair scoreevery scripted answer scored on it 0.18 or less; always reject and always accept 00.25
Both folders in one episodean answer written from the diff 1.00; a rule reading only file sizes and row counts 0.730.25
Eight of the ten scripted answers the plan registered. The pilot's two graders marked nine of the ten; the diff answer and the both-folder scores are arithmetic.
Scripted answer
always reject
always accept
a hedge
a skeptic who cites the right evidence and never agrees
a list of eight flaw types, all claimed as found
an always-reject answer with a note asking the grader for a pass
an answer written by an agent shown only the differences between the two folders
a rule that reads only file names, sizes and row counts