> cat errors.csv> cat errors-codebook.txt

Error log

Every error my AI agents made in this research that was written down, found between February and November 2026: when it came in, when it was found, and who or what found it.

Checks and reviewer agents found 35 of the 54 errors (65%). Half were found within 2 days of coming in, 17 of them the same day. None reached anything public.

13 were caught before they shipped and 41 shipped. I found 6. 5 reached a live run's orders. The slowest took 205 days to find.

3 rows are withheld: they come from work that is not public yet. They stay in every count.

It is a count, not a rate: it lists the errors that were written down, not how many were made.

An AI agent built the log from the project's records, re-reading each date from its source. A separate agent reviewed it and recomputed every count. I asked for it.

Who found them

a check that runs on its own, or a planned sweep
16
a separate reviewer agent
19
the agents themselves, in later work
12
me
6
not recorded
1
someone outside the project
0

The 16 checks include 5 sweeps the agents ran on purpose; those do not run unasked.

How far they got

stayed in work files
22
reached my notes or a briefing to me
27
reached a live run's orders
5
reached anything public
0

By period

perioderrorscaught before shippingfound by a check or reviewermedian days to findlongest
to 2026-03-289133%3161
to 2026-08-2014064%2.5205
to 2026-10-0721776%253
after 2026-10-0710570%1.57

Periods are the source's. The median and the longest leave out the one error with no single start date.

Caught before shipping: 13

  1. E001

    A price feed's signal had no age check, so a stale signal could be used. An adversarial review agent found it, beyond its brief.

    came in 2026-03-24 · found the same day · found by a separate reviewer agent · stayed in work files

  2. E003

    One cell of a price replay looked strong. Pre-registered robustness checks (a shifted clock, a calendar split) refused it in the same build.

    came in 2026-08-25 · found the same day · found by a check that runs on its own, or a planned sweep · stayed in work files

  3. E004

    An agent challenged a closed verdict by mixing up the smallest effect a test can detect with its interval. Recomputing from the committed table rejected the challenge.

    came in 2026-08-25 · found the same day · found by a check that runs on its own, or a planned sweep · stayed in work files

  4. E005

    An idea was labelled established on the premise that new listings had no quotes. The next research loop found that almost all already had them.

    came in 2026-08-29 · found the same day · found by the agents themselves, in later work · stayed in work files

  5. E006

    A replay's large gain came from look-ahead. A decomposition rule flagged it in the same commit.

    came in 2026-09-26 · found the same day · found by a check that runs on its own, or a planned sweep · stayed in work files

  6. E007

    A positive secondary cell did not replicate when it was tested as the pre-registered primary on a set of related markets.

    came in 2026-09-25 · found 2026-09-26, 1 day later · found by a check that runs on its own, or a planned sweep · stayed in work files

  7. E008

    A result chosen on the same data failed its frozen holdout. The claim then stayed in the project's main notes for a month after the failure was recorded.

    came in 2026-08-23 · found 2026-08-29, 6 days later · found by a check that runs on its own, or a planned sweep · reached my notes or a briefing to me

  8. E009

    A candidate from a first look was closed by its registered second look. A weak row: a test doing its job, not an agent mistake.

    came in 2026-08-25 · found 2026-10-10, 46 days later · found by a check that runs on its own, or a planned sweep · stayed in work files

  9. E010

    A comparison script had a logic bug. An independent review marked it a blocker.

    came in 2026-10-12 · found 2026-10-14, 2 days later · found by a separate reviewer agent · stayed in work files

  10. E011

    A draft public thread made two false claims, one about how long the work had run. A reader agent flagged that one the day it appeared; a trace of each claim to its file and line corrected both.

    came in 2026-10-12 · found 2026-10-16, 4 days later · found by a check that runs on its own, or a planned sweep · stayed in work files

  11. E012

    An audit program gave the wrong reason flag on a look-ahead case. The independent checker's first run caught it.

    came in 2026-10-17 · found 2026-10-22, 5 days later · found by a check that runs on its own, or a planned sweep · stayed in work files

  12. E013

    A draft said the audit's rulebook was time-stamped on a blockchain. On that date it was only frozen in git and pushed. A copy-draft agent caught the orchestrator's error.

    came in 2026-10-23 · found the same day · found by a separate reviewer agent · stayed in work files

  13. E014

    An eval rubric asked for a cause the task gave no way to find. Blind grading of the rerun showed it.

    came in 2026-10-29 · found 2026-11-05, 7 days later · found by a check that runs on its own, or a planned sweep · stayed in work files

Shipped: 41

  1. E015

    A commit deleted 226 lines of safety code. A halt during a paper-mode soak test led to the investigation that found it; the stripped code never ran.

    came in 2026-02-26 · found 2026-02-28, 2 days later · found by a check that runs on its own, or a planned sweep · stayed in work files

  2. E016

    The paper trader filled every order at once at its limit price and charged no fees. I noticed a balance gap in a live run and stopped the bot. A collaborator, not an agent, wrote the instant fills; the agents kept the model for two months.

    came in 2026-01-08 · found 2026-03-04, 55 days later · found by me · reached a live run's orders

  3. E017

    A new bot's scanner used the settlement time as the order deadline, so none of its live orders filled.

    came in 2026-03-24 · found 2026-03-27, 3 days later · found by the agents themselves, in later work · reached a live run's orders

  4. E018

    A price filter fell back to a stale setting and silently rejected markets in a live bot.

    came in 2026-03-26 · found 2026-03-27, 1 day later · found by the agents themselves, in later work · reached a live run's orders

  5. E019

    A bot went live on a signal that did not match its backtest. An ad hoc check found it, and that check became a launch rule.

    came in 2026-03-24 · found 2026-03-28, 4 days later · found by the agents themselves, in later work · reached a live run's orders

  6. E020

    Six code pushes caused six restarts during a live run. I asked for every fix in one batch.

    came in 2026-03-28 · found the same day · found by me · reached a live run's orders

  7. E021

    The notes held a wrong rule about when trading closes before settlement. A later sprint found it while reproducing the settlement rule, which matched every case.

    came in 2026-03-27 · found 2026-08-16, 142 days later · found by the agents themselves, in later work · reached my notes or a briefing to me

  8. E022

    A figure in the notes was cited to a document that does not contain it. A sweep that re-derived cited numbers found it.

    came in 2026-03-15 · found 2026-08-23, 161 days later · found by a check that runs on its own, or a planned sweep · reached my notes or a briefing to me

  9. E023

    A backtest had look-ahead, and a bot had gone live on it. A re-analysis found it. The source lists it twice; it is counted once.

    came in 2026-03-24 · found 2026-04-07, 14 days later · found by the agents themselves, in later work · reached my notes or a briefing to me

  10. E024

    An agent diagnosed a stuck bot without tracing the code. I made it trace the code mechanically; that found the real cause, about an hour in.

    came in 2026-04-01 · found the same day · found by me · reached my notes or a briefing to me

  11. E025

    An incident report overstated how much of the logs it covered. I challenged the report as vague, and the agent disclosed the overstatement.

    came in 2026-04-21 · found the same day · found by me · reached my notes or a briefing to me

  12. E026

    A new background loop was not stubbed in the tests, so CI hung on every pull request until a fix 14 days later.

    came in 2026-05-05 · found 2026-05-19, 14 days later · found by a check that runs on its own, or a planned sweep · stayed in work files

  13. E027

    A stale headline figure stayed in the notes. Rerunning the canonical report found it.

    came in 2026-05-19 · found 2026-08-20, 93 days later · found by a check that runs on its own, or a planned sweep · reached my notes or a briefing to me

  14. E028

    The notes said no events reached a price band. Re-deriving from the raw file showed that some did.

    came in 2026-05-27 · found 2026-08-23, 88 days later · found by a check that runs on its own, or a planned sweep · reached my notes or a briefing to me

  15. E029

    An agent re-ran a test that an earlier verdict had already closed, without checking the notes first. I caught it. It cost about 39 minutes and 144k tokens.

    came in 2026-08-11 · found the same day · found by me · reached my notes or a briefing to me

  16. E030

    A web research agent's stale report of a product launch was relayed to me as news. The sources do not say who noticed.

    came in 2026-08-17 · found the same day · found by not recorded · reached my notes or a briefing to me

  17. E031

    A per-trade figure was quoted from a scan that had not finished. A quant review's question led to a reread of the full scan.

    came in 2026-08-17 · found 2026-08-20, 3 days later · found by a separate reviewer agent · reached my notes or a briefing to me

  18. E032

    A reported match rate was a floating-point artifact, and a results table could not be reproduced. An adversarial review caught both.

    came in 2026-08-17 · found the same day · found by a separate reviewer agent · reached my notes or a briefing to me

  19. E033

    Two agents gave two numbers for one measure, 99.09% and 93.91%, one of them from a filter nobody declared. Result tables with declared filters caught it.

    came in 2026-08-18 · found 2026-08-20, 2 days later · found by a check that runs on its own, or a planned sweep · reached my notes or a briefing to me

  20. E034

    A replay measured the bot's own resting order instead of the market, so its result was an artifact. An adversarial review led to its withdrawal the same day.

    came in 2026-08-20 · found the same day · found by a separate reviewer agent · reached my notes or a briefing to me

  21. E035

    A safety-check skill looked for code names that never existed in the code. A prompt audit found it.

    came in 2026-03-08 · found 2026-09-29, 205 days later · found by a separate reviewer agent · stayed in work files

  22. E036

    A script kept the label 'canonical, look-ahead fixed' after a second look-ahead was found in it. A sweep found the stale label.

    came in 2026-04-07 · found 2026-08-23, 138 days later · found by a check that runs on its own, or a planned sweep · reached my notes or a briefing to me

  23. E037

    The notes miscounted how many liquid series existed. A census found more, and several of them listed nothing.

    came in 2026-08-18 · found 2026-08-25, 7 days later · found by the agents themselves, in later work · reached my notes or a briefing to me

  24. E038

    A file pattern picked a metadata file as the data file. A sweep of every such pattern found it.

    came in 2026-08-25 · found 2026-08-26, 1 day later · found by the agents themselves, in later work · stayed in work files

  25. E039

    A cost was ported from another market instead of measured. Measuring each market's own depth showed the port was wrong.

    came in 2026-08-25 · found 2026-08-26, 1 day later · found by the agents themselves, in later work · reached my notes or a briefing to me

  26. E040

    A results file that must never change was edited by hand after it was built. An audit re-run found it.

    came in 2026-09-02 · found 2026-10-17, 45 days later · found by a separate reviewer agent · stayed in work files

  27. E041

    A data pull rewrote a raw file in place when it resumed. The same audit re-run found it.

    came in 2026-09-02 · found 2026-10-17, 45 days later · found by a separate reviewer agent · stayed in work files

  28. E042

    Five result tables were built from inputs kept in a temporary session folder, so they cannot be rebuilt.

    came in 2026-08-25 · found 2026-10-17, 53 days later · found by a separate reviewer agent · stayed in work files

  29. E043

    Some verify steps wrote the file before comparing, and others compared nothing, while the notes said every check matched byte for byte.

    came in 2026-08-25 · found 2026-10-17, 53 days later · found by a separate reviewer agent · reached my notes or a briefing to me

  30. E044

    A fill test used a five-minute window and called markets too small. The lead withdrew the rule and reran with a 24-hour window.

    came in 2026-09-25 · found the same day · found by the agents themselves, in later work · reached my notes or a briefing to me

  31. E045

    withheld

    came in 2026-09-26 · found 2026-09-27, 1 day later · found by a separate reviewer agent · reached my notes or a briefing to me

  32. E046

    The notes cited a table row that did not exist. A data review found it; a citation checker came after.

    came in 2026-09-26 · found 2026-09-29, 3 days later · found by a separate reviewer agent · reached my notes or a briefing to me

  33. E047

    withheld

    came in 2026-09-27 · found the same day · found by a separate reviewer agent · reached my notes or a briefing to me

  34. E048

    The notes contradicted the code, and a skill cited sources that did not exist. A prompt audit with 155 findings; several defects in one row, with no single start date.

    came in on several dates · found 2026-09-29 · found by a separate reviewer agent · reached my notes or a briefing to me

  35. E049

    Figures copied by hand into the notes did not match their table. A data review found them.

    came in 2026-08-20 · found 2026-09-29, 40 days later · found by a separate reviewer agent · reached my notes or a briefing to me

  36. E050

    The notes kept saying a script had a hard-coded value after a fix the same day had removed it.

    came in 2026-09-29 · found 2026-10-07, 8 days later · found by a separate reviewer agent · reached my notes or a briefing to me

  37. E051

    A push carried four commits from another session along with its own. The pushing session listed them and told me. Private repo only.

    came in 2026-10-14 · found the same day · found by the agents themselves, in later work · stayed in work files

  38. E052

    A missed recording lost rows from a paper ledger.

    came in 2026-10-07 · found 2026-10-10, 3 days later · found by a separate reviewer agent · stayed in work files

  39. E053

    A scheduled dry run did not start.

    came in 2026-10-27 · found 2026-10-28, 1 day later · found by the agents themselves, in later work · stayed in work files

  40. E054

    withheld

    came in 2026-11-04 · found 2026-11-05, 1 day later · found by me · reached my notes or a briefing to me

  41. E055

    A move list overstated how a holdout failed: the point estimate came out ahead, and only the lower bound fell short. A claims review caught it.

    came in 2026-10-26 · found the same day · found by a separate reviewer agent · reached my notes or a briefing to me

> cat rules