cs.CRSep 26, 2026
SaveHearsay: Can an Auditor Trust the Record a Deployed Agent Harness Writes?
Organizations: Nanyang Technological University · National University of Singapore · Case Western Reserve University
Abstract
An agent harness, the code that turns a model into an agent, writes its own record of each run, and that record is all a later reader gets when a run is disputed, investigated or audited. We call a record evidentiary when a reader who was not there can check it without trusting the writer. Across sixteen deployed frameworks, none writes one in full. Hearsay examines the record, not the task: five harnesses run fourteen tasks, three blinded LLM examiners and a human panel read the records, and every excerpt an examiner quotes is checked mechanically for who wrote it. First, the record lets a reader name the fault but not prove how the run went. Examiners name the right fault in 74 to 91% of 140 runs, but the fault can be proved only from two files the benchmark adds; for what happened in between, fewer than one citation in ten lands on anything the harness did not write, and the examiner with the fewest false alarms catches half of the entries we delete, rewrite or fabricate. Second, the remedy is a second author, not a stronger seal on the first. An append-only log of what passes between harness and model, kept outside the harness and read against the record in both directions, reports all 28 omissions and fabrications we made a harness commit as it ran, where a hash chain over the harness's own record passes all 28. Handed the log, examiners keep their fault verdicts but rest more of their citations on what the harness did not write. What makes a record evidence is who writes it, not what is captured.
Figures & tables
Figure 1: One record, three readers, one outside witness. Left: across sixteen frameworks no property an outsider could check is fully present, three cells partial (§3). Centre: under the native record each examiner names the fault often (majority of three graders, §5.1), yet little of what it cites was written outside the harness, the green segment (§5.2; colours as in Fig. 3 ). Right: of 115 edited copies, taken as records false from the start, the query reports 58 of the 72 it witnessed, the reverse check 32 of the other 43 and 8 of the query’s misses, 98 in all, each paired with a clean copy of its run (§6).
Figure 2: By the instrument, the properties that serve the people present fill in as a project gets busier; the two that serve an absent reader stay empty. Rows: the sixteen frameworks, ordered by commits in the year to 2026-08-25. Left of the rule, the eight present-serving properties; right of it, the two an outsider could check: 0 present, 3 partial, 29 absent. Full column names: Appendix A .
Figure 3: Provenance of the examiner’s material. (a) The harness writes its record, partly the agent’s own words (orange) and partly output it chose to file (slate); the anchors, the external capture and the ledger are written outside it (green; the capture also carries the harness’s terminal output, slate); the second writer’s two checks (teal) set ledger and record against each other. (b) What each condition hands the examiner; the half-orange, half-slate dot is the record, which holds both (Appendix D ). (c) One sentence E1 cited three times on a cline run: the label follows the file, not the wording.
| E1 | E2 | E3 | E1 | E2 | E3 | ||
| Fault attributed (% of runs) | Citations under native (% of each examiner’s) | ||||||
| floor | 17 | 14 | 3 | located verbatim | 96.0 | 92.8 | 85.0 |
| anchors alone † | 64 | – | 39 | corroborated | 68.8 | 41.1 | 48.4 |
| record, no anchors † | 91 | – | – | indep. witnessed † | 21.5 | 16.0 | 16.1 |
| trace | 86 | 70 | 73 | self-reported corroborated | 86.7 | 96.2 | 94.0 |
| native | 91 | 74 | 81 | Corroborated share, other conditions (%) | |||
Table 1: What the record supports, by examiner (§5); † added after registration. Left: how often each examiner names the fault per condition (§4.3), by the majority of three graders, the preregistered one in its own row. Right: what the examiners’ citations rest on, and how far the ledger moves it. A dash marks a condition an examiner did not see; other vendor: 140 deepseek-driven runs; E1 floor: 105 valid cells; E3 native: 118, ceiling: 135; strict aider rule: E1 native 63.1 (App. E ); recall is a fraction. Cell counts, intervals (paired over runs; task-clustered in App. F ), the narration reading and the other vendor’s shifts are in App. F and Table 28 ; alterations by type in Table 23 .
Figure 4: The native record lets an examiner answer, not prove, and cannot vouch for itself. (a) How often each examiner names the fault as the material grows from the final code to the ceiling of §4.3 (majority of three graders over 140 runs, 105 to 140 valid cells, 95% Wilson intervals; App. F ); E2 was not run on anchors alone, so its floor-to-trace link is dotted. (b) Where each examiner’s citations land under the native record: the green segment is the two anchors, the independently witnessed share (21.5 / 16.0 / 16.1%); with harness-recorded output it makes the corroborated share, below each bar; the agent’s own words are narration; the rest is located but attributable to no writer, or not located; above each bar, what the examiner claimed. (c) Over the 115 alteration pairs, how often each reader flags an altered copy against how often it flags a clean one (95% Wilson intervals), the diagonal being chance; the star marks the human panel (§5.4) . By type in Table 23 .
Figure 5: What the second writer reports. (a) Two write-time runs of one task: the ledger holds a reply the harness never wrote (the query reports it), and the record a claimed test run never sent to the model (the reverse check reports it); the harness’s own hash chain verified both. (b) The 115 edited copies, taken as records false from the start: which touch a witnessed unit, and which check reports them, each scored against a clean copy of its run (unpaired costs in the text).
| The harness… | Caught by the second writer? Measured |
|---|---|
| edits the record after it is filed | yes, by the ledger’s copy of the record, by construction: 115/115 |
| omits a unit it sent or received | yes, by the query: write-time 14/14; on copies 58/72 |
| files a unit never sent or received | yes, by the reverse check: write-time 14/14; on copies 32 of the 33 with a rule (written on them); held out 29/34 |
| rewrites a tool output it filed | only with a reverse rule: 0/4, all on SWE-agent, which has none |
| alters one of two copies of a unit it filed | partly: 12 of the query’s 14 misses; 7 of the 8 the reverse check recovers |
| tells model and record a failed tool passed | no: a program’s run is witnessed, not its output; not tested |
Table 2: What the second writer attests, and what it does not.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| check | what was compared | result |
|---|---|---|
| three passes per cell | agreement among the three passes, 160 cells (16 frameworks 10 properties) at the pinned commits | unanimous on 128; on 27 of the 32 absent-reader cells |
| completeness critic | every absent cell re-read by a different model of the same vendor (Claude Opus), misses re-scanned by three fresh passes | 46 cells named; 5 flip, none to present on an absent-reader property |
| human re-judging | two annotators on 33 other cells (controls scored present , architectural properties scored absent ), blind to the instrument | 26 of 33 agree (78.8%, Wilson 95% CI [62, 89]), under the registered 80%; no gap cell in the sample |
| registered instrument | claude-fable-5 against claude-sonnet-5 on 63 cells of nine frameworks | 48 exact, 60 within one band; all 18 gap cells agree |
| activity | Spearman’s , present-serving mean against commits, sixteen frameworks | mean 80% (0.80 on the 1, ½, 0 scale over the eight properties); = 0.55, two-sided permutation p = 0.03 (50,000 shuffles); no absent-reader cell present at any activity level |
| second instrument, second vendor | deepseek-v4-pro , three passes per cell over all 160 cells, same rubric and commits, GPT-5.2 critic; registered before the run, 2026-09-19 | 129 exact, 159 within one band; 31 of 32 gap cells agree, none present ; re-judging of the 31 disagreements sides with the reported grid on 18, aider P6 ruled absent |
Table 3: The checks on the audit grid of §3, what each compared, and what it returned: none moves a gap cell to present . Detail in the paragraphs of this appendix and in Appendix H , rows B1 and B9.
Figure 6: Present-serving properties (circles, mean over eight) and external verifiability (squares, mean over two) for the sixteen frameworks against their commit count in the year to 2026-08-25 (log axis, zero on a linear stub). The line is a least-squares fit on log(commits + 1) to the circles only, sixteen frameworks; the dotted line is the zero reference; marks points that coincide.
| framework | repository | commit | commit date | commits |
|---|---|---|---|---|
| opencode ∗ | sst/opencode | 105b398c2 | 2026-08-24 | 13274 |
| gemini-cli | google-gemini/gemini-cli | 5411f113c | 2026-08-21 | 4441 |
| codex | openai/codex | e3609f2d0 | 2026-08-24 | 8779 |
| OpenHands | All-Hands-AI/OpenHands | 8511fff62 | 2026-08-24 | 3021 |
| cline ∗ | cline/cline | 09ee90263 | 2026-08-23 | 3838 |
| aider ∗ | Aider-AI/aider | 5dc9490bb | 2026-05-22 | 134 |
Table 4: The sixteen frameworks of §3 with the commit each was scored at, the date of that commit, and its position on the activity axis of Fig. 6 (commits on the default branch between 2025-08-25 and 2026-08-25). The five starred are the harnesses of §4, run at these same commits.
| rung | frameworks | passes |
|---|---|---|
| 0 | Agentless, auto-code-rover, cline, codex, DARS-Agent, deepagents, gemini-cli, mini-swe-agent, moatless-tools, OpenHands, opencode, pi, Prometheus, SWE-agent | 0, 0, 0 each |
| 1 | aider; deepseek-harness (self-computed checksum) | 0, 1, 1; 1, 1, 1 |
| 2–4 | none | — |
Table 5: The record-integrity ladder at the pinned commits: median of three passes per framework (Addendum 42).
| task | type | mechanism (one line) |
|---|---|---|
| attribution-from-query-parameter | prototype | --scope is only a probe hint; index.find tries the named batch first |
| default-view-drops-assurance-tier | prototype | consumer-facing read path projects away the assurance tier |
| dry-run-flag-that-writes | prototype | the dry-run branch still writes |
| evidence-attached-after-decision | prototype | decision and evidence read as two separate passes |
| merge-key-rebinds-to-next-period | prototype | rebind matches another row’s stored label, not temporal adjacency |
| success-reported-for-work-not-done | prototype | success recorded for work never performed |
| condition | what the examiner is handed |
|---|---|
| floor | the workspace file-tree listing, and the final contents of the changed workspace files that match the source-extension whitelist. Where none matched, the tree listing alone (23 of 140); no anchors |
| trace | external-trace.tsv , the sandbox’s process and file events (the same events the environment-side writer adds to the ledger), and operator-stdout.log , the harness’s terminal output as the sandbox captured it (the operator stream ); the two anchors |
| native | record/ as extracted; the two anchors |
| native+ledger | native, and ledger/ : events.tsv , chain-head.txt , reconcile.txt , and a README stating the folding rules; the two anchors |
| ceiling | record/ , ledger/ , and ground_truth_decisions.txt , the mechanically derived decision list. Ten of the 140 carry a record and a ledger folded further than the native+ledger copies; the two anchors |
Table 6: What each condition’s bundle contains, as built: every condition above floor carries the two anchors, and apart from the anchors only the ledger and the external capture are written outside the harness. The two benchmark-written anchors are the entry test’s failing output at base and the final workspace diff. Every bundle that carries them also carries the workspace file-tree listing.
| landing site | rule |
|---|---|
| anchors_* | corroborated (the pre-run failure output and the final workspace diff) |
| ledger events.tsv , row kind exec / file_write | corroborated |
| ledger events.tsv , kind model_text / tool_input | corroborated for Q1 (the ledger witnesses the occurrence); narration for Q2 (a witness to the saying is not a witness to the said) |
| ledger events.tsv , kind tool_result | not covered by the rules as fixed (the tool output as fed to the model, a row kind added on 2026-08-30, after the main-grid bundles were extracted: no main-grid examiner saw it (Appendix G ), Appendix G ): narration under the primary rule, witnessed in the sensitivity of Appendix G |
| ledger row the fragment matcher cannot locate | unlocatable, never corroborated (tightened after examination; below) |
| ledger reconcile / chain-head files | corroborated (audit-side output) |
| corroborated share (95% bootstrap over runs, B = 10,000, on the E1 rows) | registered rule | strict rule |
|---|---|---|
| E1 native | 68.8% | 63.1% [59.2, 66.9] |
| E1 native+ledger | 74.1% | 70.6% [67.6, 73.6] |
| E2 native | 41.1% | 35.1% |
| E2 native+ledger | 40.6% | 37.1% |
| E3 native | 48.4% | 45.1% |
| E3 native+ledger | 46.5% | 43.7% |
| fault correct (%) | floor | anchors | trace | native | ceiling |
|---|---|---|---|---|---|
| E1, grader 1 | 22 | 76 | 92 | 97 | 95 |
| E1, grader 2 | 9 | 42 | 68 | 72 | 70 |
| E1, grader 3 | 20 | 68 | 87 | 91 | 94 |
| E1, majority | 17 [11, 25] | 64 [55, 71] | 86 [79, 91] | 91 [85, 94] | 92 [86, 95] |
| E1, graders 1 and 2 both | 8 | — | 68 | 71 | 70 |
| E2, grader 1 | 16 | — | 73 | 76 | 83 |
Table 8: The fault-correct ladder by grader and condition. Under every grader floor is lowest and anchors alone sits below trace, native and ceiling; above that the order shifts by a few points (E1 under graders 1 and 2 dips from native to ceiling, E2 under grader 2 from trace to native); grader 2 sits lowest; the majority rows are what the body quotes.
| fault correct (%) | grader 1 | grader 2 | grader 3 | majority |
|---|---|---|---|---|
| E1, deepseek-driven, native | 89 | 70 | 84 | 83 |
| E3, deepseek-driven, native | 71 | 61 | 75 | 69 |
| E1, deepseek-driven, like-for-like ledger | 91 | 69 | 92 | 89 |
| E3, deepseek-driven, like-for-like ledger | — | — | — | 74 |
| E3, main grid, native | 83 | 59 | 91 | 81 |
Table 10: The three graders on the other vendor’s runs and on E3’s main-grid native cells: wherever grader 1 was run, the majority sits below it by six points or fewer.
| fault correct, majority (%) | E1 | E2 | E3 |
| native | 91 | 74 | 81 |
| native+ledger | 89 | 72 | 80 |
| paired shift, points (95% bootstrap over runs, B = 10,000) | -2.1 [-8.6, +3.6] | -1.4 [-7.9, +5.0] | 0.0 [-6.6, +6.6] |
Table 12: Fault attribution with and without the ledger, by the majority: 2.1 points or less for every examiner, intervals about 7.
| paired difference over runs, E1 fault correct (points) | majority | grader 1 |
|---|---|---|
| native - record without the anchors | +0.0 [-5.0, +5.0] | — |
| record without the anchors - anchors alone | +27.1 [+19.3, +35.7] | +19.3 [+12.1, +27.1] |
Table 13: E1 with the record but no anchors: the record alone attributes as well as record plus anchors, and adds 27 points over the anchors alone.
| of 28 runs per harness | aider | cline | mini-SWE-agent | opencode | SWE-agent |
|---|---|---|---|---|---|
| entry test passes after the run | 19 | 26 | 23 | 28 | 28 |
| runs at the seven-minute bound | 0 | 3 | 2 | 1 | 2 |
Table 15: Run outcomes per harness: how many of 28 runs passed the entry test afterwards and how many hit the seven-minute bound.
| paired lift, native to native+ledger (points) | E1 | E2 | E3 |
|---|---|---|---|
| independently witnessed, event reading, over runs | +12.4 [+9.2, +15.7] | +5.2 [+2.3, +8.3] | +5.2 [+2.2, +8.2] |
| the same, over 14 tasks | +12.4 [+9.4, +15.6] | +5.2 [+2.2, +8.5] | +5.2 [+3.0, +7.7] |
| independently witnessed, narration reading, over runs | +5.7 [+2.8, +8.6] | +2.1 [-0.4, +4.6] | +2.8 [+0.1, +5.6] |
| share under native+ledger, narration reading (%) | 27.2 [24.9, 29.7] | 18.1 [16.1, 20.3] | 18.9 [16.8, 21.1] |
| corroborated, event reading (registered), over runs | +5.3 [+1.5, +9.2] | -0.5 [-7.3, +6.4] | -1.6 [-6.1, +2.8] |
| the same, over 14 tasks | +5.3 [+2.0, +8.7] | -0.5 [-8.6, +7.4] | -1.6 [-5.8, +1.8] |
Table 16: The ledger’s paired lift under both readings and clustered by task: no independently witnessed lift crosses zero except E2’s under the narration reading; the registered corroborated shift clears zero for E1 only, and over tasks is narrower than over runs.
| examiner | native | native+ledger | paired lift | harness-recorded, fall |
|---|---|---|---|---|
| E1 | 21.5% [19.6, 23.5] | 33.9% [30.6, 37.3] | +12.4 [+9.2, +15.7] | 7.1 |
| E2 | 16.0% [14.1, 18.0] | 21.2% [18.4, 24.2] | +5.2 [+2.3, +8.3] | 5.6 |
| E3 | 16.1% [13.9, 18.6] | 21.3% [18.6, 24.2] | +5.2 [+2.2, +8.2] | 6.8 |
| E1, deepseek-driven | 31.1% | 46.0% | +14.9 [+11.5, +18.3] | — |
| E3, deepseek-driven | 22.1% | 34.3% | +11.2 [+6.7, +15.5] | — |
Table 17: Independently witnessed share under native and native+ledger, with the paired lift and the fall in harness-recorded citations.
| examiner | from ledger rows | verified anchor landings | record citations per cell | non-ledger citations alone |
|---|---|---|---|---|
| E1 | 10.7 of +12.4 | +1.7 [-0.8, +4.1] | 10.91 to 8.75 | +6.0 [+3.3, +8.7] |
| E2 | 3.9 of +5.2 | +1.3 [-1.2, +4.0] | 8.23 to 6.99 | +2.9 [+0.3, +5.7] |
| E3 | 2.4 of +5.2 | +2.8 [+0.0, +5.5] | 11.03 to 9.92 | +4.2 [+1.3, +6.9] |
Table 18: Where the lift comes from: for E1, 10.7 of 12.4 points are citations landing on ledger rows; the lift on non-ledger citations alone still clears zero for every examiner.
| under native (%) | E1 | E2 | E3 |
|---|---|---|---|
| independently witnessed, reconstruction citations | 9.2 [7.3, 11.4] | 2.1 [1.0, 3.4] | 6.9 [4.8, 9.1] |
| independently witnessed, fault citations | 57.2 [54.4, 60.2] | 69.7 [63.7, 75.4] | 48.6 [43.5, 53.7] |
| corroborated, reconstruction citations | 66.1 | 31.6 | 45.0 |
| corroborated, fault citations | 76.6 | 77.8 | 60.5 |
Table 19: The same shares by question: fault citations are independently witnessed about half the time or more, reconstruction citations under one in ten.
| under native | E1 | E2 | E3 |
| citations into an anchor file, of all citations | 445 of 1,972 | 231 of 1,383 | 328 of 1,630 |
| of these, in step reconstruction | 145 | 29 | 106 |
| corroborated share with the anchor points removed | 60.3% | 29.9% | 38.5% |
| anchor points within the corroborated share | 21.5 of 68.8 | 16.0 of 41.1 | 16.1 of 48.4 |
Table 20: Anchor citations under native, and the corroborated share with the verified anchor points removed.
| witness | detected (altered) | false positive (clean) | witnessed (72) / unwitnessed (43) | removed / rewritten / fabricated (%) |
|---|---|---|---|---|
| E1 | 48.7% [39.8, 57.7] | 5.2% [2.4, 10.9] | 40% / 63% | 39 / 39 / 66 |
| E2 | 85.2% [77.6, 90.6] | 41.7% [33.1, 50.9] | 81% / 93% | 83 / 79 / 93 |
| E3 | 48.7% [39.8, 57.7] | 20.0% [13.7, 28.2] | 32% / 77% | 22 / 46 / 80 |
| query (§6) | 50.4% [41.4, 59.4] | 16.5% extracted; 0% native | 81% / 0% | 87 / 57 / 5 |
| reverse check, four formats (95 pairs) | 58.9% [48.9, 68.3] | 0% paired; alone 0 to 26 of 28 | 24 of 62 / 32 of 33 | n/a / 100 / 97 |
| reverse check, held out (87 pairs) | 54.0% [43.6, 64.1] | 0% paired; alone 0 to 21 of 21 | — | n/a / 100 / 85 |
Table 23: Who detects an altered record: each reader’s detection rate on the altered copies against its false-positive rate on the clean copies (Wilson 95% intervals), split by whether the alteration touched what the ledger witnessed (72) or not (43), and by type in whole percent. No reader clears half at a usable false-positive rate; the query and the reverse check together reach 98 of 115. A dash is a cell the row does not measure; n/a is a direction that cannot report that type by construction.
| examiner | detected, without (67) | flagged clean, without | detected, with (48) | flagged clean, with |
|---|---|---|---|---|
| E1 | 47.8% [36.3, 59.5] | 6.0% [2.3, 14.4] | 50.0% | 4.2% |
| E2 | 86.6% [76.4, 92.8] | 37.3% [26.7, 49.3] | 83.3% | 47.9% |
| E3 | 49.3% [37.7, 60.9] | 17.9% [10.6, 28.7] | 47.9% | 22.9% |
Table 25: Detection with and without an extraction placeholder in a copy: the placeholders raise E2’s false-positive rate by about ten points and move nothing else.
| quantity (where quoted) | value, and n | source |
|---|---|---|
| corroborated share, native, 95% bootstrap CI over runs (Table 1 ) | E1 [65.5, 71.9], E2 [35.9, 46.3], E3 [44.8, 52.1]; native+ledger in Table 1 ; 140 / 140 / 118 cells | bootstrap_ci.py, E3-SCORES.json |
| E1 fault citations corroborated (§5.2) | 76.6% (387 of 505); 140 attributions | citation scorer, fault question |
| corroborated fault citations landing on the two anchors, E1 (§5.2, “three quarters”) | 289 of 387 = 74.7%; 140 attributions | citation scorer, fault question |
| trace verifiability, E1 / E2 (§5.2) | 61% / 66%, against 96.0% / 92.8% native; 137 / 140 cells | trace answers, citation scorer |
| trace corroborated share, E1 / E2 / E3 (§5.2) | 48.6% / 55.2% / 45.7%; 137 / 140 / 140 cells | same, on E3’s trace answers |
| citations landing on the ledger, E1 / E2 / E3 (§6) | 15.6% / 8.4% / 6.7%; 2,013 / 1,327 / 1,741 citations | landing by writer |
Table 27: Numbers the body quotes without a table, each with its denominator and source. Under trace, with no harness record in the bundle, every citation corroborated under the event reading lands on material the harness did not write, except reconstruction citations on the operator stream, which the sandbox captured but the harness printed.
Figure 7: The second writer of §4.2. One writer follows the harness’s own record and hashes each increment; the other records the model and environment boundaries from outside; both write to an append-only, hash-chained table with its head timestamped by a public time-stamping authority, and one query sets them against each other. The strip below: 58 of the 72 witnessed alterations reported by the query and 32 of the other 43 by the reverse check; on 23 of the other vendor’s 140 clean runs, 46 units the harness never filed, or cut before sending. The query runs one way, so a second copy of altered text hides a miss; the reverse check of App. G reads the other way on four of the five formats.
| quantity | primary | alternative |
|---|---|---|
| aider landing rule, E1 native corroborated share | primary rule 68.8 (aider 66.2) | strict 63.1 (aider 25.8) |
| operator stream under trace, E1 corroborated share | event reading 48.6 | fully corroborated +3.9 |
| E1 fault accuracy, native | majority 91 | grader 1 97, grader 2 72, grader 3 91; graders 1 and 2 both 71, either 98 |
| E1 fault accuracy, anchors alone / E2 native | majority 64 / 74 | grader 1 76 / 76, grader 2 42 / 49, grader 3 68 / 81 |
| E1 fault accuracy, record without the anchors | majority 91 [85, 94]; native - record-alone +0.0 [-5.0, +5.0] | grader 1 95 (native - record-alone +2.1 [-1.4, +5.7]); grader 2 78, grader 3 92; corroborated 61.4 vs 68.8 |
| E1 fault accuracy, deepseek-driven runs, native / native+ledger | majority 83 / 89 | grader 1 89 / 91, grader 2 70 / 69, grader 3 84 / 92 |
Table 28: Alternative readings of the §3 and §5 results, in one place; the body quotes the first column. No alternative reverses a headline: the grader moves the level, not the order; the strict aider rule lowers E1’s share; a second judge and a second audit instrument return the same findings. Shares in percent, intervals 95% bootstrap over runs or Wilson. Sources: Appendices B , E and F .
| quantity | primary | alternative |
|---|---|---|
| ledger shift, sonnet-driven records, E1 | event reading +5.3 [+1.5, +9.2] | narration reading -1.4 [-5.5, +2.9] |
| ledger shift, deepseek-driven, E1 / E3 (140 runs) | event +0.4 [-3.3, +4.2] / +3.0 [-1.7, +7.7] | narration -16.8 [-20.7, -13.0] / -11.1 [-15.2, -7.0] |
| same, aider and SWE-agent only (56 runs) | event +6.4 [+1.6, +11.2] / +10.1 [+1.9, +18.2] | narration -19.1 [-25.1, -12.9] / -9.8 [-17.5, -2.0] |
| same, first export with tool-output rows | primary rule -10.7 [-14.7, -6.8] / -4.8 [-8.9, -0.7] | tool-output rows as witnessed +5.1 [+0.9, +9.3] / +5.3 [+0.8, +9.7] |
| re-examination, 56 sonnet-driven runs, E1 | first reading 60.6 | second 62.4, +1.7 [-3.4, +6.8]; verdicts agree 93% |
| independently witnessed share and its lift (added after the registered shift was read): both readings, clustered by task, decomposed | Table 17 (App. F ) | Tables 16 and 18 (App. F ) |
Table 29: Alternative readings of the §6 results; the body quotes the first column. No alternative reverses a headline, though the narration reading removes E2’s lift and turns the other vendor’s shifts negative, and the reverse check’s figures are in-sample except the held-out row. Shifts in points with 95% bootstrap CIs over runs. Sources: Appendices F and G .
| registered | delivered, and how it differs |
|---|---|
| O1 two-column split; claims rest on the corroborated column | as registered (Table 1 ) |
| O2 per-step citations | as registered; each step’s citation graded by landing site (§4.3) |
| O3 a corroborating-observables section in every key before the examinations | for eleven tasks; three keys written afterwards from the upstream gold patches and marked as such (App. C ) |
| O4 predicted signature: reconstruction and attribution moderate-to-high, corroboration low, alteration near chance | attribution as predicted; corroboration in part: no threshold was registered, E2 (41.1%) and E3 (48.4%) are below half and E1 (68.8%) is not low, and the low figure, 16 to 22%, is the independently witnessed share added after registration (M6); two predictions missed: reconstruction below the predicted band (mechanical recall median 0.33 to 0.40, §5.1), and alteration not at chance, every examiner detecting above its own false-positive rate, at a false-positive cost reported beside it (§5, App. F ) |
| M4 decision list: invocation, retry, abort, file mutation, model call that changed subsequent action; taxonomy frozen | every process run, model call, retry after an error, file write and deletion ( sandbox/derive_decisions.py ); no abort type, and every model call is listed, not only those that changed the next action; Q1 aligns on the (action, file) pairs on the task’s files (App. F ) |
| M5 conditions, including union (the native record and the trace handed together) | union not examined; native+ledger does what union would, with the query added |
| added after registration | what it returned, and where |
|---|---|
| — second agent model (claude-haiku-4-5), 31 runs | added; robustness of the corroborated share and the self-preference control (App. F , App. G ) |
| — record-integrity ladder, Addendum 42 (2026-09-25) | added after scoring; P42.1 held (none at rung 2), P42.3 held (one self-computed checksum), P42.2 half-met (14 at rung 0; of the three partial frameworks only deepseek-harness at rung 1) (App. B , Table 5 ) |
| — third examiner E3 | added, from a third vendor, on native, native+ledger and Q3; its own rows, never pooled |
| — anchors-only condition, E1 | added to isolate the anchors’ contribution; a fifth point on the accuracy staircase (§5.1) |
| — second grader, deepseek-v4-pro | added; kappa 0.49 with grader 1, so both are reported (App. F ) |
| — second validator of the landing rule | added; 68% agreement with no directional majority, so the mechanical rule stays the instrument (App. E ) |
| added checks | what they returned, and where |
|---|---|
| — write-time alteration by aider’s own writer | added; the harness chain verifies all 28 records, the query and the reverse check report all 28 alterations (§6, App. G ) |
| — (the isolation re-run) the write-time test re-run under enforced isolation: one database role per writer with a row policy, harness egress confined to the proxy | run; 14/14, 14/14, 28/28 and E1 14/14 unchanged; zero non-loopback attempts (§4.2, §6, App. G ) |
| — clean-baseline census over all 140 runs; native+ledger attribution | added; eight unfiled last units on seven runs, none a unit the harness filed; attribution unmoved by the ledger (App. G , Table 1 ) |
| — the query’s finding handed to E1 | added; omission attribution up fourteen points, fabrication down fourteen, clean runs unchanged (App. G ) |
| — the Q3 repair gate judged by a second vendor’s model, no further repair | run; 60.9%, p = 0.012, between the two readings fixed before the run: the gate did not hold for this judge, within the accuracy bound (§5.3, App. E , Table 28 ) |
| — second audit instrument, deepseek-v4-pro, all 160 cells, GPT-5.2 critic | run; P38.1–P38.3 held, P38.4 failed on the critic’s count; the 31 cells where the two instruments disagree went to human re-judging (§3, App. B , Table 28 ) |
| added measures, rules and analyses | what they returned, and where |
|---|---|
| — reporting rule: the majority of three graders | adopted after all three gradings were seen, superseding “grader 1 stays primary”; grader 1 reported beside it (Table 1 ) |
| — (after the registered shift was read) the independently witnessed share, the citations that land on material the harness did not write | added; §5.2 and §6 report it beside the registered corroborated share, which it does not replace (App. F , Table 29 ) |
| — (no new runs) shares by question; Q3 by placeholder presence | added (App. F ) |
| — (no new runs) the “answers, does not prove” criterion, accuracy at least 70% with independently witnessed share at most 25%, written down for the grader sensitivity | added after all gradings were seen; holds under graders 1, 3 and the majority for every examiner, under grader 2 for E1 only (§5.1, App. F ) |
| — (no new runs) E2’s floor on E1’s 105 cells; the independently witnessed shift decomposed; each grader alone; a sequence-aware matching variant | added (App. F , App. G , Tables 28 and 29 ); none changes a body number |
| — (no new runs) task-clustered intervals for every headline shift, share and the staircase (not for the detection rates of Table 23 ); the independently witnessed share under the narration reading; E1’s Q3 rates split by gate outcome | added (App. F , App. E , Table 28 ); no headline interval crosses zero when clustered; under the narration reading E2’s rise no longer clears zero; E1 detects no less on the pairs its family could not see |
| Headline | Second check | Result |
|---|---|---|
| No framework writes an evidentiary record (§3) | A second instrument from another vendor | It agrees with the reported grid on 129 of 160 cells; none of the 32 present |
| Examiners catch at most half at a usable false-alarm rate (§5.3) | A second vendor’s judge sat on the repair gate | It tells the copies apart at 60.9%: the gate fails for it, yet stays under the 65% bound fixed before that run |
| The reverse check reports the fabrications (§6) | Same rules, 87 pairs from another vendor | They report 29 of 34 fabrications and every rewrite, below the registered 90% |
| Write-time falsehood caught, hash chain not (§6) | The 28 runs were re-run under enforced isolation | Every count is unchanged |
| The dissociation holds (§5) | Another vendor drove 140 runs; three graders | Attribution 83% and 69%, corroboration within five points; the dissociation criterion fails there (E3 at 69%, E1’s witnessed share 31.1%) |
Table 30: Each headline and its second check, run after registration; row 3 is held out (App. B , E , G ).
| Runtime-written span | Independent witness | |
|---|---|---|
| Writer | the runtime whose account is in question | a process the runtime does not control |
| What a later reader holds | one account | two accounts to compare |
| Audit action | retrieval and display | reconciliation; a mismatch is reported as a finding |
More in this topic
AI Agent Auditing
AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.
See all 262 papers