cs.CRSep 26, 2026

Hearsay: Can an Auditor Trust the Record a Deployed Agent Harness Writes?

Authors: Jiahong Dai, Zhuochen Yang, Pengyang Shao, Kelvin Ng, Zhongyi Liu, Chengquan Ju, Yuting He, Bo Hu

Organizations: Nanyang Technological University · National University of Singapore · Case Western Reserve University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

An agent harness, the code that turns a model into an agent, writes its own record of each run, and that record is all a later reader gets when a run is disputed, investigated or audited. We call a record evidentiary when a reader who was not there can check it without trusting the writer. Across sixteen deployed frameworks, none writes one in full. Hearsay examines the record, not the task: five harnesses run fourteen tasks, three blinded LLM examiners and a human panel read the records, and every excerpt an examiner quotes is checked mechanically for who wrote it. First, the record lets a reader name the fault but not prove how the run went. Examiners name the right fault in 74 to 91% of 140 runs, but the fault can be proved only from two files the benchmark adds; for what happened in between, fewer than one citation in ten lands on anything the harness did not write, and the examiner with the fewest false alarms catches half of the entries we delete, rewrite or fabricate. Second, the remedy is a second author, not a stronger seal on the first. An append-only log of what passes between harness and model, kept outside the harness and read against the record in both directions, reports all 28 omissions and fabrications we made a harness commit as it ran, where a hash chain over the harness's own record passes all 28. Handed the log, examiners keep their fault verdicts but rest more of their citations on what the harness did not write. What makes a record evidence is who writes it, not what is captured.

Figures & tables

Figure 1: One record, three readers, one outside witness. Left: across sixteen frameworks no property an outsider could check is fully present, three cells partial (§3). Centre: under the native record each examiner names the fault often (majority of three graders, §5.1), yet little of what it cites was written outside the harness, the green segment (§5.2; colours as in Fig. 3 ). Right: of 115 edited copies, taken as records false from the start, the query reports 58 of the 72 it witnessed, the reverse check 32 of the other 43 and 8 of the query’s misses, 98 in all, each paired with a clean copy of its run (§6).
Figure 2: By the instrument, the properties that serve the people present fill in as a project gets busier; the two that serve an absent reader stay empty. Rows: the sixteen frameworks, ordered by commits in the year to 2026-08-25. Left of the rule, the eight present-serving properties; right of it, the two an outsider could check: 0 present, 3 partial, 29 absent. Full column names: Appendix A .
Figure 3: Provenance of the examiner’s material. (a) The harness writes its record, partly the agent’s own words (orange) and partly output it chose to file (slate); the anchors, the external capture and the ledger are written outside it (green; the capture also carries the harness’s terminal output, slate); the second writer’s two checks (teal) set ledger and record against each other. (b) What each condition hands the examiner; the half-orange, half-slate dot is the record, which holds both (Appendix D ). (c) One sentence E1 cited three times on a cline run: the label follows the file, not the wording.
 E1E2E3 E1E2E3
Fault attributed (% of runs)Citations under native (% of each examiner’s)
floor17143located verbatim96.092.885.0
anchors alone †64–39corroborated68.841.148.4
record, no anchors †91––indep. witnessed †21.516.016.1
trace867073self-reported corroborated86.796.294.0
native917481Corroborated share, other conditions (%)
Table 1: What the record supports, by examiner (§5); † added after registration. Left: how often each examiner names the fault per condition (§4.3), by the majority of three graders, the preregistered one in its own row. Right: what the examiners’ citations rest on, and how far the ledger moves it. A dash marks a condition an examiner did not see; other vendor: 140 deepseek-driven runs; E1 floor: 105 valid cells; E3 native: 118, ceiling: 135; strict aider rule: E1 native 63.1 (App. E ); recall is a fraction. Cell counts, intervals (paired over runs; task-clustered in App. F ), the narration reading and the other vendor’s shifts are in App. F and Table 28 ; alterations by type in Table 23 .
Figure 4: The native record lets an examiner answer, not prove, and cannot vouch for itself. (a) How often each examiner names the fault as the material grows from the final code to the ceiling of §4.3 (majority of three graders over 140 runs, 105 to 140 valid cells, 95% Wilson intervals; App. F ); E2 was not run on anchors alone, so its floor-to-trace link is dotted. (b) Where each examiner’s citations land under the native record: the green segment is the two anchors, the independently witnessed share (21.5 / 16.0 / 16.1%); with harness-recorded output it makes the corroborated share, below each bar; the agent’s own words are narration; the rest is located but attributable to no writer, or not located; above each bar, what the examiner claimed. (c) Over the 115 alteration pairs, how often each reader flags an altered copy against how often it flags a clean one (95% Wilson intervals), the diagonal being chance; the star marks the human panel (§5.4) . By type in Table 23 .
Figure 5: What the second writer reports. (a) Two write-time runs of one task: the ledger holds a reply the harness never wrote (the query reports it), and the record a claimed test run never sent to the model (the reverse check reports it); the harness’s own hash chain verified both. (b) The 115 edited copies, taken as records false from the start: which touch a witnessed unit, and which check reports them, each scored against a clean copy of its run (unpaired costs in the text).
The harness…Caught by the second writer? Measured
edits the record after it is filedyes, by the ledger’s copy of the record, by construction: 115/115
omits a unit it sent or receivedyes, by the query: write-time 14/14; on copies 58/72
files a unit never sent or receivedyes, by the reverse check: write-time 14/14; on copies 32 of the 33 with a rule (written on them); held out 29/34
rewrites a tool output it filedonly with a reverse rule: 0/4, all on SWE-agent, which has none
alters one of two copies of a unit it filedpartly: 12 of the query’s 14 misses; 7 of the 8 the reverse check recovers
tells model and record a failed tool passedno: a program’s run is witnessed, not its output; not tested
Table 2: What the second writer attests, and what it does not.
Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

checkwhat was comparedresult
three passes per cellagreement among the three passes, 160 cells (16 frameworks ×\times 10 properties) at the pinned commitsunanimous on 128; on 27 of the 32 absent-reader cells
completeness criticevery absent cell re-read by a different model of the same vendor (Claude Opus), misses re-scanned by three fresh passes46 cells named; 5 flip, none to present on an absent-reader property
human re-judgingtwo annotators on 33 other cells (controls scored present , architectural properties scored absent ), blind to the instrument26 of 33 agree (78.8%, Wilson 95% CI [62, 89]), under the registered 80%; no gap cell in the sample
registered instrumentclaude-fable-5 against claude-sonnet-5 on 63 cells of nine frameworks48 exact, 60 within one band; all 18 gap cells agree
activitySpearman’s ρ\rho , present-serving mean against commits, sixteen frameworksmean 80% (0.80 on the 1, ½, 0 scale over the eight properties); ρ\rho = 0.55, two-sided permutation p = 0.03 (50,000 shuffles); no absent-reader cell present at any activity level
second instrument, second vendordeepseek-v4-pro , three passes per cell over all 160 cells, same rubric and commits, GPT-5.2 critic; registered before the run, 2026-09-19129 exact, 159 within one band; 31 of 32 gap cells agree, none present ; re-judging of the 31 disagreements sides with the reported grid on 18, aider P6 ruled absent
Appendix
Table 3: The checks on the audit grid of §3, what each compared, and what it returned: none moves a gap cell to present . Detail in the paragraphs of this appendix and in Appendix H , rows B1 and B9.
Figure 6: Present-serving properties (circles, mean over eight) and external verifiability (squares, mean over two) for the sixteen frameworks against their commit count in the year to 2026-08-25 (log axis, zero on a linear stub). The line is a least-squares fit on log(commits + 1) to the circles only, sixteen frameworks; the dotted line is the zero reference; ×n\times n marks points that coincide.
frameworkrepositorycommitcommit datecommits
opencode ∗sst/opencode105b398c22026-08-2413274
gemini-cligoogle-gemini/gemini-cli5411f113c2026-08-214441
codexopenai/codexe3609f2d02026-08-248779
OpenHandsAll-Hands-AI/OpenHands8511fff622026-08-243021
cline ∗cline/cline09ee902632026-08-233838
aider ∗Aider-AI/aider5dc9490bb2026-05-22134
Appendix
Table 4: The sixteen frameworks of §3 with the commit each was scored at, the date of that commit, and its position on the activity axis of Fig. 6 (commits on the default branch between 2025-08-25 and 2026-08-25). The five starred are the harnesses of §4, run at these same commits.
rungframeworkspasses
0Agentless, auto-code-rover, cline, codex, DARS-Agent, deepagents, gemini-cli, mini-swe-agent, moatless-tools, OpenHands, opencode, pi, Prometheus, SWE-agent0, 0, 0 each
1aider; deepseek-harness (self-computed checksum)0, 1, 1; 1, 1, 1
2–4none—
Appendix
Table 5: The record-integrity ladder at the pinned commits: median of three passes per framework (Addendum 42).
tasktypemechanism (one line)
attribution-from-query-parameterprototype--scope is only a probe hint; index.find tries the named batch first
default-view-drops-assurance-tierprototypeconsumer-facing read path projects away the assurance tier
dry-run-flag-that-writesprototypethe dry-run branch still writes
evidence-attached-after-decisionprototypedecision and evidence read as two separate passes
merge-key-rebinds-to-next-periodprototyperebind matches another row’s stored label, not temporal adjacency
success-reported-for-work-not-doneprototypesuccess recorded for work never performed
Appendix
Table 12
conditionwhat the examiner is handed
floorthe workspace file-tree listing, and the final contents of the changed workspace files that match the source-extension whitelist. Where none matched, the tree listing alone (23 of 140); no anchors
traceexternal-trace.tsv , the sandbox’s process and file events (the same events the environment-side writer adds to the ledger), and operator-stdout.log , the harness’s terminal output as the sandbox captured it (the operator stream ); the two anchors
nativerecord/ as extracted; the two anchors
native+ledgernative, and ledger/ : events.tsv , chain-head.txt , reconcile.txt , and a README stating the folding rules; the two anchors
ceilingrecord/ , ledger/ , and ground_truth_decisions.txt , the mechanically derived decision list. Ten of the 140 carry a record and a ledger folded further than the native+ledger copies; the two anchors
Appendix
Table 6: What each condition’s bundle contains, as built: every condition above floor carries the two anchors, and apart from the anchors only the ledger and the external capture are written outside the harness. The two benchmark-written anchors are the entry test’s failing output at base and the final workspace diff. Every bundle that carries them also carries the workspace file-tree listing.
landing siterule
anchors_*corroborated (the pre-run failure output and the final workspace diff)
ledger events.tsv , row kind exec / file_writecorroborated
ledger events.tsv , kind model_text / tool_inputcorroborated for Q1 (the ledger witnesses the occurrence); narration for Q2 (a witness to the saying is not a witness to the said)
ledger events.tsv , kind tool_resultnot covered by the rules as fixed (the tool output as fed to the model, a row kind added on 2026-08-30, after the main-grid bundles were extracted: no main-grid examiner saw it (Appendix G ), Appendix G ): narration under the primary rule, witnessed in the sensitivity of Appendix G
ledger row the fragment matcher cannot locateunlocatable, never corroborated (tightened after examination; below)
ledger reconcile / chain-head filescorroborated (audit-side output)
Appendix
Table 14
corroborated share (95% bootstrap over runs, B = 10,000, on the E1 rows)registered rulestrict rule
E1 native68.8%63.1% [59.2, 66.9]
E1 native+ledger74.1%70.6% [67.6, 73.6]
E2 native41.1%35.1%
E2 native+ledger40.6%37.1%
E3 native48.4%45.1%
E3 native+ledger46.5%43.7%
Appendix
Table 15
fault correct (%)flooranchorstracenativeceiling
E1, grader 12276929795
E1, grader 2942687270
E1, grader 32068879194
E1, majority17 [11, 25]64 [55, 71]86 [79, 91]91 [85, 94]92 [86, 95]
E1, graders 1 and 2 both8—687170
E2, grader 116—737683
Appendix
Table 8: The fault-correct ladder by grader and condition. Under every grader floor is lowest and anchors alone sits below trace, native and ceiling; above that the order shifts by a few points (E1 under graders 1 and 2 dips from native to ceiling, E2 under grader 2 from trace to native); grader 2 sits lowest; the majority rows are what the body quotes.
fault correct (%)grader 1grader 2grader 3majority
E1, deepseek-driven, native89708483
E3, deepseek-driven, native71617569
E1, deepseek-driven, like-for-like ledger91699289
E3, deepseek-driven, like-for-like ledger———74
E3, main grid, native83599181
Appendix
Table 10: The three graders on the other vendor’s runs and on E3’s main-grid native cells: wherever grader 1 was run, the majority sits below it by six points or fewer.
fault correct, majority (%)E1E2E3
native917481
native+ledger897280
paired shift, points (95% bootstrap over runs, B = 10,000)-2.1 [-8.6, +3.6]-1.4 [-7.9, +5.0]0.0 [-6.6, +6.6]
Appendix
Table 12: Fault attribution with and without the ledger, by the majority: 2.1 points or less for every examiner, intervals about ±\pm 7.
paired difference over runs, E1 fault correct (points)majoritygrader 1
native - record without the anchors+0.0 [-5.0, +5.0]—
record without the anchors - anchors alone+27.1 [+19.3, +35.7]+19.3 [+12.1, +27.1]
Appendix
Table 13: E1 with the record but no anchors: the record alone attributes as well as record plus anchors, and adds 27 points over the anchors alone.
of 28 runs per harnessaiderclinemini-SWE-agentopencodeSWE-agent
entry test passes after the run1926232828
runs at the seven-minute bound03212
Appendix
Table 15: Run outcomes per harness: how many of 28 runs passed the entry test afterwards and how many hit the seven-minute bound.
paired lift, native to native+ledger (points)E1E2E3
independently witnessed, event reading, over runs+12.4 [+9.2, +15.7]+5.2 [+2.3, +8.3]+5.2 [+2.2, +8.2]
the same, over 14 tasks+12.4 [+9.4, +15.6]+5.2 [+2.2, +8.5]+5.2 [+3.0, +7.7]
independently witnessed, narration reading, over runs+5.7 [+2.8, +8.6]+2.1 [-0.4, +4.6]+2.8 [+0.1, +5.6]
share under native+ledger, narration reading (%)27.2 [24.9, 29.7]18.1 [16.1, 20.3]18.9 [16.8, 21.1]
corroborated, event reading (registered), over runs+5.3 [+1.5, +9.2]-0.5 [-7.3, +6.4]-1.6 [-6.1, +2.8]
the same, over 14 tasks+5.3 [+2.0, +8.7]-0.5 [-8.6, +7.4]-1.6 [-5.8, +1.8]
Appendix
Table 16: The ledger’s paired lift under both readings and clustered by task: no independently witnessed lift crosses zero except E2’s under the narration reading; the registered corroborated shift clears zero for E1 only, and over tasks is narrower than over runs.
examinernativenative+ledgerpaired liftharness-recorded, fall
E121.5% [19.6, 23.5]33.9% [30.6, 37.3]+12.4 [+9.2, +15.7]7.1
E216.0% [14.1, 18.0]21.2% [18.4, 24.2]+5.2 [+2.3, +8.3]5.6
E316.1% [13.9, 18.6]21.3% [18.6, 24.2]+5.2 [+2.2, +8.2]6.8
E1, deepseek-driven31.1%46.0%+14.9 [+11.5, +18.3]—
E3, deepseek-driven22.1%34.3%+11.2 [+6.7, +15.5]—
Appendix
Table 17: Independently witnessed share under native and native+ledger, with the paired lift and the fall in harness-recorded citations.
examinerfrom ledger rowsverified anchor landingsrecord citations per cellnon-ledger citations alone
E110.7 of +12.4+1.7 [-0.8, +4.1]10.91 to 8.75+6.0 [+3.3, +8.7]
E23.9 of +5.2+1.3 [-1.2, +4.0]8.23 to 6.99+2.9 [+0.3, +5.7]
E32.4 of +5.2+2.8 [+0.0, +5.5]11.03 to 9.92+4.2 [+1.3, +6.9]
Appendix
Table 18: Where the lift comes from: for E1, 10.7 of 12.4 points are citations landing on ledger rows; the lift on non-ledger citations alone still clears zero for every examiner.
under native (%)E1E2E3
independently witnessed, reconstruction citations9.2 [7.3, 11.4]2.1 [1.0, 3.4]6.9 [4.8, 9.1]
independently witnessed, fault citations57.2 [54.4, 60.2]69.7 [63.7, 75.4]48.6 [43.5, 53.7]
corroborated, reconstruction citations66.131.645.0
corroborated, fault citations76.677.860.5
Appendix
Table 19: The same shares by question: fault citations are independently witnessed about half the time or more, reconstruction citations under one in ten.
under nativeE1E2E3
citations into an anchor file, of all citations445 of 1,972231 of 1,383328 of 1,630
of these, in step reconstruction14529106
corroborated share with the anchor points removed60.3%29.9%38.5%
anchor points within the corroborated share21.5 of 68.816.0 of 41.116.1 of 48.4
Appendix
Table 20: Anchor citations under native, and the corroborated share with the verified anchor points removed.
witnessdetected (altered)false positive (clean)witnessed (72) / unwitnessed (43)removed / rewritten / fabricated (%)
E148.7% [39.8, 57.7]5.2% [2.4, 10.9]40% / 63%39 / 39 / 66
E285.2% [77.6, 90.6]41.7% [33.1, 50.9]81% / 93%83 / 79 / 93
E348.7% [39.8, 57.7]20.0% [13.7, 28.2]32% / 77%22 / 46 / 80
query (§6)50.4% [41.4, 59.4]16.5% extracted; 0% native81% / 0%87 / 57 / 5
reverse check, four formats (95 pairs)58.9% [48.9, 68.3]0% paired; alone 0 to 26 of 2824 of 62 / 32 of 33n/a / 100 / 97
reverse check, held out (87 pairs)54.0% [43.6, 64.1]0% paired; alone 0 to 21 of 21—n/a / 100 / 85
Appendix
Table 23: Who detects an altered record: each reader’s detection rate on the altered copies against its false-positive rate on the clean copies (Wilson 95% intervals), split by whether the alteration touched what the ledger witnessed (72) or not (43), and by type in whole percent. No reader clears half at a usable false-positive rate; the query and the reverse check together reach 98 of 115. A dash is a cell the row does not measure; n/a is a direction that cannot report that type by construction.
examinerdetected, without (67)flagged clean, withoutdetected, with (48)flagged clean, with
E147.8% [36.3, 59.5]6.0% [2.3, 14.4]50.0%4.2%
E286.6% [76.4, 92.8]37.3% [26.7, 49.3]83.3%47.9%
E349.3% [37.7, 60.9]17.9% [10.6, 28.7]47.9%22.9%
Appendix
Table 25: Detection with and without an extraction placeholder in a copy: the placeholders raise E2’s false-positive rate by about ten points and move nothing else.
quantity (where quoted)value, and nsource
corroborated share, native, 95% bootstrap CI over runs (Table 1 )E1 [65.5, 71.9], E2 [35.9, 46.3], E3 [44.8, 52.1]; native+ledger in Table 1 ; 140 / 140 / 118 cellsbootstrap_ci.py, E3-SCORES.json
E1 fault citations corroborated (§5.2)76.6% (387 of 505); 140 attributionscitation scorer, fault question
corroborated fault citations landing on the two anchors, E1 (§5.2, “three quarters”)289 of 387 = 74.7%; 140 attributionscitation scorer, fault question
trace verifiability, E1 / E2 (§5.2)61% / 66%, against 96.0% / 92.8% native; 137 / 140 cellstrace answers, citation scorer
trace corroborated share, E1 / E2 / E3 (§5.2)48.6% / 55.2% / 45.7%; 137 / 140 / 140 cellssame, on E3’s trace answers
citations landing on the ledger, E1 / E2 / E3 (§6)15.6% / 8.4% / 6.7%; 2,013 / 1,327 / 1,741 citationslanding by writer
Appendix
Table 27: Numbers the body quotes without a table, each with its denominator and source. Under trace, with no harness record in the bundle, every citation corroborated under the event reading lands on material the harness did not write, except reconstruction citations on the operator stream, which the sandbox captured but the harness printed.
Figure 7: The second writer of §4.2. One writer follows the harness’s own record and hashes each increment; the other records the model and environment boundaries from outside; both write to an append-only, hash-chained table with its head timestamped by a public time-stamping authority, and one query sets them against each other. The strip below: 58 of the 72 witnessed alterations reported by the query and 32 of the other 43 by the reverse check; on 23 of the other vendor’s 140 clean runs, 46 units the harness never filed, or cut before sending. The query runs one way, so a second copy of altered text hides a miss; the reverse check of App. G reads the other way on four of the five formats.
quantityprimaryalternative
aider landing rule, E1 native corroborated shareprimary rule 68.8 (aider 66.2)strict 63.1 (aider 25.8)
operator stream under trace, E1 corroborated shareevent reading 48.6fully corroborated +3.9
E1 fault accuracy, nativemajority 91grader 1 97, grader 2 72, grader 3 91; graders 1 and 2 both 71, either 98
E1 fault accuracy, anchors alone / E2 nativemajority 64 / 74grader 1 76 / 76, grader 2 42 / 49, grader 3 68 / 81
E1 fault accuracy, record without the anchorsmajority 91 [85, 94]; native - record-alone +0.0 [-5.0, +5.0]grader 1 95 (native - record-alone +2.1 [-1.4, +5.7]); grader 2 78, grader 3 92; corroborated 61.4 vs 68.8
E1 fault accuracy, deepseek-driven runs, native / native+ledgermajority 83 / 89grader 1 89 / 91, grader 2 70 / 69, grader 3 84 / 92
Appendix
Table 28: Alternative readings of the §3 and §5 results, in one place; the body quotes the first column. No alternative reverses a headline: the grader moves the level, not the order; the strict aider rule lowers E1’s share; a second judge and a second audit instrument return the same findings. Shares in percent, intervals 95% bootstrap over runs or Wilson. Sources: Appendices B , E and F .
quantityprimaryalternative
ledger shift, sonnet-driven records, E1event reading +5.3 [+1.5, +9.2]narration reading -1.4 [-5.5, +2.9]
ledger shift, deepseek-driven, E1 / E3 (140 runs)event +0.4 [-3.3, +4.2] / +3.0 [-1.7, +7.7]narration -16.8 [-20.7, -13.0] / -11.1 [-15.2, -7.0]
same, aider and SWE-agent only (56 runs)event +6.4 [+1.6, +11.2] / +10.1 [+1.9, +18.2]narration -19.1 [-25.1, -12.9] / -9.8 [-17.5, -2.0]
same, first export with tool-output rowsprimary rule -10.7 [-14.7, -6.8] / -4.8 [-8.9, -0.7]tool-output rows as witnessed +5.1 [+0.9, +9.3] / +5.3 [+0.8, +9.7]
re-examination, 56 sonnet-driven runs, E1first reading 60.6second 62.4, +1.7 [-3.4, +6.8]; verdicts agree 93%
independently witnessed share and its lift (added after the registered shift was read): both readings, clustered by task, decomposedTable 17 (App. F )Tables 16 and 18 (App. F )
Appendix
Table 29: Alternative readings of the §6 results; the body quotes the first column. No alternative reverses a headline, though the narration reading removes E2’s lift and turns the other vendor’s shifts negative, and the reverse check’s figures are in-sample except the held-out row. Shifts in points with 95% bootstrap CIs over runs. Sources: Appendices F and G .
registereddelivered, and how it differs
O1 two-column split; claims rest on the corroborated columnas registered (Table 1 )
O2 per-step citationsas registered; each step’s citation graded by landing site (§4.3)
O3 a corroborating-observables section in every key before the examinationsfor eleven tasks; three keys written afterwards from the upstream gold patches and marked as such (App. C )
O4 predicted signature: reconstruction and attribution moderate-to-high, corroboration low, alteration near chanceattribution as predicted; corroboration in part: no threshold was registered, E2 (41.1%) and E3 (48.4%) are below half and E1 (68.8%) is not low, and the low figure, 16 to 22%, is the independently witnessed share added after registration (M6); two predictions missed: reconstruction below the predicted band (mechanical recall median 0.33 to 0.40, §5.1), and alteration not at chance, every examiner detecting above its own false-positive rate, at a false-positive cost reported beside it (§5, App. F )
M4 decision list: invocation, retry, abort, file mutation, model call that changed subsequent action; taxonomy frozenevery process run, model call, retry after an error, file write and deletion ( sandbox/derive_decisions.py ); no abort type, and every model call is listed, not only those that changed the next action; Q1 aligns on the (action, file) pairs on the task’s files (App. F )
M5 conditions, including union (the native record and the trace handed together)union not examined; native+ledger does what union would, with the query added
Appendix
Table 32
added after registrationwhat it returned, and where
— second agent model (claude-haiku-4-5), 31 runsadded; robustness of the corroborated share and the self-preference control (App. F , App. G )
— record-integrity ladder, Addendum 42 (2026-09-25)added after scoring; P42.1 held (none at rung 2), P42.3 held (one self-computed checksum), P42.2 half-met (14 at rung 0; of the three partial frameworks only deepseek-harness at rung 1) (App. B , Table 5 )
— third examiner E3added, from a third vendor, on native, native+ledger and Q3; its own rows, never pooled
— anchors-only condition, E1added to isolate the anchors’ contribution; a fifth point on the accuracy staircase (§5.1)
— second grader, deepseek-v4-proadded; kappa 0.49 with grader 1, so both are reported (App. F )
— second validator of the landing ruleadded; 68% agreement with no directional majority, so the mechanical rule stays the instrument (App. E )
Appendix
Table 33
added checkswhat they returned, and where
— write-time alteration by aider’s own writeradded; the harness chain verifies all 28 records, the query and the reverse check report all 28 alterations (§6, App. G )
— (the isolation re-run) the write-time test re-run under enforced isolation: one database role per writer with a row policy, harness egress confined to the proxyrun; 14/14, 14/14, 28/28 and E1 14/14 unchanged; zero non-loopback attempts (§4.2, §6, App. G )
— clean-baseline census over all 140 runs; native+ledger attributionadded; eight unfiled last units on seven runs, none a unit the harness filed; attribution unmoved by the ledger (App. G , Table 1 )
— the query’s finding handed to E1added; omission attribution up fourteen points, fabrication down fourteen, clean runs unchanged (App. G )
— the Q3 repair gate judged by a second vendor’s model, no further repairrun; 60.9%, p = 0.012, between the two readings fixed before the run: the gate did not hold for this judge, within the accuracy bound (§5.3, App. E , Table 28 )
— second audit instrument, deepseek-v4-pro, all 160 cells, GPT-5.2 criticrun; P38.1–P38.3 held, P38.4 failed on the critic’s count; the 31 cells where the two instruments disagree went to human re-judging (§3, App. B , Table 28 )
Appendix
Table 34
added measures, rules and analyseswhat they returned, and where
— reporting rule: the majority of three gradersadopted after all three gradings were seen, superseding “grader 1 stays primary”; grader 1 reported beside it (Table 1 )
— (after the registered shift was read) the independently witnessed share, the citations that land on material the harness did not writeadded; §5.2 and §6 report it beside the registered corroborated share, which it does not replace (App. F , Table 29 )
— (no new runs) shares by question; Q3 by placeholder presenceadded (App. F )
— (no new runs) the “answers, does not prove” criterion, accuracy at least 70% with independently witnessed share at most 25%, written down for the grader sensitivityadded after all gradings were seen; holds under graders 1, 3 and the majority for every examiner, under grader 2 for E1 only (§5.1, App. F )
— (no new runs) E2’s floor on E1’s 105 cells; the independently witnessed shift decomposed; each grader alone; a sequence-aware matching variantadded (App. F , App. G , Tables 28 and 29 ); none changes a body number
— (no new runs) task-clustered intervals for every headline shift, share and the staircase (not for the detection rates of Table 23 ); the independently witnessed share under the narration reading; E1’s Q3 rates split by gate outcomeadded (App. F , App. E , Table 28 ); no headline interval crosses zero when clustered; under the narration reading E2’s rise no longer clears zero; E1 detects no less on the pairs its family could not see
Appendix
Table 35
HeadlineSecond checkResult
No framework writes an evidentiary record (§3)A second instrument from another vendorIt agrees with the reported grid on 129 of 160 cells; none of the 32 present
Examiners catch at most half at a usable false-alarm rate (§5.3)A second vendor’s judge sat on the repair gateIt tells the copies apart at 60.9%: the gate fails for it, yet stays under the 65% bound fixed before that run
The reverse check reports the fabrications (§6)Same rules, 87 pairs from another vendorThey report 29 of 34 fabrications and every rewrite, below the registered 90%
Write-time falsehood caught, hash chain not (§6)The 28 runs were re-run under enforced isolationEvery count is unchanged
The dissociation holds (§5)Another vendor drove 140 runs; three gradersAttribution 83% and 69%, corroboration within five points; the dissociation criterion fails there (E3 at 69%, E1’s witnessed share 31.1%)
Appendix
Table 30: Each headline and its second check, run after registration; row 3 is held out (App. B , E , G ).
 Runtime-written spanIndependent witness
Writerthe runtime whose account is in questiona process the runtime does not control
What a later reader holdsone accounttwo accounts to compare
Audit actionretrieval and displayreconciliation; a mismatch is reported as a finding
Appendix
Table 37
More in this topic

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 262 papers

Explore similar work

CardsList
Aug 19, 2026

LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents

Large language model (LLM) agents can now carry out long-horizon technical workflows involving complex tool use, code execution, file edits, and generated artifacts. As agents do more work faster, the productivity bottleneck shifts from producing outputs to auditing whether those outputs are correct and trustworthy. Agent observability systems make fine-grained execution events visible, but visibility alone still leaves reviewers to reconstruct which actions, artifacts, and validation steps matter for a particular conclusion. We introduce LEDGER - Layered Evidence and Decision Graphs for Execution Review, a tracing and review system that builds layered trace graphs over observed agent sessions. LEDGER preserves Trace Records while grouping them into Evidence Nodes and Workflow Nodes, representing artifacts as evidence anchors, and adding typed semantic edges that connect claims to supporting actions, artifacts, and checks. Through data-analysis and coding examples, we show how the resulting traces expose workflow decisions, artifact lineage, repair steps, validation coverage, and claim-support paths for evidence-centered audit.

Daehong Kim, Haichao Miao, Shusen Liu

Carnegie Mellon University Pittsburgh, PA 15213 · Lawrence Livermore National Laboratory Livermore, CA 94550

AI Agent AuditingData Provenance
AskPDF
Oct 15, 2026

Agentic Interpretation: Harness-Governed Evidence Construction for Software Agents

Software agents answer behavioral questions about code, such as whether a patch passes tests or which lines caused a failure. These tasks require repository-wide evidence, yet most scaffolds leave intermediate findings in free-form trajectories, making them hard to reuse and easy to disconnect from final answers. We present Agentic Interpretation (AgInt), a framework for harness-governed evidence construction. The model proposes typed records, while a deterministic harness admits, grades, and persists them, reports unresolved requirements, and licenses answers under task-specific rules. The resulting certificate gives the model persistent external state and leaves an auditable decision record. Formally, append-only certificate states form a grounding lattice ordered by record inclusion, where each admitted record extends the state by union. We evaluate AgInt with four models on SWE-bench-Live and DeepSWE for patch acceptance and Defects4J and GitBug-Java for fault localization. AgInt outperforms the plain agent and Agentic Code Reasoning on all reported patch-acceptance metrics and ranks responsible lines higher while inspecting fewer irrelevant lines in fault localization. For GLM-5, reject-class F1 rises from 0.256 to 0.543 on SWE-bench-Live, All@1 rises from 0.140 to 0.400 on Defects4J, and wasted inspected lines fall from 31.5 to 11.7 on GitBug-Java. A matched-template control separates the mechanism: the persistent scaffold drives most fault-localization gains, while the certificate body and gate govern patch-acceptance calibration. With execution licensed, AgInt produces harness-reproduced witnesses for 293 of 475 null-dereference candidates versus 253 for a matched shell agent, and 62 of 117 resource-leak candidates versus 58. Across reported efficiency measurements, it uses 35% fewer executions and reduces cost per candidate by 59%.

Yifeng He, Yanjun Wang, Siwei Cui +2

University of California, Davis, Davis, California, USA · AWS AI Labs, USA

Software Engineering AgentsLLM Agent HarnessesSoftware Engineering
AskPDF
May 27, 2026

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +10

Peking University · Qiyuan Tech

Computer-Use Agent BenchmarksAI Agent ReliabilityLLM Agent EvaluationAI Agent BenchmarksLLM Agent HarnessesLLM Agent Reliability
AskPDF