Verifying numerical claims in research written by AI agents is an important but challenging problem. Recent benchmarks show that many agent-written papers fail integrity checks, but such verdicts, often produced by a single LLM judge, identify failure without explaining its nature, and existing checks either test a paper for internal consistency or test whether a reported value appears somewhere in an agent's output. We re-analyze all 540 released judgments from a recent public benchmark of AI research agents (nine agents, fifteen open research problems) and manually label 256 numerical claims from 18 submissions. Failures concentrate on alignment between the paper and the underlying work rather than overt cheating, and about four in five unsupported claims correspond to values in the agent's own files that come from the wrong experiment, metric, or run. We introduce a deterministic claim-to-evidence checker that scores whether the surrounding context identifies the claimed result and calibrates each claim against numerical decoys. With its rules fixed before labeling, it certifies claims with over 90% precision against blinded annotators (78 of 85; Cohen's kappa = 0.89), while verifying only a subset of claims; most claims it cannot verify are nevertheless traceable, so failure to verify is not evidence of falsehood. An LLM verifier given the same evidence recovers more traceable claims but accepts about three times as many misattributed ones, so the two are best used together. Applied to 99 submissions, the checker reveals large differences in traceability across agents, and a second judge from a different vendor preserves the official agent ranking while passing twice as many papers. These findings motivate provenance-first auditing of AI-produced research.
Figures & tables
Approach
Evidence
Unit
Separates
Human- validated
Judge robustness
statcheck, GRIM ( Nuijten et al., 2016 ; Brown and Heathers, 2017 )
Paper itself
Statistic
–
✓
–
FEVER, SciFact, SciTab, QuanTemp
Curated text or tables
Claim
–
✓
–
Attribution metrics ( Rashkin et al., 2023 ; Gao et al., 2023 ; Liu et al., 2023a )
Retrieved passages
Sentence
–
✓
–
MLR-Bench judge ( Chen et al., 2025a )
Paper, code, logs (LLM)
Submission
–
(✓)
–
Into the Unknown judge ( Kim et al., 2026 )
File system (LLM)
Rubric node
–
–
–
InnovationEval ( Owen, 2026 )
Transcripts (expert)
Submission
(✓)
–
–
Table 1: Positioning of this work relative to representative approaches. Evidence : what a claim is checked against. Separates : whether misattribution (value exists, wrong source) is distinguished from fabrication (value absent). Human-validated : checked against human labels on held-out data. Judge robustness : whether conclusions are tested under a second, independent judge. ✓ = yes; – = no or not applicable; (✓) = partially.
Agent
Vendor group
G1 passes
Hours
Rint
astra-6
OpenAI
11
1.3
97.8
sol-5.6
OpenAI
6
2.9
88.7
opus-5.5
Anthropic
4
5.3
90.4
opus-5
Anthropic
0
7.5
59.5
opus-4.8
Anthropic
0
4.8
65.3
fable-5
Anthropic
0
7.0
66.5
Table 2: Agents in the benchmark. G1: submissions (of 15) passing the integrity gate under the official consensus verdict. Hours: median wall-clock duration of a run. Rint : mean share of integrity nodes passed (%).
Figure 1: Overview of the study. The checker is developed on a development sample and evaluated, with its rules frozen, on held-out submissions against blinded human labels.
Quantity
Definition and interpretation
Precision
Share of certified claims that annotators labeled traceable. How far a positive verdict can be trusted.
Recall
Share of traceable claims that the checker certifies. How much of the verifiable content it finds.
Coverage
Share of all real claims that the checker certifies. How much of a paper it settles automatically.
Traceable among uncertified
Share of claims the checker did not certify that annotators labeled traceable. How often a negative verdict would be a false accusation if read as one.
Decoy acceptance
Share of held-out decoy values (offset +7 ) that would be certified. A label-free estimate of false certification.
Table 3: Quantities used to evaluate the checker. “Real claims” excludes claims labeled not a claim .
Figure 2: Failure rates of integrity (G1) rubric families under the official judge’s consensus verdicts, with failed and counted nodes in parentheses. Criteria comparing the write-up with the work (blue) fail far more often than criteria targeting overt misconduct or selection (orange).
Label
Development
Confirmation
Total
Traceable
112
46
158
Wrong source
35
12
47
Not found
4
2
6
Not a claim
33
12
45
Total labeled
184
72
256
Wrong source / unsupported
90%
86%
89%
Table 4: Human labels. Development: first annotator, ten submissions (after excluding practice items). Confirmation: first annotator, eight held-out submissions. “Wrong source / unsupported” is the share of wrong-source claims among wrong-source plus not-found claims, raw and reweighted to the population of claims.
Pattern
Reported in the paper
Where the value actually occurs
Wrong statistic
“0.045 vs. 0.026” relative L2 error of the proposed encoder
A standard deviation ( rel_l2_std =0.0453 ) of a different ablation run
Wrong model or arm
Fidelity 0.9992 for the proposed method
final_fid =0.99920 of the unconstrained baseline run
Wrong aggregation
Mean relative L2 over three seeds
A single seed’s value, or the absolute rather than relative error
Wrong condition
Error 0.144 for the raw-input control at coefficient (2,1)
rel_mean =0.1437 of a different (learned-encoder) arm
Unrelated line
CNOT overhead multiplier 1.57
A timing line, one_step 1.57 s
Setting, not result
Encoder result 0.1000
A training-profile configuration value
Table 5: Recurring patterns among wrong-source claims, with one example of each from the labeled samples. In every case the reported value exists in the agent’s files, but not as the quantity the paper describes.
Development (test split, n=69 )
Confirmation ( n=60 )
Decoy
Setting
Precision
Recall
Precision
Recall
acceptance
Level 1 (value match)
0.77
1.00
0.69
1.00
88%
Strict
0.97 [0.81, 1.00]
0.39 [0.24, 0.54]
0.95 [0.85, 1.00]
0.34 [0.24, 0.47]
1.3%
Balanced
0.84 [0.58, 0.96]
0.49 [0.38, 0.84]
0.88 [0.70, 0.98]
0.51 [0.33, 0.68]
3.8%
Broad
0.86 [0.61, 0.98]
0.71 [0.58, 0.91]
0.89 [0.74, 0.98]
0.55 [0.41, 0.71]
8.7%
Table 6: Checker performance against the first annotator’s labels (real claims only). Precision and recall are population-weighted with 95% cluster-bootstrap intervals. Decoy acceptance is computed over all 1,539 development claims. Operating points were frozen before the confirmation sample was drawn.
Figure 3: Precision and recall of the checker on the held-out confirmation sample, with 95% cluster-bootstrap intervals. Value matching (gray) finds every traceable claim but accepts most untraceable ones; the strict operating point trades recall for high precision.
Sample
Real claims
Certified
Traceable
Precision [exact 95% CI]
Development (all)
151
56
51
0.91 [0.80, 0.97]
Confirmation (held-out)
60
29
27
0.93 [0.77, 0.99]
Pooled
211
85
78
0.92 [0.84, 0.97]
Table 7: Precision of the strict checker with counts. Exact intervals are Clopper–Pearson 95% intervals on the unweighted counts; they ignore clustering by submission and are reported as a conservative complement to the weighted, cluster-bootstrap estimates in Table 6 .
Rule
Certified
Traceable
Precision [95% CI]
Recall
Decoy acceptance
Value appears anywhere (Level 1)
208
158
0.76 [0.70, 0.82]
1.00 (1.00)
87.4%
+ ignore bulk data
206
158
0.77 [0.70, 0.82]
1.00 (1.00)
71.7%
+ context score S(c)≥τ (no decoys)
103
95
0.92 [0.85, 0.97]
0.60 (0.54)
18.9%
Decoy test only, p(c)≤ρ (no threshold)
120
104
0.87 [0.79, 0.92]
0.66 (0.51)
3.8%
Full strict: score + decoy test
85
78
0.92 [0.84, 0.97]
0.49 (0.35)
1.3%
Table 8: Component ablation on all 211 labeled real claims (158 traceable), pooling development and confirmation samples. Precision with exact 95% intervals on unweighted counts; recall unweighted (population-weighted in parentheses). Decoy acceptance over all 1,539 development claims.
Method
Certified
Traceable
Precision [95% CI]
Recall
Decoy acceptance
Value appears anywhere (Level 1)
127
94
0.74 [0.65, 0.81]
1.00
–
Embedding score (no decoys)
21
20
0.95 [0.76, 1.00]
0.21
4.7%
Embedding score + decoy test
54
40
0.74 [0.60, 0.85]
0.43
3.3%
LLM verifier (Claude Opus 5.5)
102
91
0.89 [0.82, 0.94]
0.97
–
Checker and LLM agree
52
51
0.98 [0.90, 1.00]
0.54
–
Checker (strict)
57
51
0.89 [0.78, 0.96]
0.54
1.4%
Table 9: Comparison with baselines on the 129 labeled real claims not used for tuning (development test split and confirmation sample; 94 traceable). Precision with exact 95% intervals on unweighted counts. Decoy acceptance on the decoys of the 211 labeled claims; not defined for the LLM verifier, which sees only the top-ranked entries.
Figure 4: Share of claims certified by the strict checker against the official integrity score. Small markers are submissions ( n=99 ); large markers are agent means, with the number of checked submissions in parentheses. Certified shares are lower bounds on traceability, since the checker certifies about a third of traceable claims.
Agent
Subm.
Claims
Certified
Untraced
Rint
Judge NT fail
astra-6
4
113
75.6%
0.8%
98.5
0.0%
sol-5.6
7
729
62.8%
0.8%
92.4
8.3%
kimi-k3
15
1,950
17.3%
6.1%
57.4
58.3%
glm-5.3
11
1,746
16.7%
5.1%
61.0
38.9%
opus-5.5
8
1,206
13.8%
1.0%
92.9
13.9%
opus-5
14
3,301
11.5%
4.5%
59.9
58.3%
Table 10: Benchmark-wide traceability by agent (strict checker, 99 submissions). Certified: mean share of a submission’s claims the checker certifies (a lower bound on traceability). Untraced: mean share of claims whose value occurs in no result file. Judge NT fail: official judge’s failure rate on number-traceability nodes (all 15 submissions per agent).
Agent vendor group
Submissions
Official judge
Second judge
d (points)
OpenAI
8
3.5%
1.3%
−2.2
Anthropic
13
16.7%
8.1%
−8.6
Open-weight
9
27.6%
23.9%
−3.8
All
30
−5.4
Table 11: G1 node failure rates by judge and agent vendor group on the 30 re-graded submissions (means over submissions). d : second judge minus official judge.
Figure 5: G1 node failure rates under the official judge (black) and the second judge (colored by the agent’s vendor group) on the 30 re-graded submissions, by agent. The second judge is more lenient for every agent, but both judges order the agents similarly.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
(1) No fixed effects
(2) Agent FE
(3) Agent + task FE
logh (hours)
−5.21
−0.33
−0.61
(1.28)
(0.99)
(1.12)
p -value
<0.001
0.74
0.59
Agent fixed effects
–
✓
✓
Task fixed effects
–
–
✓
R2
0.11
0.61
0.77
Appendix
Table 1: Integrity score and run length: OLS estimates of Eq. ( 1 ) with HC1 standard errors ( n=135 submissions).
Cause
Claims
What happens
Score just below threshold
13
Correct entry found and beats every decoy; context overlap diluted by sentences reporting several numbers
Decoys match equally well
6
Dense grids of similar results with similar keys; p(c)>0.1
Another entry scores best
18
A log line, a generic key, or a near-duplicate entry outranks the correct one
Total
37
Appendix
Table 2: Why the strict checker misses traceable claims: the 37 traceable claims it did not certify on the development tuning split.
AI scientist systems are described as tools, coauthors, or founders, but we evaluate them as if only the final answer matters. This position paper argues that outcome-only evaluation is insufficient, and that task outcome, mechanism fidelity, and epistemic honesty must be measured separately. Our evidence comes from 28 episodes of a coding agent attempting to rediscover a known particle identification observable in a Geant4 simulation, including an 8-episode probe across two additional frontier models. In 4/20 primary-model and 3/8 cross-model episodes, agents reach right-looking results through incorrect reasoning that breaks when conditions change, which we call Correct Answer, Wrong Mechanism (CAWM). Honesty and mechanism fidelity dissociate within a single agent trajectory. When given a partially misleading prior, all five agents reject the false component on evidence, yet one defends its chosen observable with physics inconsistent with its own data. In the simulation-based discovery setting studied here, coding agents prove reliable tools but unreliable scientific co-authors for open-ended claim-making, where co-author trust requires mechanism-fidelity verification they do not reliably self-apply. The failure is detectable, and we propose a lightweight test. A one-step regime-shift check needs only the agent's claim and flags the over-generalized cases. A companion recomputation flags the remaining cases when the correct observable is known. Together, these checks flag every CAWM case in this study.
Steven Young Eulig
Department of Physics and Laboratory for Particle Physics and Cosmology (LPPC), Harvard University, Cambridge, MA 02138, USA.
AI agents can draft claims faster than authors can check whether the cited or retrieved evidence supports them. We study evidence-ledger adjudication: a claim-evidence traceability workflow that pairs each claim with an evidence packet, assigns a support relation, and routes unsupported, contradicted, or mixed-evidence claims back to the author. The empirical core is a 2,335-row blind benchmark built from independent external labels in AVeriTeC, CLIMATE-FEVER, and SciFact. Gold relations and source evidence labels are hidden during prediction and joined only for scoring. On this benchmark, the agent evidence-ledger condition achieves 0.676 relation accuracy and 0.601 macro-F1, compared with 0.383 accuracy and 0.303 macro-F1 for the best non-agent baseline. It also routes 1270/1435 claims whose gold labels indicate contradiction, missing evidence, or mixed evidence, while routing 295/900 supported claims. These results show that evidence-ledger adjudication can turn heterogeneous evidence packets into an auditable traceability layer for AI-assisted writing.
Large language model agents are increasingly capable of conducting research autonomously, producing research documents alongside the code and experiments that ostensibly support them. Yet whether the reported findings are consistently supported by corresponding implementations and execution evidence remains largely unexplored: existing review practices primarily assess textual quality and cannot reliably identify inconsistencies such as hard-coded metrics, unimplemented methods, or unsupported experimental results. We present ReAgent, an automated auditing framework for assessing the consistency between agent-generated research documents and their associated repositories. ReAgent constructs structured representations of scientific claims from research documents and uses them to guide repository analysis and evidence collection. Static auditing examines whether claimed methodologies, implementations, and experimental configurations are consistently reflected in the repository, while dynamic auditing executes relevant experiments and collects execution evidence to assess empirical findings. By combining static analysis with dynamic evidence, ReAgent identifies inconsistencies that may remain hidden under either perspective alone, such as experiments that reproduce reported numbers while deviating from the claimed methodology. The collected evidence and audit decisions are organized into a structured repository-level audit report, enabling transparent evidence traceability. We evaluate ReAgent on a manually curated benchmark of agent-generated research document--repository pairs and compare it against representative static and reproduction-based baselines. Experimental results demonstrate that ReAgent effectively identifies inconsistencies between reported research findings and their supporting repository evidence.