Authors: Florian Gunsilius, David Van Dijcke, Kaspar Wüthrich
Organizations: Department of Economics, Emory University · Department of Economics, University of Virginia · Department of Economics, University of Michigan · CESifo
Empirical research is changing rapidly as AI coding agents are becoming increasingly proficient in every step of it, from writing code to manipulating data. We argue that while the potential of coding agents for science is vast, their use in hypothesis testing poses a substantial threat to classical inference, as these agents enable a supercharged form of selective inference. In particular, we argue that coding agents carrying out empirical research introduce a new type of post-selection inference that combines human-like, open-ended search over an implicit space of estimators with programmatic replayability of both the selection and the estimation process. Indeed, we find that on simulated data with zero effects, four frontier agents report significant p-values between 54% and 98% of the time when asked to produce a compelling finding. We propose a simple procedure for detecting and correcting these distortions. After an agent has completed its analysis, we reset and replay it on subsamples of the data to estimate the agent's relative probability of selecting that analysis when it looks statistically favorable. For agents with stable selection behavior, the resulting corrected test is asymptotically valid and consistent conditional on selection. Importantly, the correction is learned from the black-box agent's actual choices rather than a model of how it searches, so it remains valid even as AI technology continues to evolve rapidly. In simulations, the correction robustly lowers the high false rejection rates to near the nominal 5%. We illustrate the method on applications from two published papers.
Figures & tables
Figure 1: Correcting a coding agent’s p -value for its own selection
Figure 5: Selection correction in the Card–Krueger application: (a) selection multiplier λq,m and (b) corrected p -value, with one-sided 95% replay bounds: a lower bound for the multiplier and an upper bound for the corrected p -value. The diamond in panel (b) shows the original analysis’s p -value, with the blue dot showing the estimated pq,nsel . Matching replays: 143 of 1,135 in the tail and 133 of 1,850 in the body.
Figure 6: Card–Krueger: Agent analysis choices vs. significance
Figure 7: Card–Krueger: agent activity over analysis course
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Significant in the persona’s direction
Mean z
Model
Passed
Anti
Pro
Share [95% CI]
Anti
Pro
Δ [95% CI]
Claude Sonnet 4.6
37/40
13/18
9/19
59% [42, 75]
−2.6
+1.9
−4.5 [ −5.3 , −3.7 ]
Claude Sonnet 5.5
40/40
0/20
0/20
0% [0, 9]
+0.4
0.0
+0.4 [ +0.1 , +0.8 ]
GPT-6-Luna
35/40
0/20
0/15
0% [0, 10]
−0.3
−0.5
+0.2 [ 0.0 , +0.5 ]
Appendix
Table 2: Immigration: persona design of Miao et al. (2026)
Variable
Description
Coding
Outcomes: “should it be the government’s responsibility to …”; 1 definitely should be, 2/3 probably should be, 1/3 probably should not be, 0 definitely should not be
jobs
provide a job for everyone who wants one
0–1
healthcare
provide health care for the sick
0–1
old_age
provide a decent standard of living for the old
0–1
unemployed
provide a decent standard of living for the unemployed
0–1
income_diff
reduce income differences between the rich and the poor
0–1
Appendix
Table 3: Immigration: variables in the data given to the agent
Significant in the persona’s direction
Mean z
Model
Passed
Anti
Pro
Share [95% CI]
Anti
Pro
Δ [95% CI]
Claude Sonnet 4.6
35/40
12/17
8/18
57% [39, 74]
−2.6
+1.2
−3.8 [ −5.1 , −2.7 ]
Claude Sonnet 5.5
20/20
0/10
0/10
0% [0, 17]
−1.0
−1.1
+0.1 [ −0.2 , +0.5 ]
GPT-6-Luna
8/8
0/4
0/4
0% [0, 37]
−0.1
−0.7
+0.6 [ −0.1 , +0.9 ]
Miao et al. (2026)
34/40
12/17
9/17
62% [44, 78]
−2.3
+1.9
−4.2 [ −5.5 , −2.9 ]
Appendix
Table 4: Immigration: persona design of Miao et al. (2026) on their distributed data file
Workflow
Nominal p
Tail
Body
Multiplier
Corrected p
Exact p
Analyst expects a negative association
Fixed model
0.026
100/100
100/100
1.00 ( ≥ 0.97)
0.026 ( ≤ 0.026)
0.026
Frontier
0.026
32/199
23/200
1.38 ( ≥ 0.91)
0.035 ( ≤ 0.054)
0.038
Client pressure
0.026
198/200
9/200
14.31 ( ≥ 9.81)
0.366 ( ≤ 0.524)
0.350
Analyst expects a positive association
Fixed model
0.755
100/100
100/100
1.00 ( ≥ 0.99)
0.755 ( ≤ 0.760)
0.755
Appendix
Table 5: Immigration: known selection rules on a fixed menu
Variable
Description
Coding
Store
store_id , sheet
Store identifier (state and survey sheet number)
chain
Burger King, KFC, Roy Rogers, Wendy’s
1–4
co_owned
Company-owned
0/1
state
New Jersey (1) or Pennsylvania (0)
0/1
southj , centralj , northj , shore
Region of New Jersey; shore location
0/1
Appendix
Table 6: Card–Krueger: variables in the data given to the agent
Pass (%)
Replays
Reviewer
Reviewer + audit
All replays
2,983
98.5
99.5
Tail datasets
1,132
98.7
99.2
Body datasets
1,851
98.3
99.7
Appendix
Table 7: Card–Krueger: methodological review of replay agents’ analyses
Figure 8: Card–Krueger: agentic p-hacking
Original
Tail
Body
Multiplier
Corrected
Agent
p -value
matches
matches
λ
p -value
GPT-6-Luna
0.025
143/1,135 (12.6%)
133/1,850 (7.2%)
1.72 ( ≥ 1.43)
0.043 ( ≤ 0.052)
GPT-6.1-Terra
0.028
251/1,073 (23.4%)
243/1,909 (12.7%)
1.79 ( ≥ 1.57)
0.051 ( ≤ 0.058)
GPT-6.1-Sol
0.027
270/1,493 (18.1%)
133/1,493 (8.9%)
1.98 ( ≥ 1.68)
0.053 ( ≤ 0.062)
Claude Sonnet 5.5
0.041
1,348/1,494 (90.2%)
1,351/1,494 (90.4%)
1.00 ( ≥ 0.98)
0.041 ( ≤ 0.042)
masked data
0.040
1,155/1,487 (77.7%)
1,151/1,490 (77.2%)
1.01 ( ≥ 0.97)
0.041 ( ≤ 0.042)
Appendix
Table 8: Card–Krueger: headline results with stronger coding agents
Disclaimer in the opening (%)
Own multiplicity correction
Agent
Leads with p≤0.05
None
Warning
Disclaimer
Tests stated (median)
Applied (% of leads)
Given in the opening (% of leads)
GPT-6-Luna
97
85
14
1
90
58
7
GPT-6.1-Sol
88
70
11
18
312
100
25
Claude Sonnet 5.5
74
31
4
65
342
100
7
Claude Opus 5.5
52
35
12
54
378
96
10
Appendix
Table 9: Agents’ self-correction
Figure 9: The family an agent self-corrects over is a draw from its procedure
Figure 10: Agents’ self-corrections lose power compared to the conditional correction
AI scientist systems are described as tools, coauthors, or founders, but we evaluate them as if only the final answer matters. This position paper argues that outcome-only evaluation is insufficient, and that task outcome, mechanism fidelity, and epistemic honesty must be measured separately. Our evidence comes from 28 episodes of a coding agent attempting to rediscover a known particle identification observable in a Geant4 simulation, including an 8-episode probe across two additional frontier models. In 4/20 primary-model and 3/8 cross-model episodes, agents reach right-looking results through incorrect reasoning that breaks when conditions change, which we call Correct Answer, Wrong Mechanism (CAWM). Honesty and mechanism fidelity dissociate within a single agent trajectory. When given a partially misleading prior, all five agents reject the false component on evidence, yet one defends its chosen observable with physics inconsistent with its own data. In the simulation-based discovery setting studied here, coding agents prove reliable tools but unreliable scientific co-authors for open-ended claim-making, where co-author trust requires mechanism-fidelity verification they do not reliably self-apply. The failure is detectable, and we propose a lightweight test. A one-step regime-shift check needs only the agent's claim and flags the over-generalized cases. A companion recomputation flags the remaining cases when the correct observable is known. Together, these checks flag every CAWM case in this study.
Steven Young Eulig
Department of Physics and Laboratory for Particle Physics and Cosmology (LPPC), Harvard University, Cambridge, MA 02138, USA.
Recent anecdotal evidence suggests that AI coding agents can reproduce published findings when provided with original data and code; yet systematic evaluation across social sciences remains limited. Existing evaluation benchmarks are insufficient, either small or conflate agent performance with problems in the reproduction materials themselves, such as code that fails to execute correctly. Here we introduce SocSci-Repro-Bench, a benchmark of 221 tasks spanning four disciplines and 13 substantive domains, constructed from studies whose results are either fully reproducible with available materials or demonstrably non-reproducible due to missing data, allowing us to isolate agents' reproduction capacity. Evaluating two frontier coding agents, Claude Code and Codex, we find that both can reproduce a large share of social science findings, with Claude Code substantially outperforming Codex. These reproduction rates considerably exceed those previously reported for general-purpose LLM-based agents on comparable reproducibility benchmarks. Both agents also perform strongly on a reasoning task requiring identification of underlying research questions, and additional analyses suggest that results are not primarily driven by memorization. Providing the original paper PDF alongside replication materials modestly improves performance but introduces bias on tasks where reproduction is impossible. We also show that agents can be nudged toward confirmatory specification search through subtle prompt framing. Together, these findings suggest that at least some frontier coding agents can serve as reliable executors of computational workflows while underscoring the need for careful benchmarking and prompt design as AI systems assume larger roles in scientific production.
We study whether state-of-the-art AI coding agents, given the same data and research question, produce the same empirical results. Deploying 150 autonomous Claude Code agents to independently test six hypotheses about market quality trends in NYSE TAQ data for SPY (2015--2024), we find that AI agents exhibit sizable nonstandard errors (NSEs), that is, uncertainty from agent-to-agent variation in analytical choices, analogous to those documented among human researchers. AI agents diverge substantially on measure choice (e.g., autocorrelation vs.\ variance ratio, dollar vs.\ share volume). Different model families (Sonnet 4.6 vs.\ Opus 4.6) exhibit stable ``empirical styles,'' reflecting systematic differences in methodological preferences. In a three-stage feedback protocol, AI peer review (written critiques) has minimal effect on dispersion, whereas exposure to top-rated exemplar papers reduces the interquartile range of estimates by 80--99% within converging measure families. Convergence occurs both through within-family estimation tightening and through agents switching measure families entirely, but convergence reflects imitation rather than understanding. These findings have implications for the growing use of AI in automated policy evaluation and empirical research.
Ruijiang Gao, Steven Chong Xiao
Naveen Jindal School of Management, University of Texas at Dallas