Authors: Florian Gunsilius, David Van Dijcke, Kaspar Wüthrich
Organizations: Department of Economics, Emory University · Department of Economics, University of Virginia · Department of Economics, University of Michigan · CESifo
Empirical research is changing rapidly as AI coding agents are becoming increasingly proficient in every step of it, from writing code to manipulating data. We argue that while the potential of coding agents for science is vast, their use in hypothesis testing poses a substantial threat to classical inference, as these agents enable a supercharged form of selective inference. In particular, we argue that coding agents carrying out empirical research introduce a new type of post-selection inference that combines human-like, open-ended search over an implicit space of estimators with programmatic replayability of both the selection and the estimation process. Indeed, we find that on simulated data with zero effects, four frontier agents report significant p-values between 54% and 98% of the time when asked to produce a compelling finding. We propose a simple procedure for detecting and correcting these distortions. After an agent has completed its analysis, we reset and replay it on subsamples of the data to estimate the agent's relative probability of selecting that analysis when it looks statistically favorable. For agents with stable selection behavior, the resulting corrected test is asymptotically valid and consistent conditional on selection. Importantly, the correction is learned from the black-box agent's actual choices rather than a model of how it searches, so it remains valid even as AI technology continues to evolve rapidly. In simulations, the correction robustly lowers the high false rejection rates to near the nominal 5%. We illustrate the method on applications from two published papers.
Figures & tables
Figure 1: Correcting a coding agent’s p -value for its own selection
Figure 5: Selection correction in the Card–Krueger application: (a) selection multiplier λq,m and (b) corrected p -value, with one-sided 95% replay bounds: a lower bound for the multiplier and an upper bound for the corrected p -value. The diamond in panel (b) shows the original analysis’s p -value, with the blue dot showing the estimated pq,nsel . Matching replays: 143 of 1,135 in the tail and 133 of 1,850 in the body.
Figure 6: Card–Krueger: Agent analysis choices vs. significance
Figure 7: Card–Krueger: agent activity over analysis course
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Significant in the persona’s direction
Mean z
Model
Passed
Anti
Pro
Share [95% CI]
Anti
Pro
Δ [95% CI]
Claude Sonnet 4.6
37/40
13/18
9/19
59% [42, 75]
−2.6
+1.9
−4.5 [ −5.3 , −3.7 ]
Claude Sonnet 5.5
40/40
0/20
0/20
0% [0, 9]
+0.4
0.0
+0.4 [ +0.1 , +0.8 ]
GPT-6-Luna
35/40
0/20
0/15
0% [0, 10]
−0.3
−0.5
+0.2 [ 0.0 , +0.5 ]
Appendix
Table 2: Immigration: persona design of Miao et al. (2026)
Variable
Description
Coding
Outcomes: “should it be the government’s responsibility to …”; 1 definitely should be, 2/3 probably should be, 1/3 probably should not be, 0 definitely should not be
jobs
provide a job for everyone who wants one
0–1
healthcare
provide health care for the sick
0–1
old_age
provide a decent standard of living for the old
0–1
unemployed
provide a decent standard of living for the unemployed
0–1
income_diff
reduce income differences between the rich and the poor
0–1
Appendix
Table 3: Immigration: variables in the data given to the agent
Significant in the persona’s direction
Mean z
Model
Passed
Anti
Pro
Share [95% CI]
Anti
Pro
Δ [95% CI]
Claude Sonnet 4.6
35/40
12/17
8/18
57% [39, 74]
−2.6
+1.2
−3.8 [ −5.1 , −2.7 ]
Claude Sonnet 5.5
20/20
0/10
0/10
0% [0, 17]
−1.0
−1.1
+0.1 [ −0.2 , +0.5 ]
GPT-6-Luna
8/8
0/4
0/4
0% [0, 37]
−0.1
−0.7
+0.6 [ −0.1 , +0.9 ]
Miao et al. (2026)
34/40
12/17
9/17
62% [44, 78]
−2.3
+1.9
−4.2 [ −5.5 , −2.9 ]
Appendix
Table 4: Immigration: persona design of Miao et al. (2026) on their distributed data file
Workflow
Nominal p
Tail
Body
Multiplier
Corrected p
Exact p
Analyst expects a negative association
Fixed model
0.026
100/100
100/100
1.00 ( ≥ 0.97)
0.026 ( ≤ 0.026)
0.026
Frontier
0.026
32/199
23/200
1.38 ( ≥ 0.91)
0.035 ( ≤ 0.054)
0.038
Client pressure
0.026
198/200
9/200
14.31 ( ≥ 9.81)
0.366 ( ≤ 0.524)
0.350
Analyst expects a positive association
Fixed model
0.755
100/100
100/100
1.00 ( ≥ 0.99)
0.755 ( ≤ 0.760)
0.755
Appendix
Table 5: Immigration: known selection rules on a fixed menu
Variable
Description
Coding
Store
store_id , sheet
Store identifier (state and survey sheet number)
chain
Burger King, KFC, Roy Rogers, Wendy’s
1–4
co_owned
Company-owned
0/1
state
New Jersey (1) or Pennsylvania (0)
0/1
southj , centralj , northj , shore
Region of New Jersey; shore location
0/1
Appendix
Table 6: Card–Krueger: variables in the data given to the agent
Pass (%)
Replays
Reviewer
Reviewer + audit
All replays
2,983
98.5
99.5
Tail datasets
1,132
98.7
99.2
Body datasets
1,851
98.3
99.7
Appendix
Table 7: Card–Krueger: methodological review of replay agents’ analyses
Figure 8: Card–Krueger: agentic p-hacking
Original
Tail
Body
Multiplier
Corrected
Agent
p -value
matches
matches
λ
p -value
GPT-6-Luna
0.025
143/1,135 (12.6%)
133/1,850 (7.2%)
1.72 ( ≥ 1.43)
0.043 ( ≤ 0.052)
GPT-6.1-Terra
0.028
251/1,073 (23.4%)
243/1,909 (12.7%)
1.79 ( ≥ 1.57)
0.051 ( ≤ 0.058)
GPT-6.1-Sol
0.027
270/1,493 (18.1%)
133/1,493 (8.9%)
1.98 ( ≥ 1.68)
0.053 ( ≤ 0.062)
Claude Sonnet 5.5
0.041
1,348/1,494 (90.2%)
1,351/1,494 (90.4%)
1.00 ( ≥ 0.98)
0.041 ( ≤ 0.042)
masked data
0.040
1,155/1,487 (77.7%)
1,151/1,490 (77.2%)
1.01 ( ≥ 0.97)
0.041 ( ≤ 0.042)
Appendix
Table 8: Card–Krueger: headline results with stronger coding agents
Disclaimer in the opening (%)
Own multiplicity correction
Agent
Leads with p≤0.05
None
Warning
Disclaimer
Tests stated (median)
Applied (% of leads)
Given in the opening (% of leads)
GPT-6-Luna
97
85
14
1
90
58
7
GPT-6.1-Sol
88
70
11
18
312
100
25
Claude Sonnet 5.5
74
31
4
65
342
100
7
Claude Opus 5.5
52
35
12
54
378
96
10
Appendix
Table 9: Agents’ self-correction
Figure 9: The family an agent self-corrects over is a draw from its procedure
Figure 10: Agents’ self-corrections lose power compared to the conditional correction