Nonstandard Errors in AI Agents

Ruijiang Gao; Steven Chong Xiao

cs.AIMar 17, 2026

Nonstandard Errors in AI Agents

Authors: Ruijiang Gao, Steven Chong Xiao

Organizations: Naveen Jindal School of Management, University of Texas at Dallas

Abstract

We study whether state-of-the-art AI coding agents, given the same data and research question, produce the same empirical results. Deploying 150 autonomous Claude Code agents to independently test six hypotheses about market quality trends in NYSE TAQ data for SPY (2015--2024), we find that AI agents exhibit sizable nonstandard errors (NSEs), that is, uncertainty from agent-to-agent variation in analytical choices, analogous to those documented among human researchers. AI agents diverge substantially on measure choice (e.g., autocorrelation vs.\ variance ratio, dollar vs.\ share volume). Different model families (Sonnet 4.6 vs.\ Opus 4.6) exhibit stable ``empirical styles,'' reflecting systematic differences in methodological preferences. In a three-stage feedback protocol, AI peer review (written critiques) has minimal effect on dispersion, whereas exposure to top-rated exemplar papers reduces the interquartile range of estimates by 80--99% within converging measure families. Convergence occurs both through within-family estimation tightening and through agents switching measure families entirely, but convergence reflects imitation rather than understanding. These findings have implications for the growing use of AI in automated policy evaluation and empirical research.

Figures & tables

Panel of Figure 1 — Figure 1: Experimental design overview. 150 AI agents independently analyze the same NYSE TAQ data (Stage 1), receive AI peer review (Stage 2), and see the five highest-rated exemplar papers (Stage 3). At each stage, agents face a “garden of forking paths” ( Steegen et al., 2016 ) : measure choice, functional form, and other decision forks. Peer review causes undirected movement (IQR unchanged); exemplar exposure causes correlated imitation (IQR collapses within measure families).

Hyp	$N$	Q10	Q25	Median	Q75	Q90	IQR	IDR
H1	150	$-$ 1.65	$-$ 0.74	$-$ 0.06	1.70	2.86	2.43	4.51
H2	150	$-$ 6.74	$-$ 6.63	$-$ 6.21	$-$ 6.21	$-$ 6.19	0.43	0.55
H3	150	1.17	2.33	4.75	7.61	9.21	5.28	8.05
H4	150	$-$ 4.71	$-$ 4.60	5.84	6.09	6.09	10.69	10.80
H5	150	2.98	3.00	3.08	3.54	3.67	0.54	0.69
H6	150	$-$ 14.97	$-$ 13.23	$-$ 10.31	$-$ 2.89	$-$ 1.35	10.34	13.62

Table 1: Stage 1 effect size distribution (%/yr) across 150 AI agents

Figure 2: Effect size distributions (%/yr) by stage. Boxes show IQR (Q25–Q75) with IQR values annotated; whiskers extend to Q10 and Q90; horizontal line is the median. H2 and H5 have small IQR in S1, driven almost entirely by the log-vs-level specification fork (within either specification, agents agree to within 0.01%/yr). H4 is bimodal in S1 (dollar vs. share volume). H5 collapses in S3 as 96% of agents adopt year dummies. H6 converges in S3 as 99% of agents adopt trade-level price impact.

Hypothesis	$N$	Median	IQR	IDR
H1:autocorrelation	87	$-$ 0.33	2.73	7.40
H1:variance_ratio	63	$-$ 0.05	1.45	2.70
H2 (all)	150	$-$ 6.21	0.43	0.55
H3 (all)	150	4.75	5.28	8.05
H4:dollar_volume	90	6.09	0.25	0.82
H4:share_volume	60	$-$ 4.60	0.11	0.55

Table 2: Stratified Stage 1 NSE: H1 and H4 by measure family

Hyp	Overall IQR	Dominant source	Within-source IQR
H1	2.43	Hypothesis abstraction (AC vs VR) + sub-forks (abs/signed, 1min/5min)	AC: 2.74, VR: 1.45. Within AC sub-cells: 0.02–0.34.
H2	0.43	Specification (log vs level OLS)	Level: 0.003. Log: 0.11
H3	5.28	Specification (trade weighting)	Equal-wtd: 3.26. Vol-wtd: 2.04. Dollar-wtd: 1.26
H4	10.69	Hypothesis abstraction (dollar vs share volume)	Dollar: 0.25. Share: 0.11
H5	0.54	Specification (log vs level OLS)	Level: 0.005. Log: 0.12
H6	10.34	Hypothesis abstraction (price impact vs Amihud vs Kyle $\lambda$ )	Price impact: 2.59. Amihud: 1.35

Table 3: Sources of AI NSE by hypothesis (Stage 1)

Hypothesis	Fork	Sonnet 4.6 ( $N$ =100)	Opus 4.6 ( $N$ =50)	Test
H1	Autocorrelation	87%	0%	10.97***
	Variance ratio	13%	100%
	Level OLS	96%	36%
	Log OLS	4%	64%
H2	Level OLS	90%	14%	37.64***
H2	Log OLS	10%	86%	37.64***

Table 4: Model-specific empirical styles (Stage 1)

Hyp	S1 IQR	S2 IQR	S3 IQR	S1 $\to$ S2	S2 $\to$ S3	S1 $\to$ S3
H1	2.43	2.69	5.68	$+$ 10.7%	$+$ 111.1%	$+$ 133.6%
H2	0.43	0.43	0.01	$+$ 0.2%	$-$ 98.1%	$-$ 98.1%
H3	5.28	5.56	3.01	$+$ 5.4%	$-$ 46.0%	$-$ 43.0%
H4	10.69	10.69	10.69	$+$ 0.0%	$+$ 0.0%	$+$ 0.0%
H5	0.54	0.54	0.96	$+$ 0.2%	$+$ 78.1%	$+$ 78.4%
H6	10.34	10.44	2.09	$+$ 1.0%	$-$ 80.0%	$-$ 79.8%

Table 5: IQR convergence across stages

Figure 3: IQR by stage for each hypothesis. Blue = S1, orange = S2, green = S3. The transition from S1 to S2 (peer review) produces minimal change. The transition from S2 to S3 (top papers) drives convergence for H2, H3, and H6, and divergence for H1 and H5.

Figure 4: Methodology choice convergence across stages. Each stacked bar shows the fraction of agents using each method. H5 year dummies: adoption rises from 1% (S1) to 96% (S3). H6 impact measure: price impact rises from 56% to 99%. H1 measure: autocorrelation drops from 58% to 17% as agents switch to variance ratio after seeing top papers.

Family	$N_{\text{S1}}$	$N_{\text{S3}}$	S1 IQR	S3 IQR	$\Delta$ IQR	$p$ -value
Panel A: All agents (S1 and S3 populations may differ)
H1:autocorrelation	87	25	2.74	0.13	$-$ 95.4%
H1:variance_ratio	63	125	1.45	5.51	$+$ 281.5%
H4:dollar_volume	90	45	0.25	0.007	$-$ 97.0%
H4:share_volume	60	88	0.11	0.001	$-$ 99.3%
Panel B: Stayers only (same agents in S1 and S3) and unstratified

Table 6: Stratified IQR convergence across stages

Category	Fork	Distribution
Measure	H1 measure family	autocorrelation 58%, variance ratio 42%
	H1 abs vs. signed	absolute 73%, signed 27%
	H4 measure family	dollar volume 60%, share volume 40%
	H6 measure type	price impact 56%, Amihud 40%, Kyle $\lambda$ 4%
Estimation	Functional form	level OLS 56%, log OLS 44%
	SE correction	HAC 59%, NW 21%, robust/HC3 11%, none 9%

Table 7: Methodology fork taxonomy (Stage 1, 150 agents)

Hypothesis	Fork	$R^{2}$	$F$ -stat	Sig.
H1: Efficiency	abs_or_signed	0.165	29.3
	measure_family	0.165	29.1
	llm (Sonnet/Opus)	0.132	22.5
	scale (log/level)	0.085	4.5
H2: Quoted Spread	scale (log/level)	0.882	1110.8
	llm (Sonnet/Opus)	0.446	119.0

Table 8: Variance decomposition: Which forks explain AI NSE?

Property	Human ( Menkveld et al., 2024 )	AI (this paper)
Paradigm diversity	High (linear trend 35%, relative changes 58%, log-diff 6%)	Low (level regression 56%, log regression 44%; no relative changes)
Measure diversity	Moderate	High (model-dependent)
Jensen’s inequality	Major source of outliers	Absent
Feedback response	Gradual, each stage contributes	Binary: exemplars work, critiques do not
PE quality $\to$ NSE	Higher quality $\to$ $-$ 33% IQR	Mixed (Appendix B ): significant for H4, H6; not for H1, H2, H5
Convergence S1 $\to$ S3/S4	$-$ 47% IQR (4 stages)	$-$ 80% to $-$ 99% stratified (3 stages)

Table 9: AI NSE in context: comparison with Menkveld et al. (2024)

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix


	Mean	Median	Min	Max
Sonnet S1	3.54	2.85	1.30	14.03
Sonnet S2	2.13	1.58	0.69	13.63
Sonnet S3	3.12	2.56	1.27	10.52
Opus S1	5.59	4.19	1.83	18.70
Opus S2	3.21	2.57	1.31	19.05

Appendix

Table 10: Cost, running time, and turns per agent-stage

Hyp	Spearman $r$	$p$ -value	Low-rated IQR	Mid-rated IQR	High-rated IQR
H1	$-$ 0.040	0.628	2.45	2.31	1.09
H2	$+$ 0.125	0.127	0.06	0.53	0.38
H3	$-$ 0.189	0.021**	5.83	3.81	3.18
H4	$-$ 0.325	$<$ 0.001***	10.60	10.70	0.82
H5	$-$ 0.099	0.229	0.54	0.54	0.48
H6	$-$ 0.601	$<$ 0.001***	5.77	11.88	2.44

Appendix

Table 11: PE rating vs. NSE: Spearman correlation and IQR by rating tercile (Stage 1)

AI Agent Reliability

AI agent reliability studies whether tool-using systems act correctly, consistently, safely, and recoverably across changing tasks and environments.

See all 338 papers

AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable
Jun 9, 2026Meysam Alizadeh, Fabrizio Gilardi, Mohsen Mosleh +1AI Agent Reliability AI Coding Agents
University of Oxford · University of Zurich · Technical University of Munich
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Jul 29, 2026Peter Kirgis, Sayash Kapoor, Andrew Schwartz +21AI-Assisted Scientific Research AI Agent Evaluation
Princeton University · Cornflower Labs · UK AI Security Institute +8
Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study
May 19, 2026Priyansh Trivedi, Olivier SchmittCoding Agents AI Agent Evaluation
SonarSource

Jun 9, 2026J/K move · Enter open · S save

AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable

Meysam Alizadeh, Fabrizio Gilardi, Mohsen Mosleh, Enkelejda Kasneci

University of Oxford · University of Zurich · Technical University of Munich

AI Agent Reliability AI Coding Agents LLM Agent Evaluation Computational Social Science Language Model Bias Evaluation

Open paper AskPDF arXiv

The deployment of LLM-based agents in scientific analysis raises opposing concerns: that agents may reduce methodological diversity, or that they may amplify the analytic flexibility through which researchers reach motivated conclusions. We argue these worries target two empirically separable layers: a design layer of methodological choices, and a verdict layer in which a decision rule maps estimates to a substantive claim. We test both by running 20 independent executions of Claude Code and Codex on a prominent immigration and social-policy against a many-analysts human baseline. At the design layer, Codex matches human methodological diversity and Claude Code produces nearly three times as many specifications; both agents' effect estimates remain broadly aligned with the human consensus, and no agent model exactly matches any human model. A prompt-induced anti-immigration researcher prior reorganizes each agent's methodological decisions but, unlike for biased human analysts in the same data, does not shift aggregate estimates or final verdicts; nor do agents reroute along the methodological axes humans use to bias their estimates. At the verdict layer, an explicit confirmatory prompt flips Claude Code's verdicts from 10% to 90% support while leaving its coefficient distribution essentially unchanged, operating through rule omission rather than rule softening. AI agents can rival or exceed human methodological diversity at the design layer while remaining vulnerable at the verdict layer. In our setting, the locus of AI bias is not estimation but interpretation.

Abstract

AI Agent Reliability

Explore similar work

AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable

Can AI agents conduct open-ended AI research? Early evidence from two case studies

Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study