Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments

Zonghan Li; Feng Ji

cs.CYApr 2, 2026

Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments

Authors: Zonghan Li, Feng Ji

Organizations: Department of Applied Psychology and Human Development, University of Toronto, ON, Canada

Abstract

Large language models (LLMs) are increasingly used to simulate human responses and estimate treatment effect of interventions when real-world experiments are costly or infeasible. The treatment-effect estimates are often evaluated using statistical realism, the degree to which simulated responses reproduce properties of observed human responses, although whether realism predicts treatment-effect accuracy remains unknown. Here we test this proxy relationship by jointly measuring statistical realism and treatment-effect accuracy on the same simulated responses in a cross-national experiment with 59,508 participants from 62 countries using three LLMs. The correlation between statistical realism and treatment-effect accuracy is weak, and optimizing for statistical realism can even worsen treatment-effect accuracy when selecting models, prompts, and target populations. The pattern replicates in two additional cross-national experiments spanning 12 and 27 countries with 20,785 participants. The divergence between the two reflects distinct error structures and is larger for behavioral outcomes, where models appear to extrapolate behavioral effects from attitudinal patterns. Because this divergence may remain hidden in deployment, errors can propagate into simulation-informed decisions. We introduce a diagnostic framework for LLM-generated synthetic data and discuss how treatment-effect validation should proceed under varying availability of experimental benchmarks. Simulated responses and simulated treatment effects are distinct estimation targets, and evidence for one does not certify the other.

Figures & tables

Figure 1: Statistical realism and treatment-effect accuracy are distinct evaluation targets that can diverge. a. The two targets are defined over different objects. A response is what a respondent says or does under the intervention(s) (left). A treatment effect is the change in responses produced by an intervention, obtained by contrasting conditions (right). Statistical realism evaluates the responses, and treatment-effect accuracy evaluates the contrast. b. Validation in practice assumes a proxy relationship between the two criteria (diagonal), in which higher realism indicates more accurate effect estimates. Observed simulations can instead fall in the lower-right region, where they pass realism-based validation while misestimating treatment effects.

Figure 2: Statistical realism does not imply treatment-effect accuracy. Statistical realism is measured by individual-level mean absolute error (MAE), computed after averaging item-level responses to construct scores. Treatment-effect accuracy is measured by the absolute difference between the predicted and the observed average treatment effect (ATE error). Results are from the ICPC dataset (11 interventions, 62 countries,

n

= 59,508. a . Similar MAE can coexist with very different ATE accuracy. Comparing the three LLMs with two supervised baselines (OLS and LASSO), MAE is nearly identical (1.07-fold) while absolute ATE error differs more than twofold (2.10-fold). Fold values are the ratio of the mean error across the three LLMs to the mean error across the two supervised baselines, with errors pooled over the three outcomes before the ratio is taken. b . Optimizing MAE does not necessarily improve ATE accuracy. Relative to baseline, reductions in MAE coincide with increases in absolute ATE error, so optimizing statistical realism can move treatment-effect accuracy in the opposite direction. c . MAE and ATE error are only weakly related (Spearman

\rho

= 0.10).

Figure 3: Errors of statistical realism and treatment-effect accuracy follow different structures across models and datasets . a . In ICPC dataset, outcome type dominates MAE, but ATE error redistributes across method, country, and their interactions. b . The structural shift replicates in both additional datasets, although the dimensions involved vary with experimental design. c . Intraclass correlation coefficients (ICC) show that country-level heterogeneity is amplified in treatment-effect accuracy across all model-dataset combinations.

Figure 4: Attitudinal estimates track the direction of persuasive force, and models tighten the association between behavioral and attitudinal estimates more than human data do . Results shown for GPT; other models in Supplementary Figs. S2 and S3 . a . Models are sensitive to persuasive force in intervention text for attitudinal outcomes, but action estimates do not respond coherently to the same perturbations. Bars show the change in absolute ATE error relative to the original intervention text under text perturbations. Error bars indicate 95% CI. b . LLM-generated data tighten the association between attitudinal and behavioral outcomes relative to human data. Lower triangles show individual-level response correlations. Upper triangles show intervention-level ATE correlations. c . Instructing models to weaken this association does not consistently reduce action ATE error. Corrections improve some interventions but overcorrect others.

Figure 5: Prompting creates unequal gains in statistical realism but not in treatment effect accuracy . Results shown for GPT; other models in Supplementary Information. a-b . Statistical realism gains concentrated in OECD countries and in countries with higher internet penetration, English proficiency, and lower inequality, but these country-level gradients had no counterpart in treatment effect accuracy gains. Error bars show 95% CI. c-d . Demographic patterns are weaker and more method-dependent than country-level gradients for both statistical realism and treatment effect accuracy gains. Error bars show 95% CI. e-f . Feature importance profiles for statistical realism and treatment effect accuracy gains. Statistical realism gains are largely associated with statistical properties (e.g., true mean, original MAE), while treatment effect accuracy gains show a more heterogeneous mix of statistical, demographic, and country-level features. Signed SHAP percentage values from random forest models (100-seed averaged) predicting prompting-induced changes in MAE and ATE error across outcomes. Positive values (red) indicate larger error increases; negative values (blue) indicate larger error reductions. Values are defined relative to the baseline prompt.

Metric	OLS	LASSO	LLM (GPT)	LLM (Gemini)	LLM (Claude)
Individual MAE	22.96	23.65	24.25	24.53	26.11
Belief	15.26	15.40	15.98	16.22	17.96
Policy	13.92	14.05	16.11	15.23	17.22
Action	39.71	41.50	40.65	42.13	43.15
Absolute ATE error	4.37	4.29	9.39	8.72	9.14
Belief	3.04	3.01	7.80	6.47	12.51

Table S1: Individual-level MAE and absolute ATE error of conventional statistical models and LLMs. All outcomes are reported on a 0–100 scale, and pooled values are equal-weight means of Belief, Policy, and Action. LLM values use baseline zero-shot predictions. OLS and LASSO values use five-fold out-of-fold predictions from demographic models fitted to the pooled participant records across the 12 experimental conditions. Individual MAE is averaged over respondents within outcome. Absolute ATE error is calculated for each country and treatment-versus-control comparison, averaged across treatments within country, and then averaged equally across countries.

		External information			Internal inference
Metric	Baseline	Few-shot	Country context	Distributional stats	CoT	VBN-CoT
Individual MAE	23.81	22.86	24.09	23.67	24.91	23.76
Belief	15.79	15.80	15.59	15.90	15.59	15.68
Policy	15.41	14.84	15.19	14.56	16.41	16.39
Action	40.21	37.92	41.48	40.56	42.72	39.21

Table S2: Development-set performance of five enhancement candidates used for method selection (

n

= 5,093). Few-shot supplied responses from demographically similar respondents retrieved within the same country and experimental condition using

K

-nearest neighbor clustering (

k

= 6). VBN-CoT used a two-stage procedure in which the model first inferred a Value-Belief-Norm profile and then generated survey responses conditioned on that intermediate plan. Country-context and distributional-statistics prompts appended country-level contextual summaries or target-variable distributional benchmarks, respectively. Generic chain-of-thought prompting used a two-stage reasoning plan without VBN-specific structure. Based on development-set performance in individual-level MAE, we selected few-shot and VBN-CoT for full evaluation in the main analyses. Complete implementation details and exact prompt templates are available in the analysis code.

Dataset	Method	Individual MAE	MAE rank	Absolute ATE error	ATE-error rank
Spampatti	Baseline	28.07	2	26.52	3
Spampatti	Few-shot	21.09	1	19.28	2
Spampatti	VBN-CoT	29.70	3	4.31	1
Većkalov	Baseline	10.74	2	3.20	2
Većkalov	Few-shot	10.73	1	3.22	3
Većkalov	VBN-CoT	11.51	3	2.91	1

Table S3: Prompting-method comparison in the Spampatti and Većkalov replication datasets.

Category	Intervention	Description
Deductive reasoning	Collective action	Features examples of successful collective action that have had meaningful effects on climate policies (e.g., protests) or have solved past global issues (e.g., the restoration of the ozone layer).
	Scientific consensus	Informs participants that “99% of expert climate scientists agree that Earth is warming and climate change is happening, mainly because of human activity.”
	Pluralistic ignorance	Presents real public opinion data collected by the United Nations that show what percentage of people in each participant’s country agree that climate change is a global emergency.
Situational simulation	Letter to future generations	Emphasizes how one’s current actions affect future generations by asking participants to write a letter to a socially close child who will read it in 25 years when they are an adult, describing current actions toward ensuring a habitable planet.
	Future self-continuity	Emphasizes the future self-continuity by asking each participant to project themselves into the future and write a letter addressed to themselves in the present, describing the actions they would have wanted to take regarding climate change.
	Psychological distance	Frames climate change as a proximal risk using examples of recent natural disasters caused by climate change in each participant’s nation and prompts them to write about the climate impacts on their community.

Table S4: Overview of the 11 climate-psychology interventions used in the ICPC simulations. Adapted from Figure 1 in Vlasceanu et al. (2024).

Original	Surface paraphrase	Force removal	Force reverse
Did you know that 99% of expert climate scientists agree that the Earth is warming and climate change is happening, mainly because of human activity (for example, burning fossil fuels) ?	Did you know that 99% of expert climate scientists agree that global warming is accelerating and the climate is changing, primarily driven by human actions (for example, fossil fuel combustion)?	Did you know that 99% of expert climate scientists agree that the Earth is warming and climate change is happening, while its causes are still under study (for example, temperature records) ?	Did you know that 99% of expert climate scientists agree that the Earth is warming and climate change is happening, mainly because of natural cyclical processes (for example, solar cycles) ?

Table S5: Perturbed texts for the Scientific Consensus intervention. Underlined bold text indicates key modified phrases.

Original	Surface paraphrase	Force removal	Force reverse
People in {country} and around the world are changing: more and more people are concerned about climate change, and are now taking action across multiple fronts. Since 2013, concerns about climate change have increased in most countries surveyed.	People in {country} and around the world are shifting: a growing number of people are worried about climate change, and are now acting on multiple fronts. Since 2013, worries about climate change have risen in most countries surveyed.	People in {country} and around the world hold a range of views on climate change, and show different patterns of engagement across multiple fronts. Since 2013, views on climate change have been recorded across most countries surveyed.	People in {country} and around the world are changing: fewer and fewer people are concerned about climate change, and fewer are taking action across multiple fronts. Since 2013, concerns about climate change have declined in most countries surveyed.
[Chart: Pew Research Center, Spring 2013 & 2018 Global Attitudes Surveys. For each country, paired data points connected by a line show that concern has grown over the period.]	[Chart: Pew Research Center, Spring 2013 & 2018 Global Attitudes Surveys. For each country, paired data points connected by a line show that worry has risen over the period.]	[Chart: Pew Research Center, Spring 2013 & 2018 Global Attitudes Surveys. For each country, paired data points connected by a line show how views have been tracked over the period.]	[Chart: Pew Research Center, Spring 2013 & 2018 Global Attitudes Surveys. For each country, paired data points connected by a line show that concern has declined over the period.]
What kinds of actions are people taking right now? More than ever before, people are making changes to their lifestyles, supporting policies to address climate change, and are giving the issue more time and attention. For example, more and more people from around the world are now: [Practice unchanged]	What kinds of actions are people taking right now? More than before, people are adjusting their lifestyles, backing policies to address climate change, and are devoting more time and attention to the issue. For example, a rising number of people from around the world are now: [Practice unchanged]	What kinds of patterns can be observed? People show different patterns in their lifestyles, views on policies related to climate change, and the time and attention they give to the issue. For example, people from around the world show varied patterns in : [Practice unchanged]	What kinds of actions are people taking right now? Compared with before, fewer people are making changes to their lifestyles, supporting policies to address climate change, and giving the issue time and attention. For example, fewer and fewer people from around the world are now: [Practice unchanged]
There’s also been a notable increase in support for climate change mitigation policy–some of the most popular policies include: [Policies unchanged]	There’s also been a notable increase in support for climate change mitigation policy–some of the most popular policies include: [Policies unchanged]	Views on climate change mitigation policy also vary across contexts –some of the most discussed policies include: [Policies unchanged]	There’s also been a notable decline in support for climate change mitigation policy–some of the least popular policies include: [Policies unchanged]

Table S6: Perturbed texts for the Dynamic Social Norm intervention. Omitted passages are identical to the original. Underlined bold text indicates key modified phrases.

Original	Surface paraphrase	Force removal	Force reverse
[Climate change description unchanged]	[Climate change description unchanged]	[Climate change description unchanged]	[Climate change description unchanged]
[Climate and human activity unchanged]	[Climate and human activity unchanged]	[Climate and human activity unchanged]	[Climate and human activity unchanged]
Please put yourself in the year 2030 - eight years from now. Take a few moments to imagine your life in that future. Imagine how you will look, where you will be, and who you are with. In the year 2030, it will be clear whether keeping climate change under 2 ∘ C is still possible . It will be clear whether the necessary change occurred fast enough to match the speed of the changing climate. As the Earth’s atmosphere continues to heat up, the effects of climate change will be more apparent : the “highest observed temperature” records will keep being updated, heatwaves and the draughts will become more common, species will continue to become extinct.	Now picture yourself in the year 2030 - eight years into the future. Take a few moments to envision your life at that point in time. Think about how you will look, where you will be living, and who will be around you. By 2030, it will be evident whether limiting warming to 2 ∘ C remains an achievable goal . It will be clear whether the necessary changes happened quickly enough to keep pace with the changing climate. As temperatures continue to climb, the effects of climate change will become more apparent : temperature records will keep being broken, heatwaves and droughts will intensify and spread, and more species will face the threat of extinction.	Please put yourself in the year 2030 - eight years from now. Take a few moments to imagine your life in that future. Imagine how you will look, where you will be, and who you are with. In the year 2030, it will be clearer how likely keeping climate change under 2 ∘ C still appears to be . It will be clear whether various factors played out as projected in current climate models . As the Earth’s atmosphere continues to heat up, the long-term effects of climate change will become clearer over time : the “highest observed temperature” records will keep being updated, heatwaves and the droughts will become more common, species will continue to become extinct.	Please put yourself in the year 2030 - eight years from now. Take a few moments to imagine your life in that future. Imagine how you will look, where you will be, and who you are with. In the year 2030, the trajectory of climate change will have been shaped by forces beyond any individual’s control . It will be clear whether the changing climate was shaped mainly by forces beyond individual action . As the Earth’s atmosphere continues to heat up, the effects of climate change will unfold regardless of individual choices : the “highest observed temperature” records will keep being updated, heatwaves and the droughts will become more common, species will continue to become extinct.

Table S7: Perturbed texts for the Future-Self Continuity intervention. Omitted passages are identical to the original. Underlined bold text indicates key modified phrases.

Original	Coupling removal	Coupling attenuation
Here is your profile: {persona} — {intervention} — {question}	Here is your profile: {persona} — {intervention} Please note, people’s views on climate change and their everyday behaviors are related but distinct, and should be inferred separately. — {question}	Here is your profile: {persona} — {intervention} Please note, even when people’s views on climate change shift, changes in everyday behavior are often smaller/slower because of habit/cost/social context. — {question}

Table S8: Prompt framings used in the attitude-behavior coupling probe. Underlined bold text indicates the inserted framing statement.

Dataset	Prompt input variables	Outcome variables
ICPC (62 countries, 11 interventions)	Age, gender, country, education, socioeconomic status (direct material-asset measure and indirect subjective-status measure), political orientation (social and economic issues), perceived scientific consensus	Climate belief (4 items, 0–100) Policy support (9 items, 0–100) Pro-environmental action (WEPT, 0–8 $\rightarrow$ 0–100)
Spampatti et al. (12 countries, 6 interventions)	Age, gender, country, education, political ideology, cognitive reflection (CRT-2)	Climate belief (9 items, 1–5 $\rightarrow$ 0–100) Pro-environmental action (WEPT, 0–8 $\rightarrow$ 0–100)
Većkalov et al. (27 countries, 2 interventions)	Age, gender, country, urbanicity, student status, college-degree group, political ideology, trust in climate scientists	Perceived consensus (2 items, 0–100) Belief and concern (4 items, 1–7 $\rightarrow$ 0–100) Policy support (1 item, 1–7 $\rightarrow$ 0–100)

Table S9: Overview of participant profile variables (prompt inputs) and outcome variables across datasets . All outcomes were rescaled to a common 0–100 metric for cross-study comparison.

		GPT		Gemini		Claude
Group	Contrast	MAE	ATE error	MAE	ATE error	MAE	ATE error
OECD	OECD vs. Non-OECD	$+1.22^{\dagger}$	$-1.25^{\dagger}$	$+0.35$	$+0.13$	$+0.90^{\dagger}$	$-2.26^{***}$
Internet rate	High vs. Low	$+1.97^{*}$	$-1.57^{\dagger}$	$+0.90$	$+0.23$	$+1.80^{**}$	$-2.43^{**}$
English proficiency	High vs. Low	$+1.76^{\dagger}$	$-1.44$	$+1.23^{\dagger}$	$+0.85^{\dagger}$	$+1.33^{\dagger}$	$-1.76^{*}$
Gini	High vs. Low	$-2.29^{**}$	$+0.87$	$-0.84^{\dagger}$	$+0.10$	$-0.53$	$+1.09^{\dagger}$
Gender	Male vs. Female	$+2.53^{***}$	$+2.12^{***}$	$+2.29^{***}$	$+1.76^{**}$	$+1.45^{***}$	$+1.83^{***}$

Table S10: Pairwise group comparisons under the baseline condition. Entries report between-group differences (first group minus second group) in individual-level absolute prediction error (MAE) and absolute treatment-effect error (ATE error). All entries are in percentage points and are based on pairwise Mann–Whitney U tests. Significance markers are based on BH-adjusted

p

-values across the eight prespecified contrasts, separately for each model and validation target: ∗∗∗

p_{\mathrm{adj}}<0.001

, ∗∗

p_{\mathrm{adj}}<0.01

, ∗

p_{\mathrm{adj}}<0.05

, †

p_{\mathrm{adj}}<0.1

. Unmarked cells were not significant after BH correction.

		GPT		Gemini		Claude
Group	Contrast	Few-shot	VBN-CoT	Few-shot	VBN-CoT	Few-shot	VBN-CoT
OECD	OECD vs. Non-OECD	$+0.64^{*}$	$+1.62^{***}$	$-0.12$	$+1.51^{*}$	$+0.54$	$+2.27^{***}$
Internet rate	High vs. Low	$+1.00^{**}$	$+1.95^{***}$	$+0.10$	$+1.53^{*}$	$+0.87$	$+2.74^{***}$
English proficiency	High vs. Low	$+0.67^{\dagger}$	$+1.68^{***}$	$+0.22$	$+2.10^{*}$	$+0.32$	$+2.48^{***}$
Gini	High vs. Low	$-1.46^{**}$	$-1.64^{**}$	$-0.18$	$-2.82^{**}$	$+0.05$	$-2.38^{***}$
Gender	Male vs. Female	$+0.11$	$-0.07$	$-0.60^{*}$	$+1.67^{***}$	$-0.98^{***}$	$+0.63^{***}$

Table S11: Pairwise group comparisons for reductions in MAE. Entries report between-group differences (first group minus second group) in reductions in individual-level absolute prediction error relative to baseline. All entries are in percentage points and are based on pairwise Mann–Whitney U tests. Significance markers are based on BH-adjusted

p

-values across the eight prespecified contrasts, separately for each model and prompting method: ∗∗∗

p_{\mathrm{adj}}<0.001

, ∗∗

p_{\mathrm{adj}}<0.01

, ∗

p_{\mathrm{adj}}<0.05

, †

p_{\mathrm{adj}}<0.1

. Unmarked cells were not significant after BH correction.

		GPT		Gemini		Claude
Group	Contrast	Few-shot	VBN-CoT	Few-shot	VBN-CoT	Few-shot	VBN-CoT
OECD	OECD vs. Non-OECD	$+0.51$	$-0.77$	$+3.22^{*}$	$+1.37^{*}$	$+0.92$	$-1.27^{*}$
Internet rate	High vs. Low	$+0.33$	$-0.94$	$+3.14^{*}$	$+1.66^{*}$	$+1.33$	$-1.36^{*}$
English proficiency	High vs. Low	$-0.44$	$-0.46$	$+4.77^{*}$	$+2.15^{*}$	$+1.07$	$-1.13^{\dagger}$
Gini	High vs. Low	$-0.76$	$+0.48$	$-4.91^{**}$	$-0.58$	$-2.96^{*}$	$+0.76$
Gender	Male vs. Female	$+0.28$	$+0.02$	$+0.49$	$+0.14$	$+1.27^{*}$	$+0.22$

Table S12: Pairwise group comparisons for reductions in ATE error. Entries report between-group differences (first group minus second group) in reductions in absolute treatment-effect error relative to baseline. All entries are in percentage points and are based on pairwise Mann–Whitney U tests. Significance markers are based on BH-adjusted

p

-values across the eight prespecified contrasts, separately for each model and prompting method: ∗∗∗

p_{\mathrm{adj}}<0.001

, ∗∗

p_{\mathrm{adj}}<0.01

, ∗

p_{\mathrm{adj}}<0.05

, †

p_{\mathrm{adj}}<0.1

. Unmarked cells were not significant after BH correction.

Figure S1: Additional measures of statistical realism and treatment-effect accuracy. a . Mean error (ME) by model and outcome under baseline conditions. b . Mean absolute error (MAE) by model and outcome. c . Country-level Spearman rank correlation and KNN overlap. d . Ratio of predicted to observed variance across models and outcomes. e . Enhancement effect on country-level mean error with prompting strategies. f . Enhancement effect on individual-level MAE with prompting strategies. g . Enhancement effect on country-level Spearman correlation and KNN overlap with prompting strategies. h . Enhancement effect on Jensen-Shannon divergence by outcome with prompting strategies. i . Mean absolute ATE error by outcome and prompting strategy for each LLM.

Figure S2: Replication analyses of the relationship between statistical realism and treatment-effect accuracy. Results are similar with Fig. 2 but from two additional cross-national experiments: Spampatti et al. and Većkalov et al. a . Comparison of baseline LLM prompting and supervised baselines. The relative performance of LLMs and supervised models differed across datasets: LLMs showed larger errors in both MAE and ATE error in Spampatti et al., whereas their performance was comparable to supervised models in Većkalov et al. b . Changes in MAE and ATE error after prompt enhancement relative to baseline prompting. The horizontal and vertical axes indicate changes in MAE and ATE error, respectively. Points in the lower-left region indicate simultaneous reductions in both errors. c . Association between baseline statistical realism and treatment-effect accuracy across countries. Each point represents a country-level observation. Spearman correlations were estimated separately for each dataset, with confidence intervals accounting for country-level clustering. Across datasets, lower MAE was not consistently associated with lower ATE error. Note: We excluded perceived scientific consensus in the Većkalov dataset in this analysis because it was the construct directly targeted by the consensus-communication intervention, rather than a downstream outcome comparable to belief/concern and policy support.

Figure S3: Intervention-level treatment-effect bias across interventions and prompting strategies. Each point represents the signed ATE error (

\Delta

ATE = predicted ATE - observed ATE) for one intervention under baseline (red), few-shot (blue), or VBN-CoT (purple) prompting. Positive values indicate overestimation and negative values indicate underestimation of treatment effects. a . GPT. b . Gemini. c . Claude. Crosses indicate interventions for which the estimated treatment-effect direction was opposite to the observed human effect, and open circles indicate interventions whose human ATE 95% confidence intervals crossed zero. The three interventions selected for perturbation analyses in Fig. 4 (scientific consensus, dynamic social norm, and future-self continuity) represent distinct theoretical categories and different ranges of treatment-effect bias. The results show substantial heterogeneity in treatment-effect errors across interventions, outcomes, and prompting strategies.

Figure S4: LLM simulations alter the association structure among behavioral outcomes. a . Gemini. b. Claude. Results for GPT are provided in Fig. 4 b in the main text. Lower triangles show individual-level response correlations. Upper triangles show intervention-level ATE correlations. Human data show weak associations between attitudinal and behavioral outcomes, whereas LLM-generated data exhibit stronger coupling, particularly under inference-based prompting. Like the results of GPT, these altered association patterns suggest that LLM simulations may not preserve the relationship structure between attitudes and behaviors observed in human data.

Figure S5: Divergence of country-level descriptive and causal error and their inequality profiles for GPT . a . Country-level scatterplots of baseline MAE against ATE error for each outcome. b . Countries ranked by MAE (top and bottom 10). c . Countries ranked by ATE error (top and bottom 10).

Figure S6: Divergence of country-level descriptive and causal error and their inequality profiles for Gemini . a . Country-level scatterplots of baseline MAE against ATE error for each outcome. b . Countries ranked by MAE (top and bottom 10). c . Countries ranked by ATE error (top and bottom 10). d . Between-group differences in MAE by country characteristics. e . Between-group differences in ATE error by country characteristics. f . Between-group differences in MAE by demographic characteristics. g . Between-group differences in ATE error by demographic characteristics.

Figure S7: Divergence of country-level descriptive and causal error and their inequality profiles for Claude . a . Country-level scatterplots of baseline MAE against ATE error for each outcome. b . Countries ranked by MAE (top and bottom 10). c . Countries ranked by ATE error (top and bottom 10). d . Between-group differences in MAE by country characteristics. e . Between-group differences in ATE error by country characteristics. f . Between-group differences in MAE by demographic characteristics. g . Between-group differences in ATE error by demographic characteristics.

Explore similar work

Feb 17, 2026

This human study did not involve human subjects: Validating LLM simulations as behavioral evidence

A growing literature uses large language models (LLMs) as synthetic participants to generate cost-effective and nearly instantaneous responses in social science experiments. However, there is limited guidance on when such simulations support valid inference about human behavior. We contrast two strategies for obtaining valid estimates of causal effects and clarify the assumptions under which each is suitable for exploratory versus confirmatory research. Heuristic approaches seek to establish that simulated and observed human behavior are interchangeable through prompt engineering, model fine-tuning, and other repair strategies designed to reduce LLM-induced inaccuracies. While useful for many exploratory tasks, heuristic approaches lack the formal statistical guarantees typically required for confirmatory research. In contrast, statistical calibration combines auxiliary human data with statistical adjustments to account for discrepancies between observed and simulated responses. Under explicit assumptions, statistical calibration preserves validity and provides more precise estimates of causal effects at lower cost than experiments that rely solely on human participants. Yet the potential of both approaches depends on how well LLMs approximate the relevant populations. We consider what opportunities are overlooked when researchers focus myopically on substituting LLMs for human participants in a study.

Jessica Hullman, David Broska, Huaman Sun +1

Northwestern University · Stanford University

Large Language Model-Based Social Simulation Human Behavior Simulation Causal Inference

Ask PDF

Mar 8, 2026

Evaluating LLMs as Human Surrogates in Controlled Experiments

Large language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data support the same experimental inferences as human data. We evaluate this by directly comparing off-the-shelf LLM-generated responses with human responses from a canonical survey experiment on accuracy perception. Each human observation is converted into a structured prompt, and models generate a single 0--10 outcome variable without task-specific training; identical statistical analyses are applied to human and synthetic responses. We find that LLMs reproduce several directional effects observed in humans, but effect magnitudes and moderation patterns vary across models. Off-the-shelf LLMs therefore capture aggregate belief-updating patterns under controlled conditions but do not consistently match human-scale effects, clarifying when LLM-generated data can function as behavioral surrogates.

Adnan Hoq, Tim Weninger

University of Notre Dame Notre Dame, Indiana, USA

Human Behavior Simulation Large Language Model-Based Social Simulation LLM Evaluation Belief Updating in LLMs

Ask PDF

Sep 26, 2026

Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study

Large language models (LLMs) are increasingly used to generate synthetic survey respondents and digital twins of real people, but whether their output preserves real human statistical structure, rather than surface plausibility, remains unresolved, and most existing evidence comes from proprietary models rather than open-weight ones. We evaluate three open-weight LLM families on a cross-instrument calibration task: conditioning personas on real respondents' verbatim answers to one psychometric instrument and measuring them on a second, construct-distance-controlled instrument, checked against a 2,058-person human panel. Across a 139-pair grid, the simulated cross-instrument correlation tracks the real human correlation at r = 0.70 - 0.73 in every model, driven mainly by correct sign rather than precise magnitude and concentrated in pairs of moderate construct distance. A correlation of this magnitude, obtained from untuned open-weight models conditioned only on individual-level survey data, is a substantively encouraging result for LLM-based behavioral simulation and digital-twin applications: specific model families and releases already reproduce a meaningful share of real human cross-instrument structure without any fine-tuning. This capability does not, however, improve monotonically across model releases: on a matched panel, the newest of three tested Llama releases performs worst on two of three headline metrics, so realizing its promise in practice requires release-specific, distance-aware verification rather than a one-time benchmark.

Grandee Lee, Wang Yue

Singapore University of Social Sciences Singapore, Singapore

Large Language Model-Based Social Simulation LLM Evaluation Language Model Calibration

Ask PDF