When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels

Robson Tigre; Hugo Gobato Souto

cs.HCJun 3, 2026

When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels

Authors: Robson Tigre, Hugo Gobato Souto

Organizations: Recast · Dell

Abstract

Large language models are increasingly used to simulate consumers, yet agreement with aggregate survey outcomes can conceal errors in the subgroup estimates that guide decisions. We develop a framework for deciding when to trust, correct, or abandon a synthetic consumer panel by distinguishing population mismatch from differences in conditional responses. The framework combines diagnostics for covariate overlap, conditional calibration, and stability across models with an augmented inverse probability weighting estimator that treats LLM predictions as auxiliary information. Correction requires a complete invitation frame, response indicators for everyone invited, and human outcomes for respondents, together with assumptions of missing at random and positivity. We evaluate the framework using simulations and controlled nonresponse experiments constructed from paired human and LLM responses in ANES and Twin-2K-500. In identified simulation regimes, point bias remains small when either the response model or the outcome model is correct, but nominal 95% confidence intervals cover only 88--94% in some regimes. When nonresponse depends on missing human outcomes, substantial bias persists and coverage approaches zero. Controlled masking of the paired datasets reproduces this contrast between successful point correction under imposed missing at random and failure under nonignorable nonresponse. These findings show why validation must address the intended population and task, and why the scope for correcting synthetic estimates depends on the available human data and the assumptions that identify the target.

Figures & tables

Evidence	Source	Key Finding
Within-profile compression + coefficient divergence	Bisbee et al. (2024)	The synthetic data exhibit substantially smaller within-profile variation; 48% of coefficients differ significantly, and among those cases, 32% flip sign
Subgroup collapse	Argyle et al. (2023)	$\rho=0.90$ – $0.94$ aggregate but $\rho\approx 0.02$ for political Independents (2020 wave)
WTP overestimation	Brand et al. (2025)	For one built-in laptop-projector feature, baseline GPT-3.5 overestimated WTP by approximately $3\times$ ; fine-tuning substantially reduced the difference, and GPT-4o estimates were similar to human estimates
Aggregate attainment, subgroup caution	Maier et al. (2025)	SSR correlation attainment is 90.2% for GPT-4o; gender, region, and ethnicity patterns are not consistently replicated, and the authors caution against interpreting subgroup analyses from synthetic panels
Subgroup and novel-item error	Morris et al. (2025)	Within ${\sim}4$ pp for one frequently polled polarized item; 9.8–18.2pp topline differences on novel immigration items; subgroup mean absolute error above 10pp
Persona conditioning has heterogeneous effects	Taday Morocho et al. (2026)	Near-zero aggregate improvement; effects concentrated in a small subset of items and low-support strata across 70,000+ instances

Table 1: Evidence of systematic failures in LLM synthetic panels.

Paper	Covariate	Concept	DR	Diagnostics	Consumer	Key Limitation
This paper	✓	✓	✓	✓	✓	—
Guerdan et al. (2025)	✓	—	✓	—	—	Assumes no concept drift
Wang et al. (2025)	—	Partial	—	—	—	Same-population assumption
Krsteski et al. (2025)	—	Partial	—	—	—	Same-population assumption
Zou et al. (2026)	✓	—	—	—	—	Assumes $C\perp\!\!\!\perp Y\|X$
Huang et al. (2025)	✓	✓	—	Partial	—	No corrected estimates

Table 2: Competitive positioning. Each column indicates whether the paper addresses the corresponding dimension. Only our paper occupies the four-way intersection.

Figure 1: Propensity score distributions under three covariate shift regimes from the controlled DGP. Left: Good overlap (

\mathrm{ESS}/N=0.884

)—source and target populations are nearly indistinguishable, reweighting is reliable. Center: Marginal overlap (

\mathrm{ESS}/N=0.047

)—moderate separation, reweighting is feasible but high-variance. Right: Poor overlap (

\mathrm{ESS}/N=0.001

)—severe separation, nearly all importance weight concentrates on a few observations, making reweighted estimates unreliable.

Diagnostic	Metric	Proceed	Caution / Correct	Walk Away
Covariate overlap	$\mathrm{ESS}/N$	$>0.10$	$[0.01,0.10]$	$<0.01$
Conditional calibration	$\beta_{g}$	$[0.5,1.5]$ , same sign	Outside range	Sign flip
Cross-LLM stability	$\min(\rho_{jk})$	$>0.80$	$[0.50,0.80]$	$<0.50$

Table 3: Diagnostic decision thresholds (operational defaults). Each diagnostic returns an interpretable metric that maps to one of three actions. Threshold sensitivity is examined in Online Appendix C.5.

Panel of Figure 2 — Figure 2: Decision framework: Trust, Correct, correction-unavailable, or Walk Away. The pipeline is ordered from cheapest (Step 1: demographics only) to most expensive (Step 4: invitation-frame data). Practitioners can exit early at Steps 1–2 without collecting real data; Step 3 requires a human calibration sample; Step 4 requires the full invitation-frame structure before frame-AIPW ( Section 5 ) may run.

Dataset	$N_{\text{real}}$	Outcome Type	LLM(s)	Role
ANES (Bisbee/Argyle)	2,286 a	Ordinal (0–100)	GPT-3.5, GPT-4, Falcon-40B	Proof of concept: detect known failures
Twin-2K-500 (Toubia et al.)	2,058 b	Binary (buy/not buy)	GPT-4.1-mini	Consumer domain: pricing & behavioral economics

Table 4: Summary of empirical datasets.

DGP	$\pi$ arm	$\mu$ arm	Reps	Bias	MCSE	Coverage [95% MC]	Status
Prob. sampling	correct	correct	2,000	0.0009	0.0011	95.15% [94.12, 96.01]	compatible
Prob. sampling	correct	wrong	2,000	$-0.0144$	0.0022	99.35% [98.89, 99.62]	conservative
Prob. sampling	wrong	correct	2,000	0.0004	0.0009	88.15% [86.66, 89.49]	failure
Prob. sampling	wrong	wrong	2,000	0.2706	0.0013	0.35% [0.17, 0.72]	failure
MAR missing	correct	correct	2,000	0.0003	0.0008	95.10% [94.06, 95.96]	compatible
MAR missing	correct	wrong	2,000	$-0.0026$	0.0010	99.85% [99.56, 99.95]	conservative

Table 5: Frame-AIPW simulation battery: point bias and interval coverage by DGP class and nuisance arm. Bias is in each DGP’s outcome scale. Coverage failures are reported as failures; “compatible” means the Wilson 95% Monte Carlo interval includes 0.95.

Dataset	Outcome	Regime	Reportable	Bias	MCSE	Coverage [95% MC]	Status
ANES	Dem. thermometer	Mild MAR	1,000/1,000	0.0149	0.0130	95.80% [94.37, 96.88]	compatible
ANES	Dem. thermometer	Strong MAR	992/1,000	0.0041	0.0251	94.90% [93.36, 96.10]	compatible
ANES	Dem. thermometer	MNAR	1,000/1,000	5.2855	0.0166	0.00% [0.00, 0.38]	failure
ANES	Rep. thermometer	Mild MAR	1,000/1,000	0.0061	0.0130	95.70% [94.26, 96.79]	compatible
ANES	Rep. thermometer	Strong MAR	995/1,000	$-0.0254$	0.0232	96.10% [94.71, 97.13]	compatible
ANES	Rep. thermometer	MNAR	999/1,000	5.7040	0.0302	0.40% [0.16, 1.02]	failure

Table 6: Frame-AIPW under controlled masking: bias and coverage by outcome and response regime (1,000 replications per cell). Bias is among reportable estimates, in each outcome’s scale; coverage uses all attempts, counting unavailable estimates as failures to cover.

Thermometer	Democrats	Independents	Republicans	Decision
Dem. Party	$-18.7$ pp	$-3.7$ pp	$+3.0$ pp	Walk away
Rep. Party	$+1.4$ pp	$-8.2$ pp	$-11.0$ pp	Walk away

Table 7: ANES conditional calibration: party subgroup bias in Falcon-40B thermometers.

Target	Therm.	$\tau_{\text{real}}$	$\|\text{Bias}_{\text{naive}}\|$	$\|\text{Bias}_{\text{IPW}}\|$	$\|\text{Bias}_{\text{DR}}\|$	Red.	CI
Young Dem.	Dem. Party	72.8	33.1	21.6	0.2	99.5%	$\checkmark$
Older Rep.	Rep. Party	66.2	28.8	14.5	2.0	92.9%	$\checkmark$

Table 8: ANES estimator comparison on subsampled targets (Falcon-40B synthetic panel).

Subgroup	Human (ANES)	GPT-3.5	GPT-4	Falcon-40B
Democrat	73.4	90.3	84.2	55.1
Independent	42.2	58.7	56.6	39.3
Republican	21.7	19.0	33.0	24.7

Table 9: Cross-LLM subgroup estimates: feeling thermometer toward Democratic Party.

Analysis	ESS/N	$\beta_{g}$ range	Decision	$\|\text{Bias}_{\text{naive}}\|$	Red.	CI
Full sample (by age)	1.000	$[0.86,0.90]$	Trust	0.010	99.2%	✓
Full sample (by race)	1.000	$[0.80,0.93]$	Trust	0.010	99.2%	✓
Full sample (by income)	1.000	$[0.87,0.90]$	Trust	0.010	99.2%	✓
High-income target	0.401	$[0.21,0.28]$	Correct	0.019	83.3%	✓
Young consumers (18–29)	0.203	$[0.16,0.54]$	Correct	0.040	94.0%	✓

Table 10: Twin-2K-500 product pricing: diagnostic results. “Red.” columns report bias reduction from the legacy two-sample estimator analyzed in Section 5; these values do not provide evidence for frame-AIPW.

Block	$\rho$	$\min(\beta_{g})$	Decision	Interpretation
Product Pricing	0.860	$[0.80,0.93]$	Trust	Purchase patterns tracked
WTP/WTA (Thaler)	0.587–0.692	0.46–0.61	Correct/Trust	Risk framing matters
Proportion Dominance	0.605–0.689	0.47–0.61	Correct/Trust	Moderate concept shift
False Consensus	0.857	0.74	Trust	Demographic patterns tracked
Conjunction Fallacy	0.387–0.392	0.25–0.28	Correct	LLM hyper-rational
Anchoring Effects	0.337–0.500	0.28–0.37	Correct	LLM anchor-resistant

Table 11: Twin-2K-500: behavioral economics diagnostics (selected blocks).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

DGP	$\pi$	$\mu$	Estimator	Bias	RMSE	Coverage [95% MC]	Status
Probability sampling	correct	correct	Naive synthetic	0.0011	0.0310	99.65% [99.28, 99.83]	conservative
Probability sampling	correct	correct	Hájek IPW	$-0.0087$	0.0926	99.80% [99.49, 99.92]	conservative
Probability sampling	correct	correct	Plain AIPW	0.0005	0.0473	93.90% [92.76, 94.87]	failure
Probability sampling	correct	correct	PPI	0.0011	0.0339	100.00% [99.81, 100.00]	conservative
Probability sampling	correct	correct	PPI++	0.0943	0.1016	73.05% [71.06, 74.95]	failure
Probability sampling	correct	correct	Legacy	0.2707	0.2769	91.65% [90.36, 92.78]	failure

Appendix

Table C.1: Complete seven-estimator simulation battery. Bias and RMSE are in each DGP outcome scale; MC is the Wilson 95% Monte Carlo interval for coverage.

Data	Outcome	Regime	Estimator	Avail.	Bias	RMSE	Coverage [95% MC]	Status
ANES	Dem. thermometer	Mild MAR	Naive synthetic	1,000/1,000	12.0796	12.0796	0.00% [0.00, 0.38]	failure
ANES	Dem. thermometer	Mild MAR	Hájek IPW	1,000/1,000	0.0611	0.4215	100.00% [99.62, 100.00]	conservative
ANES	Dem. thermometer	Mild MAR	Plain AIPW	1,000/1,000	0.0039	0.3962	95.30% [93.81, 96.45]	compatible
ANES	Dem. thermometer	Mild MAR	PPI	1,000/1,000	0.0257	0.3672	100.00% [99.62, 100.00]	conservative
ANES	Dem. thermometer	Mild MAR	PPI++	1,000/1,000	$-0.2573$	0.4707	99.80% [99.27, 99.95]	conservative
ANES	Dem. thermometer	Mild MAR	Legacy	1,000/1,000	$-0.7310$	0.9159	100.00% [99.62, 100.00]	conservative

Appendix

Table C.2: Complete seven-estimator controlled-masking results. “Avail.” is reportable estimates over 1,000 attempts. Coverage includes unavailable frame-AIPW attempts as noncoverage.

Thermometer	18–29	30–44	45–59	60+	Decision
Dem. Party	0.796	0.800	0.832	0.980	Proceed
Rep. Party	0.611	0.609	0.667	0.756	Proceed

Appendix

Table D.1: ANES conditional calibration: age subgroup slopes for Falcon-40B thermometers.

Analysis	$\hat{\Delta}_{\text{total}}$	$\hat{\Delta}_{\text{cov}}$	$\hat{\Delta}_{\text{con}}$	Cov %	Con %
Full sample, Dem. Party	$+5.92$	$-0.20$	$+6.12$	3.2%	96.8%
Full sample, Rep. Party	$+6.20$	$+0.17$	$+6.03$	2.7%	97.3%
Young Dems, Dem. Party	$+33.06$	$+11.49$	$+21.57$	34.8%	65.2%
Older Reps, Rep. Party	$+28.81$	$+14.26$	$+14.55$	49.5%	50.5%

Appendix

Table D.2: ANES bias decomposition across analysis settings.

Thermometer	Outcome Model	$\|\textbf{Bias}_{\text{DR}}\|$	Bias Red.	CI
Dem. Party	Linear	0.029	99.5%	$\checkmark$
Dem. Party	Random forest	0.156	97.4%	$\checkmark$
Rep. Party	Linear	0.022	99.6%	$\checkmark$
Rep. Party	Random forest	0.263	95.8%	$\checkmark$

Appendix

Table D.3: ANES estimator comparison: logistic propensity with linear vs. random forest outcome models.

Subgroup	Human (ANES)	GPT-3.5	GPT-4	Falcon-40B
Democrat	24.0	19.3	36.2	25.0
Independent	40.6	46.5	53.2	32.7
Republican	66.1	86.2	81.9	54.4

Appendix

Table D.4: Cross-LLM subgroup estimates: feeling thermometer toward Republican Party.

Available data	Correction status	Action
Complete invitation frame: $W_{i}$ , $f_{i}$ , $a_{i}$ , and $R_{i}$ for all invited units; $Y_{i}$ for respondents	Eligible for frame-AIPW only after credible MAR and positivity checks	Fit cross-fitted nuisance models, apply the prespecified propensity floor, and report the result under the declared finite-frame or superpopulation reading.
Respondent-only survey, with no record of invited nonrespondents	Correction unavailable	Use a design-valid human estimator if the sampling design supports the target; otherwise reconstruct the frame or collect better human data.
Aggregate population benchmarks only	Correction unavailable	Use the benchmarks for marginal diagnostics or sensitivity analysis, not frame-AIPW correction.
Invitation frame exists, but MAR is not credible or positivity fails	Point correction unsupported	Report bounds or sensitivity analysis where possible, collect better human data, or Walk Away.

Appendix

Table H.1: Correction availability depends on the observed response frame, not a fixed respondent count.

Dataset or target	Diagnostic evidence	Scoped action
ANES full sample	ESS/N $=0.954$ ; subgroup slopes $[-0.004,0.274]$ with a sign reversal	Walk Away for affected outcomes.
ANES young Democrats	ESS/N $=0.110$ ; no within-subgroup sign flip	Outside Trust; correction unavailable without an invitation frame.
ANES older Republicans	ESS/N $=0.260$ ; no within-subgroup sign flip	Outside Trust; correction unavailable without an invitation frame.
Twin-2K full sample	ESS/N $=1.000$ ; subgroup slopes $[0.80,0.93]$	No tested diagnostic failure for this task, model, and sample.
Twin-2K high-income target	ESS/N $=0.401$ ; subgroup slopes $[0.21,0.28]$	Outside Trust; correction unavailable without an invitation frame.
Twin-2K ages 18–29	ESS/N $=0.203$ ; subgroup slopes $[0.16,0.54]$	Outside Trust; correction unavailable without an invitation frame.

Appendix

Table K.1: Cross-dataset diagnostic comparison and scoped action.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 239 papers

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses
Jul 28, 2026Zihan Chen, Di Zhu, Lei Nico ZhengLLM Evaluation Large Language Model-Based Social Simulation
Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA
Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study
Sep 26, 2026Grandee Lee, Wang YueLarge Language Model-Based Social Simulation LLM Evaluation
Singapore University of Social Sciences Singapore, Singapore
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
Jul 6, 2026Mantas Lukauskas, Viktorija ŠarkauskaitėPsychometric Validation Synthetic Data Evaluation
Hostinger · Kaunas University of Technology · AI Insight Lab +1

Jul 28, 2026J/K move · Enter open · S save

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

Zihan Chen, Di Zhu, Lei Nico Zheng

Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA

LLM Evaluation Large Language Model-Based Social Simulation Demographic Bias in Language Models Synthetic Data Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions. We ask when this substitution is valid and when it fails, and package the answer as an evaluation framework for intelligent synthetic-user systems. A single protocol, run across four models spanning two families and an 8B-to-frontier capability range, is applied to two independent domains of real human-response data: U.S. general social attitudes (General Social Survey) and cross-cultural values (World Values Survey). Every model is benchmarked against a suite of non-LLM baselines fit on held-out human data. Under demographic prompting and the survey-simulation protocols we test, two failures replicate across both domains, all four models, and both families. First, at the individual level no LLM beats even the strongest baseline; on cross-cultural values every model falls well below it, and the gap survives distance-aware and proper scoring. Second, models systematically over-determine demographics, treating identity as far more predictive of attitudes than it is among real people, a distortion present for nearly every question-group combination and robust to a coding-invariant measure. Neither failure is remedied by a larger, more capable model. A decision-impact analysis shows why this matters in practice: on a segment-targeting task the models inflate between-segment gaps two to fourfold, would direct a team to the wrong segment in half of U.S. and most cross-cultural cases, and manufacture segment splits that do not exist in real people. We publicly release the cross-domain benchmark and validation framework, including all code and data, so that teams can determine in advance when synthetic-user evidence is safe for decision support and when it is not.

Abstract

Synthetic Data Evaluation

Explore similar work

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents