Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Mantas Lukauskas; Viktorija Šarkauskaitė

cs.CYJul 6, 2026

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Authors: Mantas Lukauskas, Viktorija Šarkauskaitė

Organizations: Hostinger · Kaunas University of Technology · AI Insight Lab · Independent Researcher

Abstract

Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations mostly ask whether individual answers look plausible. We ask a stricter, psychometric question: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce an open Lithuanian organisational-psychology dataset (n=263 employees; three validated instruments; 68 items, 12 subscales) and condition 37 LLMs on real respondent profiles under a persona-disclosure ladder, presentation, reasoning-effort and cross-language ablations, counterfactual demographic swaps, and a memorization probe. The resulting Psychometric Similarity Score (PSS) is anchored against five statistical baselines, a held-out human ceiling, and the real answers of the same respondents. LLMs reproduce the direction of human psychometric relationships, but statistical baselines with no language model match or beat them. The best LLM (PSS 0.81) is level with a Gaussian copula fit only on other respondents (0.82) and below the real answers of the same 100 people (0.89). A simple classifier separates every LLM sample from humans (median AUC 0.99), and verbatim recall of the items is rare (at most 4.7%) and unrelated to PSS. Counterfactual swaps move answers in the direction of human group differences and by a similar amount (role 0.49 and education 0.33 human SD, vs. 0.46 and 0.44 in humans). Gender swaps move them by 0.06 SD. Tucker's phi on UWES falls inside an item-permutation null for 10 of 37 models. Synthetic respondents compress the response scale (within-respondent SD 0.79 vs. 1.19), lose predictive validity on held-out humans (mean R^2=-0.14 vs. 0.28), and produce significant indirect effects on 4 of 7 mediation paths that are null in humans. LLM samples are therefore not a drop-in replacement for human survey data.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the benchmark. (A) A real Lithuanian organisational-psychology sample ( $n{=}263$ , 68 items, 12 subscales) provides the ground truth, the persona profile cards, a held-out human-vs-human PSS ceiling, and the real answers of the 100 headline respondents. (B) 37 LLMs are conditioned on the profile cards under a persona-disclosure ladder, presentation, reasoning-effort, cross-language, and counterfactual-swap factors, producing ${\sim}65$ k validated synthetic questionnaires. (C) Each (model, condition) cell is scored on a six-dimensional Psychometric Similarity Score anchored against five statistical baselines and the human ceiling. The anchoring is what produces the key result: the best LLM only matches a Gaussian copula that has no language model and is fit on other respondents, and it stays below the real answers of the same people.

Path	$a$ (X $\rightarrow$ M)	$b$ (M $\rightarrow$ Y)	$c$ (X $\rightarrow$ Y)	$c^{\prime}$ (X $\rightarrow$ Y $\|$ M)	indirect ( $ab$ )
Coeff.	0.42 (0.06)	0.39 (0.06)	0.37 (0.06)	0.21 (0.06)	0.16 [0.10, 0.24]

Table 1: Standardised path coefficients of the human mediation model, replicated by our pipeline. SE in parentheses.

Model / baseline	PSS	$d$	$c$	$r$	$m$	$g$
human, same 100 respondents	0.886	0.996	0.923	0.990	0.707	0.672
human, held-out split	0.825	0.981	0.762	0.986	0.720	0.480
baseline-mvn	0.949	0.990	0.950	0.994	0.995	0.664
baseline-copula	0.936	0.994	0.954	0.986	1.000	0.522
baseline-copula, fit on other 163	0.824	0.985	0.805	0.970	0.722	0.382
baseline-stratum	0.589	0.989	0.169	0.336	0.805	0.714

Table 2: Anchored PSS leaderboard at the C3 (full profile) condition under single_call_all presentation, sorted within each block.

d

= distributional,

c

= correlation,

r

= reliability,

m

= mediation,

g

= demographic. Human references, statistical baselines, then LLMs. All rows are scored with the same five components against all 263 humans. LLM rows, the same-respondent reference, and the copula fit on the other 163 humans have

n{=}100

; the other baselines are generated for all 263 respondents. Top 5 LLMs, ensemble-mean, and worst LLM shown; full table in Appendix F .

Figure 4: PSS leaderboard with statistical baselines (orange), human-vs-human held-out ceiling (green dashed), and LLMs (blue). Respondent-bootstrap 95% CIs ( llm_pss_bootstrap_ci.csv ) have mean width

0.042

across the LLM lineup; gaps exceeding

{\sim}0.05

are statistically reliable (Holm–Bonferroni MDE

d{=}0.48

n{=}100

, Appendix E ).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Instrument	$n$ models	mean $\varphi$	mean null $\varphi$	$\#$ significant ( $p<.05$ )
Dunham ATC	37	0.770	0.730	16 / 37
Change engagement	37	0.802	0.708	20 / 37
UWES-17	37	0.859	0.764	27 / 37
IWPQ	37	0.828	0.365	30 / 37

Appendix

Table 3: Tucker’s

\varphi

permutation test per instrument at C3. “mean null

\varphi

” is the mean congruence under 500 random item permutations of the LLM loading matrix and gives the chance level of the alignment search; the rightmost column counts the models whose observed

\varphi

exceeds the 95th percentile of that null.

Scenario	$n$	$\alpha$	$1-\beta$	MDE
paired_t_pss_delta_n100	100	0.05	0.80	0.280
paired_t_pss_delta_n100_holm	100	7.5e-05	0.80	0.480
paired_t_counterfactual_subscale_n100	100	0.05	0.80	0.280
paired_t_counterfactual_subscale_n100_holm	100	3.2e-05	0.80	0.500
paired_t_reasoning_ablation_n100_holm	100	0.000595	0.80	0.428
paired_t_cross_language_n100	100	0.05	0.80	0.280

Appendix

Table 4: Post-hoc power analysis: minimum detectable effect (MDE) at

\alpha

and target power, for every paired contrast in the paper. Holm-Bonferroni rows control the family-wise error rate across all pairwise comparisons in the relevant lineup (37 models for the leaderboard).

	gender		role		education
Model	$\|d_{z}\|$	$\|$ shift $\|$	$\|d_{z}\|$	$\|$ shift $\|$	$\|d_{z}\|$	$\|$ shift $\|$
gemini-3.1-flash-lite	0.26	0.11	1.66	0.98	0.77	0.42
gemini-3.1-pro	0.20	0.10	1.13	0.65	0.93	0.64
deepseek-v4-pro	0.04	0.04	0.81	0.77	0.62	0.56
gemini-2.5-pro	0.12	0.06	1.25	0.71	0.83	0.51
mistral-large-3	0.21	0.07	1.61	0.78	0.76	0.32

Appendix

Table 5: Counterfactual swaps, mean over the 12 composite subscales.

|d_{z}|

is the paired Cohen’s

d

, with each pair difference signed by the direction of its swap.

|

shift

|

is the mean absolute shift toward the first group of the human contrast (woman, managerial, university) in units of the human SD. The last row gives the mean

|d|

of the observed human contrast; the education contrast rests on 15 respondents with secondary education. Models are sorted by the mean shift over the three axes.

Model	mean $\|d\|$	max $\|d\|$	n subsc.
gpt-5.4	0.370	0.597	12
claude-opus-4-7	0.308	0.649	12
grok-4-20	0.172	0.245	12
gemini-3.1-pro	0.157	0.447	12
deepseek-v4-pro	0.156	0.283	12
grok-4-20-reasoning	0.083	0.158	12

Appendix

Table 6: Reasoning-effort ablation: paired Cohen’s

d

between “minimal” and “high” reasoning at the composite-subscale level. Mean and max absolute

d

across 12 subscales per model.

Model	mean $\|\Delta\|$	max $\|\Delta\|$	$r_{\textsf{LT,EN}}$	n items
grok-4-1-fast-non-reasoning	0.797	1.600	0.688	68
grok-4-1-fast-reasoning	0.544	1.533	0.811	68
qwen3-7-plus	0.442	1.954	0.845	68
minimax-m2-7	0.379	1.462	0.906	68
mistral-large-3	0.335	1.267	0.781	68
nova-2-lite	0.332	0.867	0.862	68

Appendix

Table 7: Cross-language ablation: same model, same persona, Lithuanian vs. English questionnaire (C3 condition, 30 stratified respondents).

\Delta

is the per-item EN

-

LT mean response on the Likert scale;

r_{\textsf{LT,EN}}

is the Pearson correlation between the model’s per-item mean responses in LT and EN, with reverse-keyed items reverse-scored.

Model	per_instrument	per_subscale	mean
llama-4-maverick	0.286	1.257	0.771
deepseek-v4-pro	0.278	0.564	0.421
gpt-5.4	0.343	0.443	0.393
claude-opus-4-7	0.354	0.309	0.332
gemini-3.1-pro	0.235	0.284	0.259

Appendix

Table 8: Presentation-mode ablation: mean

|\text{paired }d|

across 12 composite subscales relative to the single_call_all reference, per (model, mode). Larger values mean the same model produces a more different psychometric distribution when the questionnaire is split across calls.

Model	age	education	gender	max
minimax-m2-7	0.043	0.122	0.068	0.122
claude-haiku-4-5	0.055	0.101	0.053	0.101
nemotron-3-super	0.038	0.095	0.049	0.095
kimi-k2-7-code	0.041	0.091	0.069	0.091
llama-4-scout	0.000	0.028	0.085	0.085
qwen3-5-397b	0.014	0.084	0.026	0.084

Appendix

Table 9: Cohort-stratified PSS disparity. Each cell is

\max_{stratum}\,\text{PSS}-\min_{stratum}\,\text{PSS}

on a single demographic axis (computed at C3 / single_call_all). The “max” column is the largest disparity any axis produces for that model.

Model	PSS 3	95% CI	$d$	$c$	$r$
minimax-m2-7	0.586	[0.568, 0.600]	0.825	0.738	0.977
gemini-3-flash	0.584	[0.558, 0.601]	0.839	0.750	0.935
grok-4-1-fast-non-reasoning	0.583	[0.555, 0.602]	0.892	0.699	0.929
grok-4-1-fast-reasoning	0.583	[0.555, 0.600]	0.894	0.692	0.931
gemini-2.5-pro	0.581	[0.557, 0.599]	0.861	0.710	0.945
gemini-3.1-pro	0.580	[0.559, 0.597]	0.822	0.737	0.954

Appendix

Table 10: PSS leaderboard with respondent-bootstrap 95% confidence intervals (

n_{\text{boot}}=1000

). The reported total is the 3-component PSS (distribution + correlation + reliability), since those components are sensitive to respondent resampling; mediation and demographic components depend on a structural model and are held fixed at 0.5 inside the bootstrap.

Model	Violations	Rate (%)	max HTMT
Human (reference)	4/12	33.3	0.95
mistral-large-3	11/12	91.7	1.02
grok-4-1-fast-reasoning	11/12	91.7	1.00
gemini-3.5-flash	10/12	83.3	1.01
grok-4-1-fast-non-reasoning	10/12	83.3	1.00
gpt-oss-120b	10/12	83.3	0.97

Appendix

Table 11: Discriminant validity (HTMT) by model at the headline condition (C3 / single_call_all). “Violations” counts the within-instrument subscale pairs (of the 12 in the battery) with

\text{HTMT}>0.85

, i.e. subscales that are not empirically distinct; pairs whose correlations are undefined for a model are counted as non-violations. A higher count means more construct over-coherence. The human sample is the reference row. Sorted by violation count.

Model	Acq. $\Delta$ (SD)	SD $\Delta$	Extreme $\Delta$	Midpoint $\Delta$
nova-2-lite	+1.26	-0.50	-0.26	+0.29
gemini-3-flash	+0.91	-0.29	-0.16	+0.07
gpt-5.4	+0.89	-0.37	-0.23	+0.10
minimax-m2-7	+0.80	-0.39	-0.22	+0.14
gpt-5.4-nano	+0.69	-0.50	-0.26	+0.28
gpt-5.5	+0.69	-0.39	-0.24	+0.17

Appendix

Table 12: Response style relative to the human sample at C3 / single_call_all (model minus human). Acquiescence is the balanced agreement index on the two instruments with both item keys (Dunham ATC and change engagement): the mean of the forward-keyed and reverse-keyed items on the as-answered, 0–1 rescaled responses; the difference is in units of the human between-respondent SD of the index. SD is the within-respondent SD across all items (Likert points). Extreme and midpoint are the shares of endpoint and scale-centre answers. Sorted by acquiescence.

Model	Synthetic $R^{2}$	$\Delta R^{2}$ vs human
qwen3-7-plus	+0.303	+0.026
gemini-2.5-pro	+0.296	+0.019
gemini-3.5-flash	+0.290	+0.013
minimax-m2-7	+0.284	+0.007
gemini-3-flash	+0.268	-0.010
deepseek-v4-pro	+0.258	-0.019

Appendix

Table 13: Downstream predictive validity at C3 / single_call_all. An OLS regression of the performance total on the attitudes and engagement totals is fit on each model’s synthetic respondents and tested on a fixed 30% split of the humans; we report the out-of-sample

R^{2}

and its gap to the same regression fit on the other 70% of humans (

R^{2}_{\text{human}}=0.277

\Delta R^{2}<0

means the synthetic data is a worse training set than real human data;

R^{2}_{\text{synth}}<0

means it predicts worse than the mean. Sorted by synthetic

R^{2}

Placebo path ( $x\to m\to y$ )	Human ind.	Pooled ind.	Fabricated?	Models
UWES-ded $\to$ ATC-beh $\to$ IWPQ-task	+0.040	+0.126	yes	10/37
ATC-cog $\to$ ATC-aff $\to$ UWES-abs	+0.092	+0.218	yes	20/37
ATC-aff $\to$ IWPQ-cprod $\to$ ATC-beh	-0.025	-0.032	no	6/37
IWPQ-task $\to$ ATC-beh $\to$ IWPQ-cprod	-0.017	-0.111	yes	14/37
ATC-total $\to$ IWPQ-task $\to$ UWES-abs	+0.015	+0.075	no	17/37
ATC-aff $\to$ IWPQ-task $\to$ ATC-beh	+0.011	+0.061	no	5/37

Appendix

Table 14: Mediation negative control at C3. Each row is a placebo

x\to m\to y

path between composites of attitudes toward change (ATC), work engagement (UWES) and work performance (IWPQ), whose human indirect effect has a bootstrap CI that includes zero and

|ab|<0.10

. The 3 drawn paths in which a total score contains another variable of the path are left out, because they do not test mediation. “Pooled” is the indirect effect on all LLM respondents stacked (

n=3673

); “Fabricated” flags paths where that CI excludes zero and

|ab|>0.10

. The last column counts the models that fabricate the path on their own sample (

n\approx 100

each). Per model, the median number of fabricated paths is 2 of 7.

Model	ICC(1)	Top-line $\rho$	$\Delta$ vs human	$n/k$
gpt-5.4	0.93	0.93	+0.15	28/20
qwen3-7-plus	0.93	0.90	+0.15	21/20
claude-sonnet-4-6	0.93	0.93	+0.15	28/20
gemini-3.5-flash	0.93	0.88	+0.14	30/20
mistral-large-3	0.92	0.91	+0.14	27/20
gemma4-31b	0.92	0.93	+0.13	27/20

Appendix

Table 15: Test–retest reliability of LLM synthetic respondents. Each model answered the same stability panel

k

independent times (fresh per-call sampling); ICC(1) is the scale-averaged one-way intra-class correlation of the composite scores (reverse-keyed items reverse-scored), treating the repeat index as an interchangeable rater. “Top-line

\rho

” is the mean inter-repeat Spearman correlation across all 68 items.

\Delta

vs human is the gap to the pooled published human test–retest reference (

\overline{\text{ICC}}_{\text{human}}=0.78

);

n/k

is the number of respondents with complete repeats and the number of repeats. Sorted by ICC(1). Models with fewer than 10 complete-repeat respondents are omitted ( gpt-oss-120b ).

Generator	Mean AUC	Indistinguishable?
Gaussian copula	0.39	yes
Multivariate normal	0.52	yes
Marginal	0.68	no
Stratum mean	0.69	no
$k$ -NN	0.89	no
Midpoint	0.99	no

Appendix

Table 16: Human-vs-synthetic distinguishability. A logistic-regression and a random-forest discriminator are trained to separate real human respondents from each generator’s synthetic respondents; we report the mean held-out AUC (chance

=0.5

). “Indistinguishable” flags generators that a discriminator cannot beat (

\text{AUC}\leq 0.55

). The LLM row gives the median and range across models.

Generator	$n{=}10$	$n{=}25$	$n{=}50$	$n{=}100$	$n{=}200$
Gaussian copula	0.694	0.751	0.785	0.806	0.819
Multivariate normal	0.684	0.744	0.778	0.804	0.817
Best LLM (minimax-m3)	0.617	0.665	0.686	0.695	0.702
Gap (copula $-$ best LLM)	+0.077	+0.086	+0.100	+0.111	+0.117

Appendix

Table 17: PSS 3 (the three sample-driven components; mediation and demographic held at 0.5) as a function of the number of synthetic respondents

n

(bootstrap means, 60 draws). LLM rows use the C3 cell, which has about 100 respondents per model, so the LLM curve stops at

n{=}100

. The best LLM is the top model of the leaderboard.

Predictor	Coef.	SE
Intercept	+0.281	0.030
Instrument: Dunham ATC	-0.008	0.019
Instrument: IWPQ	+0.071	0.020
Instrument: UWES-17	+0.155	0.021
Item length (chars)	-0.000	0.000
Reverse-keyed	+0.049	0.019

Appendix

Table 18: Which items are hard to reproduce? OLS of the per-item mean Jensen–Shannon divergence between the LLM and human response distributions (averaged over the LLM cells at C3) on item features,

n{=}68

items (

R^{2}=0.61

). Higher divergence means worse reproduction, so a positive coefficient marks items that are reproduced worse.

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

Instrument	Human ECV	LLM ECV	$\Delta$ ECV	$\Delta\omega_{h}$
Dunham ATC	0.67	0.76	+0.08	-0.00
Change engagement	0.62	0.71	+0.09	-0.02
UWES-17	0.70	0.80	+0.10	-0.00
IWPQ	0.45	0.58	+0.12	+0.09

Appendix

Table 19: Bifactor structure, LLM mean at C3 (37 models) vs human. ECV is the share of common variance explained by the general factor;

\omega_{h}

is the general-factor (hierarchical) omega.

\Delta

is LLM minus human.

Condition	PSS	PSS ρ	PSS dist	SD	Acq.	ERS	MRS
Human reference	n/a	n/a	n/a	1.19	0.00	0.27	0.21
Zero-shot (C3)	0.664	0.682	0.764	0.82	+0.21	0.04	0.37
+ Debias instruction	0.682	0.663	0.780	0.99	+0.33	0.10	0.32
+ Few-shot ( $k{=}3$ humans)	0.658	0.664	0.770	0.81	+0.18	0.04	0.38

Appendix

Table 20: Steerability. Mean over the 6-model subset ( claude-sonnet-4-6 , gemini-3.1-pro , gpt-5.5 , glm-5-1 , llama-4-maverick , qwen3-5-397b ) at

n{=}100

matched respondents. C13 adds an explicit instruction to answer like a real, varied respondent; C14 adds three human exemplars. PSS ρ and PSS dist are the correlation and distribution components. SD is the within-respondent SD; Acq. is the balanced acquiescence index relative to humans, in human SD units; ERS/MRS are the extreme and midpoint response rates.

Generator	PSS	PSS w/o $d$	$d$	$c$	$r$	$m$	$g$
copula (unconditional)	0.936	0.688	0.994	0.954	0.986	1.000	0.522
cond-copula (in-sample)	0.968	0.719	0.996	0.950	0.996	0.978	0.871
cond-copula (5-fold OOS)	0.929	0.680	0.995	0.938	0.976	0.864	0.773
MVN baseline	0.949	0.702	0.990	0.950	0.994	0.995	0.664
best LLM (minimax-m3)	0.814	0.636	0.713	0.764	0.966	0.998	0.520

Appendix

Table 21: Conditioning a statistical baseline on the LLM’s profile. The unconditional Gaussian copula ignores the persona; the conditional copula ridge-regresses each item on the same profile fields the LLM receives and keeps the residual correlation structure, in-sample and under 5-fold cross-validation. All rows are scored with the same PSS. “PSS w/o

d

” sets the distribution component, which the bounded-Likert copulas reproduce by construction, to zero and keeps the other weights unchanged; it is computed the same way for every row.

Model / baseline	PSS	$d$	$c$	$r$	$m$	$g$
human, same 100 respondents	0.886	0.996	0.923	0.990	0.707	0.672
human, held-out split	0.825	0.981	0.762	0.986	0.720	0.480
baseline-mvn	0.949	0.990	0.950	0.994	0.995	0.664
baseline-copula	0.936	0.994	0.954	0.986	1.000	0.522
baseline-copula, fit on other 163	0.824	0.985	0.805	0.970	0.722	0.382
baseline-stratum	0.589	0.989	0.169	0.336	0.805	0.714

Appendix

Table 22: Anchored PSS leaderboard at the C3 (full profile) condition under single_call_all presentation, sorted within each block.

d

= distributional,

c

= correlation,

r

= reliability,

m

= mediation,

g

n{=}100

; the other baselines are generated for all 263 respondents.

Figure 7: Hierarchical clustering of the 37 LLMs by inter-model PSS (distance

1{-}\text{PSS}

, leaf colour = model vendor). Most models form one large cluster that mixes vendors; llama-4-maverick , gpt-5.4-mini , nova-2-lite , gpt-5.4-nano and mistral-large-3 lie outside it.

Figure 8: Cohort-stratified PSS: model $\times$ stratum heatmap (left) and worst-axis disparity per model (right).

Figure 9: Memorization probe: per-model verbatim-recall rate (left) and PSS vs. recall scatter (right). Worst-case high-recall rate is

4.7\%

;

22/37

models are exactly zero. Spearman rank correlation between recall rate and PSS is

{-}0.04

Explore similar work

Oct 10, 2026

Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

Language models are increasingly used to create synthetic survey samples, also called silicon samples. Decision models are a new class of language models that return a probability for each point of a rating scale instead of text. We ask whether these models, or the newest generative LLMs, can stand in for human respondents on a psychometric survey. We test fourteen decision models from eleven providers and 47 generative LLMs on an open Lithuanian organisational-psychology survey (n=263, 68 items), with 100 real respondent profiles, the Psychometric Similarity Score (PSS) and statistical baselines. The best generative model, MiniMax M3, reaches PSS 0.813. This is level with a Gaussian copula fit only on other respondents (0.824) and below a copula fit in sample (0.927). A second sample of 100 real people scores below the best models, so at this size PSS cannot rank them against real respondents. In 5 of 10 product lines the newest generative release scores below an earlier release of the same line beyond resampling noise, and none gains beyond it. With the default rounded readout, 13 of 14 decision models score below the generative reference gpt-5.4-mini. Of the 47 generative models, 33 reproduce the pattern of item correlations better than any decision model. Averaged over respondents, the probability vectors are much closer to the human item marginals than rounded answers. They also rank respondents about as well as a profile regression fit on 163 real people. As forecasts of each person's own answers, however, every decision model is worse than the answer frequencies of other respondents, mainly because its average vector differs from them. The profile regression improves on these frequencies by only 1.0%, so the profile fields say little about how a person answers. Synthetic respondents built on such profiles should therefore aim at sample-level fidelity.

Mantas Lukauskas, Viktorija Šarkauskaitė

Hostinger; Kaunas University of Technology; AI Insight Lab · AI Insight Lab; Independent Researcher

Synthetic Data Generation Synthetic Data Evaluation Psychometric Validation Synthetic Data Generation for Language Models Language Model Generation Evaluation

Ask PDF

Sep 26, 2026

Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study

Large language models (LLMs) are increasingly used to generate synthetic survey respondents and digital twins of real people, but whether their output preserves real human statistical structure, rather than surface plausibility, remains unresolved, and most existing evidence comes from proprietary models rather than open-weight ones. We evaluate three open-weight LLM families on a cross-instrument calibration task: conditioning personas on real respondents' verbatim answers to one psychometric instrument and measuring them on a second, construct-distance-controlled instrument, checked against a 2,058-person human panel. Across a 139-pair grid, the simulated cross-instrument correlation tracks the real human correlation at r = 0.70 - 0.73 in every model, driven mainly by correct sign rather than precise magnitude and concentrated in pairs of moderate construct distance. A correlation of this magnitude, obtained from untuned open-weight models conditioned only on individual-level survey data, is a substantively encouraging result for LLM-based behavioral simulation and digital-twin applications: specific model families and releases already reproduce a meaningful share of real human cross-instrument structure without any fine-tuning. This capability does not, however, improve monotonically across model releases: on a matched panel, the newest of three tested Llama releases performs worst on two of three headline metrics, so realizing its promise in practice requires release-specific, distance-aware verification rather than a one-time benchmark.

Grandee Lee, Wang Yue

Singapore University of Social Sciences Singapore, Singapore

Large Language Model-Based Social Simulation LLM Evaluation Language Model Calibration

Ask PDF

Jul 28, 2026

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions. We ask when this substitution is valid and when it fails, and package the answer as an evaluation framework for intelligent synthetic-user systems. A single protocol, run across four models spanning two families and an 8B-to-frontier capability range, is applied to two independent domains of real human-response data: U.S. general social attitudes (General Social Survey) and cross-cultural values (World Values Survey). Every model is benchmarked against a suite of non-LLM baselines fit on held-out human data. Under demographic prompting and the survey-simulation protocols we test, two failures replicate across both domains, all four models, and both families. First, at the individual level no LLM beats even the strongest baseline; on cross-cultural values every model falls well below it, and the gap survives distance-aware and proper scoring. Second, models systematically over-determine demographics, treating identity as far more predictive of attitudes than it is among real people, a distortion present for nearly every question-group combination and robust to a coding-invariant measure. Neither failure is remedied by a larger, more capable model. A decision-impact analysis shows why this matters in practice: on a segment-targeting task the models inflate between-segment gaps two to fourfold, would direct a team to the wrong segment in half of U.S. and most cross-cultural cases, and manufacture segment splits that do not exist in real people. We publicly release the cross-domain benchmark and validation framework, including all code and data, so that teams can determine in advance when synthetic-user evidence is safe for decision support and when it is not.

Zihan Chen, Di Zhu, Lei Nico Zheng

Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA

LLM Evaluation Large Language Model-Based Social Simulation Demographic Bias in Language Models Synthetic Data Evaluation

Ask PDF