Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

Mantas Lukauskas; Viktorija Šarkauskaitė

cs.AIOct 10, 2026

Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

Authors: Mantas Lukauskas, Viktorija Šarkauskaitė

Organizations: Hostinger; Kaunas University of Technology; AI Insight Lab · AI Insight Lab; Independent Researcher

Abstract

Language models are increasingly used to create synthetic survey samples, also called silicon samples. Decision models are a new class of language models that return a probability for each point of a rating scale instead of text. We ask whether these models, or the newest generative LLMs, can stand in for human respondents on a psychometric survey. We test fourteen decision models from eleven providers and 47 generative LLMs on an open Lithuanian organisational-psychology survey (n=263, 68 items), with 100 real respondent profiles, the Psychometric Similarity Score (PSS) and statistical baselines. The best generative model, MiniMax M3, reaches PSS 0.813. This is level with a Gaussian copula fit only on other respondents (0.824) and below a copula fit in sample (0.927). A second sample of 100 real people scores below the best models, so at this size PSS cannot rank them against real respondents. In 5 of 10 product lines the newest generative release scores below an earlier release of the same line beyond resampling noise, and none gains beyond it. With the default rounded readout, 13 of 14 decision models score below the generative reference gpt-5.4-mini. Of the 47 generative models, 33 reproduce the pattern of item correlations better than any decision model. Averaged over respondents, the probability vectors are much closer to the human item marginals than rounded answers. They also rank respondents about as well as a profile regression fit on 163 real people. As forecasts of each person's own answers, however, every decision model is worse than the answer frequencies of other respondents, mainly because its average vector differs from them. The profile regression improves on these frequencies by only 1.0%, so the profile fields say little about how a person answers. Synthetic respondents built on such profiles should therefore aim at sample-level fidelity.

Figures & tables

Reference points
Source	Dist.	Corr.	Rel.	Med.	Demo.	PSS	SE
Same 100 humans (oracle)	1.00	0.92	0.99	0.71	0.67	0.886	0.033
Held-out human split, original audit	0.98	0.76	0.99	0.72	0.48	0.825
Gaussian copula, in-sample	0.99	0.89	0.99	0.97	0.66	0.927	0.019
Gaussian copula, out-of-sample	0.98	0.81	0.97	0.72	0.38	0.824	0.028
Independent marginals	0.99	0.00	0.00	0.67	0.49	0.430

Table 1: Aligned PSS components at C3 on the same 100 respondents, scored against all 263 humans. Generative rows show the three best of 47 generative cells, the best and the worst of the 11 models released after the original audit (marked ∗ ), GPT-6 Luna and the fixed reference gpt-5.4-mini; Appendix Table 15 lists all of them. The median row takes the median of each column separately. Decision models use the rounded expected score. SE is the bootstrap standard error of PSS over 200 draws that resample respondents and humans. The held-out split row is the ceiling reported in the original audit (53 humans scored against 210). The out-of-sample copula and the profile regression are fit on the 163 humans outside the headline sample; the profile regression predicts each item from the profile fields and is read out like a decision model.

Figure 1: The best language models are level with a copula fit only on other respondents, and none reaches the in-sample copula or the oracle (the real answers of the same 100 people). Left: PSS of every language-model cell at C3. Right: the correlation component. Each point is one model, and vertical jitter only separates points. Vertical lines mark the out-of-sample copula, the in-sample copula and the oracle.

Source	Readout	$\Delta$ Correlation	$\Delta$ Mediation	$\Delta$ PSS
Same 100 humans (oracle)		+0.16 [+0.08, +0.23]	$-$ 0.29 [ $-$ 0.62, +0.04]	+0.07 [0.00, +0.15]
Gaussian copula (in-sample)		+0.12 [+0.05, +0.20]	$-$ 0.03 [ $-$ 0.24, +0.18]	+0.11 [+0.06, +0.17]
Gaussian copula (out-of-sample)		+0.04 [ $-$ 0.06, +0.15]	$-$ 0.28 [ $-$ 0.55, $-$ 0.01]	+0.01 [ $-$ 0.07, +0.09]
MiniMax M2.7		+0.04 [ $-$ 0.01, +0.10]	$-$ 0.24 [ $-$ 0.62, +0.14]	0.00 [ $-$ 0.09, +0.08]
Kimi K2.7 Code		+0.04 [ $-$ 0.01, +0.09]	$-$ 0.20 [ $-$ 0.53, +0.12]	$-$ 0.01 [ $-$ 0.08, +0.06]
gpt-5.4-mini		$-$ 0.16 [ $-$ 0.24, $-$ 0.09]	$-$ 0.07 [ $-$ 0.29, +0.15]	$-$ 0.07 [ $-$ 0.13, $-$ 0.01]

Table 2: Paired differences to the best generative cell, MiniMax M3 (aligned). Brackets give the point estimate

\pm 1.96

bootstrap standard errors of the paired difference over 200 draws, clipped to

[-1,1]

. Every draw resamples the 100 respondents and the 263 humans once and applies the same draw to both sources. R = rounded expected score, A = most probable category. An interval that contains zero means the difference is within resampling noise. The reference was chosen as the highest point estimate among 47 cells, so the differences to it are biased away from zero.

	PSS			Correlation			Distribution			Mediation
Model	R	A	S	R	A	S	R	A	S	R	A	S
Jev 1.13	0.681	0.786	0.700	0.66	0.67	0.35	0.77	0.78	0.91	0.50	0.97	0.83
Microsoft-Decision-1	0.667	0.622	0.665	0.40	0.37	0.27	0.68	0.68	0.85	0.79	0.65	0.86
Decider v1.1 27B	0.660	0.579	0.517	0.34	0.34	0.07	0.63	0.66	0.95	0.95	0.55	0.73
Drex v1.5	0.659	0.682	0.654	0.46	0.32	0.23	0.70	0.72	0.92	0.72	1.00	0.85
Decider v1 27B	0.659	0.560	0.519	0.35	0.35	0.07	0.63	0.66	0.95	0.93	0.44	0.73

Table 3: The same probability vectors read out three ways (aligned, C3,

n{=}100

): R is the expected score rounded to the nearest category, A is the most probable category, S is one category drawn from the vector per item (mean of five draws; the SD of PSS across draws is at most 0.040). Bold marks the best readout per model and metric. For reference, the median of 47 generative LLMs is 0.695 for PSS, 0.73 for correlation, 0.75 for distribution and 0.46 for mediation; the best generative cell (MiniMax M3) has PSS 0.813. The last row reads the forecasts of the profile regression, fit on the 163 humans outside the sample, in the same three ways.

Figure 2: PSS of each decision model under three readouts of the same probability vectors. Bars show the rounded expected score with

\pm 1.96

bootstrap standard errors. Vertical lines mark the median and the best of the 47 generative LLMs, the out-of-sample copula and the real answers of the same 100 people.

Figure 3: Sampling raises the distribution component and lowers the correlation component for every decision model. Each colour is one decision model, and lines connect its three readouts. Grey points are the 47 generative LLMs, the star is gpt-5.4-mini, and the crosses mark the out-of-sample copula and the real answers of the same 100 people.

	Entropy							JS to human
Model	Fwd.	Rev.	Max $p$	Between	Person	SD ratio	Unique	Rounded	Mixture
Mercury Decide	0.61	0.61	0.58	0.30	0.49	0.59	99	0.218	0.085
Jev 1.13	0.67	0.74	0.53	0.16	0.49	0.46	100	0.231	0.078
Tev1 4B	0.81	0.87	0.40	0.10	0.40	0.44	100	0.296	0.082
Clef Flash	0.82	0.79	0.42	0.12	0.39	0.46	94	0.300	0.068
Drex v1.5	0.82	0.84	0.39	0.12	0.38	0.45	99	0.298	0.076

Table 4: What the probability vectors contain (aligned, C3,

n{=}100

). Entropy is normalised by the log of the number of categories, for forward and reverse-keyed items. Between is the share of the model’s answer variance (between-respondent variance of the expected score plus mean within-respondent variance of the vector) that depends on the respondent. Person is the share of respondent-by-item variance in rounded answers that is not an item main effect. SD ratio is the median item SD relative to all 263 humans. Unique counts distinct 68-item answer vectors. The last two columns give the mean item Jensen-Shannon divergence to the human marginal for the rounded answers and for the average probability vector.

Source	Mean $r$	$\lambda_{1}$	W $-$ B	$\alpha$	$\varphi$	Null	$\lambda_{1}$ (E)
Humans ( $n{=}263$ , reference)	0.22	0.28	0.16	0.81	1.00	4
Same 100 humans (oracle)	0.25	0.33	0.15	0.81	0.97	4
Copula, in-sample	0.21	0.27	0.12	0.79	0.96	4
Copula, out-of-sample	0.15	0.20	0.14	0.79	0.91	3
MiniMax M3 (best LLM)	0.37	0.44	0.16	0.89	0.87	3
gpt-5.4-mini	0.23	0.29	0.19	0.72	0.69	2

Table 5: Internal structure of each sample (aligned answers, rounded readout for language models, the same 100 profiles). Mean

r

is the mean inter-item correlation over all 68 items.

\lambda_{1}

is the share of the largest eigenvalue of the item correlation matrix. W

-

B is the mean within-subscale minus the mean between-subscale item correlation inside an instrument, averaged over the four instruments.

\alpha

is the median Cronbach’s

\alpha

of the 12 subscales.

\varphi

is the median over the four instruments of Tucker’s

\varphi

against the 263 humans, with three varimax factors per instrument. Null counts the instruments on which

\varphi

is above an item-permutation null (

K{=}500

p{<}0.05

). Items without variance in a sample are left out of its

\varphi

\lambda_{1}

(E) is the same eigenvalue share for the expected scores of a decision model. The oracle is part of the 263-person reference, which favours it.

	Retest		Profile			JS to human
Model	Same	Repl.	Shift	PSS C0	PSS C3	C0	C3
Jev 1.13	95	0.99	0.14	0.40	0.70	0.154	0.078
Microsoft-Decision-1	95	0.98	0.16	0.39	0.66	0.206	0.137
Decider v1.1 27B	100	1.00	0.04	0.43	0.52	0.046	0.040
Drex v1.5	99	1.00	0.18	0.40	0.65	0.156	0.076
Decider v1 27B	100	1.00	0.04	0.43	0.52	0.046	0.040

Table 6: Control grids for the decision models. Same is the percentage of rounded answers that are identical when 20 respondents are sent again. Repl. is the share of between-respondent variance of the expected score that is not run-to-run noise. Shift is the mean absolute difference between the C3 expected score and the no-profile (C0) expected score on a 0–1 scale. PSS uses the sampled readout, which is the only readout that gives varying answers at C0, averaged over the same five draws as in Table 3 . The JS columns give the mean item Jensen-Shannon divergence between the human marginal and the average probability vector. The generative row uses the C0 and C3 cells of the original audit. GPT-6 Luna Decisions returned HTTP 502 for every request without a profile.

	Education		Role		Gender
Model	$\|$ Shift $\|$	$\|d\|$	$\|$ Shift $\|$	$\|d\|$	$\|$ Shift $\|$	$\|d\|$
Jev 1.13	0.33	1.11	0.22	0.92	0.01	0.11
Microsoft-Decision-1	0.30	1.36	0.33	1.54	0.03	0.17
Decider v1.1 27B	0.15	1.00	0.18	1.43	0.03	0.34
Drex v1.5	0.37	1.39	0.28	1.28	0.08	0.54
Decider v1 27B	0.15	1.01

Table 7: Counterfactual swaps of one profile field, averaged over the 12 composites.

|

Shift

|

is the mean absolute shift toward the first group of the human contrast (university, managerial, woman) in human SD units.

|d|

is the mean absolute paired

d

of the original audit, with each difference signed by the direction of the swap. Decision models use the rounded readout. Generative rows use the swap grids of the original audit. The human row gives the mean

|d|

of the observed contrast between the two groups among all 263 humans, which is on the same scale as

|

Shift

|

. The education contrast rests on 15 respondents with secondary education, and the gender contrast is null. Section 5.8 gives bootstrap intervals for the medians and the human contrasts. Decider v1 27B had no available endpoint when the role and gender swaps were run.

Forecaster	Skill (%)	Shrunk	Own avg.	C0	Rounded	$\rho$	Entropy
Clef	$-$ 6 [ $-$ 10, $-$ 2]	0.6	$-$ 7	$-$ 9	$-$ 48	0.15	1.40
Liquid d1	$-$ 8 [ $-$ 12, $-$ 3]	0.9	$-$ 9	$-$ 16	$-$ 51	0.16	1.35
Decider v1.1 27B	$-$ 8 [ $-$ 12, $-$ 4]	0.1	$-$ 10	$-$ 10	$-$ 55	0.14	1.40
Decider v1 27B	$-$ 8 [ $-$ 12, $-$ 4]	0.1	$-$ 10	$-$ 10	$-$ 55	0.14	1.41
Clef Omni	$-$ 16 [ $-$ 23, $-$ 10]	0.8	$-$ 17	$-$ 16	$-$ 58	0.18	1.29
Clef Flash	$-$ 18 [ $-$ 26, $-$ 11]	0.7	$-$ 18	$-$ 75	$-$ 65	0.19	1.39

Table 8: Probability vectors scored as forecasts of the answers of the same 100 people. Skill is

100\,(1-\mathrm{RPS}/\mathrm{RPS}_{\text{pop}})

, where RPS is the ranked probability score averaged over items and respondents and

\mathrm{RPS}_{\text{pop}}=0.128

is the score of the population distribution of the 163 humans outside the sample. Negative values are worse than this distribution. Brackets give 95% paired bootstrap intervals over respondents. Shrunk mixes each vector with the population distribution, with the weight chosen by leave-one-respondent-out cross-fitting. Own average gives every respondent the model’s mean vector over all respondents. C0 uses the vectors returned without a profile, and Rounded scores the rounded expected score as a certain forecast.

\rho

is the mean item-level Spearman correlation between the expected score and the person’s answer. Entropy is the mean entropy of the forecast in nats. The profile regression is a multinomial logistic regression of each item on the profile fields, fit on the same 163 humans.

Figure 4: No language model forecasts a person’s answers better than the answer frequencies of other respondents. Forecast skill on the 26 to 30 respondents of the test-retest panel, relative to the population distribution of the 163 humans outside the headline sample. Each point is one model. Vertical lines mark the population distribution (zero), the profile regression and the uniform distribution. The second row draws 20 answers from each decision vector and scores them like the generative repeats.

Line	Releases (PSS)	$\Delta$	95% interval	No M
Anthropic Opus	Claude Opus 4.7 0.79 $\rightarrow$ Claude Opus 4.8 0.74 $\rightarrow$ Claude Opus 5.5 0.61	$-$ 0.13	[ $-$ 0.20, $-$ 0.07]	$-$ 0.02
Anthropic Sonnet	Claude Sonnet 4.6 0.59 $\rightarrow$ Claude Sonnet 5 0.74 $\rightarrow$ Claude Sonnet 5.5 0.73	0.00	[ $-$ 0.08, +0.07]	0.00
OpenAI flagship	GPT-5.4 0.61 $\rightarrow$ GPT-5.5 0.61 $\rightarrow$ GPT-6.1 Sol 0.51	$-$ 0.10	[ $-$ 0.17, $-$ 0.03]	$-$ 0.10
OpenAI small	gpt-5.4-mini 0.74 $\rightarrow$ GPT-6 Luna 0.62	$-$ 0.12	[ $-$ 0.19, $-$ 0.05]	$-$ 0.03
Google Flash	Gemini 3 Flash 0.70 $\rightarrow$ Gemini 3.5 Flash 0.74 $\rightarrow$ Gemini 3.6 Flash 0.64 $\rightarrow$ Gemini 3.8 Flash 0.71	$-$ 0.02	[ $-$ 0.10, +0.05]	$-$ 0.01
xAI Grok	Grok 4.3 0.71 $\rightarrow$ Grok 4.7 0.61	$-$ 0.10	[ $-$ 0.19, 0.00]	$-$ 0.04

Table 9: Aligned PSS of successive releases in each product line with a release after the original audit, oldest first.

\Delta

compares the two releases in bold: the newest release with the newest release of the same line in the original audit. Brackets give the 95% paired bootstrap interval. In 8 of 10 lines the newest release scores lower, 1 is level and 1 gains. In 5 lines the interval lies below zero. No M is

\Delta

for PSS without the mediation component, with the other weights rescaled; it is negative in 9 lines. The DeepSeek, Moonshot Kimi and OpenAI small lines change tier as well as version. MiniMax has no release after the original audit, and Qwen 3.7 Plus has too few complete respondents to be scored.

Source	$\Delta$ ARS	Fwd	Rev	Keying gap	Extreme	Midpoint	SD ratio
Humans ( $n{=}263$ )	0.00	0.00	0.00	2.00	0.27	0.21	1.00
gpt-5.4-mini (LLM)	+0.21	+0.35	+0.07	1.51	0.02	0.33	0.28
GPT-6 Luna (LLM)	+0.53	$-$ 0.66	+1.71	1.12	0.00	0.45	0.39
Median of 47 generative LLMs	+0.21	$-$ 1.20	+1.40	1.20	0.03	0.38	0.54
Solar Decide Flash	+3.54	+3.41	+3.68	1.05	0.20	0.35	0.35
Microsoft-Decision-1	+2.68	$-$ 0.85	+6.20	0.21	0.06	0.27	0.48

Table 10: Response style on a common raw orientation (answers to the original wording for humans and models). ARS is the balanced acquiescence index: the mean of the forward-item and reverse-item means on the 0–1 scale, minus 0.5, over the two instruments with reverse-keyed items.

\Delta

ARS is the difference to humans in units of the human ARS standard deviation. Fwd and Rev are the differences of the forward-item mean and of the reverse-item mean in the same units, so that

\Delta

ARS is their average. The keying gap is the “acquiescence” index of the original audit (forward mean minus reverse mean), which grows with content consistency. Extreme and midpoint rates are shares of answers at the scale ends and the centre. SD ratio is the median item standard deviation relative to humans.

Source	$R^{2}$ (aligned)	$\Delta R^{2}$	$R^{2}$ (raw)	Null paths
Gaussian copula	0.29	+0.01		0/7
gpt-5.4-mini (LLM)	$-$ 1.79	$-$ 2.07	$-$ 0.61	6/7
MiniMax M3 (best LLM)	$-$ 0.24	$-$ 0.51		1/7
GPT-6 Luna (LLM)	$-$ 0.26	$-$ 0.54		2/7
Median of 47 generative LLMs	0.10	$-$ 0.18		2/7
Jev 1.13	0.30	+0.02	0.03	3/7

Table 11: Downstream use (C3,

n{=}100

, rounded readout).

R^{2}

: an OLS regression of performance on attitudes and engagement is fit on the synthetic sample and tested on a fixed 30% split of the humans; the same regression fit on the other 70% of humans reaches

R^{2}{=}0.28

. The raw column repeats the test without aligning reverse-keyed items, which is how the original audit scored it. Null paths: number of the 7 mediation triples with no human indirect effect for which the synthetic sample shows a significant indirect effect above 0.10. Of the 10 triples drawn, 3 are left out because a total score in them contains another of its variables.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Original audit, v1	Revised	Cause
PSS, gpt-5.4-mini (C3)	0.714	0.743	A
Rank of gpt-5.4-mini among the models of the original audit	1 of 37	10 of 36 †	A
Best LLM of the original audit	gpt-5.4-mini, 0.714	MiniMax M3, 0.813	A
Correlation component, gpt-5.4-mini	0.522	0.602	A
PSS, Gaussian copula	0.688	0.927 (in-sample), 0.824 (out-of-sample)	B
In-sample copula minus best LLM	$-$ 0.026	+0.114	A, B

Appendix

Table 12: Numbers from the first version (v1) of the original audit [ 33 ] that change under the revised scoring, computed on the same 100 respondents. The revised version (v2) of that paper uses the same code. Causes: (A) model answers to reverse-keyed items were not reversed before scoring; (B) the distribution component was missing from the baseline totals; (C) the published index measured keying consistency rather than agreement; (D) the published test pooled all LLM cells; (E) null triples in which a total score contains another variable of the triple are left out. † Qwen 3.7 Plus has fewer than 90 complete respondents at C3 and is not ranked. ∗ Pooled over 37 models and all conditions (

n{=}24{,}267

Figure 5: PSS of the 36 C3 cells of the original audit under the published scoring (raw) and the revised scoring (aligned). The dashed line marks equal scores. gpt-5.4-mini, the published leader, is shown in red.

Model	Resolved version	Calls	$n$	Input tokens	Price	Cost ( $10^{-3}$ $)
Jev 1.13	typesafe/jev-1.13-20260917	1	100	7,800	0.042	0.33
Decider v1 27B	perplexity/pplx-decider-v1-27b-20261001	1	100	17,817	0.040	0.71
Decider v1.1 27B	perplexity/pplx-decider-v1.1-27b-20261006	1	100	17,817	0.020	0.36
Liquid d1	liquid/d1-20260930	1	100	15,953	0.040	0.64
Kev 4B	jaredpalmer/kev-4b-20260924	1	100	5,607	0.042	0.24
Tev1 4B	togethercomputer/tev1-4b-experimental-20260923	1	100	18,264	0.042	0.77

Appendix

Table 13: Decision models in the study. All were called through the OpenRouter decisions endpoint with one score question per item. Calls is the number of requests per respondent: the three Clef models and Drex accept at most 64 questions, so they receive the profile twice (64 items, then 4). Price is the listed input price in USD per million tokens; output tokens are free for this endpoint. Cost is the observed mean cost per synthetic respondent in thousandths of a dollar. The C3 grid of the scored models cost

1.27, and the incomplete Solar Decide run another

0.08.

Source	Forward items	Reverse items (aligned)	Reverse items (raw)
gpt-5.4-mini (LLM)	0.243	0.356	0.728
Jev 1.13	0.132	0.173	0.524
Decider v1 27B	0.263	0.411	0.602
Decider v1.1 27B	0.265	0.413	0.604
Liquid d1	0.168	0.288	0.561
Kev 4B	0.248	0.470	0.555

Appendix

Table 14: Mean item Jensen-Shannon divergence to the human marginal on the two instruments with reverse-keyed items (rounded readout). The raw column compares unflipped model answers with reverse-coded human answers, which is what the original audit did for every reverse-keyed item.

#	Model	Vendor	$n$	PSS	Rank	Dist.	Corr.	Rel.	Med.	Demo.	PSS	SE	Null
				Raw		Aligned
1	MiniMax M3	Open weights	100	0.679	7	0.71	0.76	0.97	1.00	0.51	0.813	0.023	1
2	MiniMax M2.7	Open weights	100	0.695	3	0.83	0.81	0.98	0.76	0.52	0.809	0.039	2
3	Kimi K2.7 Code	Open weights	100	0.669	8	0.80	0.81	0.95	0.80	0.52	0.802	0.030	1
4	Claude Opus 4.7	Anthropic	100	0.614	18	0.72	0.77	0.96	0.96	0.34	0.789	0.033	2
5	Gemini 3.1 Flash-Lite	Google	100	0.687	4	0.86	0.72	0.95	0.75	0.42	0.779	0.047	5

Appendix

Table 15: All 47 generative cells at C3 (single call, minimal reasoning, same 100 respondents), ordered by aligned PSS. ∗ Released after the original audit and run for this study (11 models). Raw PSS is the original scoring path and Raw rank its order among the models of the original audit. The aligned columns are the components, the total and its bootstrap SE. Null is the number of the 7 null mediation triples with a fabricated path (Table 11 ). Qwen 3.7 Plus (75 complete respondents) and the six cells run with high reasoning effort are not shown.

Source	PSS	D	C	M	G	PSS
	All 263	Headline 100
Second human sample (mean of 20)	0.92	0.99	0.73	0.56	0.53	0.79
Copula, out-of-sample (reported draw)	0.82	0.97	0.61	0.43	0.32	0.71
Copula, out-of-sample (mean of 20)	0.86	0.97	0.67	0.56	0.35	0.75
Copula, out-of-sample (2000 rows)	0.89	0.98	0.77	0.53	0.30	0.77
Copula, in-sample	0.93	0.99	0.83	0.67	0.49	0.83

Appendix

Table 16: PSS against two reference samples. All 263 is the reference of the main text. Headline 100 scores every source against the real answers of the 100 people whose profiles were simulated. The 163 training people of the out-of-sample copula are not part of this reference, while the in-sample copula is fit on all 263 people and still overlaps with it. The second human sample draws 100 of the 163 people outside the headline sample. The 2000-row copula repeats each headline profile 20 times and shows the copula without its own sampling noise. D, C, M and G are the distribution, correlation, mediation and demographic components against the headline 100.

Figure 6: Pearson correlation between the upper triangles of the $68\times 68$ item correlation matrices (aligned, rounded readout). The human matrix uses all 263 respondents, and the copula is the in-sample copula. Every other source uses the same 100 profiles.

Explore similar work

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
Jul 6, 2026Mantas Lukauskas, Viktorija ŠarkauskaitėPsychometric Validation Synthetic Data Evaluation
Hostinger · Kaunas University of Technology · AI Insight Lab +1
Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data
Feb 10, 2026Jason Miklian, Kristian Hoelscher, John E. KatsosHuman Behavior Simulation Synthetic Data Generation
University of Oslo · Peace Research Institute Oslo · American University of Sharjah
When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses
Jul 28, 2026Zihan Chen, Di Zhu, Lei Nico ZhengLLM Evaluation Large Language Model-Based Social Simulation
Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA

Jul 6, 2026J/K move · Enter open · S save

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Mantas Lukauskas, Viktorija Šarkauskaitė

Hostinger · Kaunas University of Technology · AI Insight Lab+1

Psychometric Validation Synthetic Data Evaluation Synthetic Data Generation Large Language Model-Based Social Simulation Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations mostly ask whether individual answers look plausible. We ask a stricter, psychometric question: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce an open Lithuanian organisational-psychology dataset (n=263 employees; three validated instruments; 68 items, 12 subscales) and condition 37 LLMs on real respondent profiles under a persona-disclosure ladder, presentation, reasoning-effort and cross-language ablations, counterfactual demographic swaps, and a memorization probe. The resulting Psychometric Similarity Score (PSS) is anchored against five statistical baselines, a held-out human ceiling, and the real answers of the same respondents. LLMs reproduce the direction of human psychometric relationships, but statistical baselines with no language model match or beat them. The best LLM (PSS 0.81) is level with a Gaussian copula fit only on other respondents (0.82) and below the real answers of the same 100 people (0.89). A simple classifier separates every LLM sample from humans (median AUC 0.99), and verbatim recall of the items is rare (at most 4.7%) and unrelated to PSS. Counterfactual swaps move answers in the direction of human group differences and by a similar amount (role 0.49 and education 0.33 human SD, vs. 0.46 and 0.44 in humans). Gender swaps move them by 0.06 SD. Tucker's phi on UWES falls inside an item-permutation null for 10 of 37 models. Synthetic respondents compress the response scale (within-respondent SD 0.79 vs. 1.19), lose predictive validity on held-out humans (mean R^2=-0.14 vs. 0.28), and produce significant indirect effects on 4 of 7 mediation paths that are null in humans. LLM samples are therefore not a drop-in replacement for human survey data.

Abstract

Explore similar work

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses