Behavioral History Outperforms Descriptions of the Person for LLM Synthetic Personas

Khashayar Pourtaheri; Ahmad Zareei

cs.AIOct 2, 2026

Behavioral History Outperforms Descriptions of the Person for LLM Synthetic Personas

Authors: Khashayar Pourtaheri, Ahmad Zareei

Organizations: University of Washington · Berkeley

Abstract

Large language models (LLMs) are increasingly used as synthetic personas representing survey respondents. Their validity as substitutes for particular respondents depends on whether they reproduce individuals' decisions. We examine what information helps synthetic respondents predict each individual's later choices, using five conditions that add progressively richer information: no personal information, demographics, personality traits, cognitive scores, and finally the respondent's earlier survey choices as behavioral history. We use a two-wave panel of 845 US adults who completed measures of 14 behavioral biases (spanning risk, time preferences, overconfidence, and reasoning), so each respondent's earlier answers provide a human test-retest benchmark; in the behavioral-history condition, all items that score the target bias are withheld. At the population level, the average number of biases per respondent in every condition is close to the human average (7.1-8.1 biases, against 7.1 for humans). This aggregate similarity masks differences in variance: persona descriptions recover only 53-67% of human between-person variation, whereas adding behavioral history restores it to approximately the human level. At the individual level, description-based personas achieve only 7-12% of the informedness observed in human test-retest responses, while adding behavioral history raises this to 28%. The condition including behavioral history has the highest estimated informedness in all 17 demographic groups, whereas description-based conditions provide little or no information for some groups. Synthetic responses also exhibit stronger education- and income-related differences than human responses. For LLM synthetic personas, a respondent's past answers add more to individual-level prediction than a description of who they are.

Figures & tables

	Condition	Prompt information
A1	No identity	the survey question alone
A2	Demographics	adds 16 wave-2 demographic fields
A3	Personality	adds the BFI-10 Big Five, with scale notes
A4	Cognitive	adds four cognitive scores, each with its scale
A5	Behavioral history	adds the person’s wave-1 responses, excluding all items used to score the target bias

Table 1: Five nested prompt conditions. Each condition adds one information block to the previous condition. A5 includes respondent-specific behavioral evidence, but the target bias block is withheld.

Figure 1: How many of the 14 biases each respondent shows, and how the distribution differs by condition. Panel (a) shows the bias-count distribution, and panel (b) reports each condition’s standard deviation and mean; colors are the same in both panels.

n=696

respondents with all 14 biases available in every condition.

Condition	Bias	Person	Residual	Person/HUMAN	Bias/Person
HUMAN (wave 2)	0.191	0.074	0.735	1.00	2.6
A1 No identity	0.834	0.011	0.155	0.15	72.8
A2 Demographics	0.605	0.049	0.345	0.67	12.2
A3 Personality	0.567	0.046	0.386	0.62	12.3
A4 Cognitive	0.614	0.039	0.347	0.53	15.7
A5 Behavioral history	0.395	0.078	0.527	1.05	5.1

Table 2: Variance decomposition of the person-by-bias matrix. The complete

n\times B

binary matrix is separated into between-bias, between-person, and residual shares of total variance (here

n=518

respondents and

B=14

biases).

Figure 2: Prevalence of each bias across the information conditions. Each panel shows one bias family: HUMAN (gray column), then A1–A5. Dashed lines mark each bias’s human wave-2 prevalence, so the vertical distance from a dashed line is the aggregate error. Shaded bands mark prevalence near zero or one. Each bias uses its available cases (

n=629

–845). EGB is exponential-growth bias and LLN the law of large numbers.

	HUMAN retest	A1 No identity	A2 Demographics	A3 Personality	A4 Cognitive	A5 Behavioral history
Mean accuracy	0.709	0.550	0.558	0.564	0.568	0.569
Mean informedness $J$	0.283	0.001	0.020	0.024	0.033	0.078
Share of HUMAN	1.00	0.00	0.07	0.08	0.12	0.28
Mean $\|\mathrm{TPR}-\mathrm{TNR}\|$	0.26	0.90	0.70	0.71	0.72	0.53
Ratio to HUMAN	1.00	3.42	2.65	2.69	2.74	2.00

Table 3: Behavioral history improves individual-level prediction. Each synthetic verdict is compared with the same respondent’s observed wave-2 verdict, and results are averaged across the 14 biases on each bias’s common retest sample (

n=526

–843). HUMAN compares the same person’s wave-1 and wave-2 verdicts.

Attribute	Group	$n$	HUMAN retest	A1 No identity	A2 Demographics	A3 Personality	A4 Cognitive	A5 Behavioral history	A5 $-$ A4
income	low	308	0.269	0.010	0.018	0.020	0.025	0.071	0.047
	mid	308	0.285	$-0.012$	$-0.002$	0.003	0.037	0.089	0.052
	high	192	0.263	0.010	0.019	0.027	0.027	0.030	0.003
education	high school or less	129	0.255	0.001	0.001	0.020	0.047	0.057	0.010
	some college	289	0.283	$-0.001$	0.002	0.005	0.022	0.059	0.037
	bachelor’s	223	0.281	$-0.005$	0.006	0.023	0.039	0.087	0.048

Table 4: Mean informedness

J

by demographic group. We compute

J=\mathrm{TPR}+\mathrm{TNR}-1

within each group-by-bias cell and then average across estimable biases. HUMAN is the group’s wave-1–wave-2 benchmark, and

n

is the median valid sample across included biases. Other is the source panel’s own category, not a merger of small groups. The last column is A5 minus A4; bold values exceed the mean gain across groups (0.039).

Figure 3: Behavioral history yields the highest informedness across demographic groups. Each thin line shows mean informedness

J

for one of the 17 demographic groups, and the bold line shows the average across groups. HUMAN retest is each group’s wave-1–wave-2 benchmark; the dashed line marks

J=0

Condition	Best in	%	Worst in	%
A1 No identity	22	10.6%	90	43.5%
A2 Demographics	19	9.2%	48	23.2%
A3 Personality	19	9.2%	25	12.1%
A4 Cognitive	43	20.8%	24	11.6%
A5 Behavioral history	104	50.2%	20	9.7%

Table 5: Best- and worst-performing conditions by bias and demographic subgroup. For each of the 207 estimable bias-by-subgroup cells, A1–A5 are ranked by informedness

J

. “Best in” and “worst in” report how often each condition has the highest or lowest point estimate.

Figure 4: Persona conditions reshape demographic disparities relative to humans. Each point shows the largest gap across an attribute’s groups in prediction accuracy (a) or attributed bias prevalence (b); the black line marks the human gap. Red shading marks gaps larger than the human gap and green shading smaller ones, and larger markers lie farther from the human gap.

Figure 5: Synthetic respondents misrepresent how skill varies with education. (a) Share answering the compound-interest item wrongly. (b) Average number of the three numeracy questions answered correctly (0–3).

Construct	HUMAN (wave 2)	A2 Demographics	A3 Personality	A4 Cognitive	A5 Behavioral history
Age	$0.09$	$0.21$	$0.34$	$1.01$	$0.12$
Gender	$\textbf{1.56}^{*}$	$\textbf{0.87}^{*}$	$0.45$	$\textbf{1.65}^{*}$	$\textbf{1.71}^{*}$
Ethnicity & Hispanic origin	$0.84$	$0.59$	$0.47$	$1.10$	$1.33$
Education	$1.64$	$\textbf{22.69}^{***}$	$\textbf{16.91}^{***}$	$\textbf{12.53}^{***}$	$\textbf{13.80}^{***}$
Income	$1.29$	$\textbf{11.31}^{***}$	$\textbf{8.89}^{***}$	$\textbf{7.25}^{**}$	$\textbf{7.26}^{**}$
Big Five (5 traits)	$\textbf{1.93}^{*}$	$0.80$	$\textbf{3.81}^{***}$	$\textbf{3.34}^{***}$	$1.44$

Table 6: Decomposition of explained variation in human and synthetic bias counts. Cells are Shapley contributions to

R^{2}

(percentage points) from regressing each column’s bias count on respondent attributes (757 respondents; Shapley, 1953 ). Blocks absent from a condition’s prompt (the Big Five and cognitive scores for A2; the cognitive scores for A3) measure correlation with the synthetic answers, not use of that information. Stars mark jointly significant blocks (Wald test with Benjamini–Hochberg correction, Benjamini and Hochberg, 1995 ;

{}^{*}q<.05

{}^{**}q<.01

{}^{***}q<.001

); every condition’s education-and-income share exceeds the human share (

q<.001

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Bias	Instrument	Biased if	Rate	$n$	$n_{\text{pair}}$
Present bias
Present bias — money	24 convex time-budget allocations	less patient when the earlier date is today	0.248	819	782
Present bias — consumption	two snack choices, now vs later	unhealthy now and healthy later	0.168	829	821
Inconsistent choice
Narrow bracketing	Rabin–Weizsäcker pairs, shown jointly	dominated pair, or not the both-safe pair	0.624	837	820
Risk

Appendix

Table 7: The 14 biases, scoring rules, and coverage. Rate is the human wave-2 prevalence. EGB is exponential-growth bias and LLN the law of large numbers.

Condition	Pairs	$r$	$p$	Pairs	$r$	$p$
	All pairs			Without the confounded pair
A1	55	0.623	0.004	54	0.209	0.138
A2	78	0.176	0.288	77	$-0.129$	0.462
A3	91	0.363	0.016	90	$-0.033$	0.820
A4	91	0.278	0.046	90	0.010	0.950
A5	91	0.429	0.006	90	0.244	0.118

Appendix

Table 8: Mantel test of the correlation structure among biases.

r

is the correlation between the human and synthetic between-bias Spearman correlation matrices, computed on the source study’s percentile (tile) bias scores; for present bias on consumption, which has no graded score, the tile of its binary verdict is used. The two-sided

p

comes from 5,000 permutations of the bias labels. Biases whose synthetic scores do not vary have undefined correlations, so A1 and A2 use fewer pairs.

Bias	HUMAN	A1	A2	A3	A4	A5
Present bias — money	0.248	0.719	0.663	0.662	0.621	0.587
Present bias — consumption	0.168	1.000	0.900	0.802	0.853	0.557
Narrow bracketing	0.624	1.000	0.980	0.996	0.927	0.885
Loss aversion	0.602	0.006	0.464	0.625	0.581	0.447
Certainty preference	0.762	0.878	0.903	0.812	0.712	0.254
Ambiguity aversion	0.764	1.000	0.870	0.827	0.859	0.798

Appendix

Table 9: Prevalence of each bias by condition (values behind Figure 2 ). Each bias uses its available cases (

n=629

–845). Bold marks the condition closest to HUMAN.

(a) Accuracy
Attribute	HUMAN	A1	A2	A3	A4	A5
Age	0.027	0.045	0.040	0.026	0.014	0.026
Gender	0.012	0.013	0.002	0.008	0.007	0.003
Ethnicity & Hispanic origin	0.026	0.034	0.057	0.047	0.059	0.043
Income	0.010	0.059	0.067	0.076	0.079	0.050
Education	0.021	0.052	0.063	0.054	0.049	0.053

Appendix

Table 10: Largest gap between groups (values behind Figure 4 ). Each cell is the highest minus the lowest group value within an attribute: (a) accuracy, on respondents with at least 8 valid biases (

n=818

); (b) attributed bias prevalence, averaged over the 14 biases, on all 845 respondents. Ethnicity groups are defined in Appendix E .

(a) Compound-interest item wrong
Education	$n$	HUMAN	A1	A2	A3	A4	A5
High school or less	141	0.647	0.028	0.872	0.596	0.397	0.894
Some college	302	0.473	0.017	0.666	0.434	0.238	0.775
Bachelor’s	230	0.340	0.017	0.209	0.126	0.078	0.426
Postgraduate	172	0.262	0.029	0.023	0.006	0.029	0.285
(b) Numeracy score

Appendix

Table 11: Compound-interest errors and numeracy by education (values behind Figure 5 ). (a) Share answering the compound-interest item wrongly. (b) Mean number of the three numeracy questions answered correctly (0–3).

n

is the number of respondents in each group.

	Classifier AUC		Residual correlation
Condition	Primary	Complete	Primary	Complete
HUMAN	0.476 [0.451, 0.500]	0.510 [0.476, 0.544]	0.287 [0.266, 0.308]	0.297 [0.267, 0.325]
A1	1.000 [1.000, 1.000]	1.000 [1.000, 1.000]	$-0.004$ [ $-0.026$ , 0.017]	—
A2	0.996 [0.993, 0.998]	—	0.008 [ $-0.013$ , 0.029]	—
A3	0.994 [0.991, 0.996]	—	0.022 [0.001, 0.044]	—
A4	0.994 [0.991, 0.996]	—	0.023 [0.002, 0.043]	0.040 [0.012, 0.067]

Appendix

Table 12: Profile-level checks on the primary and complete-profile samples. Classifier AUC measures how easily a respondent-grouped classifier separates synthetic from human bias profiles; 0.5 is indistinguishable and closer to 0.5 is better. Residual correlation measures agreement with human wave 2 after removing person and bias means; higher is better. HUMAN compares wave 1 with wave 2. Brackets are 95% respondent-bootstrap intervals. The primary sample has

n=829

and the complete-profile sample

n=389

. Dashes mark values not reported.

Condition	All-pool AUC [95% interval]	Matched AUC [95% interval]
HUMAN	0.718 [0.699, 0.737]	0.706 [0.685, 0.726]
A1	0.514 [0.495, 0.534]	0.513 [0.492, 0.535]
A2	0.518 [0.498, 0.538]	0.513 [0.491, 0.534]
A3	0.524 [0.505, 0.543]	0.520 [0.499, 0.542]
A4	0.533 [0.514, 0.553]	0.530 [0.509, 0.551]
A5	0.552 [0.534, 0.572]	0.538 [0.514, 0.560]

Appendix

Table 13: Identification on the primary sample. Identification AUC is the probability that the correct person’s profile is closer than an alternative, with half credit for tied distances. All-pool comparisons use

n=828

profiles; demographic-matched comparisons use

n=788

. Brackets are 95% respondent-bootstrap intervals. These are the estimates discussed in § 3.2 ; the complete-profile sensitivity estimate is reported in the text of this appendix.

Bias	HUMAN	A1	A2	A3	A4	A5
Present bias — money	0.485	0.507	0.462	0.523	0.479	0.512
Loss aversion	0.694	0.502	0.459	0.505	0.464	0.548
Narrow bracketing	0.659	0.515	0.511	0.532	0.558	0.545
Certainty preference	0.564	0.505	0.463	0.434	0.541	0.521
Ambiguity aversion	0.628	0.534	0.475	0.520	0.462	0.537
Overconfidence — level	0.663	0.503	0.554	0.545	0.506	0.602

Appendix

Table 14: Agreement on the continuous scores underlying 13 verdicts. AUC measures how well each condition’s continuous score discriminates the respondent’s wave-2 binary verdict; HUMAN uses the person’s wave-1 score. A5 reaches 0.33 of the human benchmark. Per-bias common samples range from

n=526

n=843

Measure	A5 compared with	$n$	Better	Same	Worse	Mean gain [95% interval]
Binary accuracy	A1	837	46.2	21.0	32.7	2.48 [1.45, 3.49]
Binary accuracy	A4	837	41.3	20.4	38.2	0.69 [ $-0.32$ , 1.62]
Graded error	A1	821	50.2	0.0	49.8	0.0036 [ $-0.0010$ , 0.0082]
Graded error	A4	821	59.2	0.0	40.8	0.0153 [0.0108, 0.0199]

Appendix

Table 15: Respondent-level changes under the behavioral-history condition (A5). Better, Same, and Worse are the percentages of respondents for whom A5 has lower, equal, or higher prediction error than the comparison condition. Binary gains are percentage-point changes in accuracy across 14 biases; graded gains are reductions in normalized absolute error across 12 biases. Intervals are 95% respondent-bootstrap intervals from 2,000 draws.

Attribute	Group	$n$	HUMAN	A1	A2	A3	A4	A5
Income	low income	328	0.555 [0.503, 0.607]	0.009	0.826	0.869	0.820	0.509
	middle income	318	0.604 [0.547, 0.657]	0.006	0.330	0.582	0.516	0.450
	high income	198	0.677 [0.611, 0.742]	0.000	0.081	0.293	0.293	0.343
Education	high school or less	141	0.518 [0.440, 0.596]	0.000	0.766	0.872	0.865	0.560
	some college	302	0.596 [0.540, 0.649]	0.010	0.609	0.732	0.639	0.460
	bachelor’s	230	0.643 [0.578, 0.704]	0.004	0.309	0.522	0.483	0.448

Appendix

Table 16: Loss-aversion prevalence by income and education. HUMAN is the wave-2 rate with a 95% bootstrap interval. Income and education groups are those used in Figure 4 (Appendix E ). Overlapping human intervals mean that the human gradient should be treated as descriptive.

Large Language Model-Based Social Simulation

Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.

See all 257 papers

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
Jul 6, 2026Mantas Lukauskas, Viktorija ŠarkauskaitėPsychometric Validation Synthetic Data Evaluation
Hostinger · Kaunas University of Technology · AI Insight Lab +1
Evaluating LLMs as Human Surrogates in Controlled Experiments
Mar 8, 2026Adnan Hoq, Tim WeningerHuman Behavior Simulation Large Language Model-Based Social Simulation
University of Notre Dame Notre Dame, Indiana, USA
How Well Do Large Language Models Capture Human Personality?
May 12, 2026Aanisha Bhattacharyya, Yaman Kumar Singla, Rajiv Ratn Shah +2Persona-Conditioned Generation Representation Collapse
Adobe Media and Data Science Research (MDSR) · IIIT-Delhi · SUNY at Buffalo

Jul 6, 2026J/K move · Enter open · S save

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Mantas Lukauskas, Viktorija Šarkauskaitė

Hostinger · Kaunas University of Technology · AI Insight Lab+1

Psychometric Validation Synthetic Data Evaluation Synthetic Data Generation Large Language Model-Based Social Simulation Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations mostly ask whether individual answers look plausible. We ask a stricter, psychometric question: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce an open Lithuanian organisational-psychology dataset (n=263 employees; three validated instruments; 68 items, 12 subscales) and condition 37 LLMs on real respondent profiles under a persona-disclosure ladder, presentation, reasoning-effort and cross-language ablations, counterfactual demographic swaps, and a memorization probe. The resulting Psychometric Similarity Score (PSS) is anchored against five statistical baselines, a held-out human ceiling, and the real answers of the same respondents. LLMs reproduce the direction of human psychometric relationships, but statistical baselines with no language model match or beat them. The best LLM (PSS 0.81) is level with a Gaussian copula fit only on other respondents (0.82) and below the real answers of the same 100 people (0.89). A simple classifier separates every LLM sample from humans (median AUC 0.99), and verbatim recall of the items is rare (at most 4.7%) and unrelated to PSS. Counterfactual swaps move answers in the direction of human group differences and by a similar amount (role 0.49 and education 0.33 human SD, vs. 0.46 and 0.44 in humans). Gender swaps move them by 0.06 SD. Tucker's phi on UWES falls inside an item-permutation null for 10 of 37 models. Synthetic respondents compress the response scale (within-respondent SD 0.79 vs. 1.19), lose predictive validity on held-out humans (mean R^2=-0.14 vs. 0.28), and produce significant indirect effects on 4 of 7 mediation paths that are null in humans. LLM samples are therefore not a drop-in replacement for human survey data.

Abstract

Large Language Model-Based Social Simulation

Explore similar work

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Evaluating LLMs as Human Surrogates in Controlled Experiments

How Well Do Large Language Models Capture Human Personality?