Understanding Reliability in LLM-based Human Behavior Simulation

Pei Wang; Lei Wang; Yuanzi Li; Xu Chen

cs.CLSep 16, 2026

Understanding Reliability in LLM-based Human Behavior Simulation

Authors: Pei Wang, Lei Wang, Yuanzi Li, Xu Chen

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China

Abstract

Large language models (LLMs) are increasingly used to simulate human survey responses and behavioral reactions, yet unreliable simulations can mislead social science conclusions. However, existing evaluations focus on end-to-end scores, leaving it unclear how different aspects of the simulation process interact to determine reliability. We propose ReliMap, which decomposes LLM-based human behavior simulation into three structured layers and evaluates reliability at both the individual level (R1) and population level (R2) across three configuration dimensions: model capacity, profile completeness, and population coverage. Through experiments across four simulation tasks and eleven LLMs, we find that all models exhibit substantial distributional bias without profile conditioning. Profile conditioning reduces this bias with diminishing returns. Larger models benefit more, and attribute informativeness matters more than quantity. Critically, R1 gains do not reliably transfer to R2--individual and population-level reliability can move in opposite directions. At the population layer, increasing coverage reduces variance but not systematic bias, with R2 stabilizing at around 50-100 individuals. These findings highlight that reliable simulation cannot be achieved by optimizing any single layer in isolation, but requires coordinated improvement across all three.

Figures & tables

Figure 1: LLM-based Human Behavior Simulation As a Hierarchical Structure.

Figure 2: ReliMap Overview: Three-Layer Structure with Configurable Dimensions and Two-Level Reliability Evaluation.

Figure 3: Effect of profile completeness on population simulation reliability.

Proprietary
Model	Party	Immig.	Relig.	Media
GPT-4.1	$0.670$	$0.368$	$0.542$	$0.747$
Gemini-2.5-Pro	$0.589$	$0.297$	$0.549$	$0.527$
Open-source: DeepSeek
DeepSeek-V3.2-Exp	$0.483$	$0.128$	$0.342$	$0.613$
DeepSeek-R1	$0.664$	$0.579$	$0.345$	0.480

Table 1: TVD

\downarrow

between LLM-induced base response distributions and human survey distributions (Layer I analysis; motivates the profile-conditioning study of Findings 1–5).

Panel of Figure 4 — Figure 4: Changes in individual-level simulation accuracy as profile completeness $c$ increases across models. The x-axis $c$ denotes the profile attribute coverage rate, i.e., the percentage of attributes provided to the model out of all available attributes in the dataset. At each level of c, attributes are sampled randomly.

	Party	Immigration	Religion
# attributes	100	200	76
ASPG median	0.017	0.010	0.014
Gini	0.511	0.503	0.443
Top 20% info	53.5%	53.3%	49.7%
Bot 50% info	15.0%	16.9%	21.7%
Info@ $c=50\%$ (rand.)	45%	44%	47%

Table 2: Attribute informativeness statistics supporting Finding 3. Gini measures ASPG concentration across attributes (0 = uniform, 1 = maximally concentrated). Top 20% info and Bot 50% info denote the cumulative predictive information contributed by the top 20% and bottom 50% of attributes, respectively. Info@c=50% reports cumulative information captured at half profile completeness under random (rand.) and informativeness-weighted (wtd.) sampling.

Model		Party	Immigration	Religion
GPT4.1	$r_{Q}$	0.9177	0.8368	0.9604
	$r_{I}$	0.9928	0.9662	0.9607
	$\Delta$	+0.0751	+0.1294	+0.0003
Gemini-2.5-Pro	$r_{Q}$	0.7835	-0.0386	-0.5336
	$r_{I}$	0.9624	0.0539	-0.7305
	$\Delta$	+0.1789	+0.0925	-0.1969

Table 3: Pearson correlations between accuracy and two profile completeness metrics, supporting Finding 4.

r_{Q}

: quantity-based completeness;

r_{I}

: information-based completeness.

\Delta=r_{I}-r_{Q}

Figure 5: Population simulation reliability versus population coverage. TVD between the aggregated simulated distribution and the target population distribution as a function of the number of simulated individuals

N

. The dashed curve shows the GT sampling baseline: human ground-truth responses aggregated at varying sampling sizes, serving as a reference for the sampling variance of real human responses. The

c

is fixed at 100%.

Party
Dataset	GPT-4.1		DeepSeek-V3.2-Exp		Qwen3-235B-A22B
Dataset	$\Delta$ ACC	$\Delta$ TVD	$\Delta$ ACC	$\Delta$ TVD	$\Delta$ ACC	$\Delta$ TVD
20	+26.50	-55.52	+44.45	-43.66	+144.57	+5.91
40	+12.08	-13.78	+7.68	-16.99	+18.68	-23.62
60	+8.94	-31.88	+9.17	+6.33	+5.39	-39.73
80	-1.19	+14.64	-3.00	-1.33	+1.51	+42.54

Table 4: Comparison of model performance across four datasets with varying profile completeness

c

\Delta

ACC and

\Delta

TVD denote the relative percentage change with respect to the previous profile-completeness level (i.e., the preceding row in each block). Blue values mark steps where ACC increases while TVD also increases (R1 improves, R2 worsens); Orange values mark steps where both decrease (R1 worsens, R2 improves). In both cases R1 and R2 move in opposite directions, illustrating the decoupling in Finding 5.

Case 1: Immigration ( $c{=}12.5$ % ${\to}25$ %)
Option	GT%	Pred%	Bias	Pred%	Bias
Very bad	7.49	5.81	-1.68	5.48	-2.01
Quite bad	14.80	30.47	+15.67	23.49	+8.70
Neither	35.71	21.97	-13.74	23.32	-12.39
Quite good	31.17	35.17	+4.00	44.23	+13.06
Very good	10.83	6.57	-4.25	3.47	-7.35

Table 5: Option-level prediction distributions for two decoupling cases (Qwen3-235B-A22B), illustrating Finding 5. GT% denotes the occurrence rate of each option in the ground-truth human data. Bias values are colored by sign: red = over-prediction, blue = under-prediction. Case 1 (Immigration):

c{=}12.5

{\to}25

%, ACC

\uparrow

, TVD

\uparrow

. Case 2 (Media):

c{=}66.7

{\to}83.3

%, ACC

\downarrow

, TVD

\downarrow

Findings

Simulation Layers

Configurations

Evaluation

Work

III

Cap.

Com.

Cov.

Hu et al. (2025)

–

\times

✓

\times

\times

\times

✓

Wang et al. (2025c)

–

\times

✓

\times

✓

\times

\times

✓

\times

Dominguez-Olmedo et al. (2024)

–

✓

\times

✓

\times

\times

\times

✓

Meister et al. (2024)

–

✓

\times

\times

\times

✓

Geng et al. (2024)

–

\times

✓

\times

\times

✓

Table 6: Comparison with existing evaluation studies. (a) Coverage of our six findings F1–F6 (§ 3 ): “R” = related prior evidence, “–” = not discussed. (b) Coverage of ReliMap’s design space across simulation layers, configuration dimensions (Cap.: model capacity, Com.: profile completeness, Cov.: population coverage), and evaluation levels (R1/R2).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Party	Immigration	Religious	Media
profile_attr_num	100	200	80	30
population_num	1,000	9,994	500	150
question	Which political party do you feel closest to? (Germany)	How would you evaluate the impact of immigrants on your country’s development?	Do churches and religious organizations have too much or too little power?	Analyze this user’s attitude toward the ‘Luckin Coffee stock price crash’ incident.

Appendix

Table 7: Task settings and question texts.

Party (options)	Immigration (options)	Religious (options)	Media (options)
1. CDU/CSU 2. SPD 3. The Left (Die Linke) 4. Alliance 90/The Greens 5. FDP 6. AFD 7. Free Voters 8. dieBasis 9. Die PARTEI	1. Very bad 2. Quite bad 3. Neither good nor bad 4. Quite good 5. Very good	1. Far too much power 2. Too much power 3. About the right amount 4. Too little power 5. Far too little power	1. Loyal Advocate 2. Opportunist 3. Moral Outrage 4. Rational Evaluator 5. Bystander Entertainer

Appendix

Table 8: Answer options for each task.

Panel of Figure 6 — Figure 6: Prompt template for Party, Immigration, Religion.

Panel of Figure 7 — Figure 7: Prompt template for Media Task (translated from Chinese).

Component	Content
$s$ (scenario)	Question : Which political party do you feel closest to? (Germany) Options : 1. CDU/CSU, 2. SPD, 3. The Left (Die Linke), 4. Alliance 90/The Greens, 5. FDP, 6. AFD, 7. Free Voters, 8. dieBasis, 9. Die PARTEI
$u$ (individual)	Respondent_1
$x$ (profile)	{“What the country needs most is loyalty towards its leaders.”: “Disagree strongly”, “How worried are you about climate change?”: “Not at all worried”, “Have you felt unfairly treated by the police because you are a man/woman?”: “No”, …}
Prompt	### Instruction: You are participating in the International Social Survey Programme. Assume the role of a real individual with the following personal information. Fully immerse yourself in this persona and answer the question truthfully, based solely on the provided personal information and your previous answers to prior questions. ### Personal Information and Previous Answers: {content from $x$ } ### Question: {content from $s$ } ### Options: {content from $s$ }
Response	{“reason”: “Based on my previous answers, I am strongly in favor of gender equality in business leadership […] SPD is also traditionally supportive of worker rights and moderate reforms, which matches my practical, balanced approach. Therefore, I feel closest to the SPD.”, “option”: “2”}
Ground Truth	2

Appendix

Table 9: A concrete example of the simulation pipeline for the Party task, illustrating how

s

u

, and

x

are instantiated in practice.

Figure 8: Distribution of ASPG values across attributes for each task. Most attributes carry low predictive information individually, while a small minority contributes disproportionately large predictive gain.

Panel of Figure 9 — Figure 9: TVD Scaling with Population Coverage $N$ under Varying Profile Completeness $c$ .

Model	$c$	TVD	KL	JSD	EMD
GPT-4.1	0	0.6704	10.2946	0.2970	1.5623
	20	0.2982	1.1940	0.0877	0.5902
	40	0.2571	0.6908	0.0641	0.5909
	60	0.1751	0.4742	0.0398	0.4494
	80	0.2008	0.5499	0.0446	0.3849
	100	0.1551	0.6279	0.0441	0.3967

Appendix

Table 10: R2 metrics under varying profile completeness

c

on the Party task. All four metrics exhibit the same qualitative trends as TVD.

Model	$N$	TVD	KL	JSD	EMD
GPT-4.1	10	0.2258	3.5638	0.0701	0.4924
	50	0.1909	3.5109	0.0618	0.4939
	100	0.1739	2.5949	0.0510	0.4184
	150	0.1703	1.9460	0.0518	0.3688
	200	0.1689	1.7412	0.0514	0.3752
	250	0.1611	0.6213	0.0441	0.4321

Appendix

Table 11: R2 metrics under varying population coverage

N

on the Party task. All four metrics decrease as

N

increases and stabilize at larger

N

, consistent with the TVD-based results.

			$c$ (%)
Model	Prompt	Temperature	0	20	40	60	80	100
Qwen3-8B	Original	greedy	26.53	26.72	30.76	31.96	31.72	28.78
	Prompt_v1	greedy	28.40	31.00	32.57	34.16	32.08	33.26
	Prompt_v2	greedy	26.40	28.07	29.81	31.29	30.81	31.95
	Original	$t=0.6$	26.52	27.22	31.05	32.88	31.87	32.53
Qwen3-32B	Original	greedy	25.76	30.32	33.32	37.14	32.26	32.90

Appendix

Table 12: Individual-level simulation accuracy (ACC, %) under different prompt variants and decoding temperatures on the Party task, supporting the robustness of Findings 1 and 2. Original refers to the default prompt used in the main experiments.

Figure 10: Individual-level accuracy (R1) as a function of profile completeness

c

across all four tasks, with

\pm

1 standard deviation shaded regions computed over 5 fixed random seeds (2025–2029). Variance stems from attribute subset sampling. Despite variability in some tasks, the overall trends remain consistent across models.

Human Behavior Simulation

Human behavior simulation models how individuals and groups decide, interact, move, and adapt in social, economic, educational, and physical settings.

See all 239 papers

LLM-based Human Simulations Have Not Yet Been Reliable
Jan 15, 2025Qian Wang, Jiaying Wu, Zichen Jiang +6Human Behavior Simulation
National University of Singapore · The Hong Kong University of Science and Technology · University of Chinese Academy of Sciences
Will Scaling Improve Social Simulation with LLMs?
Jul 2, 2026Caleb Ziems, William Held, Su Doga Karaca +3Opinion Dynamics Language Model Scaling Laws
Stanford University · Open Athena
Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management
Sep 30, 2025Runze Zhang, Xiaowei Zhang, Mingyang ZhaoLLM Evaluation Human Behavior Simulation
Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong

Jan 15, 2025J/K move · Enter open · S save

LLM-based Human Simulations Have Not Yet Been Reliable

Qian Wang, Jiaying Wu, Zichen Jiang, Zhenheng Tang+5

National University of Singapore · The Hong Kong University of Science and Technology · University of Chinese Academy of Sciences

Human Behavior Simulation

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly employed for simulating human behaviors across diverse domains. However, our position is that current LLM-based human simulations remain insufficiently reliable, as evidenced by significant discrepancies between their outcomes and authentic human actions. Our investigation begins with a systematic review of LLM-based human simulations in social, economic, policy, and psychological contexts, identifying their common frameworks, recent advances, and persistent limitations. This review reveals that such discrepancies primarily stem from inherent limitations of LLMs and flaws in simulation design, both of which are examined in detail. Building on these insights, we propose a systematic solution framework that emphasizes enriching data foundations, advancing LLM capabilities, and ensuring robust simulation design to enhance reliability. Finally, we introduce a structured algorithm that operationalizes the proposed framework, aiming to guide credible and human-aligned LLM-based simulations. To facilitate further research, we provide a curated list of related literature and resources at https://github.com/Persdre/awesome-llm-human-simulation.

Abstract

Human Behavior Simulation

Explore similar work

LLM-based Human Simulations Have Not Yet Been Reliable

Will Scaling Improve Social Simulation with LLMs?

Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management