Rethinking the Evaluation and Optimization of LLM-Based Social Simulation

Pei Wang; Xu Chen; Ji-Rong Wen

cs.AIAug 20, 2026

Rethinking the Evaluation and Optimization of LLM-Based Social Simulation

Authors: Pei Wang, Xu Chen, Ji-Rong Wen

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China

Abstract

LLM-based social simulation is a promising complement to traditional methods such as surveys and behavioral experiments. A core question is how to evaluate the fidelity of LLM-simulated human behavior and optimize LLMs toward it. Prevailing practice evaluates by accuracy, checking whether the model selects the single response observed from a human, and trains the LLM to reproduce this hard label. However, human behavior is inherently subjective: the same person in the same situation may reasonably act differently, so an observed response is only one draw from an underlying response distribution, rendering accuracy-based evaluation unreliable and hard-label training misleading. To address these problems, we first introduce the subjectivity coefficient, an entropy-based quantity distinguishing objective tasks such as coding from subjective ones such as social simulation, and use it to systematically analyze how accuracy-based evaluation and hard-label training fail as subjectivity grows. Based on the subjectivity coefficient, we propose Subjectivity-Adaptive soft-Label Training (SALT): it pools observed outputs from semantically nearby inputs into soft distributional labels, with an aggregation radius adapted to the estimated subjectivity of each input; in the near-objective limit the neighborhood shrinks, so SALT naturally falls back to standard single-label training. Moreover, since existing datasets record only single observed responses and cannot support distributional evaluation, we construct SUBJSIM, a benchmark of 19,300 contexts covering 193 annotators and 100 subjective questions. Since real-world data typically provide only a single observation per input, our experiments train models from single observed outputs while evaluating them against the full response distributions, verifying feasibility in realistic settings. Results on SUBJSIM demonstrate the advantages of our method.

Figures & tables

	Economy				Political				Technology
Method	KL $\downarrow$	JSD $\downarrow$	TVD $\downarrow$	MMD $\downarrow$	KL $\downarrow$	JSD $\downarrow$	TVD $\downarrow$	MMD $\downarrow$	KL $\downarrow$	JSD $\downarrow$	TVD $\downarrow$	MMD $\downarrow$
Pretrained	4.6285	0.2851	0.5735	0.6346	4.1453	0.3044	0.6206	0.6586	4.2015	0.2547	0.5335	0.5502
SFT	1.3543	0.1542	0.3951	0.3283	1.2865	0.1568	0.4145	0.3167	1.3409	0.1537	0.3956	0.3215
DPO	6.9962	0.2752	0.5573	0.6049	7.2926	0.2834	0.5816	0.5861	6.9174	0.2580	0.5305	0.5450
PPO	4.0161	0.2672	0.5495	0.5863	4.4698	0.2726	0.5724	0.5684	4.2959	0.2502	0.5251	0.5310
DSA	3.2032	0.2332	0.4920	0.4337	3.4679	0.2442	0.5161	0.4436	3.4744	0.2337	0.4888	0.4275

Table 1: Main results on SubjSim. All metrics are divergences (lower is better). Best values are in bold on a blue background; second-best values are underlined on a sand background.

Figure 1: Subjectivity-stratified comparison between SFT and SALT on SubjSim , over low-, mid-, and high-subjectivity questions. All metrics are divergences (lower is better), and percentages give SALT’s relative change over SFT.

Figure 2: Ablation over neighbor selection. Top-

N

varies

N

(teal, bottom axis); fixed

K_{\text{eff}}

substitutes a constant for

\hat{K}_{\text{eff}}(x)

(navy, top axis). Both share the

y

axis and reference lines; SFT lies far above the plotted range and is reported as text.

Figure 3: Sensitivity of SALT to the radius scale

C

(divergences, lower is better).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Concept	Notation	SubjSim realization	Protocol role
Decision context	$x=(u,s)$	Annotator persona paired with a survey question; each persona has 30 demographic attributes.	Input to training and evaluation.
Action space	$\mathcal{A}(x)$	Candidate response options for the survey question.	Common support for distributional evaluation within each question.
Latent response-propensity target	$\phi^{*}(\cdot\mid x)$	Probability-ball empirical distribution $\hat{\phi}^{\mathrm{ball}}_{u}(\cdot\mid x)$ , used as an elicited distributional proxy rather than a repeated-choice frequency.	Hidden during training; used only as the evaluation target.
Hard observation	$a(x)$	Modal response under $\hat{\phi}^{\mathrm{ball}}_{u}(\cdot\mid x)$ .	Hard-label proxy for SFT, DPO, PPO, and SALT targets.
Action-space group	$\mathcal{G}_{g}$	Persona-question contexts from the same survey question.	Restricts aggregation to comparable candidate options.
SALT neighborhood	$\mathcal{N}(x)$	Nearest contexts within the adaptive embedding radius, using Qwen3-embedding-8b embeddings.	Defines which hard observations SALT pools.

Appendix

Table 2: Mapping between theoretical concepts, notation, and their SubjSim implementation.

		Policy models					PPO aux.
Group	Parameter	SALT	SFT	DPO	PPO	DSA	RM
General	Epochs	4	5	4	4	4	4
	Global batch size	128	128	96	128	128	128
	Learning rate	5e-6	5e-6	5e-7	1e-6	5e-6	1e-5
	LR schedule	Cosine	Cosine	Cosine	Cosine	Cosine	Cosine
	Warmup ratio	0.1	0.1	0.1	0.1	0.1	0.1

Appendix

Table 3: Main training and method-specific hyperparameters for all methods.

Panel of Figure 4 — Figure 4: Per-domain results on the Economy domain, mirroring Figure 1 , Figure 2 , and Figure 3 of the main text. All metrics are divergences (lower is better).

Panel of Figure 5 — Figure 5: Per-domain results on the Political domain, mirroring Figure 1 , Figure 2 , and Figure 3 of the main text. All metrics are divergences (lower is better).

Panel of Figure 6 — Figure 6: Per-domain results on the Technology domain, mirroring Figure 1 , Figure 2 , and Figure 3 of the main text. All metrics are divergences (lower is better).

Panel of Figure 7 — Figure 7: Per-domain results on the Social domain, mirroring Figure 1 , Figure 2 , and Figure 3 of the main text. All metrics are divergences (lower is better).

Panel of Figure 8 — Figure 8: Per-domain results on the Culture domain, mirroring Figure 1 , Figure 2 , and Figure 3 of the main text. All metrics are divergences (lower is better).

Panel of Figure 9 — Figure 9: Per-domain results on the Health domain, mirroring Figure 1 , Figure 2 , and Figure 3 of the main text. All metrics are divergences (lower is better).

Panel of Figure 10 — Figure 10: Per-domain results on the Environment domain, mirroring Figure 1 , Figure 2 , and Figure 3 of the main text. All metrics are divergences (lower is better).

Panel of Figure 11 — Figure 11: Per-domain results on the Education domain, mirroring Figure 1 , Figure 2 , and Figure 3 of the main text. All metrics are divergences (lower is better).

Method	KL	JSD	TVD	MMD
Pretrained	$4.2801\ [4.1628,4.3949]$	$0.2851\ [0.2785,0.2912]$	$0.5745\ [0.5646,0.5833]$	$0.6350\ [0.6161,0.6524]$
SFT	$1.2871\ [1.2281,1.3439]$	$0.1524\ [0.1474,0.1573]$	$0.3953\ [0.3869,0.4034]$	$0.3194\ [0.3060,0.3324]$
DPO	$6.8819\ [6.7226,7.0539]$	$0.2690\ [0.2625,0.2752]$	$0.5497\ [0.5396,0.5590]$	$0.5774\ [0.5595,0.5935]$
PPO	$4.1393\ [4.0256,4.2464]$	$0.2588\ [0.2524,0.2647]$	$0.5403\ [0.5302,0.5490]$	$0.5578\ [0.5407,0.5743]$
DSA	$3.3904\ [3.2932,3.4851]$	$0.2343\ [0.2284,0.2403]$	$0.4937\ [0.4857,0.5019]$	$0.4418\ [0.4281,0.4567]$
SALT (Ours)	$\mathbf{0.2880}\ [0.2749,0.3009]$	$\mathbf{0.0825}\ [0.0788,0.0863]$	$\mathbf{0.2720}\ [0.2653,0.2787]$	$\mathbf{0.1510}\ [0.1440,0.1582]$

Appendix

Table 4: Point estimates and 95% paired-bootstrap confidence intervals on the full test set (1,000 resamples over contexts). All metrics are divergences (lower is better).

Figure 12: Overview of the SubjSim dataset. (a) Topics. (b) Option count

K

. (c) Subjectivity coefficient.

Source	Count	Proportion
ATP	35	35.0%
GSS	19	19.0%
WVS	14	14.0%
CGSS	9	9.0%
ANES	9	9.0%
ESS	9	9.0%

Appendix

Table 5: Source distribution of the 100 subjective questions in SubjSim.

Domain	Question	Response Options
Demographic Attributes
Demographic	What is your current academic year?	1. Freshman/Sophomore; 2. Junior/Senior; 3. Master’s; 4. PhD; 5. Not a student
	Are you an only child?	1. Yes; 2. No
	What is your current employment status?	1. Full-time; 2. Part-time; 3. Self-employed; 4. Retired; 5. Homemaker; 6. Student; 7. Unemployed; 8. Other
Subjective Questions
Political System	Over the next 30 years, which social trend concerns you the most?	1. AI replacing jobs; 2. Social stratification; 3. Misinformation; 4. Weakening family structures

Appendix

Table 6: Examples of Demographic Attributes and Subjective Questions

Figure 13: Demographic distributions of annotators across gender, age, academic status, and field of study.

Figure 14: Screenshot of the annotation interface. The platform was deployed in Chinese for native Chinese-speaking annotators; the interface shown here is an English translation for presentation purposes.

Large Language Model-Based Social Simulation

Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.

See all 257 papers

Feb 26, 2026

Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction

The transition of Large Language Models (LLMs) from exploratory tools to active "silicon subjects" in social science lacks extensive validation of operational validity. This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces. This framework enables a rigorous evaluation of current LLM capabilities with respect to the simulation of social media user behavior. We evaluated open-weight 8B models (Llama3.1, Qwen3, Ministral) in English, German, and Luxembourgish language scenarios. By systematically comparing prompting strategies (explicit vs. implicit) and the impact of Supervised Fine-Tuning (SFT), we identify a critical form vs. content decoupling in low-resource settings: while SFT aligns the surface structure of the text output (length and syntax), it degrades semantic grounding. Furthermore, we demonstrate that explicit conditioning (generated biographies) becomes redundant under fine-tuning, as models successfully perform latent inference directly from behavioral histories. Our findings challenge current "naive prompting" paradigms and offer operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.

Nils Schwager, Simon Münker, Alistair Plum +1

Trier University, Trier, Germany · University of Luxembourg, Esch-sur-Alzette, Luxembourg

Large Language Model-Based Social Simulation Language Model Generation Evaluation LLM Post-Training

Ask PDF

Sep 16, 2026

Understanding Reliability in LLM-based Human Behavior Simulation

Large language models (LLMs) are increasingly used to simulate human survey responses and behavioral reactions, yet unreliable simulations can mislead social science conclusions. However, existing evaluations focus on end-to-end scores, leaving it unclear how different aspects of the simulation process interact to determine reliability. We propose ReliMap, which decomposes LLM-based human behavior simulation into three structured layers and evaluates reliability at both the individual level (R1) and population level (R2) across three configuration dimensions: model capacity, profile completeness, and population coverage. Through experiments across four simulation tasks and eleven LLMs, we find that all models exhibit substantial distributional bias without profile conditioning. Profile conditioning reduces this bias with diminishing returns. Larger models benefit more, and attribute informativeness matters more than quantity. Critically, R1 gains do not reliably transfer to R2--individual and population-level reliability can move in opposite directions. At the population layer, increasing coverage reduces variance but not systematic bias, with R2 stabilizing at around 50-100 individuals. These findings highlight that reliable simulation cannot be achieved by optimizing any single layer in isolation, but requires coordinated improvement across all three.

Pei Wang, Lei Wang, Yuanzi Li +1

Gaoling School of Artificial Intelligence, Renmin University of China

Human Behavior Simulation Large Language Model-Based Social Simulation LLM Reliability

Ask PDF

Mar 11, 2026

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full

τ

-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Xuhui Zhou, Weiwei Sun, Qianou Ma +8

Carnegie Mellon University, Language Technologies Institute

User Simulation Sim-to-Real Transfer LLM Agent Evaluation Human-in-the-Loop Evaluation

Ask PDF

Abstract

Large Language Model-Based Social Simulation

Explore similar work

Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction

Understanding Reliability in LLM-based Human Behavior Simulation

Mind the Sim2Real Gap in User Simulation for Agentic Tasks