cs.CYApr 8, 2026

Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approach

Authors: Xiaoyou Qin, Zhihong Li, Xiaoxiao Cheng

Organizations: School of Journalism, Fudan University · College of Media and International Culture, Zhejiang University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) are increasingly used to simulate social attitudes and behaviors, offering scalable "silicon samples" that can approximate human data. However, current simulation practice often collapses diversity into an "average persona," masking subgroup variation that is central to social reality. This study introduces audience segmentation as a systematic approach for restoring heterogeneity in LLM-based social simulation. Using U.S. climate-opinion survey data, we compare six segmentation configurations across two open-weight LLMs (Llama 3.1-70B and Mixtral 8x22B), varying segmentation identifier granularity, parsimony, and selection logic (theory-driven, data-driven, and instrument-based). We evaluate simulation performance with a three-dimensional evaluation framework covering distributional, structural, and predictive fidelity. Results show that increasing identifier granularity does not produce consistent improvement: moderate enrichment can improve performance, but further expansion does not reliably help and can worsen structural and predictive fidelity. Across parsimony comparisons, compact configurations often match or outperform more comprehensive alternatives, especially in structural and predictive fidelity, while distributional fidelity remains metric dependent. Identifier selection logic determines which fidelity dimension benefits most: instrument-based selection best preserves distributional shape, whereas data-driven selection best recovers between-group structure and identifier-outcome associations. Overall, no single configuration dominates all dimensions, and performance gains in one dimension can coincide with losses in another. These findings position audience segmentation as a core methodological approach for valid LLM-based social simulation and highlight the need for heterogeneity-aware evaluation and variance-preserving modeling strategies.

Figures & tables

DimensionTheoretical focusKey metrics
Distributional fidelityOverall distribution alignmentAccuracy, Precision, Recall, F1 score; Mean Absolute Error (MAE); Kullback-Leibler divergence (KLD)
Structural fidelitySubgroup variance preservationWithin-group: Standard Deviation (SD), Coefficient of Variation (CV); Between-group: Normalized Earth mover’s distance (nEMD), Multidimensional scaling (MDS), Procrustes distance
Predictive fidelityRelational pattern correspondenceCramér’s V
Table 1: Three-dimensional framework for evaluating heterogeneity restoration
Segmentation configurationGranularityParsimonyIdentifier selection logicAddressed RQs
DemoLow (single-dimensional)High (5 identifiers)Empirical defaultBaseline for comparison
Demo+Theory-59Very high (multidimensional)Low (59 identifiers)Theory-drivenRQ1, RQ2
Demo+Theory-15High (multidimensional)Medium (15 identifiers)Theory-drivenRQ1–RQ3
Data-drivenHigh (multidimensional)Medium (15 identifiers)Data-driven (GBM)RQ3
Item-15Medium (attitude-focused)Medium (15 identifiers)Pre-validated instrument-basedRQ2, RQ3
Item-4Medium (attitude-focused)High (4 identifiers)Pre-validated instrument-basedRQ2, RQ3
Table 2: Segmentation configurations
Figure 1: Comparison of distributional fidelity metrics across segmentation configurations and LLMs.
Figure 2: Comparison of structural fidelity metrics across segmentation configurations and LLMs.
Figure 3: MDS maps of empirical and simulated subgroup structures across segmentation configurations and LLMs.
Segmentation configurationHuman benchmarkLlamaMixtral
Demo.19.27.08
Demo+Theory-59.25.19.21
Demo+Theory-15.26.23.18
Data-driven.28.29.33
Item-15.34.24.21
Item-4.39.40.44
Table 3: Summary of Cramér’s V values
Figure 4: Comparison of Cramér’s V across segmentation configurations and LLMs. Brackets and numbers indicate the differences from the human benchmark.
 Distributional fidelityStructural fidelityPredictive fidelity
GranularityMore ≠\neq betterMore ≠\neq betterMore ≠\neq better
ParsimonyMixed (metric-dependent)Fewer generally betterFewer generally better
Identifier selection logicInstrument-based bestData-driven and instrument-based bestData-driven best
Table 4: Summary of simulation performance
More in this topic

Large Language Model-Based Social Simulation

Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.

See all 257 papers

Explore similar work

CardsList
Jun 24, 2025

LLM-Based Social Simulations Require a Boundary

This position paper argues that LLM-based social simulations require clear boundaries to make meaningful contributions to social science. While Large Language Models (LLMs) offer promising capabilities for simulating human behavior, their tendency to produce homogeneous outputs, acting as an "average persona", fundamentally limits their ability to capture the behavioral diversity essential for complex social dynamics. We examine why heterogeneity matters for social simulations and how current LLMs fall short, analyzing the relationship between mean alignment and variance in LLM-generated behaviors. Through a systematic review of representative studies, we find that validation practices often fail to match the heterogeneity requirements of research questions: while most papers include ground truth comparisons, fewer than half explicitly assess behavioral variance, and most that do report lower variance than human populations. We propose that researchers should: (1) match validation depth to the heterogeneity demands of their research questions, (2) explicitly report variance alongside mean alignment, and (3) constrain claims to collective-level qualitative patterns when variance is insufficient. Rather than dismissing LLM-based simulation, we advocate for a boundary-aware approach that ensures these methods contribute genuine insights to social science.

Zengqing Wu, Run Peng, Takayuki Ito +2

University of Osaka · University of Michigan · Kyoto University +1

LLM EvaluationHuman Behavior SimulationComputational Social ScienceLarge Language Model-Based Social Simulation
AskPDF
Mar 17, 2026

Parametric Social Identity Injection and Diversification in Public Opinion Simulation

Large language models (LLMs) have recently been adopted as synthetic agents for public opinion simulation, offering a promising alternative to costly and slow human surveys. Despite their scalability, current LLM-based simulation methods fail to capture social diversity, producing flattened inter-group differences and overly homogeneous responses across demographic groups. We identify this limitation as a Diversity Collapse phenomenon in LLM hidden representations, where distinct social identities become increasingly indistinguishable across layers. Motivated by this observation, we propose Parametric Social Identity Injection (PSII), a general framework that injects explicit, parametric representations of demographic attributes and value orientations directly into intermediate hidden states of LLMs. Unlike prompt-based persona conditioning, PSII enables fine-grained and controllable identity modulation at the representation level. Extensive experiments on the World Values Survey using multiple open-source LLMs show that PSII significantly improves distributional fidelity and diversity, reducing KL divergence to real-world survey data while enhancing overall diversity. This work provides new insights into representation-level control of LLM agents and advances scalable, diversity-aware public opinion simulation.

Hexi Wang, Yujia Zhou, Bangde Du +2

DCST, Tsinghua University Beijing, China · Quancheng Laboratory Jinan, Shandong, China

Large Language Model-Based Social SimulationRepresentation Geometry in Language ModelsActivation Steering
AskPDF
Jul 2, 2026

Will Scaling Improve Social Simulation with LLMs?

Large Language Model (LLM) social simulations are a promising research method, but they are not yet faithful enough to be adopted widely. In this work, we investigate whether the current scaling paradigm in language modeling is likely to close these gaps, or whether simulation fidelity is orthogonal to general capabilities and therefore deserving of more research attention. We use scaling laws to study the relationship between LLMs' compute scale, general capability benchmarks, and the fidelity of social simulation in three representative sub-domains: opinion modeling, behavioral simulation, and longitudinal forecasting. Surprisingly, we discover strong compute scaling in all three settings, using a suite of 85 transformer LLMs with the Qwen3 architecture pre-trained on the DCLM web text corpus under fixed-compute budgets from 101810^{18} to 102010^{20} FLOPs. Then we evaluate 35 larger and more capable open-weight models up to 70B parameters, allowing us to predict downstream accuracy from loss. This reveals that the majority of behavioral and opinion simulation tasks will rapidly improve with scale, particularly when they involve populations that are well-represented in English web corpora. Longitudinal forecasting and underrepresented opinions scale more slowly, especially when they are less correlated with general knowledge and reasoning benchmarks like MMLU. In behavior simulation, scaling fails to improve model calibration with human cognitive biases like risk aversion, as well as human heuristics like learning correlated rewards from related tasks. On these tasks, even fine-tuned models fail to noticeably scale up performance from 0.5B to 8B parameters. Taken together, we conclude that scale will improve social simulations in most settings, but outliers exist, and improvements will be less reliable in low-resource domains.

Caleb Ziems, William Held, Su Doga Karaca +3

Stanford University · Open Athena

Opinion DynamicsLanguage Model Scaling LawsComputational Social ScienceLanguage Model CalibrationLarge Language Model-Based Social Simulation
AskPDF