Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates

Daehwan Ahn; Chengfeng Mao; Dokyun Lee

cs.CLAug 29, 2026

Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates

Authors: Daehwan Ahn, Chengfeng Mao, Dokyun Lee

Organizations: College of Family and Consumer Sciences, University of Georgia, Athens, GA 30602 · MIT Sloan School of Management, Cambridge, MA 02142 · Questrom School of Business & Computing and Data Sciences, Boston University, Boston, MA 02215

Abstract

LLMs are increasingly used as human surrogates, often on the premise that richer persona data could make them substitutes or exploratory tools for specific individuals. We test this premise across four datasets covering more than 400,000 participants and more than 6,000 survey items and experimental outcomes. LLMs perform well at the aggregate level: their average responses closely align with average human responses to the same items. But this success largely reflects predicting each item's average human response. Once each item's human mean is removed, LLM predictions explain only 3.05% of the remaining respondent-specific variation, far below the 53.6% human test-retest benchmark. Richer personas, model variants, and fine-tuning do not close this gap. In variance analyses, once item means are removed, the reliable remaining signal is person-by-item. It captures how a respondent departs from the mean on a particular item and is about 8.9x larger than the stable person effect. Persona data encode the respondent, but not this item-specific deviation. LLM responses also compress human response distributions, using less spread, fewer response categories, and distorted distributional shapes. We call this pattern item-mean surrogacy. Current LLM surrogates can approximate item averages, but not the distributions or respondent-specific deviations needed to replace individual humans. We propose four empirical tests for LLM-based human-surrogate claims.

Explore similar work

Mar 8, 2026

Evaluating LLMs as Human Surrogates in Controlled Experiments

Large language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data support the same experimental inferences as human data. We evaluate this by directly comparing off-the-shelf LLM-generated responses with human responses from a canonical survey experiment on accuracy perception. Each human observation is converted into a structured prompt, and models generate a single 0--10 outcome variable without task-specific training; identical statistical analyses are applied to human and synthetic responses. We find that LLMs reproduce several directional effects observed in humans, but effect magnitudes and moderation patterns vary across models. Off-the-shelf LLMs therefore capture aggregate belief-updating patterns under controlled conditions but do not consistently match human-scale effects, clarifying when LLM-generated data can function as behavioral surrogates.

Adnan Hoq, Tim Weninger

University of Notre Dame Notre Dame, Indiana, USA

Human Behavior Simulation Large Language Model-Based Social Simulation LLM Evaluation Belief Updating in LLMs

Ask PDF

Jun 8, 2026

Beyond Averages: Evaluating LLMs on Human Survey Replication at the Distributional Level

LLMs are increasingly used to simulate human survey responses, but prior work has mainly evaluated replication using mean-level or aggregate agreement, offering limited insight into whether LLMs reproduce the variability of human behavior. We evaluate LLM-based survey replication at the distributional level using a non-public 2010 consumer choice experiment on Korean instant noodle purchases, a setting unlikely to overlap with model training data. We evaluate three response variables of differing statistical type: binary purchase incidence, categorical brand choice, and count purchase quantity. For each, we compare human and LLM responses at mean-level, pattern, and distributional alignment, and against reference baselines from the human data alone. LLMs reproduce condition-level patterns reasonably well but fail to capture distributional structure: for purchase quantity, no model beats a condition-insensitive baseline that simply matches the pooled human distribution. Because models that match human means well can still produce distributions further from humans than this baseline, mean-based evaluation alone can be actively misleading. Replication also varies with input configuration, with structured personas and multimodal inputs improving alignment while explicit reasoning prompting degrades it monotonically.

Jeonghyeon Moon, Jiwon Kim, Yeheum Lah +2

Ewha Womans University

LLM Evaluation Human Behavior Simulation

Ask PDF

May 11, 2026

When Can Digital Personas Reliably Approximate Human Survey Findings?

Digital personas powered by Large Language Models (LLMs) are increasingly proposed as substitutes for human survey respondents, yet it remains unclear when they can reliably approximate human survey findings. We answer this question using the LISS panel, constructing personas from respondents' background variables and pre-2023 survey histories, then testing them against the same respondents' held-out post-cutoff answers. Across four persona architectures, three LLMs, and two prediction tasks, we assess performance at the question, respondent, distributional, equity, and clustering levels. Digital personas improve alignment with human response distributions, especially in domains tied to stable attributes and values, but remain limited for individual prediction and fail to recover multivariate respondent structure. Retrieval-augmented architectures provide the clearest gains, but performance depends more on human response structure than on model choice: personas perform best for low-variability questions and common respondent patterns, and worst for subjective, heterogeneous, or rare responses. Our results provide practical guidance on when digital personas could be appropriate for survey research and when human validation remains necessary.

Mumin Jia, Yilin Chen, Divya Sharma +1

Department of Mathematics and Statistics, York University, Toronto, Ontario M3J 1P3 · Department of Biostatistics, University Health Network, Toronto, ON M5G 2C4

Human Behavior Simulation User Modeling

Ask PDF