cs.AIJul 27, 2026

Do Synthetic Personas Predict Real Audience Response? A Headline-Ranking Study with a Generic-Reader Baseline

Authors: Alexandre Cristovão Maiorano

Abstract

Do demographic personas improve an LLM's prediction of audience engagement? We compare a fixed ten-persona panel with a generic-reader baseline on Upworthy headlines, with supplementary news and Reddit comparisons. The primary experiment collects Gemini 3.1 Flash-Lite and GPT-4.1 observations on the same 190-package exploratory cohort, selected with simultaneous winner screening. It allocates thirty scores per arm and variant within model, retains exact raw responses, and reports metric-specific valid and paired counts. Complete-package coverage is 188 for Gemini and 188 for GPT-4.1. The baseline-minus-panel Kendall τbτ_b gap is +0.222 (CI [+0.061, +0.385]) for Gemini and +0.257 (CI [+0.119, +0.391]) for GPT-4.1, on respectively 136 and 188 valid rank pairs. Expected top-1 is 39.1% versus 54.8% for Gemini and 42.0% versus 61.7% for GPT-4.1 (panel versus baseline). These intervals condition on the selected cohort and recorded generations and do not adjust for dependence between related experiments. The Upworthy archive documents a randomization-risk period. Excluding that period in a post-hoc sensitivity retains positive expected top-1 and lift gaps in both configurations, while Gemini tau-b and Spearman intervals include zero. Supplementary Reddit top-1 and captured lift favor the panel; these aggregate comparisons use unreconciled collection protocols and do not isolate dataset effects. A raw-replay artifact supports offline reproduction of the primary experiment. The results concern this panel, cohort, and protocol; they are not a held-out evaluation or evidence of universal persona harm. Provider request protocols differ across models, and no causal mechanism is isolated.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Sep 29, 2026

MetaPersona: Task-Grounded Synthetic Populations from Empirical Social Science

Personas used to seed LLM social simulations face a cold-start problem: existing methods lack a principled basis for deciding which attributes to include and how to assign their values. As a result, synthetic populations may misrepresent the demographic composition, latent attributes, and dependency structure that shape downstream behavior. We introduce MetaPersona-DB, a dataset of 11,000+ empirical human-subjects studies annotated with task-relevant variables, reported relationships, and aggregate-level population statistics. Building on this resource, we propose MetaPersona, a framework that retrieves task-relevant evidence, constructs literature-derived persona dependency graphs, and samples synthetic populations from empirical priors linking demographics, latent attributes, and outcomes. Across three downstream case studies, three baselines, and three frontier models, results vary by task and model: MetaPersona performs strongly on misinformation belief and AI-tool sentiment, while results on income redistribution are mixed. It also reduces persona-construction cost to under $0.5 per task using GPT-5.2. Finally, we present MetaPersona-Studio, a prototype interactive interface for empirically grounded persona generation.
Jul 23, 2026

Distributional Validity of a Korean Synthetic Persona Panel: Evidence From the Korea Media Panel Survey

Large language model (LLM) personas are proposed as survey respondents, yet validation outside English-speaking contexts is scarce. We evaluate how well a Korean synthetic persona panel used to condition Gemini 3.5 Flash and EXAONE reproduces digital and artificial intelligence (AI) service-use distributions of the Korea Media Panel Survey. About 8,000 personas per model answered eight service-use items and eight attitudinal constructs; responses were compared with weighted survey estimates. The overall mean absolute error (MAE) was 14-19 percentage points (pp), with binary item-mean correlations of 0.70-0.91 across waves. Segment error across five axes was 14-18 pp, with between-group signed-error ranges of 49.6/34.7 pp (Gemini/EXAONE; 39.5/31.2 without the non-comparable teen cells). Errors were model-specific: an age stereotype (Gemini) versus an acquiescence-consistent level bias (EXAONE). Generative-AI overestimation was consistent with temporal misalignment; short-form underestimation was framing-sensitive and persisted under randomized order (both shown for Gemini). Post-hoc holdout calibration on 30% of the real data, with the correction form selected inside the calibration set, cut cell MAE from 18.3/15.1 to 4.9/4.4 pp, yet direct estimation from that subsample was more accurate than the calibrated panel (3.6 pp), a synthetic-informed shrinkage estimator beat its real-only counterpart by at most 0.7 pp, and the correction did not transfer competitively across waves. The calibrated panel kept an advantage only below roughly 250-860 real responses (at most 2.3 pp over a real-only shrinkage estimator) or, for one model, on unobserved segments. In this setting, synthetic panels are not survey substitutes; their value is diagnostic.
May 11, 2026

When Can Digital Personas Reliably Approximate Human Survey Findings?

Digital personas powered by Large Language Models (LLMs) are increasingly proposed as substitutes for human survey respondents, yet it remains unclear when they can reliably approximate human survey findings. We answer this question using the LISS panel, constructing personas from respondents' background variables and pre-2023 survey histories, then testing them against the same respondents' held-out post-cutoff answers. Across four persona architectures, three LLMs, and two prediction tasks, we assess performance at the question, respondent, distributional, equity, and clustering levels. Digital personas improve alignment with human response distributions, especially in domains tied to stable attributes and values, but remain limited for individual prediction and fail to recover multivariate respondent structure. Retrieval-augmented architectures provide the clearest gains, but performance depends more on human response structure than on model choice: personas perform best for low-variability questions and common respondent patterns, and worst for subjective, heterogeneous, or rare responses. Our results provide practical guidance on when digital personas could be appropriate for survey research and when human validation remains necessary.