Do Synthetic Personas Predict Real Audience Response? A Headline-Ranking Study with a Generic-Reader Baseline
Abstract
Do demographic personas improve an LLM's prediction of audience engagement? We compare a fixed ten-persona panel with a generic-reader baseline on Upworthy headlines, with supplementary news and Reddit comparisons. The primary experiment collects Gemini 3.1 Flash-Lite and GPT-4.1 observations on the same 190-package exploratory cohort, selected with simultaneous winner screening. It allocates thirty scores per arm and variant within model, retains exact raw responses, and reports metric-specific valid and paired counts. Complete-package coverage is 188 for Gemini and 188 for GPT-4.1. The baseline-minus-panel Kendall gap is +0.222 (CI [+0.061, +0.385]) for Gemini and +0.257 (CI [+0.119, +0.391]) for GPT-4.1, on respectively 136 and 188 valid rank pairs. Expected top-1 is 39.1% versus 54.8% for Gemini and 42.0% versus 61.7% for GPT-4.1 (panel versus baseline). These intervals condition on the selected cohort and recorded generations and do not adjust for dependence between related experiments. The Upworthy archive documents a randomization-risk period. Excluding that period in a post-hoc sensitivity retains positive expected top-1 and lift gaps in both configurations, while Gemini tau-b and Spearman intervals include zero. Supplementary Reddit top-1 and captured lift favor the panel; these aggregate comparisons use unreconciled collection protocols and do not isolate dataset effects. A raw-replay artifact supports offline reproduction of the primary experiment. The results concern this panel, cohort, and protocol; they are not a held-out evaluation or evidence of universal persona harm. Provider request protocols differ across models, and no causal mechanism is isolated.
Figures & tables
| Metric | Panel [95% CI] | Baseline [95% CI] | Paired gap [95% CI] | p/b/pair |
|---|---|---|---|---|
| Goodman–Kruskal | 0.082 [-0.031, 0.192] | 0.381 [0.253, 0.501] | 0.268 [0.101, 0.434] | 185/137/136 |
| Kendall | 0.082 [-0.030, 0.191] | 0.336 [0.214, 0.450] | 0.222 [0.061, 0.385] | 185/137/136 |
| Spearman | 0.100 [-0.016, 0.214] | 0.371 [0.248, 0.488] | 0.236 [0.069, 0.405] | 185/137/136 |
| Expected top-1 | 39.1% [32.2, 46.3] | 54.8% [49.4, 60.2] | 15.7 pp [7.1, 24.0] | 188/188/188 |
| Expected bottom-1 | 43.4% [36.4, 50.3] | 47.5% [41.4, 53.6] | 4.2 pp [-4.1, 12.3] | 188/188/188 |
| Captured lift | 0.458 [0.394, 0.522] | 0.591 [0.541, 0.641] | 0.133 [0.054, 0.210] | 188/188/188 |
| Metric | Panel [95% CI] | Baseline [95% CI] | Paired gap [95% CI] | p/b/pair |
|---|---|---|---|---|
| Goodman–Kruskal | 0.088 [-0.026, 0.200] | 0.345 [0.242, 0.445] | 0.257 [0.119, 0.391] | 188/188/188 |
| Kendall | 0.088 [-0.026, 0.200] | 0.345 [0.242, 0.445] | 0.257 [0.119, 0.391] | 188/188/188 |
| Spearman | 0.107 [-0.010, 0.222] | 0.373 [0.265, 0.475] | 0.266 [0.124, 0.404] | 188/188/188 |
| Expected top-1 | 42.0% [35.1, 48.9] | 61.7% [54.8, 68.6] | 19.7 pp [10.6, 28.7] | 188/188/188 |
| Expected bottom-1 | 43.1% [36.2, 50.0] | 52.1% [44.7, 59.0] | 9.0 pp [0.0, 17.6] | 188/188/188 |
| Captured lift | 0.467 [0.400, 0.534] | 0.654 [0.590, 0.719] | 0.187 [0.103, 0.271] | 188/188/188 |
| Model (complete packages) | Metric | Paired gap [95% CI] | |
|---|---|---|---|
| Gemini (148) | Goodman–Kruskal | 0.201 [0.010, 0.396] | 109 |
| Gemini (148) | Kendall | 0.157 [-0.030, 0.346] | 109 |
| Gemini (148) | Spearman | 0.170 [-0.022, 0.367] | 109 |
| Gemini (148) | Expected top-1 | 11.4 pp [1.7, 21.0] | 148 |
| Gemini (148) | Expected bottom-1 | 3.1 pp [-6.4, 12.6] | 148 |
| Gemini (148) | Captured lift | 0.091 [0.001, 0.181] | 148 |
| Method | Top-1 [95% CI] | Goodman–Kruskal [95% CI] | Spearman [95% CI] |
|---|---|---|---|
| No-persona baseline | 49.2% [44.2, 54.3] | 0.361 [0.289, 0.430] | 0.350 [0.281, 0.417] |
| Persona panel | 34.6% [29.8, 39.1] | 0.084 [0.020, 0.147] | 0.104 [0.035, 0.172] |
| Dataset | p/b | Persona | Baseline | Gap | Lift p/b |
|---|---|---|---|---|---|
| Upworthy exploratory (2013–15) | 399/398 | 0.079 | 0.361 | 0.45/0.58 | |
| Upworthy holdout (2013–15) | 402/401 | 0.100 | 0.398 | 0.48/0.56 | |
| Upworthy confirmatory (2013–15) | 1866/1864 | 0.070 | 0.393 | 0.44/0.55 | |
| MIND news (2019) | 74/74 | 0.177 | 0.214 | 0.47/0.51 | |
| Reddit titles (2008–12) | 600/599 | -0.009 | -0.026 | 0.45/0.33 |
| Aggregation | Goodman–Kruskal [95% CI] | Top-1 [95% CI] |
|---|---|---|
| No-persona baseline | 0.361 [0.289, 0.430] | 49.2% [44.2, 54.3] |
| Equal-persona mean | 0.084 [0.020, 0.147] | 34.8% [30.1, 39.3] |
| Panel median | 0.141 [0.076, 0.205] | 39.3% [34.6, 44.1] |
| Trimmed mean (20%) | 0.081 [0.017, 0.144] | 34.3% [29.6, 38.8] |
| Per-persona -mean | 0.064 [0.001, 0.128] | 33.6% [29.1, 38.3] |
| Borda (rank fusion) | 0.081 [0.018, 0.146] | 35.3% [30.8, 40.1] |
| Metric | Panel [95% CI] | Baseline [95% CI] | Paired gap [95% CI] | p/b/pair |
|---|---|---|---|---|
| Goodman–Kruskal | 0.049 [-0.061, 0.159] | 0.325 [0.223, 0.429] | 0.270 [0.132, 0.407] | 190/189/189 |
| Kendall | 0.049 [-0.061, 0.159] | 0.325 [0.222, 0.429] | 0.270 [0.132, 0.407] | 190/189/189 |
| Expected top-1 | 39.5% [32.6, 46.3] | 60.6% [53.7, 67.7] | 20.9 pp [11.1, 30.4] | 190/189/189 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Top-1 | Goodman–Kruskal | Spearman |
|---|---|---|---|
| gemini-2.5-flash | 0.390 [0.29, 0.49] | 0.116 [-0.01, 0.24] | 0.133 [-0.01, 0.27] |
| gemini-3.1-flash-lite | 0.330 [0.24, 0.42] | 0.035 [-0.10, 0.17] | 0.051 [-0.09, 0.19] |
| gemini-3.5-flash | 0.390 [0.29, 0.49] | 0.122 [-0.01, 0.25] | 0.139 [-0.00, 0.28] |
| Model | Persona | Baseline | Gap |
|---|---|---|---|
| gemini-3.1-flash-lite | 0.035 | 0.401 | |
| gemini-2.5-flash | 0.116 | 0.219 | |
| gemini-3.5-flash | 0.122 | 0.404 |
| Real winning headline | CTR | #var |
|---|---|---|
| Bookmark This For When You’re Sad And Instantly Be 100 Times Happier | 1.38% | 4 |
| 10 Amazing International Photos That Will Give You Some Perspective | 1.92% | 4 |
| If You Saw Someone Bullying A Kid For Being Gay Would You Speak Out… | 1.11% | 2 |
| If You Watch Hilary Swank Annoy A Whiny Senator, $1 Will Go To Pun… | 1.67% | 3 |
| Seven Wonderful People Share Secrets That Are Killing Them. This C… | 1.99% | 4 |
| A 9-Year-Old Asks McDonald’s CEO A Simple Question. She’s Not Lovin… | 3.23% | 4 |
| ID | Demographics | News engagement | Behavioural description (verbatim) |
|---|---|---|---|
| p01 | 18-29, F, Facebook | incidental | 29-year-old woman, scrolls Facebook on her phone during breaks; clicks human-interest and feel-good stories shared by friends; rarely seeks news directly. |
| p02 | 18-29, M, Twitter | active | 24-year-old man, heavy Twitter user, follows journalists and culture accounts; clicks surprising or contrarian takes; low tolerance for clickbait he’s seen before. |
| p03 | 30-49, F, Facebook | incidental | 38-year-old working mother, checks Facebook in the evening; drawn to headlines about family, health, and local impact; shares uplifting or outrage-inducing stories. |
| p04 | 30-49, M, Reddit | active | 41-year-old man, browses Reddit and Facebook; skeptical of sensational headlines, clicks when a headline promises concrete information or a strong narrative payoff. |
| p05 | 50-64, F, Facebook | incidental | 57-year-old woman, mostly on Facebook; clicks emotionally resonant stories and anything about social causes she follows; less drawn to internet-slang headlines. |
| p06 | 50-64, M, Facebook | moderate | 60-year-old man, reads news on Facebook and cable; clicks political and economic headlines, distrusts overtly hyped ’you won’t believe’ framing. |
Human Behavior Prediction
Human behavior prediction models how people will act, decide, move, or feel from their histories, context, and observable signals.
See all 86 papers