Data-Driven Personas for Survey Simulation: Insights into Simulation Alignment Across Data-Access Regimes

Dongryeol Lee; Weronika Łajewska; Leonardo Perelli; Saab Mansour

cs.AIOct 5, 2026

Data-Driven Personas for Survey Simulation: Insights into Simulation Alignment Across Data-Access Regimes

Authors: Dongryeol Lee, Weronika Łajewska, Leonardo Perelli, Saab Mansour

Organizations: Amazon

Abstract

However, many existing steering approaches rely on target-domain human data for fine-tuning or prompting that is costly to collect and raises privacy concerns. In this paper, we study demographic group-level survey simulation, where personas induced from heterogeneous, anonymized public behavioral data condition agents that simulate responses of individuals from specific demographic groups. We examine whether representative personas can be induced from diverse sources and analyze how the domain, scale, and granularity of the source data affect survey simulation alignment. We find that personas induced from out-of-domain sources rarely outperform simulations conditioned only on basic demographic information, largely due to population mismatch. However, when personas are accurately assigned to the target demographic groups, alignment improves substantially. Finally, personas induced from target-domain survey data generalize better as more survey question history becomes available, suggesting that richer behavioral evidence enables more stable persona trait inference that transfers to better unseen questions simulation alignment.

Figures & tables

Figure 1: Data driven persona induction and survey simulation: In our framework, we generate personas from behavioral data across different domains and predict responses for individuals in target demographic groups.

Figure 2: Persona generation examples across domains.

Baseline/Persona Source	GSS		OpinionQA W26		OpinionQA W29		SocSci210 (MP3DR)		SocSci210 (NJ5DX)
Baseline/Persona Source	TVD	WD	TVD	WD	TVD	WD	TVD	WD	TVD	WD
B1	0.169	0.322	0.189	0.299	0.171	0.262	0.311	0.474	0.241	0.678
B2	0.160	0.301	0.173	0.274	0.139	0.204	0.314	0.552	0.204	0.560
B3	0.151	0.280	0.178	0.287	0.130	0.194	0.323	0.583	0.204	0.538
Model Knowledge	0.160	0.303	0.190	0.311	0.157	0.237	0.330	0.579	0.191	0.443
Movie Reviews	0.169	0.326	0.197	0.318	0.176	0.263	0.315	0.592	0.203	0.506

Table 1: Survey simulation results on the target survey datasets in the low data-access regime. The best-performing simulation approach for each target survey is indicated in bold , with the second-best underlined .

Baseline/Persona Source	GSS		OpinionQA W26		OpinionQA W29		SocSci210 (MP3DR)		SocSci210 (NJ5DX)
Baseline/Persona Source	TVD	WD	TVD	WD	TVD	WD	TVD	WD	TVD	WD
B3	0.151	0.280	0.178	0.287	0.130	0.194	0.323	0.583	0.204	0.538
GSS	–	–	0.204	0.332	0.149	0.222	0.335	0.614	0.213	0.574
+GT assignment	–	–	0.200	0.325	0.141	0.210	0.319	0.512	0.194	0.474
+GT generation	–	–	0.208	0.336	0.149	0.236	0.317	0.482	0.200	0.495
OpinionQA W26	0.167	0.320	–	–	0.151	0.229	0.320	0.608	0.205	0.533

Table 2: Survey simulation results on the target survey datasets in mid data-access regime. The best-performing approach for each survey is indicated in bold , with the second-best underlined . GT assignment uses GT demographics only for persona–group assignment, while GT generation uses GT demographics during persona generation.

Figure 3: Survey simulation results in the in-domain setting, showing TVD for simulations using personas generated from varying fractions of GSS questions.

Figure 4: For each target demographic group, we measure the correlation between demographic mismatch score (as measured by TVD between selected personas and actual demographics) and survey TVD. Correlation coefficient: (Pearson=0.631, Spearman = 0.323)

Figure 5: TVD across different demographic-group sizes after binning groups into Small, Medium, and Large buckets by respondent count in GSS dataset.

Figure 6: Performance Heatmap (TVD) across demographic keys

Persona Source
Dataset	Domain	#Q	Avg. Resp.	#Resp.
GSS	General Survey	60	36.1	3309
OpinionQA W26	Survey: Guns	78	46.2	4168
OpinionQA W29	Survey: Gender	77	53.8	4867
SocSci210 MP3DR	Survey: Gender	39	6.9	4161
SocSci210 NJ5DX	Survey: Inequality	48	18.2	1814

Table 3: Summary statistics of persona sources and test datasets.

Figure 7: Survey simulation results in the in-domain setting, showing TVD and standard deviation (Std.) for simulations using personas generated from varying fractions of GSS questions. Results for each iteration and Std. are available in Table 6

GT Generation

Free

You are a 33-year-old Black man living in the Northeast who works in education administration at an elementary or secondary school. You’re a college graduate with a bachelor’s degree who has never married but has two children. You identify as Protestant and attend religious services weekly, indicating that faith is an important part of your life. Politically, you lean Democratic but consider yourself moderate rather than strongly liberal. You consistently vote in presidential elections, supporting Democratic candidates like Clinton and Biden. You have only some confidence in the executive branch of government, suggesting a degree of political skepticism despite your party affiliation…

You are a politically moderate Democrat who consistently votes in elections and supports Democratic presidential candidates. While you identify as middle-of-the-road politically, your policy positions tend to lean progressive on domestic issues - you support increased spending on education, healthcare, environmental protection, addressing urban problems, and improving conditions for Black Americans. You oppose the death penalty and believe premarital sex is not morally wrong. You’re fiscally more conservative on international matters, believing we spend too much on military and foreign aid. You’re civically engaged and vote regularly in presidential elections…

Table 4: GT feature vs. Free description

Simulator: Claude 3.7
Target Data	GSS		OpinionQA W26		OpinionQA W29		SocSci210 (MP3DR)		SocSci210 (NJ5DX)
	TVD	WD	TVD	WD	TVD	WD	TVD	WD	TVD	WD
B1	0.169	0.322	0.189	0.299	0.171	0.262	0.311	0.474	0.241	0.678
B2	0.160	0.301	0.173	0.274	0.139	0.204	0.314	0.552	0.204	0.560
B3	0.151	0.280	0.178	0.287	0.130	0.194	0.323	0.583	0.204	0.538
Persona Source

Table 5: Survey simulation results using three different LLMs on the target survey datasets in the low data-access regime. The best-performing simulation approach for each target survey is indicated in bold , with the second-best underlined .

Iteration	Question Split	GSS Persona	+GT Generation	B3
1	20%	0.187	0.163	0.157
	40%	0.202	0.166	0.167
	60%	0.178	0.158	0.156
	80%	0.141	0.111	0.131
2	20%	0.178	0.154	0.149
	40%	0.173	0.151	0.150

Table 6: Survey simulation results in the in-domain setting across iterations, showing TVD for simulations using personas generated from varying fractions of GSS questions.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Performance of Claude-3.7-Sonnet and Llama-3.1-70B model on OpinionQA across different number of personas and different distribution aggregation methods

Figure 9: Average Shannon entropy of simulated response distributions under the optionselection aggregation scheme. Lower entropy indicates more peaked (less diverse)distributions, showing that option selection tends to collapse onto a single dominant option.

Figure 10: Number of answer options with zero counts under the option selection aggregation scheme, aggregated across survey questions. A high number of zero-count options indicates that minority opinions are rarely represented in the simulated distributions.

Panel of Figure 11 — Figure 11: Prompt template used for Persona Generation (Free) as described in Appendix A.4 .

Panel of Figure 12 — Figure 12: Prompt template used for Persona Generation (Free+GT) as described in Appendix A.4 .

Panel of Figure 13 — Figure 13: Test survey instance examples

Panel of Figure 14 — Figure 14: Prompt template used for Persona-conditioned Simulation.

Confidence Level	GSS	OpinionQA W26	OpinionQA W29
High	0.812	0.729	0.842
Medium	0.320	0.434	0.394
Low	0.393	0.399	0.382

Appendix

Table 7: Demographic feature extraction accuracy across different persona source datasets and confidence levels.

Feature	GSS	OpinionQA W26	OpinionQA W29
Overall	0.553	0.395	0.456
AGE	0.467	0.300	0.422
EDUCATION	0.340	0.267	0.310
INCOME	0.260	0.246	0.347
MARITAL	0.731	0.485	0.472
RACE	0.750	0.783	0.871

Appendix

Table 8: Demographic feature extraction accuracy across different persona source datasets and demographic keys.

Explore similar work

Data-Driven Persona-Conditioned Agents for A/B Test Simulation
Sep 1, 2026Ziyad Benomar, Weronika Łajewska, Leonardo Perelli +1User Simulation Data-Driven
Amazon · Luxembourg, Luxembourg · Barcelona, Spain
MetaPersona: Task-Grounded Synthetic Populations from Empirical Social Science
Sep 29, 2026Jinyi Ye, Yuangang Li, Chenxiao Yu +7Synthetic Data Demographics
University of Southern California · University of California, Irvine · 2077AI +1
How Well Do Large Language Models Capture Human Personality?
May 12, 2026Aanisha Bhattacharyya, Yaman Kumar Singla, Rajiv Ratn Shah +2Artificial Intelligence Personas Persona Consistency
Adobe Media and Data Science Research (MDSR) · IIIT-Delhi · SUNY at Buffalo

Sep 1, 2026, cs.AIJ/K move · Enter open · S save

Data-Driven Persona-Conditioned Agents for A/B Test Simulation

Ziyad Benomar, Weronika Łajewska, Leonardo Perelli, Saab Mansour

Amazon · Luxembourg, Luxembourg · Barcelona, Spain

User Simulation Data-Driven Population Dynamics

Open paper AskPDF arXiv

A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data-activity patterns, engagement signals, and inferred demographics-enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75-0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.

Abstract

Explore similar work

Data-Driven Persona-Conditioned Agents for A/B Test Simulation

MetaPersona: Task-Grounded Synthetic Populations from Empirical Social Science

How Well Do Large Language Models Capture Human Personality?