Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues

Eric Gong; Nathan E. Sanders; Bruce Schneier

cs.CYMar 6, 2026

Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues

Authors: Eric Gong, Nathan E. Sanders, Bruce Schneier

Organizations: Harvard College · Berkman-Klein Center, Harvard University · Harvard Kennedy School

Abstract

Traditional survey-based political issue polling is becoming less tractable due to increasing costs and risk of bias associated with growing non-response rates and declining coverage of key demographic groups. With researchers and pollsters seeking alternatives, Large Language Models have drawn attention for their potential to augment human population studies in polling contexts. We propose and implement a new framework for anticipating human responses on multiple-choice political issue polling questions by directly prompting an LLM to predict a distribution of responses. By comparison to a large and high quality issue poll of the US population, the Cooperative Election Study, we evaluate how the accuracy of this framework varies across a range of demographics and questions on a variety of topics, as well as how this framework compares to previously proposed frameworks where LLMs are repeatedly queried to simulate individual respondents. We find the proposed framework consistently exhibits more accurate predictions than individual querying at significantly lower cost. In addition, we find the performance of the proposed framework varies much more systematically and predictably across demographics and questions, making it possible for those performing AI polling to better anticipate model performance using only information available before a query is issued.

Figures & tables

Figure 1: Framework for curating and evaluating LLM predictions on opinion polling questions

Figure 2: Vector embedding of survey questions mapped to two dimensional space with UMAP and color coded by LLM-assigned content tag.

Panel of Figure 3 — Figure 3: Histograms of pair-wise metric differences between SI and DD Frameworks

Panel of Figure 4 — Figure 4: LLM response heterogeneity (standard deviation) for the DD and SI frameworks versus the heterogeneity of human responses. Moving average and associated 2 $\sigma$ Standard Error shown

One-Hot Encoded Features	Metrics *
	NEMD	MD	SDD
Ideology: Very conservative	0.016	0.065	-0.016
Ideology: Conservative	0.015	0.046	0.018
Ideology: Liberal	0.003	0.008	0.032
Ideology: Very liberal	-0.002	0.013	0.027
Race: non-white	-0.001	-0.004	-0.019

Table 1: Bayesian Ridge Regression Coefficients for one-hot encoded features across each metric for the DD framework

Regression Model	NEMD		MD		SDD
	Training	Testing	Training	Testing	Training	Testing
Bayesian Ridge	0.287	0.312	0.328	0.175	0.394	0.378
Bayesian with Interactions	0.474	0.526	0.473	0.268	0.732	0.687
Gradient Boosting	0.415	0.517	0.442	0.273	0.598	0.655

Table 2: Predictive abilities of various regression models for each metric as evaluated through in-sample (training) and out-of-sample (testing)

R^{2}

under the DD querying framework

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Feature Number *
Tag	87	14	97	38	35
Abortion/Reproductive Rights	0.43	-0.01	0.06	0.07	-0.16
Environment & Energy Policy	-0.03	-0.20	-0.11	-0.22	-0.06
Federal Spending & Economic Policy	-0.18	0.22	-0.40	0.30	0.03
Government Approval	0.08	0.18	0.33	0.15	0.10
Gun Policy	0.14	0.17	0.01	-0.04	0.16

Appendix

Table 3: Pearson correlation between each tag and statistically significant embedding features (as determined through the bayesian ridge model)

Regression Model	NEMD		MD		SDD
	Training	Testing	Training	Testing	Training	Testing
Bayesian Ridge	-0.310	-0.156	-0.163	-0.296	0.126	-0.158
Bayesian with Interactions	-0.559	-0.091	-0.443	-0.308	0.079	-0.213
Gradient Booster	-0.029	-0.103	-0.036	-0.448	-0.114	-0.441

Appendix

Table 4: Predictive abilities of various regression models for each metric as evaluated through in-sample (training) and out-of-sample (testing)

R^{2}

under the SI query framework

Regression Model	NEMD		MD		SDD
	Training	Testing	Training	Testing	Training	Testing
Bayesian Ridge	-0.363	-0.106	-0.336	-0.136	0.071	-0.131
Bayesian with Interactions	-0.587	-0.103	-0.477	-0.153	0.033	-0.175
Gradient Booster	-0.003	-0.187	0.026	-0.217	-0.095	-0.627

Appendix

Table 5: Predictive abilities of various regression models for metric differences (SI - DD) between frameworks, as evaluated through in-sample (training) and out-of-sample (testing)

R^{2}

Panel of Figure 5 — Figure 5: DD and SI framework SDD metric as a function of NEMD, with instances in which the LLM returned the same response across all trials (Standard Deviation of 0) for SI framework identified.

Panel of Figure 6 — Figure 6: Statistically Significant feature coefficients (p = 0.05) for Bayesian Ridge regression models predicting each metric

Embedding Features	Metrics *
	NEMD	MD	SDD
Feature 1	0.010	-0.005	0.019
Feature 2	-0.009	0.005	-0.010
Feature 5	0.010	0.005	-0.001
Feature 6	-0.009	-0.004	-0.030
Feature 10	-0.011	-0.001	0.021

Appendix

Table 6: Bayesian Ridge Regression Coefficients for embedding features across each metric, filtered to Embedding features with statistically significant impact on NEMD

Large Language Model-Based Social Simulation

Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.

See all 257 papers

Before You Poll with LLMs: A Deliberative Diagnostic Framework
Sep 14, 2026Ahmed Wali, Hassaan TayyabLLM Evaluation Political Bias in Language Models
Lahore University of Management Sciences
Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study
Aug 12, 2026Simone MungariLLM Auditing Political Bias in Language Models
Revelis s.r.l.
Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples
Sep 29, 2025Tobias Holtdirk, Dennis Assenmacher, Arnim Bleier +1LLM Fine-Tuning
LMU Munich · Munich Center for Machine Learning · GESIS - Leibniz Institute for the Social Sciences +2

Sep 14, 2026J/K move · Enter open · S save

Before You Poll with LLMs: A Deliberative Diagnostic Framework

Ahmed Wali, Hassaan Tayyab

Lahore University of Management Sciences

LLM Evaluation Political Bias in Language Models Belief Updating in LLMs

Open paper AskPDF arXiv

Can LLMs reason through new information like humans, or do they merely retrieve cached opinions? This is critical for silicon sampling, where LLM personas simulate public opinion at scale. Current evaluations test only whether personas hold the right opinions -- a static snapshot. But opinion research increasingly depends on dynamic fidelity: whether personas update beliefs in response to new arguments, as humans do during deliberation. No existing benchmark tests this. We introduce the Deliberative Polling Diagnostic Framework, which compares human and LLM belief shifts after identical informational interventions. Grounded in deliberative polling, it surfaces failures invisible to static evaluation: models that produce plausible partisan opinions can still misrepresent how those opinions change. Applying the framework to five frontier models using data from America in One Room (526 personas, 72 questions), we find that every model fails, each in a unique manner. GPT-5.1 exhibits reversal: its personas become more hostile toward the opposing party after balanced information, while humans become less so. This reversal is selective (80% on outgroup vs. 26% on policy questions) and symmetric across partisan identities. Gemini 2.0 Flash, Claude Sonnet 4.5, and Llama 3.3 70B exhibit overshoot, shifting correctly but at 5-7x human magnitude. DeepSeek V3 exhibits rigidity with near-zero change. Targeted ablations reveal that policy content triggers these failures and that they are identity-specific: GPT-5.1 reverses on outgroup questions but overshoots on ingroup; Gemini shows the inverse. We term this signature self-sycophancy: conformity to the model's internal stereotype of the persona rather than reasoning from the information provided. Our framework offers a concrete protocol: run the deliberative diagnostic before trusting LLM personas to mimic revised beliefs.

Abstract

Large Language Model-Based Social Simulation

Explore similar work

Before You Poll with LLMs: A Deliberative Diagnostic Framework

Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study

Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples