Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples
Authors: Tobias Holtdirk, Dennis Assenmacher, Arnim Bleier, Claudia Wagner
Organizations: LMU Munich · Munich Center for Machine Learning · GESIS - Leibniz Institute for the Social Sciences · RWTH Aachen University · Complexity Science Hub Vienna
Survey research increasingly relies on convenience samples as rising costs and nonresponse make representative probability sampling more difficult. This creates a central challenge for public opinion research and electoral studies: how to recover reliable population-level inferences from socially and demographically biased data. While recent work has explored the use of large language models (LLMs) for simulating survey responses, most studies focus on the U.S. population and assume relatively favourable data conditions with limited or random missingness. In this work, we investigate whether LLMs can leverage contextual knowledge about socio-demographic and political patterns to predict vote choice in the German population under sampling constraints. We systematically vary both the size and composition of training data, ranging from large representative samples to small, biased convenience samples, and compare zero-shot prompting and supervised fine-tuning against established tabular classifiers such as CatBoost. Our results show that when representative training data are available, fine-tuned LLMs achieve predictive performance comparable to strong tabular baselines and substantially outperform zero-shot prompting. Under small and biased convenience samples, however, fine-tuned LLMs more accurately recover both individual-level vote choice and population-level distributions than zero-shot approaches and often outperform tabular methods. These findings suggest that fine-tuned LLMs can combine sample-specific information with contextual knowledge acquired during pre-training, enabling more robust inference under conditions of sampling bias and missing data.
Figures & tables
Figure 1: Approach to simulate survey answers with LLMs. We prompt and fine-tune LLMs to extract their answer for the question “Which party did you vote for in the federal election?”. We compare LLM-generated answers with real survey answers. The prompt design consists of three parts: the system prompt, the user description, and the answer. For fine-tuning, we provide the real vote-choice labels of training participants as assistant completions in the depicted format. For the evaluation, we only present the system prompt and the user prompt to the model and require the model to generate the answer.
Figure 2: Experiments. Each experiment uses a different subset of the GLES data, reduced either by removing survey variables (columns) or participants (rows).
Figure 3: Sample efficiency comparison. We vary the size of the training sample (1%, 5%, 10%, 20%, 40%, 60%) of the E1a setup, drawn as random sub-samples of GLES participants. Additionally, the original 80% training sample is included. Because the sweep uses random sub-samples (not convenience-style biased ones), it isolates sample-size effects from sampling-bias effects.
Figure 4: Macro-F1 scores (higher is better) of the predicted vote choices. The left column shows results using all survey variables as features. The right column shows results without using “party identification”. The rows show different subsets of participants used for training.
Figure 5: Total variation distance (lower is better) between the predicted distribution and the true distribution of vote choices. The left column shows results using all survey variables as features. The right column shows results without using “party identification”. The rows show different subsets of participants used for training.
Figure 6: Aggregated vote share of the predicted vote choices by model across all experiments (E1–E4). The red line shows the true vote share in the complete dataset; the purple line shows the vote share within the respective biased training subset. For E1 both the tabular classifier and the fine-tuned model perform similarly well, in the case with and without having the strong predictor party identification in the data. The zero-shot models struggle with predicting non-voters in (a) and additionally overpredict more extreme parties (the AfD being the most right-wing party and Die Linke the most left-wing) in (b). For E2a, we see that fine-tuned LLMs are able to match the true distribution well. This degrades somewhat for E2b, and similarly to the CatBoost baseline, overfits to the student population by predicting more left-leaning parties and more non-voters. For E3, the training vote share distribution is more similar to the complete data. For E4, non-voters are again overrepresented in the training data, as in E2; here, however, the fine-tuned LLM does not generalise well to the overall population and overpredicts non-voters, similar to CatBoost.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Macro F1
TVD
Vanilla
IW
Vanilla
IW
Exp.
Model
E2a
LogReg
0.237±0.019
0.219±0.018
0.295±0.036
0.282±0.036
RF
0.169±0.022
0.167±0.026
0.322±0.032
0.328±0.035
CatBoost
0.186±0.019
0.199±0.038
0.349±0.031
0.407±0.068
E2b
LogReg
0.158±0.015
0.161±0.013
0.306±0.034
0.294±0.036
Appendix
Table 1: Macro-F1 and TVD ( mean±95% CI ) for vanilla and importance-weighted (IW) classifier baselines across convenience sample experiments E2–E4.
Figure 7: Macro-F1 scores of Llama and Qwen models up to 8B parameters.
Figure 8: TVD of Llama and Qwen models up to 8B parameters.
Figure 9: Feature importance for the logistic regression classifier in E1a computed using feature permutation, where each feature’s values are randomly shuffled to quantify the drop in performance. This drop indicates how much each feature contributes to the model’s predictive power.
Figure 10: Cross-tabulation of “vote choice” and “party identification”.
Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction. We propose cross-survey transfer, a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, three open-weight LLMs (27B-120B parameters), and supervised machine learning baselines, we find that: (1) zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points (pp) of a supervised random forest trained on same-population data; (2) a stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty; and (3) variance collapse and safety alignment effects-two commonly cited LLM limitations-turn out to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families. These findings clarify both the promise and boundaries of silicon sampling.
Chan-Tung Ku, Chan Hsu, Pei-Cing Huang +3
Department of Information Management National Sun Yat-sen University Kaohsiung, Taiwan · Institute of Political Science National Sun Yat-sen University Kaohsiung, Taiwan · Graduate Institute of Library and Information Science National Chung Hsing University Taichung, Taiwan
Traditional survey-based political issue polling is becoming less tractable due to increasing costs and risk of bias associated with growing non-response rates and declining coverage of key demographic groups. With researchers and pollsters seeking alternatives, Large Language Models have drawn attention for their potential to augment human population studies in polling contexts. We propose and implement a new framework for anticipating human responses on multiple-choice political issue polling questions by directly prompting an LLM to predict a distribution of responses. By comparison to a large and high quality issue poll of the US population, the Cooperative Election Study, we evaluate how the accuracy of this framework varies across a range of demographics and questions on a variety of topics, as well as how this framework compares to previously proposed frameworks where LLMs are repeatedly queried to simulate individual respondents. We find the proposed framework consistently exhibits more accurate predictions than individual querying at significantly lower cost. In addition, we find the performance of the proposed framework varies much more systematically and predictably across demographics and questions, making it possible for those performing AI polling to better anticipate model performance using only information available before a query is issued.
Eric Gong, Nathan E. Sanders, Bruce Schneier
Harvard College · Berkman-Klein Center, Harvard University · Harvard Kennedy School
Large language models (LLMs) trained on large-scale internet corpora encode extensive statistical regularities about social identities, attitudes, and political behaviour. This paper introduces and evaluates a methodological framework that leverages these latent representations to reconstruct aggregate voting behaviour from individual-level sociodemographic profiles. We operationalize LLMs as implicit sociological models by conditioning them on demographic descriptions, eliciting probabilistic turnout and party preferences, and aggregating individual outputs via a soft voting procedure. Using the 2021 Czech parliamentary election as a validation case, we demonstrate that contemporary LLMs reproduce official election outcomes with low mean absolute error, recover known political bloc structures, and align with independently established sociodemographic gradients. The contribution of this work is methodological rather than predictive: we show how LLMs can be systematically interrogated as compressed representations of social reality, offering a novel exploratory instrument for computational social science while clearly delineating its epistemic and ethical limits.
Roman Neruda, Martin Bakoš, Josef Šlerka +3
Institute of Computer Science, Czech Academy of Sciences, Prague, Czech Republic. · Faculty of Arts, Charles University, Prague, Czech Republic. · distil labs, Berlin, Germany.