Population Fidelity: Evaluating Population Representativeness in LLMs

Neemias B. da Silva; Martin Lukk; Ali Sutani; Abhishek Moturu; Harris Yang; Daniel Silver; Matt Ratto; Thiago H. Silva

cs.CLSep 28, 2026

Population Fidelity: Evaluating Population Representativeness in LLMs

Authors: Neemias B. da Silva, Martin Lukk, Ali Sutani, Abhishek Moturu, Harris Yang, Daniel Silver, Matt Ratto, Thiago H. Silva

Organizations: University of Toronto, Canada · Federal University of Technology – Paran´a, Brazil

Abstract

Large language models (LLMs) show considerable potential in simulating human attitudes and preferences. Prior work finds that LLM-generated responses can compress the range of attitudes found within populations and misrepresent particular subgroups in ways that vary across models and topics. We introduce Population Fidelity, an evaluation framework that distinguishes key conditions required for a set of LLM-generated responses to represent a population. It incorporates three dimensions: group-level accuracy, the amount of between-group variation, and the structure of that variation. We demonstrate the framework's utility in two ways. First, we reproduce a prior study of "machine bias" in LLM survey responses and apply the framework to its models and more recent ones, showing that poor representation reflects not only insufficient between-group variation but also variation assigned to the wrong groups. Second, we evaluate one proposed approach to improving models' population representativeness: cultural fine-tuning. We find that cultural fine-tuning can improve alignment with the survey center without improving the representation of within-population differences, a distinction that measures of aggregate agreement do not capture. We argue that representing a population requires models to reproduce several features of human attitudinal variation simultaneously. Our framework organizes these features and provides reusable code, data, and trained models for evaluating population fidelity across substantive domains and assessing proposed alignment methods.

Figures & tables

Figure 1: Stylized example with three subpopulations (A–C). Colored curves show each group’s share of respondents and sum to the pooled distribution (gray), which is identical in every panel (

S_{\mathrm{center}}=1

for all models); dashed lines mark group means. Calculations are in Appendix B.3 .

Panel of Figure 2 — Figure 2: Population Fidelity for happiness under FA elicitation. (a) Components, PFS, and center alignment for the indicated cell sets. (b) Fine-tuned minus as-released changes on target-country cells. Markers show three-run FA means using each run’s paired support; Table 10 , Appendix E.1 , uses the stricter all-runs common-cell intersection. All-cell results are in Figure 13 , Appendix D.5 .

Figure 3: PFS and center alignment by demographic level for happiness under FA elicitation. Boxes summarize 19 model conditions. Results for all questions: Figure 10 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Reproduction of Figure 2 from Boelaert et al. (2025) : cell-level nEMD densities for the six published model–elicitation series, shown with the demographic and random baselines and the original response-quality bands.

Panel of Figure 5 — Figure 5: The Population Fidelity framework compares survey and model distributions cell by cell ( $S_{\mathrm{acc}}$ ), in spread ( $S_{\mathrm{adapt}}$ ), and in arrangement ( $S_{\mathrm{struct}}$ ); center alignment ( $S_{\mathrm{center}}$ ) is reported separately. Section 4 defines each. A reference population provides $n$ demographic cells with survey distributions $s_{i}$ . Model responses to the corresponding demographic profiles produce distributions $m_{i}$ . Accuracy, adaptability, and structure compare the two sets of distributions and combine into the PFS by geometric mean; center alignment is reported separately. The metrics can also be recomputed within demographic subsets.

Variable	Topic	Options
d_happy	Happiness	4
d_polpos	Political ideology	10
d_religiousp	Religious attendance	7
d_trust	Social trust	2

Appendix

Table 4: Attitudinal items used in the evaluation.

Figure 7: PFS and center alignment by demographic level for the two proprietary models under FA elicitation, shown separately by question. Within each question, the upper row shows PFS and the lower row center alignment; demographic columns follow Figure 3 . Blue circles denote archived GPT-3 outputs from the original study, and green triangles denote GPT-5.6 Terra as released.

Figure 8: Cell-level error and adaptability under FA elicitation. The horizontal axis reports mean nEMD to the WVS cells, with lower values indicating greater accuracy. The vertical axis reports the adaptability ratio

A

. Values below

A=1

indicate compression of between-group variation, while values above one indicate amplification. Adaptability must be interpreted alongside structure because matching the amount of variation does not establish that it occurs between the corresponding social groups.

Figure 9: Change in PFS by country after cultural fine-tuning for happiness under FA elicitation. Each row represents a fine-tuned condition and each column a surveyed country. Green indicates improvement relative to the corresponding as-released model, and red indicates decline. For happiness under FA, target-country changes are not consistently more favorable than changes in the other surveyed countries.

Figure 10: PFS and center alignment by demographic level under FA elicitation, shown separately by question. Filled boxes show the distribution of PFS and hollow boxes the distribution of center alignment across 19 model conditions; Mixtral reference series are excluded. Demographic columns follow Figure 3 .

Figure 11: PFS and center alignment by demographic level for individual model conditions under FA elicitation, shown separately by question. Within each question, the upper row shows PFS and the lower row center alignment; demographic columns follow Figure 3 . Color and marker shape identify the model. Hollow markers denote as-released conditions, filled markers German fine-tuning, and half-filled markers Mexican fine-tuning.

Figure 12: Adaptability and structure under NTP elicitation, shown separately by question. The horizontal axis reports the adaptability ratio

A

, and the vertical axis the untransformed structure correlation

\rho

. Dashed lines mark

A=1

and

\rho=0

. Dotted connectors link each population-wide estimate to the corresponding estimate recomputed within the target-country subset: Germany for German fine-tuning and Mexico for Mexican fine-tuning. As-released conditions show both target-country estimates, whereas fine-tuned conditions show only the estimate for their target country. Conditions with

A>1

and

\rho\approx 0

exhibit amplified between-group variation without recovering the observed structure of group differences.

Panel of Figure 13 — Figure 13: Population Fidelity for happiness under FA elicitation on all retained cells. (a) Component scores for all 21 conditions, ordered by PFS; diamonds show PFS and hollow squares center alignment. (b) Fine-tuned minus as-released changes computed on the cells retained by both conditions. Figure 2 shows the corresponding target-country results. Plotting conventions are as in Figure 2 .

Panel of Figure 14 — Figure 14: Changes in PFS and center alignment after cultural fine-tuning under FA elicitation, for the cross-question summary, political ideology, religious attendance, and social trust. Each point compares a fine-tuned condition with its corresponding as-released model. Movement to the right indicates improved PFS, and upward movement indicates improved center alignment. The happiness result appears in Figure 2(b) .

Panel of Figure 15 — Figure 15: Population Fidelity components under FA elicitation, for the cross-question summary, political ideology, religious attendance, and social trust. Accuracy is generally the strongest component, whereas structure is the weakest and most often limiting; happiness is shown in Figure 2(a) and movement intervals in Table 9 . Fine-tuned conditions are labelled German ( $\mathcal{F}_{\text{de}}$ ) and Mexican ( $\mathcal{F}_{\text{mx}}$ ).

Question	$S_{\mathrm{acc}}$	$S_{\mathrm{adapt}}$	$S_{\mathrm{struct}}$	PFS	$S_{\mathrm{center}}$	$\rho$ (PFS)
Happiness	0.001	0.013	0.015	0.019	0.001	0.960
Political ideology	0.001	0.008	0.017	0.027	0.001	0.920
Religious attendance	0.002	0.008	0.007	0.029	0.002	0.922
Social trust	0.002	0.021	0.012	0.045	0.002	0.926
All questions	0.002	0.013	0.013	0.031	0.002	0.972

Appendix

Table 9: Run-to-run uncertainty across repeated FA runs. Entries are pooled 95% CI half-widths,

h_{\mathrm{pool}}

, across the condition–question pairs in each row. The final column reports the mean of the three pairwise Spearman rank correlations in PFS between repeated runs; for “All questions,” correlations are computed after pooling the condition–question observations across the four questions. Smaller CI half-widths and larger rank correlations indicate greater stability across repeated elicitation.

Panel of Figure 16 — Figure 16: Population Fidelity and center alignment for five Qwen3-VL-2B-Thinking conditions: as released, $\mathcal{F}_{\text{de}}$ , $\mathcal{F}_{\text{mx}}$ , distribution-matched, and subgroup-matched. (a,b) Component profiles on all retained cells, German cells, and Mexican cells for social trust under NTP and happiness under FA, respectively. Diamonds show PFS and hollow squares center alignment. (c,d) Adapted minus as-released changes on all retained cells for the same two settings.

Explore similar work

Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data
Jun 27, 2026Eun Cheol Choi, Youngrae Kim, Prabhu Pugalenthi +2Survey Biases
University of Southern California, CA, USA.
PALMs: Using Multi Construct-Grounded Rationales for Modeling Population Preferences in LLMs
Aug 2, 2026Priyanka Dey, Brihi Joshi, Preyashi Poddar +2Large Language Model Alignment Social Reasoning
University of Southern California · Information Sciences Institute
Tastes without distinction: silicon samples and the synthetic construction of tastes
Jun 29, 2026Xiangyu Ma, Mengmi Zhang, Shannon Ang +1Large Language Model Bias Survey
Nanyang Technological University, Singapore

Jun 27, 2026, cs.CLJ/K move · Enter open · S save

Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data

Eun Cheol Choi, Youngrae Kim, Prabhu Pugalenthi, Hong-En Chen+1

University of Southern California, CA, USA.

Survey Biases Responses Fidelity Value Alignment Misinformation+1

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to simulate social survey responses, yet their outputs exhibit systematic biases: marginal distributions are skewed, response variance is poorly calibrated, and predictor-outcome relationships are attenuated. We ask a simple question: given a small pilot sample of human responses, can an LLM recover the statistical characteristics of a broader population? We decompose recovery along three axes: structural fidelity, marginal fidelity, and individual fidelity. Using a COVID-19 misinformation survey as a case study, we benchmark three families of approaches: prompting, rectification, and fine-tuning. The findings suggest that fine-tuning on small pilot samples offers a balanced approach for achieving multiple forms of fidelity, but the levels of such fidelity can vary across subsamples, potentially threatening pluralistic alignment.

Abstract

Explore similar work

Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data

PALMs: Using Multi Construct-Grounded Rationales for Modeling Population Preferences in LLMs

Tastes without distinction: silicon samples and the synthetic construction of tastes