Validating LLM-generated political networks with survey-reported partisan exposure

Marco Tonin; Michele Tizzoni; Bruno Lepri; Luca Maria Aiello

cs.SIOct 16, 2026

Validating LLM-generated political networks with survey-reported partisan exposure

Authors: Marco Tonin, Michele Tizzoni, Bruno Lepri, Luca Maria Aiello

Organizations: Department of Sociology and Social Research, University of Trento, Via Verdi 26, Trento, I-38122, Italy. · Mobile and Social Computing Lab (MobS), Fondazione Bruno Kessler, Via Sommarive 18, Trento, I-38123, Italy. · Data Science Section, IT University of Copenhagen, Rued Langgaards Vej 7, Copenhagen, DK-2300, Denmark.

Abstract

Large language models (LLMs) are increasingly used to simulate human behavior in social science research, including decisions, interactions, and social relationships. However, in network formation tasks, existing evaluations often focus on aggregate network properties, while networks with plausible structural properties may still fail to reproduce individuals' social and political environments. Here, we generate LLM-based political networks using personas sampled from the nationally representative ANES 2020--2022 Social Media Study and use respondents' survey-reported partisan exposure as an empirical benchmark for the political environments generated for the corresponding personas. We first characterize the structural properties and political assortativity of the generated networks, and then compare ego-level partisan exposure with the survey. We find that the structure and political assortativity of the generated networks vary across experimental configurations. More importantly, when comparing generated partisan exposure with the survey, some models generate stronger separation between Democratic and Republican egos than reported in the survey, with higher same-party and lower cross-party exposure, while others mainly underestimate Republican exposure across ego groups. Moreover, networks with similar political assortativity can still generate different partisan environments around the same personas, including among networks with plausible structural properties. Finally, generated networks show group-specific errors, under- or overestimating exposure to either Democrats or Republicans for specific social groups. These findings show that evaluating LLM-based social simulations only through aggregate outcomes can hide errors in the local environments through which those outcomes emerge, highlighting the importance of validation at both aggregate and individual levels.

Figures & tables

Figure 1: Workflow for generating and evaluating LLM-based social networks. (a) We sample respondents from the ANES 2020–2022 Social Media Study and construct personas using their demographic and political attributes. The survey also provides self-reported exposure to Democratic and Republican friends or family on a 1–5 scale, which is used as an empirical benchmark. (b) We generate friendship nominations using pairwise and list-based prompting methods, asking each persona to select potential friends. (c) From these choices, we construct two network representations: a directed nomination network containing all ego-to-alter friendship choices, and an undirected reciprocal network containing only mutually selected ties. (d) We evaluate the generated networks along three dimensions: structural plausibility, by comparing reciprocal networks with empirical friendship-network benchmarks; survey-based partisan exposure, by comparing generated ego-level exposure with the corresponding ANES measure; and assortativity–exposure consistency, by assessing whether networks with similar political assortativity generate similar partisan exposure for the same personas.

Figure 2: Network metrics vary across models and prompting techniques. Prompt versions contain an increasing number of candidate friends, from one (c1) to fifty (c50), and differ in the ordering of persona attributes, mentioning the presidential vote first or last (circle vs. triangle markers). Larger colored points show model averages across attribute orders and runs ( $10$ per configuration), and error bars indicate standard errors. Results are shown for temperature $T=0.1$ . The gray area represents the plausible values of the metrics according to the empirical benchmarks. (a) Structural network metrics vary most under the pairwise prompt (c1), both across models and, for some models, across runs and attribute orderings. Among the list-based prompts, larger candidate sets produce sparser networks with lower clustering and longer shortest paths, while networks remain almost entirely connected. Modularity also tends to increase with larger candidate sets, although this pattern varies across models. (b) Assortativity is limited for demographic attributes, although increasing with the number of candidates in the prompt, whereas 2020 presidential vote shows stronger assortative mixing and larger differences across models and prompting configurations.

Figure 3: Distance between partisan exposure in LLM-generated networks and ANES survey using the nomination networks. (a) Mean Signed Deviation (MSD) between partisan exposure in the LLM-generated networks and in the ANES survey across models and experimental configurations. Mistral 7B and Qwen2.5 7B tend to underestimate Republican exposure, while the two Llama models and Qwen2.5 72B to overestimate same-party exposure and underestimate cross-party exposure; (b) Cumulative distribution functions of the average Democratic and Republican exposure across models compared to the survey measure. Here, lines below the survey one suggest an overestimation of the exposure, while lines above the survey measure represent an underestimation.

Figure 4: Exposure differences among networks with similar political assortativity. Mean absolute within-person differences in partisan exposure between pairs of LLM-generated networks from the same model whose vote assortativity differs by less than

0.02

. For each pair, exposure is compared for the same personas on the 1–5 exposure scale. Results are shown using the nomination networks (a) and the reciprocal networks that meet all five plausibility criteria (b) . Boxes show the interquartile range, center lines indicate medians, and whiskers the 5th–95th percentiles. Networks can differ in prompting configuration.

Figure 5: Group-specific errors in partisan exposure in LLM-generated networks. Mean Signed Deviation (MSD) between LLM-generated and survey-reported exposure to (a) Democrats and (b) Republicans for the social groups with the largest deviations. Democratic exposure shows group-specific patterns of under- and overestimation, while Republican exposure is mainly underestimated. Different point styles represent the different prompting methods, with larger markers showing the model average. Results are shown for

T=0.1

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Characteristic	Category	Weighted ANES (%)	$N=100$	Abs. diff.
Gender	Female	51.7	51.0	0.7
	Male	48.3	49.0	0.7
Age	18–29	19.8	21.0	1.2
	30–44	25.6	26.0	0.4
	45–59	24.6	24.0	0.6
	60–74	23.5	22.0	1.5

Appendix

Table S1: Demographic and political composition of the reduced ANES sample. Comparison between the weighted distribution of eligible respondents in the ANES 2020–2022 Social Media Study and the

N=100

sample used to construct the personas. The last column reports the absolute difference in percentage points.

Figure S1: Network metrics vary across models and prompting techniques for reciprocal networks at $T=0.7$ . The pairwise configuration is indicated by c1, while c5, c10, c25, and c50 indicate the number of candidate alters in each list-based prompt. Markers represent individual generated networks, with circles and triangles indicating whether vote was positioned last or first in the persona description. Larger colored points show the models’ averages across attribute orders and runs ( $10$ for each configuration), and error bars indicate standard errors. The gray areas in panel a represent the central $95\%$ empirical reference ranges derived from the friendship and friendship-like network benchmarks. Panel (a) reports structural network measures, while panel (b) reports assortativity by demographic and political attributes.

Figure S2: Network metrics across models and prompting techniques for nomination networks. Results are shown for $T=0.1$ (a) and $T=0.7$ (b) . Prompt versions contain an increasing number of candidate friends, from one (c1) to fifty (c50), and differ in the ordering of persona attributes, with presidential vote presented first or last. Larger colored points show model averages across attribute orders and runs, and error bars indicate standard errors.

Model	$T$	Method	Order	Average degree	Clustering	LCC	Normalized path	Modularity
Llama-3.1 8B	0.1	c1	Vote last	36.742 (1.127)	0.717 (0.002)	1.000 (0.000)	0.369 (0.006)	0.342 (0.006)
Llama-3.1 8B	0.1	c1	Vote first	51.538 (1.604)	0.760 (0.006)	1.000 (0.000)	0.324 (0.004)	0.156 (0.007)
Llama-3.1 8B	0.1	c5	Vote last	24.880 (0.094)	0.598 (0.001)	1.000 (0.000)	0.433 (0.001)	0.418 (0.001)
Llama-3.1 8B	0.1	c5	Vote first	22.624 (0.103)	0.557 (0.001)	1.000 (0.000)	0.443 (0.002)	0.451 (0.002)
Llama-3.1 8B	0.1	c10	Vote last	15.118 (0.053)	0.471 (0.002)	0.970 (0.003)	0.536 (0.001)	0.404 (0.002)
Llama-3.1 8B	0.1	c10	Vote first	14.030 (0.035)	0.510 (0.002)	0.987 (0.002)	0.569 (0.002)	0.490 (0.001)

Appendix

Table S2: Structural properties of the reciprocal LLM-generated networks. Values report the mean across 10 runs for each configuration, with standard errors in parentheses.

Model	$T$	Method	Order	Gender	Race/ethnicity	Education	Religion	2020 vote
Llama-3.1 8B	0.1	c1	Vote last	-0.015 (0.001)	0.050 (0.001)	0.061 (0.001)	0.098 (0.002)	0.578 (0.016)
Llama-3.1 8B	0.1	c1	Vote first	0.044 (0.001)	0.074 (0.001)	0.049 (0.003)	0.099 (0.004)	0.215 (0.015)
Llama-3.1 8B	0.1	c5	Vote last	0.016 (0.001)	0.061 (0.001)	0.073 (0.001)	0.136 (0.001)	0.690 (0.001)
Llama-3.1 8B	0.1	c5	Vote first	0.041 (0.001)	0.102 (0.001)	0.075 (0.001)	0.144 (0.001)	0.734 (0.003)
Llama-3.1 8B	0.1	c10	Vote last	-0.022 (0.003)	0.183 (0.001)	0.117 (0.001)	0.198 (0.001)	0.770 (0.002)
Llama-3.1 8B	0.1	c10	Vote first	0.015 (0.003)	0.188 (0.001)	0.140 (0.002)	0.222 (0.001)	0.838 (0.002)

Appendix

Table S3: Attribute assortativity in the reciprocal LLM-generated networks. Values report the mean across 10 runs for each configuration, with standard errors in parentheses.

Model	$T$	Method	Order	Average degree	Clustering	LCC	Normalized path	Modularity
Llama-3.1 8B	0.1	c1	Vote last	53.338 (1.311)	0.733 (0.002)	1.000 (0.000)	0.318 (0.003)	0.225 (0.009)
Llama-3.1 8B	0.1	c1	Vote first	68.656 (1.606)	0.834 (0.006)	1.000 (0.000)	0.284 (0.004)	0.092 (0.005)
Llama-3.1 8B	0.1	c5	Vote last	51.862 (0.124)	0.674 (0.001)	1.000 (0.000)	0.321 (0.000)	0.255 (0.001)
Llama-3.1 8B	0.1	c5	Vote first	49.616 (0.131)	0.659 (0.001)	1.000 (0.000)	0.325 (0.000)	0.258 (0.001)
Llama-3.1 8B	0.1	c10	Vote last	37.422 (0.095)	0.606 (0.001)	1.000 (0.000)	0.352 (0.000)	0.286 (0.001)
Llama-3.1 8B	0.1	c10	Vote first	34.936 (0.090)	0.605 (0.001)	1.000 (0.000)	0.359 (0.000)	0.330 (0.001)

Appendix

Table S4: Structural properties of the nomination LLM-generated networks. Values report the mean across 10 runs for each configuration, with standard errors in parentheses.

Model	$T$	Method	Order	Gender	Race/ethnicity	Education	Religion	2020 vote
Llama-3.1 8B	0.1	c1	Vote last	-0.033 (0.002)	0.022 (0.001)	0.039 (0.002)	0.100 (0.001)	0.345 (0.015)
Llama-3.1 8B	0.1	c1	Vote first	0.031 (0.000)	0.038 (0.001)	0.013 (0.002)	0.053 (0.004)	0.094 (0.010)
Llama-3.1 8B	0.1	c5	Vote last	0.010 (0.001)	0.034 (0.000)	0.043 (0.001)	0.074 (0.001)	0.362 (0.002)
Llama-3.1 8B	0.1	c5	Vote first	0.001 (0.001)	0.039 (0.001)	0.041 (0.001)	0.090 (0.001)	0.384 (0.002)
Llama-3.1 8B	0.1	c10	Vote last	0.008 (0.001)	0.082 (0.001)	0.066 (0.001)	0.116 (0.001)	0.457 (0.001)
Llama-3.1 8B	0.1	c10	Vote first	0.015 (0.001)	0.095 (0.002)	0.061 (0.001)	0.117 (0.001)	0.532 (0.002)

Appendix

Table S5: Attribute assortativity in the nomination LLM-generated networks. Values report the mean across 10 runs for each configuration, with standard errors in parentheses.

Dataset	No. networks	$N$	Relation	Reference
Baerveldt Dutch Social Behavior	38	31–91	Emotional support	Houtzager and Baerveldt (1999)
Galesburg	1	31	Friendship	Coleman et al. (1957)
Hi-tech	1	36	Friendship	Krackhardt (1999)
High School 2013	1	134	Friendship	Mastrandrea et al. (2015)
Tailor	2	39	Sociational	Kapferer (1972)
Knecht class 12b	4	26	Friendship	Knecht et al. (2010)

Appendix

Table S6: Empirical friendship and friendship-like networks used to define the structural reference ranges. The benchmark contains 57 networks from 13 datasets. For datasets containing multiple networks, the number of networks and range of network sizes are reported. Values of

N

are derived from the graph objects used in the analysis; the High School 2013 graph is edge-induced, and the MSW Students graphs retain the full 145-student roster. Network-level observations are weighted so that each dataset contributes equal total weight when estimating the empirical reference ranges.

Metric	2.5th percentile	97.5th percentile
Average degree	0.679	11.436
Average clustering	0.000	0.571
Nodes in LCC	0.106	1.000
Normalized shortest path	0.484	2.141
Modularity	0.212	0.881

Appendix

Table S7: Empirical reference ranges used to assess structural plausibility. Ranges correspond to the weighted 2.5th and 97.5th percentiles across the 57 empirical networks, with equal total weight assigned to each of the 13 datasets.

Figure S3: Distance between partisan exposure in LLM-generated networks and ANES survey using reciprocal networks. Panel (a) reports the Mean Signed Deviation (MSD) between generated and survey-reported exposure across models and experimental configurations. Positive values indicate overestimation relative to the ANES survey, while negative values indicate underestimation. Panel (b) reports the corresponding cumulative distributions of partisan exposure for Democratic and Republican egos.

Model	T	Method	Order	Ego	Dem. exposure	Rep. exposure
Llama-3.1 8B	0.1	c1	First	Democratic	-0.167 (0.048)	-0.316 (0.025)
Llama-3.1 8B	0.1	c1	First	Republican	-0.126 (0.045)	-0.335 (0.037)
Llama-3.1 8B	0.1	c1	First	Other	0.318 (0.015)	-0.723 (0.005)
Llama-3.1 8B	0.1	c1	Last	Democratic	0.588 (0.058)	-1.102 (0.006)
Llama-3.1 8B	0.1	c1	Last	Republican	-1.144 (0.032)	0.591 (0.065)
Llama-3.1 8B	0.1	c1	Last	Other	-0.241 (0.034)	-1.245 (0.007)

Appendix

Table S8: Mean Signed Deviation between LLM-generated and survey-reported partisan exposure in nomination networks. Values report the unweighted mean across 10 runs for each experimental configuration, with standard errors in parentheses. Positive values indicate overestimation relative to the corresponding ANES personas, while negative values indicate underestimation. Order indicates whether vote was presented first or last in the persona description.

Model	T	Method	Order	Ego	Dem. exposure	Rep. exposure
Llama-3.1 8B	0.1	c1	First	Democratic	0.247 (0.067)	-0.516 (0.034)
Llama-3.1 8B	0.1	c1	First	Republican	-0.091 (0.035)	-0.106 (0.043)
Llama-3.1 8B	0.1	c1	First	Other	0.305 (0.021)	-0.718 (0.034)
Llama-3.1 8B	0.1	c1	Last	Democratic	1.153 (0.028)	-1.112 (0.003)
Llama-3.1 8B	0.1	c1	Last	Republican	-1.244 (0.016)	1.371 (0.036)
Llama-3.1 8B	0.1	c1	Last	Other	-0.105 (0.052)	-1.200 (0.012)

Appendix

Table S9: Mean Signed Deviation between LLM-generated and survey-reported partisan exposure in reciprocal networks. Values report the unweighted mean across 10 runs for each experimental configuration, with standard errors in parentheses. Positive values indicate overestimation relative to the corresponding ANES personas, while negative values indicate underestimation. Order indicates whether vote was presented first or last in the persona description.

Figure S4: Mean absolute within-person differences in partisan exposure between pairs of LLM-generated networks from the same model whose vote assortativity differs by less than 0.02 and 0.05. Results are shown for nomination networks (a) and for reciprocal networks meeting all five structural plausibility criteria (b) . For each pair, exposure is compared for the same personas on the 1–5 exposure scale. Boxes show the interquartile range, center lines indicate medians, and whiskers the 5th–95th percentiles.

Model	$\|\Delta r_{\mathrm{vote}}\|<0.02$	$\|\Delta r_{\mathrm{vote}}\|<0.05$	$\|\Delta r_{\mathrm{vote}}\|<0.02$	$\|\Delta r_{\mathrm{vote}}\|<0.05$
	Nomination networks—all		Reciprocal networks—plausible
Llama-3.1 8B	3,092	4,652	484	1,100
Mistral 7B	2,804	6,534	0	0
Qwen2.5 7B	4,221	7,818	1,019	2,352
Llama-3.1 70B AWQ	3,039	3,729	1,723	2,767
Qwen2.5 72B AWQ	4,300	6,124	166	415

Appendix

Table S10: Number of matched network pairs with similar political assortativity. Number of eligible pairs of LLM-generated networks from the same model under the two assortativity thresholds used in the matching analysis. Nomination-network counts use all generated networks, whereas reciprocal-network counts use only networks meeting all five structural plausibility criteria. Matched networks differ in experimental configuration.

Figure S5: Group-specific errors in partisan exposure in LLM-generated networks at

T=0.7

. Mean Signed Deviation (MSD) between LLM-generated and survey-reported exposure to (a) Democrats and (b) Republicans for the social groups with the largest deviations. Negative values indicate underestimation relative to the survey, while positive values indicate overestimation. Different point styles represent the different prompting methods, with larger markers showing the model average.

Model	$T$	Attribute	Group	$N$	Dem. exposure	Rep. exposure
Llama-3.1 8B	0.1	Race/ethnicity	black	9	0.224 (0.176)	-0.109 (0.079)
Llama-3.1 8B	0.7	Race/ethnicity	black	9	0.218 (0.163)	-0.093 (0.057)
Mistral 7B	0.1	Race/ethnicity	black	9	-0.953 (0.107)	0.477 (0.099)
Mistral 7B	0.7	Race/ethnicity	black	9	-0.937 (0.106)	0.469 (0.094)
Qwen2.5 7B	0.1	Race/ethnicity	black	9	-0.719 (0.081)	0.197 (0.063)
Qwen2.5 7B	0.7	Race/ethnicity	black	9	-0.737 (0.069)	0.212 (0.044)

Appendix

Table S11: Complete group-specific errors in LLM-generated partisan exposure. Mean Signed Deviation (MSD) between LLM-generated and survey-reported exposure to Democrats and Republicans in nomination networks for all demographic and political groups with at least five personas. Values report model averages across prompting methods and attribute orderings separately for each temperature, with standard errors across configurations in parentheses. Positive values indicate overestimation relative to the ANES survey, while negative values indicate underestimation.

Large Language Model-Based Social Simulation

Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.

See all 257 papers

From Caricature to Collective Artifact: How Persona Construction Drives Polarization in Generative Social Simulations
Oct 12, 2026Dimitris Michailidis, Petter Törnberg, Fernando P. SantosLarge Language Model-Based Social Simulation Computational Social Science
Socially Intelligent Artificial Systems, University of Amsterdam · Institute for Logic, Language and Computation, University of Amsterdam
When Do LLMs Generate Realistic Social Networks? A Multi-Dimensional Study of Culture, Language, Scale, and Method
May 13, 2026Sai Hemanth Kilaru, Sriram Theerdh Manikyala, Raghav Upadhyay +3LLM Prompting Social Bias in Language Models
University of Arizona. Tucson, AZ 85721
Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study
Aug 12, 2026Simone MungariLLM Auditing Political Bias in Language Models
Revelis s.r.l.

Oct 12, 2026J/K move · Enter open · S save

From Caricature to Collective Artifact: How Persona Construction Drives Polarization in Generative Social Simulations

Dimitris Michailidis, Petter Törnberg, Fernando P. Santos

Socially Intelligent Artificial Systems, University of Amsterdam · Institute for Logic, Language and Computation, University of Amsterdam

Large Language Model-Based Social Simulation Computational Social Science Persona-Conditioned Generation Opinion Dynamics

Open paper AskPDF arXiv

Generative Social Simulation (GSS) leverages Large Language Models (LLMs) to create sophisticated personas that define agents with demographics, emotions, and political opinions. While prior work shows that personas compress outputs into stereotypical caricatures, there is no comprehensive study of how these caricatures evolve into collective artifacts when they interact. Using a Twitter-like GSS platform with personas derived from the American National Elections Study (ANES), we ablate three layers of partisan identity: affective polarization markers, party identification, and vote choice, while retaining demographics, ideological positioning and important issues. We find that collective outcomes depend heavily on how survey responses are translated into personas. Removing a few explicitly partisan statements moves the network from near-complete segregation to cross-party ties approaching those of real Twitter networks. We argue that matching one outcome is not evidence for behavioral validity and point to salience as a possible explanation. Surveys ask for a position on every question regardless of its importance to the respondent, and persona construction turns each answer into a direct behavioral instruction. We conclude with a checklist for more comprehensive evaluation of personas in future works.

Abstract

Large Language Model-Based Social Simulation

Explore similar work

From Caricature to Collective Artifact: How Persona Construction Drives Polarization in Generative Social Simulations

When Do LLMs Generate Realistic Social Networks? A Multi-Dimensional Study of Culture, Language, Scale, and Method

Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study