Validating LLM-generated political networks with survey-reported partisan exposure
Authors: Marco Tonin, Michele Tizzoni, Bruno Lepri, Luca Maria Aiello
Organizations: Department of Sociology and Social Research, University of Trento, Via Verdi 26, Trento, I-38122, Italy. · Mobile and Social Computing Lab (MobS), Fondazione Bruno Kessler, Via Sommarive 18, Trento, I-38123, Italy. · Data Science Section, IT University of Copenhagen, Rued Langgaards Vej 7, Copenhagen, DK-2300, Denmark.
Large language models (LLMs) are increasingly used to simulate human behavior in social science research, including decisions, interactions, and social relationships. However, in network formation tasks, existing evaluations often focus on aggregate network properties, while networks with plausible structural properties may still fail to reproduce individuals' social and political environments. Here, we generate LLM-based political networks using personas sampled from the nationally representative ANES 2020--2022 Social Media Study and use respondents' survey-reported partisan exposure as an empirical benchmark for the political environments generated for the corresponding personas. We first characterize the structural properties and political assortativity of the generated networks, and then compare ego-level partisan exposure with the survey. We find that the structure and political assortativity of the generated networks vary across experimental configurations. More importantly, when comparing generated partisan exposure with the survey, some models generate stronger separation between Democratic and Republican egos than reported in the survey, with higher same-party and lower cross-party exposure, while others mainly underestimate Republican exposure across ego groups. Moreover, networks with similar political assortativity can still generate different partisan environments around the same personas, including among networks with plausible structural properties. Finally, generated networks show group-specific errors, under- or overestimating exposure to either Democrats or Republicans for specific social groups. These findings show that evaluating LLM-based social simulations only through aggregate outcomes can hide errors in the local environments through which those outcomes emerge, highlighting the importance of validation at both aggregate and individual levels.
Figures & tables
Figure 1: Workflow for generating and evaluating LLM-based social networks. (a) We sample respondents from the ANES 2020–2022 Social Media Study and construct personas using their demographic and political attributes. The survey also provides self-reported exposure to Democratic and Republican friends or family on a 1–5 scale, which is used as an empirical benchmark. (b) We generate friendship nominations using pairwise and list-based prompting methods, asking each persona to select potential friends. (c) From these choices, we construct two network representations: a directed nomination network containing all ego-to-alter friendship choices, and an undirected reciprocal network containing only mutually selected ties. (d) We evaluate the generated networks along three dimensions: structural plausibility, by comparing reciprocal networks with empirical friendship-network benchmarks; survey-based partisan exposure, by comparing generated ego-level exposure with the corresponding ANES measure; and assortativity–exposure consistency, by assessing whether networks with similar political assortativity generate similar partisan exposure for the same personas.
Figure 2: Network metrics vary across models and prompting techniques. Prompt versions contain an increasing number of candidate friends, from one (c1) to fifty (c50), and differ in the ordering of persona attributes, mentioning the presidential vote first or last (circle vs. triangle markers). Larger colored points show model averages across attribute orders and runs ( 10 per configuration), and error bars indicate standard errors. Results are shown for temperature T=0.1 . The gray area represents the plausible values of the metrics according to the empirical benchmarks. (a) Structural network metrics vary most under the pairwise prompt (c1), both across models and, for some models, across runs and attribute orderings. Among the list-based prompts, larger candidate sets produce sparser networks with lower clustering and longer shortest paths, while networks remain almost entirely connected. Modularity also tends to increase with larger candidate sets, although this pattern varies across models. (b) Assortativity is limited for demographic attributes, although increasing with the number of candidates in the prompt, whereas 2020 presidential vote shows stronger assortative mixing and larger differences across models and prompting configurations.
Figure 3: Distance between partisan exposure in LLM-generated networks and ANES survey using the nomination networks. (a) Mean Signed Deviation (MSD) between partisan exposure in the LLM-generated networks and in the ANES survey across models and experimental configurations. Mistral 7B and Qwen2.5 7B tend to underestimate Republican exposure, while the two Llama models and Qwen2.5 72B to overestimate same-party exposure and underestimate cross-party exposure; (b) Cumulative distribution functions of the average Democratic and Republican exposure across models compared to the survey measure. Here, lines below the survey one suggest an overestimation of the exposure, while lines above the survey measure represent an underestimation.
Figure 4: Exposure differences among networks with similar political assortativity. Mean absolute within-person differences in partisan exposure between pairs of LLM-generated networks from the same model whose vote assortativity differs by less than 0.02 . For each pair, exposure is compared for the same personas on the 1–5 exposure scale. Results are shown using the nomination networks (a) and the reciprocal networks that meet all five plausibility criteria (b) . Boxes show the interquartile range, center lines indicate medians, and whiskers the 5th–95th percentiles. Networks can differ in prompting configuration.
Figure 5: Group-specific errors in partisan exposure in LLM-generated networks. Mean Signed Deviation (MSD) between LLM-generated and survey-reported exposure to (a) Democrats and (b) Republicans for the social groups with the largest deviations. Democratic exposure shows group-specific patterns of under- and overestimation, while Republican exposure is mainly underestimated. Different point styles represent the different prompting methods, with larger markers showing the model average. Results are shown for T=0.1 .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Characteristic
Category
Weighted ANES (%)
N=100
Abs. diff.
Gender
Female
51.7
51.0
0.7
Male
48.3
49.0
0.7
Age
18–29
19.8
21.0
1.2
30–44
25.6
26.0
0.4
45–59
24.6
24.0
0.6
60–74
23.5
22.0
1.5
Appendix
Table S1: Demographic and political composition of the reduced ANES sample. Comparison between the weighted distribution of eligible respondents in the ANES 2020–2022 Social Media Study and the N=100 sample used to construct the personas. The last column reports the absolute difference in percentage points.
Figure S1: Network metrics vary across models and prompting techniques for reciprocal networks at T=0.7 . The pairwise configuration is indicated by c1, while c5, c10, c25, and c50 indicate the number of candidate alters in each list-based prompt. Markers represent individual generated networks, with circles and triangles indicating whether vote was positioned last or first in the persona description. Larger colored points show the models’ averages across attribute orders and runs ( 10 for each configuration), and error bars indicate standard errors. The gray areas in panel a represent the central 95% empirical reference ranges derived from the friendship and friendship-like network benchmarks. Panel (a) reports structural network measures, while panel (b) reports assortativity by demographic and political attributes.
Figure S2: Network metrics across models and prompting techniques for nomination networks. Results are shown for T=0.1 (a) and T=0.7 (b) . Prompt versions contain an increasing number of candidate friends, from one (c1) to fifty (c50), and differ in the ordering of persona attributes, with presidential vote presented first or last. Larger colored points show model averages across attribute orders and runs, and error bars indicate standard errors.
Model
T
Method
Order
Average degree
Clustering
LCC
Normalized path
Modularity
Llama-3.1 8B
0.1
c1
Vote last
36.742 (1.127)
0.717 (0.002)
1.000 (0.000)
0.369 (0.006)
0.342 (0.006)
Llama-3.1 8B
0.1
c1
Vote first
51.538 (1.604)
0.760 (0.006)
1.000 (0.000)
0.324 (0.004)
0.156 (0.007)
Llama-3.1 8B
0.1
c5
Vote last
24.880 (0.094)
0.598 (0.001)
1.000 (0.000)
0.433 (0.001)
0.418 (0.001)
Llama-3.1 8B
0.1
c5
Vote first
22.624 (0.103)
0.557 (0.001)
1.000 (0.000)
0.443 (0.002)
0.451 (0.002)
Llama-3.1 8B
0.1
c10
Vote last
15.118 (0.053)
0.471 (0.002)
0.970 (0.003)
0.536 (0.001)
0.404 (0.002)
Llama-3.1 8B
0.1
c10
Vote first
14.030 (0.035)
0.510 (0.002)
0.987 (0.002)
0.569 (0.002)
0.490 (0.001)
Appendix
Table S2: Structural properties of the reciprocal LLM-generated networks. Values report the mean across 10 runs for each configuration, with standard errors in parentheses.
Model
T
Method
Order
Gender
Race/ethnicity
Education
Religion
2020 vote
Llama-3.1 8B
0.1
c1
Vote last
-0.015 (0.001)
0.050 (0.001)
0.061 (0.001)
0.098 (0.002)
0.578 (0.016)
Llama-3.1 8B
0.1
c1
Vote first
0.044 (0.001)
0.074 (0.001)
0.049 (0.003)
0.099 (0.004)
0.215 (0.015)
Llama-3.1 8B
0.1
c5
Vote last
0.016 (0.001)
0.061 (0.001)
0.073 (0.001)
0.136 (0.001)
0.690 (0.001)
Llama-3.1 8B
0.1
c5
Vote first
0.041 (0.001)
0.102 (0.001)
0.075 (0.001)
0.144 (0.001)
0.734 (0.003)
Llama-3.1 8B
0.1
c10
Vote last
-0.022 (0.003)
0.183 (0.001)
0.117 (0.001)
0.198 (0.001)
0.770 (0.002)
Llama-3.1 8B
0.1
c10
Vote first
0.015 (0.003)
0.188 (0.001)
0.140 (0.002)
0.222 (0.001)
0.838 (0.002)
Appendix
Table S3: Attribute assortativity in the reciprocal LLM-generated networks. Values report the mean across 10 runs for each configuration, with standard errors in parentheses.
Model
T
Method
Order
Average degree
Clustering
LCC
Normalized path
Modularity
Llama-3.1 8B
0.1
c1
Vote last
53.338 (1.311)
0.733 (0.002)
1.000 (0.000)
0.318 (0.003)
0.225 (0.009)
Llama-3.1 8B
0.1
c1
Vote first
68.656 (1.606)
0.834 (0.006)
1.000 (0.000)
0.284 (0.004)
0.092 (0.005)
Llama-3.1 8B
0.1
c5
Vote last
51.862 (0.124)
0.674 (0.001)
1.000 (0.000)
0.321 (0.000)
0.255 (0.001)
Llama-3.1 8B
0.1
c5
Vote first
49.616 (0.131)
0.659 (0.001)
1.000 (0.000)
0.325 (0.000)
0.258 (0.001)
Llama-3.1 8B
0.1
c10
Vote last
37.422 (0.095)
0.606 (0.001)
1.000 (0.000)
0.352 (0.000)
0.286 (0.001)
Llama-3.1 8B
0.1
c10
Vote first
34.936 (0.090)
0.605 (0.001)
1.000 (0.000)
0.359 (0.000)
0.330 (0.001)
Appendix
Table S4: Structural properties of the nomination LLM-generated networks. Values report the mean across 10 runs for each configuration, with standard errors in parentheses.
Model
T
Method
Order
Gender
Race/ethnicity
Education
Religion
2020 vote
Llama-3.1 8B
0.1
c1
Vote last
-0.033 (0.002)
0.022 (0.001)
0.039 (0.002)
0.100 (0.001)
0.345 (0.015)
Llama-3.1 8B
0.1
c1
Vote first
0.031 (0.000)
0.038 (0.001)
0.013 (0.002)
0.053 (0.004)
0.094 (0.010)
Llama-3.1 8B
0.1
c5
Vote last
0.010 (0.001)
0.034 (0.000)
0.043 (0.001)
0.074 (0.001)
0.362 (0.002)
Llama-3.1 8B
0.1
c5
Vote first
0.001 (0.001)
0.039 (0.001)
0.041 (0.001)
0.090 (0.001)
0.384 (0.002)
Llama-3.1 8B
0.1
c10
Vote last
0.008 (0.001)
0.082 (0.001)
0.066 (0.001)
0.116 (0.001)
0.457 (0.001)
Llama-3.1 8B
0.1
c10
Vote first
0.015 (0.001)
0.095 (0.002)
0.061 (0.001)
0.117 (0.001)
0.532 (0.002)
Appendix
Table S5: Attribute assortativity in the nomination LLM-generated networks. Values report the mean across 10 runs for each configuration, with standard errors in parentheses.
Dataset
No. networks
N
Relation
Reference
Baerveldt Dutch Social Behavior
38
31–91
Emotional support
Houtzager and Baerveldt (1999)
Galesburg
1
31
Friendship
Coleman et al. (1957)
Hi-tech
1
36
Friendship
Krackhardt (1999)
High School 2013
1
134
Friendship
Mastrandrea et al. (2015)
Tailor
2
39
Sociational
Kapferer (1972)
Knecht class 12b
4
26
Friendship
Knecht et al. (2010)
Appendix
Table S6: Empirical friendship and friendship-like networks used to define the structural reference ranges. The benchmark contains 57 networks from 13 datasets. For datasets containing multiple networks, the number of networks and range of network sizes are reported. Values of N are derived from the graph objects used in the analysis; the High School 2013 graph is edge-induced, and the MSW Students graphs retain the full 145-student roster. Network-level observations are weighted so that each dataset contributes equal total weight when estimating the empirical reference ranges.
Metric
2.5th percentile
97.5th percentile
Average degree
0.679
11.436
Average clustering
0.000
0.571
Nodes in LCC
0.106
1.000
Normalized shortest path
0.484
2.141
Modularity
0.212
0.881
Appendix
Table S7: Empirical reference ranges used to assess structural plausibility. Ranges correspond to the weighted 2.5th and 97.5th percentiles across the 57 empirical networks, with equal total weight assigned to each of the 13 datasets.
Figure S3: Distance between partisan exposure in LLM-generated networks and ANES survey using reciprocal networks. Panel (a) reports the Mean Signed Deviation (MSD) between generated and survey-reported exposure across models and experimental configurations. Positive values indicate overestimation relative to the ANES survey, while negative values indicate underestimation. Panel (b) reports the corresponding cumulative distributions of partisan exposure for Democratic and Republican egos.
Model
T
Method
Order
Ego
Dem. exposure
Rep. exposure
Llama-3.1 8B
0.1
c1
First
Democratic
-0.167 (0.048)
-0.316 (0.025)
Llama-3.1 8B
0.1
c1
First
Republican
-0.126 (0.045)
-0.335 (0.037)
Llama-3.1 8B
0.1
c1
First
Other
0.318 (0.015)
-0.723 (0.005)
Llama-3.1 8B
0.1
c1
Last
Democratic
0.588 (0.058)
-1.102 (0.006)
Llama-3.1 8B
0.1
c1
Last
Republican
-1.144 (0.032)
0.591 (0.065)
Llama-3.1 8B
0.1
c1
Last
Other
-0.241 (0.034)
-1.245 (0.007)
Appendix
Table S8: Mean Signed Deviation between LLM-generated and survey-reported partisan exposure in nomination networks. Values report the unweighted mean across 10 runs for each experimental configuration, with standard errors in parentheses. Positive values indicate overestimation relative to the corresponding ANES personas, while negative values indicate underestimation. Order indicates whether vote was presented first or last in the persona description.
Model
T
Method
Order
Ego
Dem. exposure
Rep. exposure
Llama-3.1 8B
0.1
c1
First
Democratic
0.247 (0.067)
-0.516 (0.034)
Llama-3.1 8B
0.1
c1
First
Republican
-0.091 (0.035)
-0.106 (0.043)
Llama-3.1 8B
0.1
c1
First
Other
0.305 (0.021)
-0.718 (0.034)
Llama-3.1 8B
0.1
c1
Last
Democratic
1.153 (0.028)
-1.112 (0.003)
Llama-3.1 8B
0.1
c1
Last
Republican
-1.244 (0.016)
1.371 (0.036)
Llama-3.1 8B
0.1
c1
Last
Other
-0.105 (0.052)
-1.200 (0.012)
Appendix
Table S9: Mean Signed Deviation between LLM-generated and survey-reported partisan exposure in reciprocal networks. Values report the unweighted mean across 10 runs for each experimental configuration, with standard errors in parentheses. Positive values indicate overestimation relative to the corresponding ANES personas, while negative values indicate underestimation. Order indicates whether vote was presented first or last in the persona description.
Figure S4: Mean absolute within-person differences in partisan exposure between pairs of LLM-generated networks from the same model whose vote assortativity differs by less than 0.02 and 0.05. Results are shown for nomination networks (a) and for reciprocal networks meeting all five structural plausibility criteria (b) . For each pair, exposure is compared for the same personas on the 1–5 exposure scale. Boxes show the interquartile range, center lines indicate medians, and whiskers the 5th–95th percentiles.
Nomination networks—all
Reciprocal networks—plausible
Model
∣Δrvote∣<0.02
∣Δrvote∣<0.05
∣Δrvote∣<0.02
∣Δrvote∣<0.05
Llama-3.1 8B
3,092
4,652
484
1,100
Mistral 7B
2,804
6,534
0
0
Qwen2.5 7B
4,221
7,818
1,019
2,352
Llama-3.1 70B AWQ
3,039
3,729
1,723
2,767
Qwen2.5 72B AWQ
4,300
6,124
166
415
Appendix
Table S10: Number of matched network pairs with similar political assortativity. Number of eligible pairs of LLM-generated networks from the same model under the two assortativity thresholds used in the matching analysis. Nomination-network counts use all generated networks, whereas reciprocal-network counts use only networks meeting all five structural plausibility criteria. Matched networks differ in experimental configuration.
Figure S5: Group-specific errors in partisan exposure in LLM-generated networks at T=0.7 . Mean Signed Deviation (MSD) between LLM-generated and survey-reported exposure to (a) Democrats and (b) Republicans for the social groups with the largest deviations. Negative values indicate underestimation relative to the survey, while positive values indicate overestimation. Different point styles represent the different prompting methods, with larger markers showing the model average.
Model
T
Attribute
Group
N
Dem. exposure
Rep. exposure
Llama-3.1 8B
0.1
Race/ethnicity
black
9
0.224 (0.176)
-0.109 (0.079)
Llama-3.1 8B
0.7
Race/ethnicity
black
9
0.218 (0.163)
-0.093 (0.057)
Mistral 7B
0.1
Race/ethnicity
black
9
-0.953 (0.107)
0.477 (0.099)
Mistral 7B
0.7
Race/ethnicity
black
9
-0.937 (0.106)
0.469 (0.094)
Qwen2.5 7B
0.1
Race/ethnicity
black
9
-0.719 (0.081)
0.197 (0.063)
Qwen2.5 7B
0.7
Race/ethnicity
black
9
-0.737 (0.069)
0.212 (0.044)
Appendix
Table S11: Complete group-specific errors in LLM-generated partisan exposure. Mean Signed Deviation (MSD) between LLM-generated and survey-reported exposure to Democrats and Republicans in nomination networks for all demographic and political groups with at least five personas. Values report model averages across prompting methods and attribute orderings separately for each temperature, with standard errors across configurations in parentheses. Positive values indicate overestimation relative to the ANES survey, while negative values indicate underestimation.
Generative Social Simulation (GSS) leverages Large Language Models (LLMs) to create sophisticated personas that define agents with demographics, emotions, and political opinions. While prior work shows that personas compress outputs into stereotypical caricatures, there is no comprehensive study of how these caricatures evolve into collective artifacts when they interact. Using a Twitter-like GSS platform with personas derived from the American National Elections Study (ANES), we ablate three layers of partisan identity: affective polarization markers, party identification, and vote choice, while retaining demographics, ideological positioning and important issues. We find that collective outcomes depend heavily on how survey responses are translated into personas. Removing a few explicitly partisan statements moves the network from near-complete segregation to cross-party ties approaching those of real Twitter networks. We argue that matching one outcome is not evidence for behavioral validity and point to salience as a possible explanation. Surveys ask for a position on every question regardless of its importance to the respondent, and persona construction turns each answer into a direct behavioral instruction. We conclude with a checklist for more comprehensive evaluation of personas in future works.
Dimitris Michailidis, Petter Törnberg, Fernando P. Santos
Socially Intelligent Artificial Systems, University of Amsterdam · Institute for Logic, Language and Computation, University of Amsterdam
Large language models (LLMs) are increasingly used as substitutes for human subjects in behavioral simulations, including synthetic social network generation. Yet it remains unclear how their relational outputs depend on prompt design, cultural framing, prompt language, and model scale. Building on homophily theory and structural balance theory, we formalize four LLM-based tie-formation mechanisms: sequential, global, local, and iterative, and treat them as distinct conditional distributions over edge sets. Using a fixed roster of 50 demographically grounded personas, we generate 192 verified directed networks across four cultural contexts, four prompt languages, three GPT-4.1 variants, and four prompting architectures, with two seeds per condition. We find that cultural framing shifts inbreeding homophily and largest-component connectivity. Political affiliation dominates tie formation under three methods, while the global method substitutes age, showing that prompt architecture functions as a substantive sociological variable. Model scale produces a stable divergence ranking, with the smallest variant behaving qualitatively differently rather than merely noisily. Prompt language alone sharply shifts religion homophily, especially under Hindi prompting, while leaving political homophily nearly invariant. LLM-generated networks match real social graphs on clustering and modularity better than standard graph baselines, yet encode demographic biases above empirical levels. These results show that prompt choices often treated as implementation details encode substantive sociological assumptions.
Sai Hemanth Kilaru, Sriram Theerdh Manikyala, Raghav Upadhyay +3
As users increasingly turn to Large Language Models (LLMs) for information and advice on political matters, particularly during election periods, the political preferences expressed by these systems have become a matter of public interest. Prior research has shown that interactions with LLMs can influence users' political attitudes and choices, raising questions about how these models themselves evaluate political actors. In this paper, we investigate whether and how LLMs express preferences toward political parties and political leaders. We introduce a systematic and reproducible auditing framework in which multiple LLMs are prompted to evaluate parties and leaders across nine criteria. Rather than attempting to infer the models' "true" political beliefs, we focus on their observable behavior, examining consistency across evaluations, differences between models, refusal rates, and sensitivity to prompt formulation. We further investigate how these evaluations vary when models are instructed to adopt different personas. We demonstrate the framework through an Italian case study, providing a systematic analysis of LLM-generated political evaluations on italian parties and leaders.