cs.SIMar 11, 2026
SaveLLMs Can Infer Political Alignment from Online Conversations
Organizations: School of Data Science, University of Virginia, Charlottesville, Virginia, USA, 22903 · Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering Indiana University, Bloomington, Indiana, USA, 47408 · Division of Communication and Media, Ewha Womans University, Seoul, Republic of Korea, 03760
Abstract
Due to the correlational structure in our traits such as identities, cultures, and political attitudes, seemingly innocuous preferences like following a band or using a specific slang can reveal private traits. This possibility, especially when combined with massive, public social data and advanced computational methods, poses a fundamental privacy risk. As our data exposure online and the rapid advancement of AI are increasing the risk of misuse, it is critical to understand the capacity of large language models (LLMs) to exploit such potential. Here, using online discussions on DebateOrg and Reddit, we show that LLMs can reliably infer hidden political alignment, significantly outperforming traditional machine learning models. Prediction accuracy further improves as we aggregate multiple text-level inferences into a user-level prediction, and as we use more politics-adjacent domains. We demonstrate that LLMs leverage words that are highly predictive of political alignment while not being explicitly political. Our findings underscore the capacity and risks of LLMs for exploiting socio-cultural correlates.
Figures & tables
Figure 1: LLMs can reliably infer political alignment from general conversations. (a) Illustration of the political alignment inference process using LLMs applied to user-generated texts. Text categories can be political (in blue) or general (orange). (b-c) Relationship between average text-level F1 score for the political alignment inference task and LLM confidence scores for (b) DDO and (c) Reddit. Error bars represent standard error estimated via bootstrapping (n = 1,000). The average F1 score increases with the LLM’s confidence in the text-level inference. (d-e) Distribution of confidence for two datasets, (d) DDO and (e) Reddit, as reported by the two LLMs. (f-i) Accuracy of LLMs on the political alignment inference task for DDO and Reddit datasets. User-level F1 scores are calculated for three text contexts: general, political, and combined. Results are presented for two LLMs (GPT-4o and Llama-3.1-8B) across three aggregation methods (see main text): (1) majority vote, (2) confidence-weighted average, and (3) maximum-confidence average. Text-level F1 scores are also shown for comparison. Panels are divided by dataset (DDO and Reddit) and model. Across all data and LLMs, user-level aggregation methods that incorporate LLM confidence scores substantially improve inference accuracy. Among the three methods, the maximum-confidence average tends to yield the highest performance, highlighting the effectiveness of leveraging highly confident predictions during inference. All the differences among aggregation methods are statistically significant based on sample paired t-test obtained by bootstrapping ( ).
Figure 2: Inference accuracy varies systematically across topical categories. (a, b) User-level F1 scores (using the maximum-confidence method) for political alignment inference across categories in (a) DDO and (b) Reddit using GPT-4o and Llama-3.1-8B. As a reference, the gray dotted lines in this and following figures indicate F1=0.5, the expected macro F1 score from a random classifier with balanced classes. (c, d) Correlations of user-level F1 scores by category between GPT-4o and Llama-3.1-8B for (c) DDO and (d) Reddit. Circle size is proportional to the number of observations in each category. (e, f) Correlations of user-level F1 scores by category between DDO and Reddit for (e) GPT-4o and (f) Llama-3.1-8B. Prediction performance varies systematically across categories, independent of dataset or model.
Figure 3: Semantic similarity and user overlap with “Politics” predict higher inference performance across categories. Panels (a–d) show similarity measurements between general categories and the “Politics” category in the DDO dataset, based on (a) content embedding similarity, (b) debate title embedding similarity, (c) Jaccard similarity of user participation between categories, and (d) normalized pointwise mutual information (NPMI) of user participation. Content and title embedding similarities capture semantic similarity between categories, computed as the cosine similarity between category-level average embedding vectors obtained from a pre-trained Sentence-BERT model ( sentence-transformers/all-mpnet-base-v2 [ 46 ] ). In contrast, Jaccard similarity and NPMI reflect user engagement overlap, quantifying the extent to which users participate in both categories. Jaccard similarity ranges from 0 (no shared users) to 1 (identical user sets), and NPMI ranges from (no shared users) to (identical user sets), with zero indicating independence. Panels (e–h) show the same comparisons for the Reddit dataset. We use similarity in average subreddit description embeddings between categories (f) instead of title embeddings. GPT-4o is used for inference in all cases. Overall, categories with greater semantic similarity and user overlap with the “Politics” category tend to yield better inference in predicting political alignment.
Figure 4: Word-level confidence reveals politicization patterns in general-topic categories. (a, b) Six illustrative categories are shown: (a) Economics, Health, and Science from the DDO dataset, and (b) Cars, Entertainment, and Music from the Reddit dataset. Within each category, word clouds are arranged left to right by quintiles of word-level confidence, from lowest (Q1) to highest (Q5). Word-level confidence is defined as the mean confidence, averaged over all texts containing a given word, in inferring political alignment using GPT-4o. Higher confidence indicates stronger politicization signals, contributing more to LLM inference accuracy; even general-topic words in the Q5 clouds carry partisan signals. To visualize lexical distinctiveness, word size reflects the standardized log-odds -score estimated using the Dirichlet prior method [ 47 ] . Within each confidence quintile, we display up to the 100 words with the highest -score values. Word color indicates relative partisan ratio , where red ( ) indicates exclusive usage by Republicans and blue ( ) indicates exclusive usage by Democrats (see Methods). (c, d) Relationship between word-level confidence and mean F1 score for political alignment inference within each category. Each point represents one of 15 confidence quantiles, with shaded areas indicating 95% confidence intervals. Across categories, higher-confidence words consistently yield more accurate inference. These results demonstrate that LLMs effectively capture both implicit and explicit lexical cues relevant to political alignment.
Figure S1: (a) Performance of five annotators who independently inferred each user’s political alignment (Republican / Democrat) from 50 randomly sampled comments. Bars show individual accuracy and macro F1 scores, demonstrating consistently high classification performance across annotators. (b) Comparison between the mean individual accuracy and the majority-vote accuracy against the true labels. The high majority-vote accuracy (0.92) and mean annotator accuracy (0.85) indicate that the heuristic labeling of a user’s political alignment based on community activity is largely consistent with human judgment. (c) Inter-annotator agreement measured by pairwise Cohen’s coefficients, showing a moderate level of consensus among annotators (mean = 0.576).
| Model | Level | Combined | Political | General |
| TF-IDF + NB (Multinomial) | Text | 0.579 0.005 | 0.564 0.015 | 0.583 0.005 |
| User | 0.546 0.009 | 0.535 0.009 | 0.553 0.008 | |
| S-BERT + NB (Gaussian) | Text | 0.544 0.005 | 0.540 0.011 | 0.542 0.005 |
| User | 0.540 0.010 | 0.532 0.006 | 0.532 0.009 | |
| TF-IDF + Logit | Text | 0.615 0.007 | 0.623 0.015 | 0.612 0.008 |
| User | 0.574 0.014 | 0.583 0.011 | 0.577 0.015 |
Table S1: Political alignment inference results on the DDO dataset using traditional machine learning classifiers and LLMs. We report average and standard deviation of macro F1 scores over 5-fold cross-validation, with 80% of the data used for training and 20% for testing in each fold. Text representations include TF-IDF and Sentence-BERT (S-BERT) embeddings, both applied across all traditional classifiers. For LLMs (GPT-4o and Llama-3.1-8B), predictions are made only on the test set from each fold, without access to training data. This approach ensures comparability with the traditional supervised models. Results are reported for three label types (Combined, Political, and General) at both the Text and User levels. For LLMs, user-level scores are aggregated from text-level predictions using different strategies (majority, confidence-weighted, and maximum-confidence), whereas traditional classifiers are evaluated directly at both levels.
| Model | Level | Combined | Political | General |
| TF-IDF + NB (Multinomial) | Text | 0.513 0.004 | 0.595 0.020 | 0.502 0.004 |
| User | 0.465 0.008 | 0.583 0.022 | 0.459 0.008 | |
| S-BERT + NB (Gaussian) | Text | 0.585 0.002 | 0.633 0.005 | 0.579 0.003 |
| User | 0.631 0.008 | 0.640 0.014 | 0.619 0.007 | |
| TF-IDF + Logit | Text | 0.564 0.004 | 0.659 0.007 | 0.553 0.004 |
| User | 0.572 0.008 | 0.661 0.007 | 0.554 0.008 |
Table S2: Political alignment inference results on the Reddit dataset using traditional machine learning classifiers and LLMs. All results are macro F1 scores averaged over 5-fold cross-validation, with 80% of the data used for training and 20% for testing in each fold. Text representations include TF-IDF and Sentence-BERT (S-BERT) embeddings, both applied across all traditional classifiers. For LLMs (GPT-4o and Llama-3.1-8B), predictions are made only on the test set from each fold, without access to training data. This approach ensures comparability with the traditional supervised models. Results are reported for three label types (Combined, Political, and General) at both the Text and User levels. For LLMs, user-level scores are aggregated from text-level predictions using different strategies (majority, confidence-weighted, and maximum-confidence), whereas traditional classifiers are evaluated directly at both levels.
| Model | Level | Combined | Political | General |
| TF-IDF + NB (Multinomial) | Text | 0.556 0.006 | 0.559 0.007 | 0.554 0.008 |
| User | 0.560 0.009 | 0.558 0.012 | 0.557 0.009 | |
| S-BERT + NB (Gaussian) | Text | 0.544 0.006 | 0.544 0.006 | 0.541 0.027 |
| User | 0.545 0.005 | 0.535 0.023 | 0.542 0.004 | |
| TF-IDF + Logit | Text | 0.572 0.005 | 0.574 0.005 | 0.577 0.008 |
| User | 0.578 0.004 | 0.575 0.015 | 0.572 0.010 |
Table S3: Political alignment inference results on the DDO dataset using traditional machine learning classifiers and LLMs, based on an undersampled version of the data. To address concerns about hyperactive users disproportionately influencing the results and potentially leading to overfitting during model training, we conducted an additional experiment in which users with more than 10 posts were limited to a random sample of 10 observations. All results are macro F1 scores averaged over 5-fold cross-validation, with 80% of the data used for training and 20% for testing in each fold. Text representations include TF-IDF and Sentence-BERT (S-BERT) embeddings, both applied across all traditional classifiers. For LLMs (GPT-4o and Llama-3.1-8B), predictions are made only on the test set from each fold, without access to training data. This approach ensures comparability with the traditional supervised models. Results are reported for three label types (Combined, Political, and General) at both the Text and User levels. For LLMs, user-level scores are aggregated from text-level predictions using different strategies (majority, confidence-weighted, and maximum-confidence), whereas traditional classifiers are evaluated directly at both levels.
| Model | Macro F1 score | F1 Weighted |
| TF-IDF + NB (Multinomial) | 0.782 0.010 | 0.829 0.008 |
| S-BERT + NB (Gaussian) | 0.784 0.008 | 0.819 0.008 |
| TF-IDF + Logit | 0.770 0.007 | 0.826 0.005 |
| S-BERT + Logit | 0.794 0.005 | 0.841 0.004 |
| TF-IDF + SVM | 0.768 0.010 | 0.821 0.008 |
| S-BERT + SVM | 0.800 0.006 | 0.845 0.003 |
Table S4: Political label classification results on the DDO dataset using traditional machine learning classifiers. We used data from 50% of the users for model development, reserving the remaining 50% for a separate sensitivity analysis. Macro and weighted F1-scores with S.D. obtained from 5-fold cross-validation, with each fold using an 80/20 train-test split. Text representations include TF-IDF and Sentence-BERT (S-BERT) embeddings, both applied across all traditional classifiers. The best-performing classifier (in bold) was later used to assign political labels to text data in the held-out portion of the dataset.
| Model | Macro F1 score | F1 Weighted |
| TF-IDF + NB (Multinomial) | 0.507 0.008 | 0.875 0.010 |
| S-BERT + NB (Gaussian) | 0.673 0.003 | 0.861 0.005 |
| TF-IDF + Logit | 0.572 0.014 | 0.887 0.010 |
| S-BERT + Logit | 0.631 0.008 | 0.897 0.007 |
| TF-IDF + SVM | 0.612 0.011 | 0.890 0.007 |
| S-BERT + SVM | 0.626 0.007 | 0.897 0.007 |
Table S5: Political label classification results on the Reddit dataset using traditional machine learning classifiers. We used data from 50% of the users for model development, reserving the remaining 50% for a separate sensitivity analysis. Macro and weighted F1-scores with S.D. obtained from 5-fold cross-validation, with each fold using an 80/20 train-test split. Text representations include TF-IDF and Sentence-BERT (S-BERT) embeddings, both applied across all traditional classifiers. The best-performing classifier (in bold) was later used to assign political labels to text data in the held-out portion of the dataset.
Figure S2: Changes in confidence score distributions for general texts before and after filtering for sensitivity analysis, using GPT-4o and Llama-3.1-8B across two test sets (DDO and Reddit). The plots show the distributions of confidence scores assigned by the two models to general texts, both before (blue) and after (orange) filtering out texts identified as explicitly political by a trained classifier. Filtering primarily removes high-confidence texts, particularly at the upper end of the confidence scale (4 or 5), suggesting that the classifier effectively detects political content embedded within general texts.
| Model | Method | Combined | Political | General |
| GPT-4o | Text-level | 0.647 | 0.718 | 0.608 |
| User (majority) | 0.703 | 0.750 | 0.645 | |
| User (confidence-weighted) | 0.735 | 0.769 | 0.673 | |
| User (maximum-confidence) | 0.725 | 0.767 | 0.670 | |
| Llama-3.1-8b | Text-level | 0.637 | 0.690 | 0.615 |
| User (majority) | 0.686 | 0.717 | 0.645 |
Table S6: F1 scores for general texts in the DDO dataset after filtering out potentially political content using the trained classifier.
| Model | Method | Combined | Political | General |
| GPT-4o | Text level | 0.626 | 0.768 | 0.578 |
| User (majority) | 0.731 | 0.842 | 0.613 | |
| User (confidence-weighted) | 0.827 | 0.860 | 0.709 | |
| User (maximum-confidence) | 0.850 | 0.859 | 0.722 | |
| Llama-3.1-8b | Text-level | 0.542 | 0.667 | 0.528 |
| User (majority) | 0.527 | 0.676 | 0.502 |
Table S7: F1 scores for general texts in the Reddit dataset after filtering out potentially political content using the trained classifier.
Figure S3: Distribution of observations across 23 categories in the (a) DDO and (b) Reddit datasets. For Reddit, topic categories of communities (subreddits) were inferred from subreddit descriptions using GPT-4o and mapped to the same 23 categories as in DDO.
| Category | Examples |
| Arts | pics, books, comics, Cooking, DIY, suggestmeabook, woodworking, Infographics, interiordecorating, HomeDecorating, PhotoshopRequest, stories, DesignMyRoom, TattooDesigns, crochet, Tattoocoverups, tattoos, somethingimade, Broadway, Baking |
| Cars | CyberStuck, whatcarshouldIbuy, dashcams, motorcycles, MildlyBadDrivers, InfowarriorRides, electricvehicles, MechanicAdvice, cars, subaru, ToyotaTacoma, RealTesla, regularcarreviews, AskMechanics, BMW, thegrandtour, f150, Cartalk, AskAMechanic, classiccars |
| Economics | FluentInFinance, the_everything_bubble, Economics, wallstreetbets, economicCollapse, povertyfinance, jobs, unusual_whales, personalfinance, economy, FirstTimeHomeBuyer, Money, inflation, REBubble, stocks, Accounting, StockMarket, Bitcoin, austrian_economics, CryptoCurrency |
| Education | todayilearned, coolguides, explainlikeimfive, Teachers, USHistory, careerguidance, answers, Scams, careeradvice, education, AskHistory, howto, hiphop101, duolingo, highschool, WGU, cookingforbeginners, wikipedia, productivity, teaching |
| Entertainment | TikTokCringe, PublicFreakout, BoomersBeingFools, JoeRogan, nextfuckinglevel, OldSchoolCool, BeAmazed, Fauxmoi, StarWars, entertainment, popculturechat, CrazyFuckingVideos, videos, ThatsInsane, oddlysatisfying, CriticalDrinker, TwoHotTakes, NPR, Satisfyingasfuck, aww |
| Fashion | Watches, malefashionadvice, malegrooming, MakeupAddiction, mensfashion, handbags, whatthefrockk, weddingdress, fashion, Makeup, BusinessFashion, mensfashionadvice, EngagementRings, Weddingattireapproval, malehairadvice, beauty, Sneakers, OUTFITS, HairDye, fragrance |
Table S8: Top 20 representative subreddits for each inferred category (23 categories in total), ranked by participation frequency in our dataset. Categories were inferred from subreddit descriptions using GPT-4o following the same category scheme as the DDO dataset.
| Category | Examples |
| Music | Music, TaylorSwift, musicsuggestions, Guitar, hiphopheads, chappellroan, ClassicRock, KendrickLamar, MusicRecommendations, crappymusic, Bass, Eminem, beatles, LinkinPark, popheads, MetalForTheMasses, country, spotify, LetsTalkMusic, Foofighters |
| News | news, AnythingGoesNews, inthenews, worldnews, FOXNEWS, USNewsHub, UkraineWarVideoReport, OutOfTheLoop, UkrainianConflict, UpliftingNews, nytimes, southcarolina, Canada_sub, PBS_NewsHour, anime_titties, Hunting, worldnewsvideo, offbeat, fednews, newzealand |
| People | AskMen, GenX, Asmongold, daddit, Destiny, AskOldPeopleAdvice, elonmusk, AirForce, USMC, SwiftlyNeutral, EntitledPeople, IAmA, GlowUps, Faces, musked, Serverlife, AmazonFC, GenXWomen, kardashians, FamousFaces |
| Philosophy | AmItheAsshole, changemyview, DecodingTheGurus, Anarcho_Capitalism, JordanPeterson, whatif, Life, samharris, SeriousConversation, DeepThoughts, LiminalSpace, FutureWhatIf, Manipulation, ExplainBothSides, InsightfulQuestions, SimulationTheory, likeus, philosophy, TheNightFeeling, AskLibertarians |
| Places-Travel | texas, Ohio, Pennsylvania, minnesota, europe, florida, Michigan, Georgia, wisconsin, travel, ukraine, NatureIsFuckingLit, delta, California, TravelMaps, canada, missouri, Seattle, phoenix, massachusetts |
| Politics | Conservative, politics, PoliticalHumor, KamalaHarris, Presidents, LeopardsAteMyFace, walkaway, AntiTrumpAlliance, PoliticalCompassMemes, conservativeterrorism, Defeat_Project_2025, PoliticalDiscussion, trump, Firearms, ShitPoliticsSays, Fuckthealtright, libsofreddit, IntellectualDarkWeb, Law_and_Politics, Libertarian |
Table S9: Top 20 representative subreddits for each inferred category (continued)
Figure S4: The prompt used by GPT-4o for inferring a DDO user’s political alignment.
Figure S5: The prompt used by GPT-4o for inferring a Reddit user’s political alignment.
Figure S6: Prompt used by Llama-3.1-8B for inferring a DDO user’s political alignment.
Figure S7: Prompt used by Llama-3.1-8B for inferring a Reddit user’s political alignment.
Figure S8: User-level political alignment inference performance using GPT-4o across Reddit user groups with varying mean comment scores in the source communities ( r/Conservative or r/democrats ). To ensure that our preprocessing choice—retaining only users with positive mean comment scores—does not bias our results, we examine whether GPT-4o’s user-level political alignment inference performance varies across the full range of mean comment scores. This analysis tests the robustness of our findings to alternative implicit threshold values for user inclusion. During preprocessing, we retained only users who consistently received positive feedback, operationalized as having positive mean comment scores (upvotes minus downvotes) across their recent comments. We then grouped users by their mean comment scores to examine how inference performance of LLMs varies across user groups. (a–c) Macro F1 scores across mean comment score groups for three textual contexts: (a) Combined, (b) General, and (c) Politics. Each panel compares three user-level aggregation methods applied to GPT-4o predictions: (1) majority vote, (2) confidence-weighted average, and (3) maximum-confidence average. Across all three contexts, inference performance remains consistent across a user’s mean comment scores. The confidence-weighted and maximum-confidence aggregation methods show stable performance patterns throughout the entire score range, indicating that GPT-4o’s user-level inference is robust to variations in mean comment scores. Only the majority-vote method shows a decline for high–mean-score users in the Combined and General contexts, likely due to these users contributing fewer general-topic comments; however, this effect does not appear in the Politics context. (d–f) Number of participated subreddits per user across the same mean comment score groups for (d) Combined, (e) General, and (f) Politics categories. The decreasing trend in (d) and (e) demonstrates that users with high mean comment scores typically participate in a smaller, more concentrated set of subreddits, consistent with the performance drop observed for the majority-vote method in (a) and (b). In the Politics category (f), the number of participated subreddits remains relatively stable across score groups, aligning with the more uniform performance patterns observed in (c). Symbols and error bars represent mean standard error.
Figure S9: (a) Text-level F1 scores for political alignment inference across categories in the DDO dataset for GPT-4o and Llama-3.1-8B. (b) Text-level F1 scores across categories in the Reddit dataset for GPT-4o and Llama-3.1-8B. The gray dotted lines indicate an F1 score of 0.5, representing the expected macro-F1 of a random classifier.
Figure S10: Effect of semantic similarity and user overlap with politics on inference performance across categories, based on user-level F1 scores calculated using the maximum-confidence method with Llama-3.1-8B. (a–d) Similarity measurements between general categories and the Politics category in the DDO dataset, based on (a) content embedding similarity, (b) debate title embedding similarity, (c) Jaccard similarity of user participation between categories (User Jaccard similarity), and (d) normalized pointwise mutual information of user participation (User NPMI). (e–h) Same comparisons for the Reddit dataset. For Reddit, we use similarity in average subreddit descriptions embeddings between categories instead of title embeddings.
Figure S11: Correlation matrices between the four similarity measures (content embedding similarity, title or subreddit description embedding similarity, user Jaccard similarity, and user NPMI) for (a) the DDO dataset and (b) the Reddit dataset. Each similarity measure comprises 22 values corresponding to correlations between each category and the “Politics” category. Pearson correlation coefficients between pairs of similarity measures are shown, with the associated -values in parentheses.
Figure S12: DDO word clouds (Arts, Miscellaneous, Technology, Religion, Philosophy, People, Health, Politics)
Figure S13: DDO word clouds (Economics, Science, Society, Games, Entertainment, Sports, Education, Funny)
Figure S14: DDO word clouds (News, Cars, Movies, Music, Fashion, TV, Places-Travel)
Figure S15: Reddit word clouds (Arts, Miscellaneous, Technology, Religion, Philosophy, People, Health, Politics)
Figure S16: Reddit word clouds (Economics, Science, Society, Games, Entertainment, Sports, Education, Funny)
Figure S17: Reddit word clouds (News, Cars, Movies, Music, Fashion, TV, Places-Travel)
Figure S18: Relationship between word-level confidence and mean F1 score of political alignment inference across all categories for (a) the DDO and (b) the Reddit dataset. Each point represents one of 15 confidence quantiles, with shaded areas indicating 95% confidence intervals. In both (a) and (b), the bottom-right subplot shows the relationship for the entire dataset, which includes all general topic categories.
| Dataset | Total text | Answered text | Text-level answer ratio |
| DDO, GPT-4o | 22,265 | 21,389 | 0.961 |
| DDO, Llama-3.1-8B | 22,265 | 21,759 | 0.977 |
| Reddit, GPT-4o | 45,960 | 44,751 | 0.974 |
| Reddit, Llama-3.1-8B | 45,960 | 45,161 | 0.983 |
Table S10: Number and ratio of correctly formatted political orientation inference responses across datasets and models.
| Dataset | Model | Mean texts per user (Total) | Mean texts per user (Maximum-confidence) | Proportion |
| DDO (Combined) | GPT-4o | 6.342 | 2.018 | 0.318 |
| Llama-3.1-8B | 6.240 | 2.987 | 0.479 | |
| DDO (Political) | GPT-4o | 3.291 | 1.847 | 0.561 |
| Llama-3.1-8B | 3.250 | 2.348 | 0.722 | |
| DDO (General) | GPT-4o | 5.437 | 1.609 | 0.296 |
| Llama-3.1-8B | 5.342 | 2.257 | 0.422 |
Table S11: Average number of texts per user contributing to user-level inference across datasets and models. The table reports: (1) the mean number of texts available for majority or confidence-weighted aggregation (“Total”), (2) the mean number of texts retained under maximum-confidence aggregation, and (3) the proportion of retained texts relative to the total.
More in this topic
Privacy Leakage in Language Models
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.
See all 272 papers