POLAR:A Per-User Association Test in Embedding Space

Pedro Bento; Arthur Buzelin; Arthur Chagas; Yan Aquino; Victoria Estanislau; Samira Malaquias; Pedro Robles Dutenhefner; Gisele L. Pappa

cs.CLMar 16, 2026

POLAR:A Per-User Association Test in Embedding Space

Authors: Pedro Bento, Arthur Buzelin, Arthur Chagas, Yan Aquino, Victoria Estanislau, Samira Malaquias, Pedro Robles Dutenhefner, Gisele L. Pappa,

Organizations: Universidade Federal de Minas Gerais (UFMG), Belo Horizonte, Brazil

Abstract

Most intrinsic association probes operate at the word, sentence, or corpus level, obscuring author-level variation. We present POLAR (Per-user On-axis Lexical Association Re-port), a per-user lexical association test that runs in the embedding space of a lightly adapted masked language model. Authors are represented by private deterministic to-kens; POLAR projects these vectors onto curated lexicalaxes and reports standardized effects with permutation p-values and Benjamini--Hochberg control. On a balanced bot--human Twitter benchmark, POLAR cleanly separates LLM-driven bots from organic accounts; on an extremist forum,it quantifies strong alignment with slur lexicons and reveals rightward drift over time. The method is modular to new attribute sets and provides concise, per-author diagnostics for computational social science. All code is publicly avail-able at https://github.com/pedroaugtb/POLAR-A-Per-User-Association-Test-in-Embedding-Space.

Figures & tables

Figure 1: Overview of POLAR , which first learns user vectors via masked language modeling with injected tokens (Phase 1), then computes association scores through cosine similarity and permutation testing against lexical attribute sets (Phase 2).

Figure 2: PCA projection of users in the 4D POLAR-axis score space. Marker shape encodes the gold label (bot vs. human), while color indicates the dominant axis (largest

|s|

for that user). Centroids (black

\times

) summarize average positions; PCA is used for visualization, while classification is performed in the original axis space (Tables 1 – 2 ).

Figure 3: Box-and-swarm plots of per-user POLAR scores

s(u;\mathcal{A},\mathcal{B})

(y-axis; Eq. 1 ) for humans and bots across four lexical axes. Points are individual users; boxplots summarize median and interquartile range.

Axis	AUROC	PR–AUC	Brier	ECE
Scam/Finance	0.961	0.973	0.061	0.073
Newswire	0.950	0.962	0.087	0.108
Promo/CTA	0.915	0.930	0.116	0.068
Toxicity	0.868	0.878	0.145	0.054

Table 1: Single-axis OOF performance on fox8–23 (one LR on each POLAR score). Best two AUROC in bold .

Features (axes)	AUROC [95% CI]	PR–AUC	Brier
POLAR ( $k{=}1$ )	0.961 [0.952, 0.970]	0.911	0.060
POLAR ( $k{=}2$ )	0.960 [0.951, 0.969]	0.913	0.061
POLAR ( $k{=}3$ )	0.961 [0.952, 0.970]	0.918	0.060
POLAR ( $k{=}4$ )	0.961 [0.952, 0.970]	0.918	0.060

Table 2: OOF performance on fox8–23 using a 1-layer logistic regression fed only with POLAR axis scores. AUROC CIs from

2{,}000

bootstrap resamples.

Figure 4: Stormfront per-user POLAR associations on sensitive targets and policy frames.

Axis	Mean $s$	Sig. BH
Racial Slurs vs. Neutral	1.332	1.00
Gender Slurs vs. Respect	1.320	1.00
Incel Jargon vs. Relationships	1.232	1.00
LGBTQ Slurs vs. Neutral	0.707	1.00
Violence vs. Peace	0.258	0.00
Toxicity vs. Civility	0.033	0.00

Table 3: Stormfront: axis-level descriptives. Sig. BH is the share of users significant after BH–FDR at

\alpha{=}0.05

for that axis.

Figure 5: Stormfront dynamics: per-user scatterplots mark least- (blue) and most-aligned (red) accounts on four slur axes; coloured trajectories track selected users with the fastest drift toward alignment.

Social Bot Detection

See all 12 papers

LLMs Can Infer Political Alignment from Online Conversations
Mar 11, 2026Byunghwee Lee, Sangyeon Kim, Filippo Menczer +3Privacy Leakage in Language Models
School of Data Science, University of Virginia, Charlottesville, Virginia, USA, 22903 · Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering Indiana University, Bloomington, Indiana, USA, 47408 · Division of Communication and Media, Ewha Womans University, Seoul, Republic of Korea, 03760
Polar: A Benchmark for Evaluating Political Bias in LLMs
Jun 11, 2026Sangho Kim, Heejin Kim, Yoonhee Park +2Multilingual Language Model Evaluation Political Bias in Language Models
Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University
Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest
Apr 21, 2026Ramtin Davoudi, Kartik Thakkar, Nazanin Donyapour +2Social Media Analysis LLM Evaluation
Utah State University · Independent Researcher · Vanderbilt University

Mar 11, 2026J/K move · Enter open · S save

LLMs Can Infer Political Alignment from Online Conversations

Byunghwee Lee, Sangyeon Kim, Filippo Menczer, Yong-Yeol Ahn+2

School of Data Science, University of Virginia, Charlottesville, Virginia, USA, 22903 · Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering Indiana University, Bloomington, Indiana, USA, 47408 · Division of Communication and Media, Ewha Womans University, Seoul, Republic of Korea, 03760

Privacy Leakage in Language Models

Open paper AskPDF arXiv

Due to the correlational structure in our traits such as identities, cultures, and political attitudes, seemingly innocuous preferences like following a band or using a specific slang can reveal private traits. This possibility, especially when combined with massive, public social data and advanced computational methods, poses a fundamental privacy risk. As our data exposure online and the rapid advancement of AI are increasing the risk of misuse, it is critical to understand the capacity of large language models (LLMs) to exploit such potential. Here, using online discussions on DebateOrg and Reddit, we show that LLMs can reliably infer hidden political alignment, significantly outperforming traditional machine learning models. Prediction accuracy further improves as we aggregate multiple text-level inferences into a user-level prediction, and as we use more politics-adjacent domains. We demonstrate that LLMs leverage words that are highly predictive of political alignment while not being explicitly political. Our findings underscore the capacity and risks of LLMs for exploiting socio-cultural correlates.

Abstract

Social Bot Detection

Explore similar work

LLMs Can Infer Political Alignment from Online Conversations

Polar: A Benchmark for Evaluating Political Bias in LLMs

Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest