CSC: Calibrated Simplicity for Conflict-Aware Social Bot Detection in the LLM Era

Yipeng Qian; Pengjie Zhao; Chaoxi Niu

cs.LGSep 20, 2026

CSC: Calibrated Simplicity for Conflict-Aware Social Bot Detection in the LLM Era

Authors: Yipeng Qian, Pengjie Zhao, Chaoxi Niu

Organizations: Minzu University of China Beijing, China · City University of Macau Macau, China

Abstract

Social bot detection is essential for protecting online platforms from misinformation amplification, coordinated manipulation, and distorted public discourse. However, large language models have made social bots much harder to detect from text alone because semantic camouflage is now cheap, fluent, and scalable. The resulting challenge is modality conflict: an account may look human-like in semantics while remaining suspicious in graph structure, profile attributes, or cross-modal consistency. Recent graph-based detectors tackle this limitation by adding graph-side complexity, such as sparse prototype selection, adaptive gating, or architecture-specific control logic, yet our experiments suggest that complexity alone is not the most reliable way to resolve such conflict. We therefore propose CSC, a calibrated-simplicity framework for conflict-aware LLM-era social bot detection. The framework combines three design choices: a simplified prototype-guided graph expert that retains useful structural biases while removing unstable graph-side heuristics, calibrated simplex-constrained fusion that aligns heterogeneous confidence spaces before late fusion, and a lightweight inconsistency expert that models cross-modal disagreement. Experiments on TwiBot-22, TwiBot-20, and MGStBot-large show that \textsc{CSC} improves calibrated operating-point decision quality while remaining competitive across external benchmarks. Further analyses show that calibration improves confidence reliability, the inconsistency expert mainly provides localized corrections in high-conflict or near-threshold regions, and simplified graph-side control yields a better stability-cost trade-off. A targeted semantic-camouflage stress test further shows that replacing selected bot text with matched human text sharply degrades the standalone text expert while leaving graph and fused evidence stable on a balanced challenge set.

Figures & tables

Figure 1. Modality conflict in the LLM era. A target account may receive different signals from different evidence sources: human-like language leads the text-side expert to assign a low bot score

q_{i}^{T}

, while suspicious topology and profile–behavior mismatch lead the graph- and attribute-side experts to assign high bot scores

q_{i}^{G}

and

q_{i}^{D}

. Here

q_{i}^{T}

q_{i}^{G}

, and

q_{i}^{D}

denote the text-, graph-, and attribute-side expert scores for account

i

, respectively. This conflict motivates confidence calibration and inconsistency modeling. A conceptual diagram showing a target account with conflicting evidence: text appears human-like, while graph topology and profile-behavior consistency appear suspicious.

Figure 2. Overview of CSC . The framework extracts graph, text, profile, and structured evidence from a target account and obtains modality-specific expert scores. These scores are independently calibrated using validation-only Platt scaling. A lightweight inconsistency expert captures cross-branch disagreement, and all calibrated outputs are combined through simplex-constrained fusion to produce the final bot-detection decision. A publication-quality schematic of CSC showing a target account feeding multiple evidence experts, a validation-only calibration stage, a lightweight inconsistency expert, and a final simplex-constrained fusion layer that outputs the bot-detection prediction.

Method	TwiBot-22			TwiBot-20			MGStBot-large
	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC
Kantepe-LR	$0.5389{\pm}0.0453$	$0.4765{\pm}0.0519$	$0.5752{\pm}0.0625$	$0.8183{\pm}0.0032$	$0.8560{\pm}0.0021$	$0.8204{\pm}0.0012$	$0.8863{\pm}0.0072$	$0.8034{\pm}0.0102$	$0.9417{\pm}0.0013$
LightGBM	$0.6334{\pm}0.0374$	$0.5665{\pm}0.0015$	$0.7781{\pm}0.0421$	$0.8115{\pm}0.0047$	$0.8504{\pm}0.0015$	$0.8298{\pm}0.0009$	$0.9098{\pm}0.0054$	$0.8497{\pm}0.0093$	$\mathbf{0.9681{\pm}0.0015}$
RoBERTa-direct	$0.6993{\pm}0.0018$	$0.5477{\pm}0.0024$	$0.7465{\pm}0.0312$	$0.7143{\pm}0.0096$	$0.7652{\pm}0.0033$	$0.7890{\pm}0.0028$	$0.8258{\pm}0.0028$	$0.7193{\pm}0.0053$	$0.8993{\pm}0.0009$
BotSCL	$0.6364{\pm}0.0272$	$0.5216{\pm}0.0098$	$0.7007{\pm}0.0217$	–	–	–	–	–	–
HGT-strict	$0.5653{\pm}0.0059$	$0.4849{\pm}0.0156$	$0.6560{\pm}0.0012$	–	–	–	–	–	–

Table 1. Final-model comparison across the three evaluation scopes. TwiBot-22 uses the local comparison setting, TwiBot-20 uses MGTAB-style protocol-matched runs, and MGStBot-large uses three-seed external evaluation with protocol-matched non-graph runs where available. Results are reported as mean

\pm

standard deviation. Best values among the reported rows within each dataset are bolded; “–” indicates that the method was not evaluated under that scope.

Figure 3. Protocol-controlled TwiBot-22 diagnostics. Left: Platt calibration improves probability reliability under a fixed fusion setup. Right: the final fused model is threshold-sensitive and peaks near the validation-selected threshold rather than at the default threshold of $0.5$ . Two protocol-controlled diagnostic plots for TwiBot-22. The left plot compares probability reliability before and after Platt scaling. The right plot shows F1 as a function of the decision threshold, with the final fused model peaking near the validation-selected threshold rather than the default threshold of 0.5.

Figure 4. Comparison between the four base experts and the final CSC model on TwiBot-22. The figure reports Recall and F1 for classification behavior, and AUC and AP for ranking-oriented behavior. Among the base experts, the profile expert achieves the highest recall. The final CSC model obtains the strongest F1, AUC, and AP, showing that calibrated fusion improves final decision quality beyond any single expert. A two-panel grouped bar chart comparing the calibrated graph, profile, text, and structured-feature experts with the final \textsc{CSC} model on TwiBot-22. The left panel reports Recall and F1, and the right panel reports AUC and AP.

Method	F1	AUC	AP	ECE	Brier	NLL	Thr.
Raw Average	0.5887	0.7807	0.6126	0.2626	0.2538	0.7333	0.68
Raw Simplex	0.5908	0.7808	0.6122	0.2643	0.2547	0.7370	0.70
Platt Average	0.5918	0.7807	0.6094	0.0899	0.1696	0.5532	0.18
Platt Simplex	0.5953	0.7809	0.6092	0.0905	0.1696	0.5530	0.20
Platt Simplex + $C$	0.5956	0.7809	0.6092	0.0855	0.1687	0.5473	0.21

Table 2. Calibration and fusion diagnostics on TwiBot-22. Calibration mainly improves reliability and operating-point decision quality, while AUC changes only slightly.

ID	Version	F1 20	AUC 20	Thr. 20	F1 22	AUC 22	Thr. 22
A1	Text-only + GraphSAGE	0.7669	0.8102	0.3300	0.5359	0.7236	0.3300
A2	$[x_{i}^{\mathrm{text}};x_{i}^{\mathrm{struct}}]$ + single-path GraphSAGE	0.8751	0.9243	0.3400	0.5780	0.7650	0.6800
A3	$[x_{i}^{\mathrm{text}};x_{i}^{\mathrm{struct}}]$ + dual encoders	0.8772	0.9270	0.5000	0.5755	0.7664	0.6600
A4	Single prototype	0.8774	0.9241	0.5700	0.5839	0.7621	0.6700
A5	Multi-prototype, without top- $r$	0.8815	0.9252	0.5400	0.5822	0.7675	0.7400
A6	Multi-prototype + top- $r$ =2, without residual	0.8812	0.9251	0.6000	0.5799	0.7657	0.6200

Table 3. Graph-branch ablation under the shared validation-threshold protocol. Rows separate graph architectural choices from later fusion choices, testing whether additional graph-side controls consistently improve the retained graph expert.

ID	Version	F1 20	AUC 20	Thr. 20	F1 22	AUC 22	Thr. 22
S1	Best graph branch	0.8839	0.9256	0.5200	0.5802	0.7635	0.7000
S2	Best non-graph branch	0.8820	0.9327	0.4600	0.5903	0.7759	0.7100
S3	Graph + non-graph, direct average	0.8779	0.9328	0.5800	0.5907	0.7763	0.7000
S4	Graph + non-graph + Platt calibration	0.8831	0.9321	0.4900	0.5918	0.7762	0.1700
S5	Final calibrated fusion + validation threshold	0.8827	0.9334	0.4800	0.5940	0.7801	0.1900
S6	Final calibrated fusion + fixed threshold 0.5	0.8809	0.9334	0.5000	0.4362	0.7801	0.5000

Table 4. Fusion-stage ablation under the shared local protocol. S5 is the final validation-threshold system; S6 uses the same fused scores as S5 but fixes the decision threshold at

0.5

Region	$N$	Base F1	+ $C$ F1	$\Delta$ F1	Fix	Hurt
Low conflict	33,333	0.3687	0.3687	-0.0001	2	3
Medium conflict	33,333	0.6478	0.6478	-0.0000	4	9
High conflict	33,334	0.5975	0.5981	0.0006	96	38

Table 5. Conflict-region analysis on TwiBot-22. Samples are split into low-, medium-, and high-conflict terciles by disagreement score;

N

denotes the number of test samples in each region. The inconsistency expert mainly helps in the high-conflict region.

Explore similar work

May 28, 2026cs.SI

SAHG: Sector-Anisotropic Hyperbolic Graph Model for Social Bot Detection

LLM-driven social bots can generate fluent, human-like text, reducing the discriminative advantage of content-based detection alone. However, coordinated campaigns still leave relational patterns -- interactions, behavioral similarity, shared neighborhoods, community positions, and coordinated activity -- that graph-based methods can exploit. Existing graph detectors face two challenges when exploiting such evidence. First, Euclidean GNNs distort hierarchical and scale-free social graphs; while hyperbolic geometry addresses this volume-growth mismatch, fixed-curvature models still assign uniform geometric resolution to structural directions with different densities and separation needs. Second, relational evidence is not always reliable: sophisticated bots forge heterophilic connections with genuine users, causing neighborhood aggregation to mix bot and human signals and dilute account-level evidence. We propose SAHG (Sector-Anisotropic Hyperbolic Graph), addressing both challenges. SAHG learns a direction-dependent curvature field

γ(u)

that adapts geometric resolution across structural directions, and uses sector prototypes to convert angular concentration and alignment into classifier-readable features. To prevent contaminated aggregation from overwhelming account-level evidence, SAHG encodes per-account features and graph-neighborhood representations in two independent SAH channels, fusing them only at the classifier. Experiments on Fox8-23, BotSim-24, and MGTAB show that SAHG achieves the highest accuracy and F1 on all three benchmarks, outperforming feature-based, graph-based, LLM-based, and isotropic hyperbolic baselines. Ablation and geometric analyses confirm the effectiveness of the anisotropic geometry and dual-channel design.

Hanning Lu, Yingguang Yang, Jinwei Su +8

University of Leeds · University of Science and Technology of China · 3South China Normal University +6

Hypergraphs Graph Neural Networks

Ask PDF

Jun 5, 2026cs.CL

Adversarial Creation and Detection of AI-Generated Social Bot Content

The convergence of large language models and social bots allows malicious actors to manipulate the information ecosystem by generating human-like content at scale. Existing models for detecting AI-generated content often fail in the wild, primarily due to the lack of ground-truth data. We address this gap through an adversarial methodology that models the impersonation of real social media users by malicious actors. Using this methodology, we curate a multilingual, cross-platform dataset of paired human and AI-generated messages. Training on such adversarial data yields accurate detection of AI-generated text. Our approach significantly outperforms existing models for content-based bot detection in real-world, out-of-distribution data.

Mykola Trokhymovych, Ricardo Baeza-Yates, Alessandro Flammini +2

Universitat Pompeu Fabra, Barcelona, Spain · Observatory on Social Media, Indiana University, Bloomington, Indiana, USA · KTH Royal Institute of Technology, Stockholm, Sweden

Ai-Generated Content Detection Generative Artificial Intelligence Adversarial Prompts Chatbots Malicious Agents

Ask PDF

Aug 10, 2026cs.AI

Beyond Detection: Evaluating Defensive LLMs Against AI-Generated Social Engineering in Live Turn-by-Turn Interaction

Generative AI makes social-engineering attacks more fluent, adaptive, and scalable, increasing the need for LLM-based de- fenders that can protect users during ongoing interactions. We ask whether such defenders identify the structural source of risk or merely react to surface cues. We formalize trust-chain localization: identifying whether an interaction fails at actor authority, asset control, verification sufficiency, or transaction path. We construct a controlled 300-case online-housing corpus spanning 20 scenario families, legitimate cases, four structural failure modes, and three surface conditions. Five defender models are evaluated on the same corpus in state- ful turn-by-turn and one-shot static settings, yielding 1,500 model-case evaluations per protocol and 3,000 in total. No model produced explicit unsafe compliance, yet defensive effectiveness varied sharply: intervention rates ranged from 0% to 96.3%. Protective action and correct structural localization were frequently decoupled, with models sometimes intervening while identifying the wrong trust component or recognizing a structural failure without taking protective action. Asset-control failures were a major localization bottleneck, surface sensitivity varied across models, and live-static differences were model-dependent. These findings show that safe-looking behavior alone is insufficient; live scam resistance must separately measure intervention, timing, structural localization, and false-positive behavior.

Yuqiao Xu, Osama Zafar, Alexander Nemecek +1

Case Western Reserve University

Attacker Large Language Model LLM Defense Mechanisms AI Trustworthiness Generative Artificial Intelligence Beyond Retrieval

Ask PDF