Breaking and Defending LLM-Powered Social Media Bot Detection Systems

Nof Orenstein; Yoni Birman

cs.AIAug 16, 2026

Breaking and Defending LLM-Powered Social Media Bot Detection Systems

Authors: Nof Orenstein, Yoni Birman

Organizations: Department of Computer Science Reichman University Herzliya, Israel

Abstract

The rise of social media bots poses a persistent threat, enabling misinformation, opinion manipulation, and the erosion of trust in online platforms. To combat this, machine learning systems have been developed to detect and limit bot activity, but attackers continuously adapt through techniques such as adversarial learning and behavior imitation, fueling an ongoing arms race between bots and detection tools. Recent advances in large language models (LLMs) have significantly improved bot detection by enabling deeper semantic and contextual analysis of accounts and their content. However, this shift also introduces new attack surfaces, allowing adversaries to craft exploits that directly target the reasoning and generation mechanisms of LLM-based classifiers. Industry tools such as Anthropic's Claude Code Security similarly leverage LLMs for security-critical decisions, further motivating a careful study of their attack surfaces. In this work, we investigate both the offensive and defensive aspects of LLM-powered, threat-specific cybersecurity applications. While centered on the challenge of social media bot detection, our methodology and insights generalize to a broad class of LLM-powered cybersecurity systems, including phishing detection, email classification, and fraud analysis. We introduce two novel adversarial attack strategies that systematically exploit the semantic and contextual weaknesses of LLM-based classifiers, degrading their detection accuracy by up to 48%. To counter these threats, we propose a robust multi-LLM defense architecture designed to preserve detection reliability under adaptive adversarial conditions. Our solution, LSABRE (LLM-powered Social Adversarial Bot Recognition Ensemble), is a multi-LLM framework that substantially improves robustness across a range of attacks, maintaining 86% detection accuracy even under strong, adaptive adversarial pressure.

Figures & tables

Fig. 1: Training and Inference-Time Phases Diagram

Fig. 3: Classifier Guidance Rewrite Diagram

Fig. 4: Defenses Training and Inference-time Phases Diagram

The Data	Model	Acc	TPR	FPR
	llama3	0.908	0.993	0.177
Tweets per User	mistral	0.5195	0.045	0.006
	gemma	0.8925	0.865	0.08
	llama3	0.849	0.84	0.142
Tweets per User	mistral	0.512	0.027	0.003
(sanity check performed around 6 months later)	gemma	0.7945	0.649	0.06

TABLE I: Detection Prediction Results

Attack Type	Mistral	Gemma	Llama
No Attack	0.5195	0.8925	0.908
Zero Shot	0.527	0.63783	0.810167
Few Shot	0.5465	0.635167	0.8925
Feature Guidance	0.50883	0.5735	0.82467
Classifier Guidance	0.575167	0.57267	0.857

TABLE II: Average Prediction Per Rewrite Attack Type

Attack Type	Mistral	Gemma	Llama
No Attack	0.5195	0.8925	0.908
Reasoning Injection	0.5045	0.4605	0.473
Safety Injection	0.6195	0.4705	0.5685
Out Of Service Injection	0.5255	0.468	0.421

TABLE III: Average Prediction Per Injection Attack Type

Attacker Model	Mistral	Gemma	Llama
No Attack	0.5195	0.8925	0.908
Mistral	0.508125	0.561375	0.81175
Gemma	0.584	0.630375	0.894375
Llama	0.518125	0.623375	0.829625

TABLE IV: Average Prediction Per Attacker Model

Table 11

Defense Type	Mistral	Gemma	Llama
No Attack	0.5195	0.8925	0.908
Zero Shot Attack	0.527	0.63783	0.810167
Self Examination - Zero Shot Defense	0.5585	0.63083	0.793167
Self Examination - ICL Defense	0.64	0.5805	0.843
Self Examination - Feature Guidance Defense	0.57983	0.56183	0.85167

TABLE VII: Average Prediction Per Rewrite Prevention Defense Type

Fig. 7: Prevention Techniques against Content Manipulations

Defense Type	Mistral	Gemma	Llama
No Attack	0.5195	0.8925	0.908
Average Attack	0.54983	0.4663	0.4875
Self Examination - Zero Shot Defense	0.5265	0.24867	0.38883
Self Examination - ICL Defense	0.509167	0.2363	0.25083
Self Examination - Feature Guidance Defense	0.5445	0.549167	0.80067
Known Answer	0.51467	0.503	0.8625

TABLE VIII: Average Prediction Per Injection Prevention Defense Type

Fig. 8: Prevention Techniques against LLM Manipulations

Table 16

Defense Type	Mistral	Gemma	Llama
No Attack	0.499	0.5885	0.673
Naive LLM-based	0.53867	0.6785	0.734
In Context Learning LLM-based	0.5813	0.64767	0.72683
Feature Guidance LLM-based	0.66267	0.5603	0.633

TABLE XI: Average Prediction Per Rewrite Detection Type

Fig. 9: Detection Techniques against Content Manipulations

Model Attacker	Mistral	Gemma	Llama
No Attack	0.499	0.5885	0.673
Mistral	0.70583	0.696167	0.737167
Gemma	0.559167	0.568	0.6713
Llama	0.51767	0.6223	0.6853

TABLE XII: Average Prediction Per The Attacker Model

Table 20

Attack Type	Mistral	Gemma	Llama
No Attack - Baseline Average	0.0	0.0445	0.8975
Reasoning Injection	0.252125	0.50625	0.6265
Safety Injection	0.057	0.50625	0.38075
Out Of Service Injection	0.147	0.6223	0.306625

TABLE XV: Average Injection Prediction Per Injection Type

Fig. 10: Detection Techniques against LLM Manipulations

Method	Acc	TPR	FPR
Detection Before Attack	0.90025	0.929	0.1285
Detection After Attack	0.725438	0.56083	0.1105416
Ensemble Results	0.8623	0.855	0.13033

TABLE XVI: Ensemble Architecture Results 4 4 4 The Detection Before Attack and Detection After Attack results are averaged excluding Mistral, as it is not well-suited for this task.

Model	Attack	Model	Acc	TPR	FPR
		llama3	0.8165	0.783	0.15
	Zero Shot Rewrite	mistral	0.522	0.046	0.002
		gemma	0.6	0.264	0.064
		llama3	0.8485	0.847	0.15
mistral	Few Shot Rewrite	mistral	0.505	0.01	0.0
		gemma	0.565	0.194	0.064

TABLE XVII: Content Manipulations on Bot Detection LLMs

Type	Attack	Model	Acc	TPR	FPR
		llama3	0.473	0.123	0.177
	Reasoning Prompt Injection	mistral	0.5045	0.015	0.006
		gemma	0.4605	0.001	0.08
		llama3	0.5685	0.314	0.177
LLM Manipulation	Safety Alignment Prompt Injection	mistral	0.6195	0.245	0.006
		gemma	0.4705	0.021	0.08

TABLE XVIII: LLM Manipulations on Bot Detection LLMs

Technique	Model	Acc	TPR	FPR
	llama3	0.394	0.126	0.338
Self Examination - Zero Shot	mistral	0.5025	0.007	0.002
	gemma	0.357	0.003	0.289
	llama3	0.634	0.394	0.126
Self Examination - ICL	mistral	0.5175	0.036	0.001
	gemma	0.5	0.0	0.0

TABLE XIX: Prevention Technique for Defense from LLM Manipulation (Prompt Injections) on Bot Detection LLMs

Rewrites By	Technique	Model	Acc	TPR	FPR
		llama3	0.7805	0.926	0.365
	Self Examination - Zero Shot	mistral	0.5315	0.063	0.0
		gemma	0.5515	0.266	0.163
		llama3	0.83	0.902	0.242
Mistral - Zero Shot Rewrite	Self Examination - ICL	mistral	0.6115	0.345	0.122
		gemma	0.5515	0.167	0.064

TABLE XX: Prevention Technique for Defense from Content Manipulation (Zero Shot rewrites) on Bot Detection LLMs

Injection	Technique	Model	Acc	TPR	FPR
		llama3	0.9315	0.983	0.12
	Naive LLM-based	mistral	0.01	0.02	1.0
		gemma	0.5115	0.988	0.965
		llama3	0.4965	0.968	0.975
Reasoning	In Context Learning	mistral	0.0365	0.01	0.937
		gemma	0.5	0.97	0.97

TABLE XXI: Detection of LLM Manipulation (Prompt Injections) on Bot Detection LLMs as a Defense Technique

Rewrites By	Technique	Model	Acc	TPR	FPR
		llama3	0.774	0.916	0.368
	Naive LLM-based	mistral	0.6075	0.226	0.011
		gemma	0.758	0.792	0.276
		llama3	0.773	0.94	0.394
Mistral - Zero Shot Rewrite	In Context Learning	mistral	0.695	0.391	0.001
		gemma	0.7495	0.771	0.272

TABLE XXII: Detection of Content Manipulation (Rewrites) on Bot Detection LLMs as a Defense Technique

Social Bot Detection

See all 12 papers

Mar 31, 2026

Beyond Individual Mimicry: Constructing Human-Like Social network with Graph-Augmented LLM Agents

Driven by large language models (LLMs), social bot can autonomously engage in local interactions, whose human-like behaviors enable them to evade social bot detection. However, while these botnets exhibit realistic local social interactions, they fail to preserve human-like social network. This is because LLM-based bots are graph-unaware and cannot coordinate over global interactions, which makes those botnets vulnerable to graph neural network (GNN)-based detection. To address this limitation, we propose GraphMind, which equips LLM-driven social bots to explicitly learn and fit human-like social network structures. Building on this foundation, we further construct GraphMind-Botnet, a LLM-driven botnet designed to evaluate the performance of existing social bot detection algorithms. Experiments on datasets derived from GraphMind-Botnet show that both text-based and graph-based detection models show substantially degraded performance in distinguishing. Our results highlight the critical role of social link construction in LLM-driven social network generation, while exposing fundamental weaknesses in existing bot detection mechanisms.

Haoran Bu, Litian Zhang, Chuxuan Zhang +3

Cyberspace Security, Beijing University of Posts and Telecommunications · Institute of Software, Chinese Academy of Sciences

Social Bot Detection Large Language Model-Based Social Simulation Synthetic Data Generation

Ask PDF

Apr 2, 2026

TRACE-Bot: Detecting Emerging LLM-Driven Social Bots via Implicit Semantic Representations and AIGC-Enhanced Behavioral Patterns

Large Language Model-driven (LLM-driven) social bots pose a growing threat to online discourse by generating human-like content that evades conventional detection. Existing methods suffer from limited detection accuracy due to overreliance on single-modality signals, insufficient sensitivity to the specific generative patterns of Artificial Intelligence-Generated Content (AIGC), and a failure to adequately model the interplay between linguistic patterns and behavioral dynamics. To address these limitations, we propose TRACE-Bot, a unified dual-channel framework that jointly models implicit semantic representations and AIGC-enhanced behavioral patterns. TRACE-Bot constructs fine-grained representations from heterogeneous sources, including personal information data, interaction behavior data and tweet data. A dual-channel architecture captures linguistic representations via a pretrained language model and behavioral irregularities via multidimensional activity features augmented with signals from state-of-the-art (SOTA) AIGC detectors. The fused representations are then classified through a lightweight prediction head. Experiments on two public LLM-driven social bot datasets demonstrate SOTA performance, achieving accuracies of 98.46% and 97.50%, respectively. The results further indicate strong robustness against advanced bot strategies, highlighting the effectiveness of jointly leveraging implicit semantic representations and AIGC-enhanced behavioral patterns for emerging LLM-driven social bot detection.

Zhongbo Wang, Zhiyu Lin, Zhu Wang +1

School of Cyber Science and Engineering, Sichuan University, Sichuan 610207, China · School of Software and Microelectronics, Peking University, Beijing 102600, China · College of Computer Science, Sichuan University, Sichuan 610065, China +1

Social Bot Detection LLM Security

Ask PDF

Mar 17, 2026

Security Assessment and Mitigation Strategies for Large Language Models: A Comprehensive Defensive Framework

Large Language Models increasingly power critical infrastructure from healthcare to finance, yet their vulnerability to adversarial manipulation threatens system integrity and user safety. Despite growing deployment, no comprehensive comparative security assessment exists across major LLM architectures, leaving organizations unable to quantify risk or select appropriately secure LLMs for sensitive applications. This research addresses this gap by establishing a standardized vulnerability assessment framework and developing a multi-layered defensive system to protect against identified threats. We systematically evaluate five widely-deployed LLM families GPT-4, GPT-3.5 Turbo, Claude-3 Haiku, LLaMA-2-70B, and Gemini-2.5-pro against 10,000 adversarial prompts spanning six attack categories. Our assessment reveals critical security disparities, with vulnerability rates ranging from 11.9% to 29.8%, demonstrating that LLM capability does not correlate with security robustness. To mitigate these risks, we develop a production-ready defensive framework achieving 83% average detection accuracy with only 5% false positives. These results demonstrate that systematic security assessment combined with external defensive measures provides a viable path toward safer LLM deployment in production environments.

Taiwo Onitiju, Iman Vakilinia

School of Computing, University of North Florida

LLM Security Adversarial Attacks on LLMs Adversarial Example Detection

Ask PDF

Abstract

Social Bot Detection

Explore similar work

Beyond Individual Mimicry: Constructing Human-Like Social network with Graph-Augmented LLM Agents

TRACE-Bot: Detecting Emerging LLM-Driven Social Bots via Implicit Semantic Representations and AIGC-Enhanced Behavioral Patterns

Security Assessment and Mitigation Strategies for Large Language Models: A Comprehensive Defensive Framework