LLM Security

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

25 papers in the last four weeks, up 127% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 200

Apr 19, 2026cs.CR

Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systems

Rowhammer on GPU DRAM has enabled adversarial bit flips in model weights; shared KV-cache blocks in LLM serving systems present an analogous but previously unexamined target. In vLLM's Prefix Caching, these blocks exist as a single physical copy without integrity protection. Using software fault injection under ideal bit targeting, we characterize worst-case severity and identify three properties: (1) Silent divergence - 13 of 16 BF16 bit positions produce coherent but altered outputs, indistinguishable from legitimate responses without a clean baseline. (2) Selective propagation - only requests sharing the targeted prefix are affected. (3) Persistent accumulation - no temporal decay occurs, so cumulative damage grows linearly with subsequent requests. Together, these constitute a threat profile distinct from weight corruption: silent divergence and selective propagation enable detection evasion; persistent accumulation then proceeds unchecked, yielding damage amplification bounded only by how long the block remains cached. A checksum-based countermeasure detects any single-bit corruption at scheduling time, bounding cumulative damage to one batch independent of the block's cache lifetime, with negligible overhead. These results argue for integrity protection of prefix blocks before end-to-end exploitation is demonstrated.

Yuji Yamamoto, Satoshi Matsuura

Institute of Science Tokyo, Tokyo, Japan

LLM Serving Adversarial Attacks on LLMs LLM Security

Apr 18, 2026cs.CR

CASCADE: A Component Ablation and Corpus Audit of a Layered Local Defense for MCP-Based Systems

The Model Context Protocol (MCP) widens the prompt injection attack surface of large language model applications to tool descriptions, parameter schemas, and tool outputs. Defenses for it are appearing quickly, but their reported figures are not comparable: each is evaluated on a corpus of its authors' construction, under a decision convention that is rarely stated. This paper asks how much those choices decide, taking CASCADE, a fully local layered defense, as the case: three configurations on a frozen 5,000-sample corpus under a pinned revision and a fixed protocol, with that corpus audited in full. Four results follow. First, the aggregation convention dominates the headline metric: counting review referrals as positives reports an 11.70% false-positive rate where 1.51% of benign traffic would be denied without a human, and conceals that 68.5% of all traffic reaches a reviewer. Second, detection is not provenance-invariant: recall ranges from 86.20% on original material to 99.88% on template-generated material, and added false positives fall on original benign records at ten times the rate they fall on transformed ones. Third, the operating point that ran is not readable from the released configuration, which names four candidate thresholds, its deployment files selecting one that did not govern; it is recoverable from point masses the policy layer leaves in the score distribution, so record-level output is a stronger reproducibility guarantee than a parameter table. Fourth, a local review model invoked for 32.56% of requests at 2.51 s each changes no classification outcome: it returned 90 not-malicious verdicts and the policy stage admitted none, making that null a guard setting rather than a model property. The ablation is unsurprising -- the rule-based layer reaches 61.05% recall, the semantic stage 94.77% -- and that is what makes the other results the substance of the paper.

İpek Abasıkeleş Turgut, Edip Gümüş

Department of Computer Engineering, Faculty of Engineering and Natural Sciences, Iskenderun Technical University, Hatay, Türkiye · Department of Computer Engineering, Institute of Graduate Studies, Iskenderun Technical University, Hatay, Türkiye

MCP Security LLM Security Prompt Injection Defense Prompt Injection Detection

Apr 17, 2026cs.CR

SafeLM: Unified Privacy-Aware Optimization for Trustworthy Federated Large Language Models

Large language models (LLMs) are increasingly deployed in high-stakes domains, yet a unified treatment of their overlapping safety challenges remains lacking. We present SafeLM, a framework that jointly addresses four pillars of LLM safety: privacy, security, misinformation, and adversarial robustness. SafeLM combines federated training with gradient smartification and Paillier encryption for privacy, integrates defenses against training and inference-time attacks, employs contrastive grounding with calibrated decoding to reduce hallucinations, and introduces alignment-aware binarized aggregation to enhance robustness while maintaining bounded reconstruction quality. Across benchmarks on factuality, toxicity, and membership inference, SafeLM achieves 98.0% harmful content detection accuracy, reduces communication by 96.9%, and lowers gradient inversion PSNR from 31.7 dB to 15.1 dB. Ablations show that each component contributes independently, whereas their integration yields a strong privacy utility efficiency trade-off for deploying trustworthy LLMs.

Noor Islam S. Mohammad, Uluğ Bayazıt

Istanbul Technical University

LLM Hallucination Mitigation Privacy-Preserving Language Models LLM Security LLM Safety Adversarial Robustness of Language Models Federated Learning

Apr 16, 2026cs.CR

Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection

Modern Large audio-language models (LALMs) power intelligent voice interactions by tightly integrating audio and text. This integration, however, expands the attack surface beyond text and introduces vulnerabilities in the continuous, high-dimensional audio channel. While prior work studied audio jailbreaks, the security risks of malicious audio injection and downstream behavior manipulation remain underexamined. In this work, we reveal a previously overlooked threat, auditory prompt injection, under realistic constraints of audio data-only access and strong perceptual stealth. To systematically analyze this threat, we propose \textit{AudioHijack}, a general framework that generates context-agnostic and imperceptible adversarial audio to hijack LALMs. \textit{AudioHijack} employs sampling-based gradient estimation for end-to-end optimization across diverse models, bypassing non-differentiable audio tokenization. Through attention supervision and multi-context training, it steers model attention toward adversarial audio and generalizes to unseen user contexts. We also design a convolutional blending method that modulates perturbations into natural reverberation, making them highly imperceptible to users. Extensive experiments on 13 state-of-the-art LALMs show consistent hijacking across 6 misbehavior categories, achieving average success rates of 79%-96% on unseen user contexts with high acoustic fidelity. Real-world studies demonstrate that commercial voice agents from Mistral AI and Microsoft Azure can be induced to execute unauthorized actions on behalf of users. These findings expose critical vulnerabilities in LALMs and highlight the urgent need for dedicated defense.

Meng Chen, Kun Wang, Li Lu +2

The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · National University of Singapore +1

Audio-Language Models LLM Security

Mar 10, 2026cs.CR

FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation

Device-side Large Language Models (LLMs) have witnessed explosive growth, offering higher privacy and availability compared to cloud-side LLMs. During LLM inference, both model weights and user data are valuable, and attackers may even compromise the OS kernel to steal them. ARM TrustZone is the de facto hardware-based isolation technology on mobile devices, used to protect sensitive applications from a compromised OS. However, protecting LLM inference with TrustZone incurs significant overhead due to its inflexible isolation of memory and the NPU. To address these challenges, this paper introduces FlexServe, a fast and secure LLM serving system for mobile devices. It first introduces a Flexible Resource Isolation mechanism to construct Flexible Secure Memory (Flex-Mem) and Flexible Secure NPU (Flex-NPU). Both memory pages and the NPU can be efficiently switched between unprotected and protected modes. Based on these mechanisms, FlexServe designs a fast and secure LLM inference framework within TrustZone's secure world. The LLM-Aware Memory Management and Secure Inference Pipeline are introduced to accelerate inference. A Multi-Model Scheduler is proposed to optimize multi-model workflows. We implement a prototype of FlexServe and compare it with two TrustZone-based strawman designs. The results show that FlexServe achieves an average

10.05\times

speedup in Time to First Token (TTFT) compared to the strawman, and an average

2.44\times

TTFT speedup compared to an optimized strawman with pipeline and secure NPU enabled. For multi-model agent workflows, the end-to-end speedup is up to

24.30\times

and

4.05\times

compared to the strawman and optimized strawman, respectively.

Yinpeng Wu, Yitong Chen, Lixiang Wang +3

Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

LLM Serving On-Device Language Model Inference Trusted Execution Environments LLM Security LLM Inference Acceleration

Feb 13, 2026cs.CR

TensorCommitments: A Lightweight Verifiable Inference for Language Models

Most large language models (LLMs) run on external clouds: users send a prompt, pay for inference, and must trust that the remote GPU executes the LLM without any adversarial tampering. We critically ask how to achieve verifiable LLM inference, where a prover (the service) must convince a verifier (the client) that an inference was run correctly without rerunning the LLM. Existing cryptographic works are too slow at the LLM scale, while non-cryptographic ones require a strong verifier GPU. We propose TensorCommitments (TCs), a tensor-native proof-of-inference scheme. TC binds the LLM inference to a commitment, an irreversible tag that breaks under tampering, organized in our multivariate Terkle Trees. For LLaMA2, TC adds only 0.97% prover and 0.12% verifier time over inference while improving robustness to tailored LLM attacks by up to 48% over the best prior work requiring a verifier GPU.

Oguzhan Baser, Elahe Sadeghi, Eric Wang +5

Electrical and Computer Engineering, The University of Texas at Austin · Theseus AI Labs · Electrical and Computer Engineering, McGill University +1

LLM Inference LLM Security

Feb 8, 2026cs.CR

Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model

LLM inference is inherently expensive, even a modest slowdown can translate into substantial operating costs and severe availability risks. Recently, a growing body of research known as latency attacks focuses on crafting inputs to trigger worst-case output lengths. However, we report a contrary finding that these algorithmic-level latency attacks are largely ineffective against modern LLM serving systems. We reveal that system-level optimization such as continuous batching provides a logical isolation to mitigate contagious latency impact on co-located users. Thus, in this paper, we shift our focus from the algorithm to the system layer, and introduce a new Fill and Squeeze attack strategy targeting the state transition of the scheduler. Fill'' first exhausts the global KV cache to induce Head-of-Line blocking, while Squeeze'' forces the system into repetitive preemption. By manipulating output lengths using different attack prompts, and leveraging side-channel probing of memory status, we demonstrate that the attack can succeed in a practical black-box setting with much less cost. Extensive evaluations on vLLM indicate up to

75-742\times

TTFT degradation relative to benign baselines and

1.5-4\times

average slowdown on Time Per Output Token compared to existing attacks with 30-40% lower attack cost. Code: https://github.com/Phil-Fan/FS-attack

Tianyi Wang, Huawei Fan, Yuanchao Shu +2

Zhejiang University

LLM Security Adversarial Attacks on LLMs LLM Inference Serving LLM Inference Scheduling

Jan 30, 2026cs.CR

Verifiable Manifest Signing and Transparency Enforcement for Secure MCP-Based LLM Pipelines

Large Language Models (LLMs) are increasingly deployed in tool-driven environments such as healthcare analytics, financial systems, retrieval-augmented generation (RAG), and multi-agent workflows. Although the Model Context Protocol (MCP) standardizes how LLM applications expose and invoke external tools, its baseline model does not require tool-use manifests to be cryptographically authenticated, freshness-checked, policy-bound, or independently auditable before execution. As a result, MCP pipelines may remain vulnerable to manifest tampering, unauthorized tool invocation, replay of stale requests, and weak accountability. This paper presents a manifest-level enforcement layer for MCP-based LLM pipelines. It treats each MCP tool-use manifest as a first-class security object whose canonical form must be policy-validated, freshness-checked, digitally signed, verified before execution, and linked to tamper-evident audit evidence. The framework binds tool invocation to verifiable manifest integrity and fail-closed authorization, separates user-visible request parameters from execution metadata, rejects non-compliant or stale manifests before execution, and records accepted invocations in a Merkle-based transparency log. Evaluation across GPT-5.3, LLaMA-3.5, and DeepSeek-V3 using up to 50,000 manifest instances shows near-linear scalability (R^2 = 0.998), bounded verification latency (<= 9.4 ms on edge devices), and rejection of expired, malformed, replayed, and policy-violating manifests, with rejection rates above 98.7%. Experiments in healthcare, finance, RAG, and multi-agent settings show that manifest-level cryptographic enforcement enables low-overhead, traceable, and auditable execution control for heterogeneous LLM-tool pipelines.

Saeid Jamshidi, Kawser Wazed Nafi, Arghavan Moradi Dakhel +2

SWAT Laboratory, Polytechnique Montr´eal, Montr´eal, Canada · SæT Laboratory, Polytechnique Montr´eal, Montr´eal, Canada

LLM Auditing MCP Security LLM Security

Jan 30, 2026cs.CR

From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching

Semantic caching has emerged as a pivotal technique for scaling LLM applications, widely adopted by major providers including AWS and Microsoft. By utilizing semantic embedding vectors as cache keys, this mechanism effectively minimizes latency and redundant computation for semantically similar queries. In this work, we conceptualize semantic cache keys as a form of fuzzy hashes. We demonstrate that the locality required to maximize cache hit rates fundamentally conflicts with the cryptographic avalanche effect necessary for collision resistance. Our conceptual analysis formalizes this inherent trade-off between performance (locality) and security (collision resilience), revealing that semantic caching is naturally vulnerable to key collision attacks. While prior research has focused on side-channel and privacy risks, we present the first systematic study of integrity risks arising from cache collisions. We introduce CacheAttack, an automated framework for launching black-box collision attacks. We evaluate CacheAttack in security-critical tasks and agentic workflows. It achieves a hit rate of 86% in LLM response hijacking and can induce malicious behaviors in LLM agent, while preserving strong transferability across different embedding models. A case study on a financial agent further illustrates the real-world impact of these vulnerabilities. Finally, we discuss mitigation strategies.

Zhixiang Zhang, Zesen Liu, Yuchong Xie +2

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology · Fudan University

Adversarial Attacks LLM Agent Security LLM Security Semantic Caching

Jan 20, 2026cs.CR

HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation

Large language models (LLMs) are increasingly used for hardware and firmware code generation, but existing studies primarily evaluate functional correctness while largely overlooking security. However, LLM-generated code that appears functionally sound may embed security flaws which could induce catastrophic damages after deployment. This critical research gap motivates us to design a benchmark for assessing security awareness under realistic specifications. In this work, we introduce HardSecBench, a benchmark with 924 tasks spanning Verilog Register Transfer Level (RTL) and firmware-level C, covering 76 hardware-relevant Common Weakness Enumeration (CWE) entries. Each task includes a structured specification, a secure reference implementation, and executable tests. To automate artifact synthesis, we propose a multi-agent pipeline that decouples synthesis from verification and grounds evaluation in execution evidence, enabling reliable evaluation. We evaluate diverse LLMs and find that they often satisfy functional requirements while leaving security risks. We also find that security results vary with prompting. These findings highlight pressing challenges and offer actionable insights for future advancements in LLM-assisted hardware design. Our data and code are available at https://github.com/chenqirui2002/HardSecBench.

Qirui Chen, Jingxian Shuai, Shuangwu Chen +8

University of Science and Technology of China, China · Xi’an Jiaotong University, China · Nanjing University, China +1

LLM Security Secure Code Generation Code Generation Benchmarks LLM Code Generation

Dec 20, 2025cs.CR

SecureCode: A Production-Grade Multi-Turn Dataset for Training Security-Aware Code Generation Models

AI coding assistants produce vulnerable code in 45% of security-relevant scenarios~\cite{veracode2025}, yet no public training dataset teaches both traditional web security and AI/ML-specific defenses in a format suitable for instruction tuning. We present SecureCode, a production-grade dataset of 2,185 multi-turn security training examples spanning two domains: web application security (1,435 examples covering the OWASP Top 10 2021 across 11 languages and 9 frameworks, 100% grounded in documented CVEs and security incidents) and AI/ML security (750 examples covering all 10 OWASP LLM Top 10 2025 categories across more than 40 frameworks, including LangChain, OpenAI, and Hugging Face). Every example follows a 4-turn conversational structure -- feature request; vulnerable and secure implementations with attack demonstrations; advanced probing; and defense-in-depth operational guidance -- designed for direct use in instruction tuning pipelines. Quality assurance combines automated structural validation with multi-agent review from seven specialist AI perspectives (more than 10{,}500 assessments) and an 8-phase remediation pipeline, producing a rubric-calibrated mean quality score of 93.8/100 (

σ= 0.93

) for the AI/ML component. Each example provides SIEM integration strategies, infrastructure hardening recommendations, and testing approaches using production frameworks. We release the unified dataset on Hugging Face with domain-specific loading configurations (web, aiml, default), alongside eight fine-tuned open-source models (3B--20B parameters, QLoRA), and an evaluation framework with four security-specific metrics. To our knowledge, SecureCode is the first public dataset that jointly provides OWASP Top 10 2021 web coverage and OWASP LLM Top 10 2025 AI/ML coverage in a unified conversational schema suitable for instruction tuning.

Scott Thornton

Software Security LLM Security Secure Code Generation LLM Code Generation

Nov 30, 2025cs.CR

Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrieval

When answering user queries, LLMs often retrieve knowledge from external sources stored in retrieval-augmented generation (RAG) databases. These are often populated from unvetted sources, e.g. the open web, and can contain maliciously crafted data. This paper studies attacks that can manipulate the context retrieved by LLMs from such RAG databases. Prior work on such context manipulation primarily injects false or toxic content, which can often be detected by fact-checking or linguistic analysis. A more subtle threat, which we call epistemic bias injection (EBI), is where adversaries inject factually correct yet epistemically biased passages that systematically favor one side of an open-ended issue. Although linguistically coherent and truthful, such adversarial passages effectively crowd out alternative viewpoints during retrieval from the RAG and push LLM outputs towards an attack-desired stance. As a core contribution, we propose a novel characterization of the problem: We give a geometric metric that quantifies stance polarity and epistemic bias. This metric can be computed directly on embeddings of text passages. Leveraging it, we construct EBI attacks and develop a lightweight prototype defense called BiasDef for them. We evaluate them both on a comprehensive benchmark constructed from public question answering datasets. Our results show that: (1) the proposed attack induces significant stance polarity shifts, effectively evading existing retrieval-based sanitization defenses, and (2) BiasDef substantially reduces adversarial retrieval and epistemic bias in LLM's answers. Overall, this demonstrates the new threat as well as the ease of employing epistemic bias metrics for filtering in RAG-enabled LLMs.

Hao Wu, Prateek Saxena

National University of Singapore

Adversarial Attacks on LLMs LLM Security RAG Security

Oct 17, 2025cs.CR

SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses

Large Language Models (LLMs) are increasingly used as interfaces to information, code, and real-world services, making prompt-level security failures a practical concern. Although jailbreak attacks, defenses, datasets, and automated judgers have advanced rapidly, evaluation remains fragmented across threat models, access assumptions, cost budgets, datasets, and success criteria. This makes reported attack success rates and defense gains hard to compare. This SoK systematizes LLM prompt security across concepts, data, tooling, and measurement. We propose linked taxonomies for jailbreak attacks, defenses, and model vulnerabilities, while separating technical mechanisms from attacker and defender capabilities. We also formalize threat, access, and cost assumptions as explicit evaluation metadata. To support reproducible evaluation, we release JailbreakDB, PromptSecurity-Eval, and PromptSecurity, a modular platform that represents each experiment as a tuple of model, attack, defense, dataset, and judger. Using matched evaluations across models, attacks, defenses, and judgers, we show that access regime, native harmful-query behavior, attack cost, defense backfire, taxonomy subcategory, and judger choice all materially affect security conclusions. Together, these artifacts support reproducible, cost-aware, and taxonomy-grounded evaluation of LLM prompt security. Leaderboard: https://datasec-lab.github.io/PromptSecurityLeaderboard/. Dataset: https://huggingface.co/datasets/youbin2014/JailbreakDB. GitHub: https://github.com/datasec-lab/PromptSecurity.

Hanbin Hong, Shuang Wu, Shuya Feng +6

University of Connecticut · University of Alabama at Birmingham · Independent

LLM Security LLM Jailbreak Attacks LLM Safety Evaluation Prompt Injection Attacks on LLMs

Oct 11, 2025cs.CR

MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation

Unlike regular tokens derived from existing text corpora, special tokens are artificially created to annotate structured conversations during the fine-tuning process of Large Language Models (LLMs). Serving as metadata of training data, these tokens play a crucial role in instructing LLMs to generate coherent and context-aware responses. We demonstrate that special tokens can be exploited to construct four attack primitives, with which malicious users can reliably bypass the internal safety alignment of online LLM services and circumvent state-of-the-art (SOTA) external content moderation systems simultaneously. Moreover, we found that addressing this threat is challenging, as aggressive defense mechanisms-such as input sanitization by removing special tokens entirely, as suggested in academia-are less effective than anticipated. This is because such defense can be evaded when the special tokens are replaced by regular ones with high semantic similarity within the tokenizer's embedding space. We systemically evaluated our method, named MetaBreak, on both lab environment and commercial LLM platforms. Our approach achieves jailbreak rates comparable to SOTA prompt-engineering-based solutions when no content moderation is deployed. However, when there is content moderation, MetaBreak outperforms SOTA solutions PAP and GPTFuzzer by 11.6% and 34.8%, respectively. Finally, since MetaBreak employs a fundamentally different strategy from prompt engineering, the two approaches can work synergistically. Notably, empowering MetaBreak on PAP and GPTFuzzer boosts jailbreak rates by 24.3% and 20.2%, respectively.

Wentian Zhu, Zhen Xiang, Wei Niu +1

University of Georgia

Adversarial Attacks on LLMs LLM Security LLM Jailbreak Attacks

Sep 3, 2025cs.CL

From Construction to Injection: Edit-Based Fingerprints for Large Language Models

Reliable model fingerprints are essential for protecting large language models (LLMs) against unauthorized redistribution and commercial misuse. In black-box deployment, verification is hindered by defensive filtering of suspected fingerprint queries, as well as by downstream model modifications that may weaken embedded ownership evidence. These risks require fingerprints to be robust in both construction and injection. For construction, prior paradigms face an imperceptibility trade-off: natural-language fingerprints may be accidentally activated, whereas garbled fingerprints are statistically exposed and easier to filter. For injection, existing methods struggle to preserve persistent trigger--target behaviors under model modification. We propose an end-to-end injected fingerprinting framework to address these challenges. Code-mixing Fingerprints (CF) use lowest-perplexity code-mixing under a high-complexity constraint to mitigate this two-sided imperceptibility trade-off. Multi-Candidate Editing (MCEdit) constructs structurally redundant, margin-separated trigger--target mappings to enable graceful degradation under model modification. Extensive evaluations on imperceptibility, detectability, and harmlessness demonstrate robust ownership verification with negligible impact on utility.

Yue Li, Xin Yi, Dongsheng Shi +3

East China Normal University · Hasso Plattner Institute/University of Potsdam

Language Model Fingerprinting LLM Security Language Model Robustness

Aug 4, 2025cs.CR

Data Security in Large Language Models: Risks, Defense, and Directions

Large Language Models (LLMs), now a foundation in advancing natural language processing, power applications such as text generation, machine translation, and conversational systems. Despite their transformative potential, these models inherently rely on massive amounts of training data, often collected from diverse and uncurated sources, which exposes them to serious data security risks. Harmful or malicious data can compromise model behavior, leading to toxic outputs or hallucinations, while also creating vulnerabilities to data-driven attacks such as prompt injection and data poisoning. As LLMs continue to be integrated into critical real-world systems, understanding and addressing these data-centric security risks is imperative to safeguard user trust and system reliability. This survey offers a comprehensive overview of the main data security risks facing LLMs and reviews current defense strategies, including adversarial training, data cleaning, output guardrails, Reinforcement Learning from Human Feedback (RLHF), data augmentation, and Retrieval-Augmented Generation (RAG)/agent defenses. Additionally, we categorize and analyze relevant datasets used for assessing robustness and security across different domains, providing guidance for future research. Finally, we highlight key research directions that focus on data provenance and traceability, verifiable machine forgetting, secure model updates, standardized evaluation framework, explainability-driven security analysis, and effective governance frameworks, aiming to promote the safe and responsible development of LLM technology. This work seeks to inform researchers, practitioners, and policymakers, driving progress toward data security in LLMs.

Kang Chen, Xiuze Zhou, Yuanhui Yu +4

School of Computer Engineering, Jimei University, Xiamen, 361021, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · School of Informatics, Xiamen University, Xiamen, 361102, China +1

Adversarial Attacks on LLMs LLM Security Prompt Injection Attacks on LLMs Language Model Robustness Data Poisoning Attacks

Apr 24, 2025cs.LG

Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks

Large language models (LMs) offer broad generalization capabilities but require vast amounts of data and computational resources for domain-specific tasks; small models (SMs), in contrast, are more efficient and tailored to specific domains yet lack general-purpose coverage. Taking a collaborative approach, where large and small models work synergistically, can accelerate the adaptation of LLMs to private domains and unlock new potential in AI. This survey presents a comprehensive overview of recent advances and challenges in harnessing the collaborative power of large and small models for private-domain adaptation. It specifically focuses on the unique constraints of cross-boundary environments, where models belong to distinct parties, and examines the resulting tensions among data privacy, model security, integrity, and resource limitations. By analyzing the information flow between distinct model and data stakeholders, we propose a unified taxonomy that classifies research into three primary directions: downward knowledge transfer (LM to SM), upward knowledge transfer (SM to LM), and inference-time collaboration across parties. Drawing on this taxonomy, we analyze the core challenges inherent to cross-boundary information exchange, including data-privacy, model-security, and integrity threats as well as efficiency constraints, and synthesize these into a multi-objective optimization problem that governs practical deployment. Finally, we review key open challenges inherent to such hybrid approaches and outline promising directions for future research. By offering a principled, boundary-centric view of this rapidly evolving landscape, this survey aims to serve as a structured resource for researchers and practitioners advancing privacy-aware, resource-efficient AI deployment.

Yang Liu, Kejia Zhang, Bingjie Yan +11

The Hong Kong Polytechnic University, China · Institute for AI Industry Research, Tsinghua University, China · Institute for AI Industry Research, Tsinghua University, China and Shanghai Jiao Tong University, China +3

Privacy-Preserving Language Models LLM Fine-Tuning LLM Security Multi-LLM Collaboration

Mar 6, 2025cs.CR

The Challenge of Identifying the Origin of Black-Box Large Language Models

The tremendous commercial potential of large language models (LLMs) has heightened concerns over their unauthorized use. To address this, we focus on the task of identifying the origin of black-box LLMs. We further propose PlugAE, an effective and efficient identification method that proactively leverages LLM-specific adversarial embeddings and allows users to customize copyright tokens on a targeted query set. Extensive experiments demonstrate that PlugAE outperforms both state-of-the-art model watermarking and fingerprinting methods in accuracy and robustness. We further analyze its stealthiness and reliability from three complementary perspectives and conduct ablation studies under various configurations, confirming its practicality for real-world misuse detection.

Ziqing Yang, Yixin Wu, Yun Shen +3

CISPA Helmholtz Center for Information Security · NetApp · TikTok Inc.

Language Model Fingerprinting LLM Security

Feb 18, 2025cs.CL

UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models

Large Language Models (LLMs) are vulnerable to attacks like prompt injection, backdoor attacks, and adversarial attacks, which manipulate prompts or models to generate harmful outputs. In this paper, departing from traditional deep learning attack paradigms, we explore their intrinsic relationship and collectively term them Prompt Trigger Attacks (PTA). This raises a key question: Given a prompt, can we tell whether a hidden trigger is steering the model's behavior? We propose UniGuardian, to the best of our knowledge the first training-free LLM detector to jointly detect successfully activated prompt injection, backdoor, and adversarial attacks without knowing the attack type. Its shared mechanism measures how structured prompt perturbations shift the model's output distribution. Additionally, we introduce a single-forward strategy to optimize the detection pipeline, enabling simultaneous attack detection and text generation within a shared batched forward pass at each decoding step. Our experiments confirm that UniGuardian accurately and efficiently identifies trigger-activated prompts in LLMs.

Huawei Lin, Yingjie Lao, Tony Geng +2

Rochester Institute of Technology · Tufts University · University of Rochester +1

LLM Backdoor Attacks Adversarial Attacks on LLMs Backdoor Defense in LLMs LLM Security Prompt Injection Attacks on LLMs Prompt Injection Detection

Jun 3, 2024cs.CR

BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by retrieving relevant information from external knowledge bases to provide more accurate, contextually informed, and up-to-date responses. However, this reliance on external knowledge introduces significant security vulnerabilities, as many RAG systems (e.g., Google Search) rely on large and unsanitized data repositories (e.g., Reddit). In this paper, we unveil a novel threat in which attackers steer the RAG system's response by injecting malicious passages into its knowledge base. When a user's query contains attacker-specified trigger words, the RAG retrieves and refers to these malicious passages, enabling the attacker to steer the response without altering the user input or modifying the RAG weights. BadRAG operates in two phases: (i) malicious passages are optimized to be retrieved exclusively when trigger words appear in user queries; (ii) these passages are meticulously crafted to achieve adversarial generation objectives, including denial of service, sentiment manipulation, context leakage, and tool misuse. Our experiments show that injecting just 10 malicious passages (0.04% of the external corpora) achieves a 98.2% retrieval success rate and increases negative response rates from 0.22% to 72% for queries containing triggers.

Jiaqi Xue, Mengxin Zheng, Yebowen Hu +3

University of Central Florida · Emory University · Samsung Research America

LLM Backdoor Attacks RAG Poisoning Attacks LLM Security RAG Security