LLM Guardrails

LLM: Large Language Model

LLM guardrails assess prompts, generated responses, and multi-step agent actions for harmful content, policy violations, prompt injection, and unsafe tool use. Common approaches include specialized classifiers and LLM judges, hidden-state or logit probes, streaming monitors, and runtime controls such as policy-as-code, constrained decoding, and formal verification. Evaluations span moderation and jailbreak benchmarks such as HarmBench, agent-task settings such as AgentDojo and τ²-Bench, and domain-specific, multilingual, and long-context tests. The field’s central tension is balancing missed risks against over-refusal, latency, and cost, while determining whether strong benchmark scores translate into safer end-to-end systems.

Latest papers

195 papers30 papers in the last four weeks, up 43% on the four weeks before. 0.2% of all new papers.

SelfGrader: LLM Jailbreak Detection via Anchored Token-Level Logits
Apr 1, 2026Zikai Zhang, Rui Hu, Olivera Kotevska +1LLM Guardrails LLM Security
Department of Computer Science and Engineering, University of Nevada, Reno, Reno, USA · Oak Ridge National Laboratory, Oak Ridge, USA
CivicShield: A Cross-Domain Defense-in-Depth Framework for Securing Government-Facing AI Chatbots Against Multi-Turn Adversarial Attacks
Mar 30, 2026KrishnaSaiReddy PatilLLM Security Cybersecurity
Evaluating Prompt Injection Defenses for Educational LLM Tutors: Security-Usability-Latency Trade-offs
Mar 29, 2026Alexandre Cristovão MaioranoPrompt Injection Defense LLM Guardrails
Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models
Mar 26, 2026Hieu Xuan Le, Benjamin Goh, Quy Anh TangPrompt Injection Detection LLM Guardrails
GovTech, Singapore
Designing for Disagreement: Front-End Guardrails for Assistance Allocation in LLM-Enabled Robots
Mar 17, 2026Carmen NgLLM Guardrails AI Agent Governance
Technical University of Munich, Germany
Proof-of-Guardrail in AI Agents and What (Not) to Trust from It
Mar 6, 2026Xisen Jin, Michael Duan, Qin Lin +4LLM Guardrails Remote Attestation
Sahara AI · University of Southern California
Design Behaviour Codes (DBCs): A Taxonomy-Driven Layered Governance Benchmark for Large Language Models
Mar 5, 2026G. Madan Mohan, Veena Kiran Nambiar, Kiranmayee JanardhanLLM Safety Benchmarks LLM Guardrails
Founder, Yonih Ventures, Bangalore, India | Canada · Professor, Ramaiah University of Applied Sciences, Bangalore, India · Data Scientist, Bangalore, India
ExpGuard: LLM Content Moderation in Specialized Domains
Mar 3, 2026Minseok Choi, Dongjin Kim, Seungbin Yang +5LLM Guardrails Content Moderation
KAIST AI · Financial Tech Lab, KakaoBank Corp
PARCER as an Operational Contract to Reduce Variance, Cost, and Risk in LLM Systems
Mar 1, 2026Elzo Brito dos Santos FilhoContext Engineering LLM Reliability
FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderation
Feb 27, 2026Zhihao Ding, Jinming Li, Ze Lu +1Content Moderation LLM Guardrails
The Hong Kong Polytechnic University · ByteDance
Improving LLM Reliability through Hybrid Abstention and Adaptive Detection
Feb 17, 2026Ankit Sharma, Nachiket Tapas, Jyotiprakash PatraLLM Guardrails LLM Reliability
Department of Computer Science & Engineering, Chhattisgarh Swami Vivekanand Technical University, Bhilai, Chhattisgarh, India
StepShield: When, Not Whether to Intervene on Rogue Agents
Jan 29, 2026Gloria Felicia, Zitha Sasindran, Jinfeng He +3LLM Guardrails AI Agent Security
University of Virginia · Indian Institute of Science Bangalore · Indian Institute of Science, Bangalore +5
YuFeng-XGuard: A Reasoning-Centric, Interpretable, and Flexible Guardrail Model for Large Language Models
Jan 22, 2026Junyu Lin, Meizhen Liu, Xiufeng Huang +12LLM Guardrails LLM Interpretability
Alibaba-AAIG
NeuroFilter: Activation-Based Guardrails for Privacy-Conscious LLM Agents
Jan 21, 2026Saswat Das, Ferdinando FiorettoLLM Guardrails Privacy Leakage in Language Models
University of Virginia
Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection
Aug 28, 2025Harethah Abu Shairah, Hasan Abed Al Kader Hammoud, George Turkiyyah +1LLM Guardrails LLM Safety Alignment
King Abdullah University of Science and Technology (KAUST)

Apr 1, 2026J/K move · Enter open · S save

SelfGrader: LLM Jailbreak Detection via Anchored Token-Level Logits

Zikai Zhang, Rui Hu, Olivera Kotevska, Jiahao Xu

Department of Computer Science and Engineering, University of Nevada, Reno, Reno, USA · Oak Ridge National Laboratory, Oak Ridge, USA

LLM Guardrails LLM Security LLM Jailbreak Attacks

Open paper AskPDF arXiv

Large Language Models (LLMs) are powerful tools for answering user queries, yet they remain highly vulnerable to jailbreak attacks. Existing guardrail methods typically rely on internal features or textual responses to detect malicious queries, which either introduce substantial latency or suffer from randomness in text generation. To overcome these limitations, we propose SelfGrader, a lightweight guardrail method that formulates jailbreak detection as a numerical grading problem using anchored token-level logits. Specifically, SelfGrader evaluates the safety of a user query within a compact set of numerical tokens (NTs) (e.g., 0-9) and interprets their logit distribution as an internal safety signal. To align these signals with the target safety rubric, SelfGrader constructs Probably Approximately Correct-guided ICL anchor examples and introduces a dual-perspective scoring rule that considers both the maliciousness and benignness of the query, yielding a stable and interpretable score that reflects harmfulness and reduces the false positive rate simultaneously. Extensive experiments across diverse jailbreak benchmarks, adaptive attacks, benign prompt benchmarks, multiple LLMs, and state-of-the-art guardrail baselines demonstrate that SelfGrader achieves strong robustness with low false positive rates, memory overhead, and latency.