Adversarial Robustness of Language Models

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 123

All topics

CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models
Jun 1, 2026Swapnil ParekhLLM Auditing Data Contamination in Language Models
Intuit Inc.
Defenses & Enablers For Skill Injection Attacks on Terminal Based Agents
Jun 1, 2026Yoshinari Fujinuma, Varun Gangal, Traian Rebedea +4Adversarial Attacks on LLMs LLM Agent Security
Patronus AI · NVIDIA
CSULoRA: Closest Safe Update Low-Rank Adaptation
May 28, 2026Oleksandr Marchenko Breneur, Adelaide Danilov, Aria Nourbakhsh +1Adapter Tuning Low-Rank Adaptation
Department of Computer Science, University of Luxembourg Esch-sur-Alzette, Luxembourg
The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure
May 27, 2026Yubo Li, Ramayya Krishnan, Rema PadmanCoT Faithfulness Reasoning Evaluation
Carnegie Mellon University
Mitigating Adaptive Attacks against Reasoning Models with Activation Consistency Training
May 27, 2026Avidan Shah, Jannik Brinkmann, Rico AngellPrompt Injection Defense Adversarial Robustness of Language Models
New York University · Technical University Clausthal
SafeMed-R1: Clinician-Audited Safety and Ethics Alignment for Medical Large Language Models
May 27, 2026Chao Ding, Mouxiao Bian, Tianbin Li +12Healthcare LLM Auditing
Shanghai Artificial Intelligence Laboratory, Shanghai, China · Joint Laboratory of Biomedical Artificial Intelligence, Shanghai East Hospital, Tongji University School of Medicine, Shanghai, China · School of Computer Science and Technology, Tongji University, Shanghai, China +5
Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Security
May 27, 2026Xiang Fang, Wanlong FangLLM Security Prompt Injection Defense
School of Software Engineering, Huazhong University of Science and Technology · Nanyang Technological University, Singapore
Which Changes Matter? Towards Trustworthy Legal AI via Relevance-Sensitive Evaluation and Solver-Grounded Reasoning
May 26, 2026Chen Linze, Cai Yufan, Hou Zhe +1Adversarial Robustness of Language Models Language Model Robustness
National University of Singapore · Griffith University
Curriculum Learning for Safety Alignment
May 25, 2026Sandeep Kumar, Virginia Smith, Chhavi YadavLLM Alignment Direct Preference Optimization
Carnegie Mellon University · Simons Institute, UC Berkeley
From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP
May 24, 2026Most. Sharmin Sultana Samu, MD. Tanvir Ahmed Seum, Md. Rakibul IslamLLM Auditing Human-in-the-Loop Annotation
Department of Computer Science and Engineering, BRAC University, Dhaka-1212. · Department of Electrical and Electronic Engineering, Rajshahi University of Engineering and Technology, Rajshahi-6204.
Reflect-Guard: Enhancing LLM Safeguards against Adversarial Prompts via Logical Self-Reflection
May 24, 2026Lixing Lin, Juli You, Yue Li +4Language Model Safety Evaluation LLM Guardrails
yaleYale University, New Haven, CT, USA · indIndependent Researcher · columbiaColumbia University, New York, NY, USA +2
Steering Beyond the Support: Adversarial Training on Unsupervised Jailbroken Activation Simulation
May 23, 2026Luoyu Chen, Weiqi Wang, Zhiyi Tian +5Adversarial Training Adversarial Robustness of Language Models
University of Technology Sydney, Sydney, Australia · Xi’an Jiaotong University, Xi’an, China · School of Cyber Science and Engineering, Southeast University, Nanjing, China.
Harder to Defend: Towards Chinese Toxicity Attacks via Implicit Enhancement and Obfuscation Rewriting
May 21, 2026Jingyi Kang, Junyu Lu, Bo Xu +4Language Model Safety Evaluation Toxicity Detection
1Dalian University of Technology · 2The University of Tokyo · 3Singapore University of Technology and Design
On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation
May 20, 2026Andy Han, Kristina Fujimoto, Avidan Shah +5Language Model Safety Evaluation LLM Safety Alignment
1New York University
Towards Context-Invariant Safety Alignment for Large Language Models
May 20, 2026Yixu Wang, Yang Yao, Xin Wang +4RL for Language Models LLM Alignment
Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China.
REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak
May 20, 2026Jiachen Ma, Jiawen Zhang, Xiangtian Li +3LLM Self-Refinement LLM Safety
Fudan University · Shanghai Artificial Intelligence Laboratory · Zhejiang University
Adaptive Probe-based Steering for Robust LLM Jailbreaking
May 19, 2026Junxi Chen, Junhao Dong, Xiaohua XieLanguage Model Steering Adversarial Attacks on LLMs
School of Computer Science and Engineering, Sun Yat-Sen University, China · Nanyang Technological University, Singapore.
The Evaluation Game: Beyond Static LLM Benchmarking
May 19, 2026Paul Wang, Jade Garcia-Bourrée, Anne-Marie Kermarrec +1Adversarial Training LLM Auditing
Sorbonne Université, CNRS, LIP6 · École Polytechnique Fédérale de Lausanne
A Few Steps Further: Why Defenses Against Malicious Finetuning Erode Under Continued Training
May 14, 2026Itay Zloczower, Eyal Lenga, Gilad Gressel +1Language Model Safety Evaluation Adversarial Attacks on LLMs
Ben-Gurion University of the Negev · Amrita Vishwa Vidyapeetham, Amritapuri
Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation
May 14, 2026Yu Fu, Longxuan Yu, Haz Sameen Shahgir +4LLM Alignment On-Policy Self-Distillation
1UC Riverside · 2International Computer Science Institute · 3Microsoft,∗ +1
Measuring and Mitigating Toxicity in Large Language Models: A Comprehensive Replication Study
May 13, 2026Mokshit Surana, Archit Rathod, Akshaj SatishkumarLanguage Model Safety Evaluation Toxicity Detection
University of Illinois Chicago
Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigation
May 12, 2026Cristian Morasso, Anisa Halimi, Muhammad Zaid Hameed +1Adversarial Training Adversarial Prompt Generation
IBM Research · Trinity College Dublin, Ireland
Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning
May 12, 2026Chi Zhang, Haibo Qiu, Qiming Zhang +3RL for Language Model Reasoning Adversarial Attacks on LLMs
School of Computer Science, Wuhan University · Meituan Inc · Independent Researchers +1
Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories
May 9, 2026Dongcheng Zhang, Yi Zhang, Yuxin Chen +3LLM Self-Correction RL for Language Model Reasoning
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · National University of Singapore
Internalizing Safety Understanding in Large Reasoning Models via Verification
May 9, 2026Yi Zhang, Yuxin Chen, Leheng Sheng +4LLM Safety Alignment LLM Safety
University of Science and Technology of China · National University of Singapore · Shanghai Artificial Intelligence Laboratory
Latent Personality Alignment: Improving Harmlessness Without Mentioning Harms
May 8, 2026Linh Le, David Williams-King, Mohamed Amine Merzouk +2Adversarial Training Neural Network Robustness
McGill University · Mila – Quebec AI Institute · ERA +1
The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play
May 8, 2026Gabriele La Malfa, Emanuele La Malfa, Saar Cohen +4Self-Play RL Language Model Safety Evaluation
Department of Informatics, King’s College London · Department of Computer Science, University of Oxford · Institute for Decentralized AI +1
Can You Break RLVER? Probing Adversarial Robustness of RL-Trained Empathetic Agents
May 8, 2026Deeraj S K, Sadhana Devarajan, Krishna Mehra +1Emotion Recognition in Conversations Adversarial Robustness of Language Models
Department of Artificial Intelligence, Sardar Vallabhbhai National Institute of Technology, Surat, India
Research on Security Enhancement Methods for Adversarial Robust Large Language Model Intelligent Agents for Medical Decision-Making Tasks
May 7, 2026Saisai HuClinical Decision Support LLM Agent Security
Pace University · New York, NY 10038 · United States
Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks
May 7, 2026Guoxin Lu, Letian Sha, Qing Wang +4LLM Fine-Tuning LLM Safety Alignment
Nanjing University of Posts and Telecommunications, Nanjing, China.

Jun 1, 2026, cs.LGJ/K move · Enter open · S save

CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models

Swapnil Parekh

Intuit Inc.

LLM Auditing Data Contamination in Language Models Adversarial Robustness of Language Models Data Poisoning Attacks

Open paper AskPDF arXiv

Adversaries can implant latent harmful behavior by poisoning as few as 1% of fine-tuning examples. The contamination is invisible to every output-level defense: harmful behavior lies dormant in the model's hidden-state geometry and does not appear in generated text until contamination exceeds 7.5%. We introduce CANARY (Contamination Auditor via Neural Activation Representation Yield), a zero-label checkpoint auditor that detects this hidden shift directly from two forward passes over an unlabeled prompt set. CANARY projects the hidden-state difference through a Sparse Autoencoder, filtering style noise to isolate meaningful semantic drift. It achieves AUROC = 1.000 at 1% contamination (95% CI = [0.997, 1.000]; Cohen's d = 3.28) across four model architectures and two training paradigms, 7.5x below where any output-level method fires, with zero false positives on benign fine-tuning and full robustness to style-matching and gradient-noise adaptive attacks. The same SAE feature basis drives a complete governance pipeline: SAE-filtered amplification surfaces latent harm at a 5x higher rate than standard generation; score-ranked prompts yield 4.2x red-teaming lift; and suppressing a handful of contamination-specific features at inference time reduces harm from 70% to 10% with no perplexity penalty. CANARY is the first zero-label framework to detect, verify, prioritize, and remediate supply-chain contamination from hidden states alone.

Momentum

Related topics

Latest papers 123