Adversarial Prompts

Follow this topic Ask about this topic

Momentum

0 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 54

All topics

Harmful Content Generation in Text-to-Image Models: Capabilities and Moderation Limitations
Oct 5, 2026Paschalis Giakoumoglou, Manos Schinas, Symeon PapadopoulosModern Text-To-Image Models Child Sexual Abuse Material
Centre for Research and Technology Hellas (CERTH), Thessaloniki, Greece
Safe Image Generation via Reinforcement Learning
Oct 5, 2026Eungyeol Han, Jong-Seok LeeAdversarial Prompts
Yonsei University School of Integrated Technology
Gaussian Core LoRA: Distribution-Aware Dynamic Adaptation for Broad Concept Erasure
Sep 1, 2026Qinghui Gong, Xunlei Chen, Yu-Xuan Zhang +2Concept Erasure Adversarial Prompts
Southwest Jiaotong University · University of Electronic Science and Technology of China
Not an A11y: How Android Accessibility Exposes Mobile AI Agents to Indirect Prompt Injection
Aug 9, 2026Rahul Deivasigamani, Sayeda Faatin Alvi, Derqui Andrea +2Artificial Intelligence Agents Device
Radboud University · Nijmegen, The Netherlands
STEMMA: An Adversarial Multi-Agent Framework for Evaluating Self-Identity Consistency in LLMs
Aug 8, 2026Nuthakki Siva Gopala Krishna, Kanishka JainKnowledge Distillation Adversarial Prompts
BML Munjal, Haryana, India · Indian Institute of Technology, Delhi, India
Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation
Aug 5, 2026Ye Leng, Junjie Chu, Yiting Qu +3Hate Speech Hate Speech Detection
1CISPA Helmholtz Center for Information Security · 2Hewlett Packard Enterprise
Test-Time Scaling for Safe Text-Guided Image Generation via Intermediate Clean Estimates
Aug 4, 2026Jinya Sakurai, Shueicheng Yan, Xun XuText-To-Image Diffusion Models Text-To-Image
1Nanyang Technological University · 2National University of Singapore · Institute of Advanced Intelligence and Computing (IAIC), A*STAR +1
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Aug 3, 2026Nan Chen, Zhouhao Yang, Soufiane HayouTraining-Free Adversarial Prompts
Department of Applied Mathematics and Statistics Johns Hopkins University
Mind the Gap: Zero-Query Jailbreaks via Filter-Generator Discrepancy in Text-to-Image Systems
Aug 2, 2026Wanguang Li, Zhaoxin Wang, Handing WangAdversarial Prompts Jailbreak Success Rates
School of Artificial Intelligence, Xidian University, Shaanxi, China
Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed
Jul 27, 2026Xinnuo Xu, Anja Thieme, Daniela Massiceti +6Toxicity Adversarial Prompts
Microsoft Research Cambridge, UK · Microsoft Paris, France
Adversarial Prompts for Acceptance Collapse in Speculative Decoding
Jul 23, 2026Run Wang, Chaoyi Zhou, Xi Liu +7Speculative Decoding Adversarial Prompts
Clemson University · Wayne State University · University of Washington
Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models
Jul 20, 2026Dongdong Yang, Deyue Zhang, Zhao Liu +5Adversarial Prompts Jailbreak Success Rates
AI Security Lab, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China
Adversarial Prompting Framework for AI Safety Assessment
Jul 15, 2026Yash Bhatnagar, Kunal Banerjee, Anirban ChatterjeeAdversarial Prompts Artificial Intelligence Safety
Microsoft · Bengaluru, Karnataka, India · Walmart Global Tech
Mechanistic Interpretability of LLM Jailbreaks via Internal Attribution Graphs
Jul 8, 2026Anupam Wagle, Ifrat Ikhtear Uddin, Chaowei Zhang +1Large Language Model Jailbreaks Adversarial Prompts
Department of Computer Science, University of South Dakota, USA · School of Information and Artificial Intelligence, Yangzhou University, Yangzhou, China
MPSelectTune: Prompt-type Selection for Fine-tuning improves Concept Unlearning in LLMs
Jul 4, 2026Shubhadip Nag, Srinjoy Das, Agniva Saha +5Large Language Model Unlearning Adversarial Prompts
*IIT Kharagpur · †MPI-SWS · ‡HPE Labs
kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail
Jul 2, 2026Mahmoud Abdelfattah, Hamid Nasiri, Peter GarraghanLarge Language Model Safety Adversarial Prompts
Lancaster University · Mindgard
Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling
Jun 29, 2026Yoonseok Choi, Chaeyoung Oh, Hyunjun Choi +2Text-To-Image Diffusion Models Adversarial Prompts
Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · Center for Humanoid Research, KIST, Seoul, Republic of Korea
Concept Removal for Frontier Image Generative Models
Jun 24, 2026Aditya Kumar, Pierre Joly, Adam Dziedzic +1Text-To-Image Diffusion Models Diffusion Models
CISPA Helmholtz Center for Information Security, Germany.
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Jun 24, 2026Abrar Alotaibi, Raed Mughus, Moataz AhmedLarge Language Model Reliability Red-Teaming
Information and Computer Science Department, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia. · College of Computer Science and Information Technology, Imam Abdulrahman Bin Faisal University, Dammam, 31441, Saudi Arabia. · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia.
When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations
Jun 5, 2026Mahdi AlkaeedLarge Language Models Fail Adversarial Prompts
Department of Computer Science and Engineering, Doha, Qatar
Adversarial Creation and Detection of AI-Generated Social Bot Content
Jun 5, 2026Mykola Trokhymovych, Ricardo Baeza-Yates, Alessandro Flammini +2Ai-Generated Content Detection Generative Artificial Intelligence
Universitat Pompeu Fabra, Barcelona, Spain · Observatory on Social Media, Indiana University, Bloomington, Indiana, USA · KTH Royal Institute of Technology, Stockholm, Sweden
Before the Model Learns the Bug:Fuzzing RLVR Verifiers
May 31, 2026Jaideep RayReinforcement Learning With Verifiable Reward Verifiable Rewards
IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs
May 30, 2026F. Carichon, S. Sharma, M. Girard +2Creativity Large Language Model Generation
McGill University, Montreal, Canada · Mila, Montreal, Canada · Concordia University, Montreal, Canada
SNARE: Adaptive Scenario Synthesis for Eliciting Overeager Behavior in Coding Agents
May 27, 2026Yubin Qu, Yi Liu, Gelei Deng +4Coding Agents Synthetic Environments
1Griffith University · 2Quantstamp · 3Nanyang Technological University +2
Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Security
May 27, 2026Xiang Fang, Wanlong FangAdversarial Prompts Large Language Model Safety
School of Software Engineering, Huazhong University of Science and Technology · Nanyang Technological University, Singapore
PAST2HARM: A Simple Adaptive Past Tense Attack for Jailbreaking Multimodal AI
May 26, 2026Snehasis MukhopadhyayAdversarial Prompts Harms
Indian Institute of Information Technology, Kalyani
Erased but Exploitable: Black-box Embedding-Aware Prompting Against Unlearned Text-to-Image Diffusion Models
May 25, 2026Arian Komaei Koma, Seyed Amir Kasaei, AmirMahdi Sadeghzadeh +1Text-To-Image Diffusion Models Adversarial Prompts
Department of Computer Engineering Sharif University of Technology
Concept Unlearning via Cross-Attention Activation Projection for Diffusion Models
May 25, 2026Saemi Moon, Suhyeon Jun, Seoyeon Lee +1Text-To-Image Diffusion Models Diffusion Models
CSE, POSTECH · GSAI, POSTECH
Reflect-Guard: Enhancing LLM Safeguards against Adversarial Prompts via Logical Self-Reflection
May 24, 2026Lixing Lin, Juli You, Yue Li +4Large Language Model Safety Adversarial Prompts
yaleYale University, New Haven, CT, USA · indIndependent Researcher · columbiaColumbia University, New York, NY, USA +2
Detecting Fluent Optimization-Based Adversarial Prompts via Sequential Entropy Changes
May 19, 2026Mohammed Alshaalan, Miguel R. D. RodriguesAdversarial Prompts Token-Level Entropy
Department of Electronic and Electrical Engineering, University College London, London, United Kingdom.
The Evaluation Game: Beyond Static LLM Benchmarking
May 19, 2026Paul Wang, Jade Garcia-Bourrée, Anne-Marie Kermarrec +1Adversarial Evaluation Adversarial Prompts
Sorbonne Université, CNRS, LIP6 · École Polytechnique Fédérale de Lausanne
TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models
May 17, 2026Xin Wang, Yixu Wang, Jiaming Zhang +6Vision-Language Model Adaptation Adversarial Training
REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations
May 12, 2026Buyun Liang, Jinqi Luo, Liangzu Peng +6Large Language Model Hallucination Attacker Large Language Model
University of Pennsylvania
Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models
May 11, 2026Nicola Novello, Andrea M. TonelloConcept Erasure Text-To-Image Diffusion Models
University of Klagenfurt Klagenfurt, Austria
What Concepts Lie Within? Detecting and Suppressing Risky Content in Diffusion Transformers
May 11, 2026Chenyu ZhangText-To-Image Diffusion Models Diffusion Transformers
Tianjin University · Tianjin, China
When Prompts Become Payloads: A Framework for Mitigating SQL Injection Attacks in Large Language Model-Driven Applications
May 11, 2026Farzad Nourmohammadzadeh Motlagh, Mehrdad Hajizadeh, Mehryar Majd +3Instruction Injection Attacks Text-To-Sql
Hasso Plattner Institute, University of Potsdam, Germany · Chemnitz University of Technology, Chemnitz, Germany
The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
May 9, 2026Ismail Hossain, Tanzim Ahad, Md Jahangir Alam +3Jailbreak Attacks Jailbreaks
Department of Computer Science, University of Texas at El Paso, TX, USA 79902 · School of Computing, Southern Illinois University Carbondale, IL, USA 62901 · University of Illinois Urbana-Champaign, IL, USA
LLM-Agnostic Semantic Representation Attack
May 9, 2026Jiawei Lian, Jianhong Pan, Lefan Wang +4Attacker Large Language Model Adversarial Prompts
School of Electronics and Information, Northwestern Polytechnical University, Xi’an 710129, China · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR · Hong Kong Institute of AI for Science, City University of Hong Kong, Hong Kong SAR
PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI
May 7, 2026Wesley Hanwen Deng, Mingxi Yan, Sunnie S. Y. Kim +5Red-Teaming Human-Ai Collaboration
Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Apple Seattle, Washington, USA · Apple Cupertino, California, USA +1
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
May 7, 2026Dasol Choi, Eugenia Kim, Jaewon Noh +14Large Language Model Safety Cultural Awareness
AIM Intelligence · Microsoft · Korea AISI +7
On the Hardness of Junking LLMs
May 6, 2026Marco Rando, Samuel VaiterLarge Language Model Jailbreaks Jailbreak Attacks
Université Côte d’Azur, Inria, CNRS, LJAD, Nice, France · CNRS & Université Côte d’Azur, LJAD, Nice, France
Self-Mined Hardness for Safety Fine-Tuning
May 4, 2026Prakhar Gupta, Garv Shah, Donghua ZhangLarge Language Model Jailbreaks Jailbreaks
University of Michigan
MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
May 3, 2026Jialin Song, Xiaodong Liu, Weiwei Yang +4Large Language Model Safety Jailbreak Attacks
Simon Fraser University · Microsoft Research
A Theoretical Game of Attacks via Compositional Skills
May 1, 2026Xinbo Wu, Huan Zhang, Abhishek Umrawal +1Attacker Large Language Model Adversarial Prompts
University of Illinois Urbana-Champaign, Urbana, IL, USA · Stony Brook University, Stony Brook, NY, USA.
SafeReview: Defending LLM-based Review Systems Against Adversarial Hidden Prompts
Apr 29, 2026Yuan Xin, Yixuan Weng, Minjun Zhu +5Peer Review Adversarial Prompts
1CISPA · 2Westlake University · 3Southern University of Science and Technology +1
CAP-CoT: Cycle Adversarial Prompt for Improving Chain of Thoughts in LLM Reasoning
Apr 25, 2026Shuxu Chen, Yitian Zhou, Jiaquan Zhang +6Adversarial Prompts LLM Reasoning Strategies
Department of Electronics and Information Convergence Engineering, Kyung Hee University, Republic of Korea · University of Electronic Science and Technology of China, Chengdu, China · East China Jiaotong University, Nanchang, China +1
Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF
Apr 20, 2026Yuan Fang, Yiming Luo, Aimin Zhou +1Adversarial Prompts Toxicity
East China Normal University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China
The Granularity Gap: A Multi-Dimensional Longitudinal Audit of Sycophancy in Gemini Models
Apr 19, 2026Patrick KeoughSycophancy Model Auditing
Independent Researcher Bremerton, WA, USA
Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence
Apr 9, 2026Niklas Herbster, Martin Zborowski, Alberto Tosato +2Adversarial Prompts
1Tara Research · 2Technical University of Munich · 3Mila Quebec AI Institute
T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Search
Mar 21, 2026Hyomin Lee, Sangwoo Park, Yumin Choi +3Attacker Large Language Model Red-Teaming
KAIST · University of California, Los Angeles
Selective Fine-Tuning for Targeted and Robust Concept Unlearning
Feb 8, 2026Mansi, Avinash Kori, Francesca Toni +1Adversarial Prompts Diffusion Models
Department of Computing Imperial College London, United Kingdom
LoRAShield: Data-Free Editing Alignment for Secure Personalized LoRA Sharing
Jul 5, 2025Jiahao Chen, Junhao Li, Yiming Wang +6Adversarial Prompts Text-To-Image Diffusion Models
Zhejiang University College of Computer Science and Technology Hangzhou, China · Guizhou Medical University School of Biology and Engineering Guiyang, China · Zhejiang University School of Software Technology Ningbo, China +1
ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork
May 29, 2025Caroline Wang, Arrasy Rahman, Benjamin Nativi +4Heterogeneous Robot Teams Adversarial Prompts
Department of Computer Science The University of Texas at Austin · Visa AI · College of Computing and Data Science Nanyang Technological University +1
VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wild
Oct 1, 2024Junlin Fang, Wenyu Chen, Reshmi Ghosh +7Adversarial Prompts Multiple Prompt Learning Frameworks
College of Computing and Data Science Nanyang Technological University · School of Physical and Mathematical Sciences Nanyang Technological University · Microsoft Corp. +1

Oct 5, 2026, cs.CVJ/K move · Enter open · S save

Harmful Content Generation in Text-to-Image Models: Capabilities and Moderation Limitations

Paschalis Giakoumoglou, Manos Schinas, Symeon Papadopoulos

Centre for Research and Technology Hellas (CERTH), Thessaloniki, Greece

Modern Text-To-Image Models Child Sexual Abuse Material Adversarial Prompts

Open paper AskPDF arXiv

Text-to-image generative models can produce highly realistic imagery but also raise concerns about harmful misuse. While safety mechanisms exist, systematic evaluations of their effectiveness against realistic attacks remain limited. We present a systematic evaluation of harmful content generation across five open text-to-image models using an automated pipeline that transforms legitimate news captions into unsafe prompts targeting sexually explicit content, violence/gore, harmful stereotypes, self-harm, and hate speech. We evaluate both standard models with built-in safety mechanisms and community fine-tuned variants that bypass content restrictions. A human evaluation of 1,500 generated images shows high harmful-content generation rates: 89.2% for gore-related prompts, 47.6% for sexually explicit content, 43.6% for harmful stereotypes, 46.0% for hate speech, and 34.5% for self-harm, predominantly through graphic violence. Models show substantial capability for generating violent and stereotypical content, while community fine-tuned variants are particularly vulnerable to sexually explicit prompts. Generation quality is largely preserved under harmful prompting, producing imagery of sufficient fidelity to pose risks for disinformation and abuse; FLUX.1-dev produces clearly realistic harmful images in 30.9% of cases. We further evaluate automated moderation systems and find substantial detection gaps that allow unsafe images to evade filtering. Finally, we assess synthetic image detectors and show that models trained only on benign datasets perform worse on explicit content, while more diverse training data improves detection, highlighting semantic distribution gaps in current approaches. These findings expose limitations in current generation safeguards, moderation systems, and synthetic image detection, highlighting the need for stronger defenses against misuse at scale.

Momentum

Related topics

Latest papers 54