Adversarial Input

Adversarial input research focuses on developing and mitigating vulnerabilities in machine learning models, particularly large language models (LLMs) and deep neural networks (DNNs), by crafting inputs designed to elicit incorrect or harmful outputs. Current research emphasizes developing novel attack methods, such as prompt injection and image manipulation techniques, alongside robust defenses including adversarial training, invariance regularization, and prompt rewriting. This field is crucial for ensuring the safe and reliable deployment of AI systems across various applications, from autonomous vehicles to medical diagnosis, by improving model robustness and trustworthiness.

Papers

February 7, 2024

Analyzing Adversarial Inputs in Deep Reinforcement Learning
Davide Corsi, Guy Amir, Guy Katz, Alessandro Farinelli
Deep Reinforcement Learning Adversarial Input Input Perturbation

December 15, 2023

No-Skim: Towards Efficiency Robustness Evaluation on Skimming-based Language Models
Shengyao Zhang, Mi Zhang, Xudong Pan, Min Yang
Large Language Model Adversarial Input Block SkiM Efficiency Robustness

December 11, 2023

December 6, 2023

On the Robustness of Large Multimodal Models Against Image Adversarial Attacks
Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, Ser-Nam Lim
Adversarial Attack Native Robustness Large Multimodal Model Adversarial Input Adversarial Environment

December 3, 2023

Rethinking PGD Attack: Is Sign Function Necessary?
Junjie Yang, Tianlong Chen, Xuxi Chen, Zhangyang Wang, Yingbin Liang
Adversarial Attack Gradient Descent Adversarial Input Multi Step Attack Projected Gradient Descent Attack Sign Function

November 28, 2023

STR-Cert: Robustness Certification for Deep Text Recognition on Deep Learning Pipelines and Vision Transformers
Daqian Shao, Lukas Fesser, Marta Kwiatkowska
Convolutional Neural Network Vision Transformer Text Recognition Scene Text Recognition Adversarial Input Robustness Certification

November 16, 2023

Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness
Ashim Gupta, Rishanth Rajendhran, Nathan Stringham, Vivek Srikumar, Ana Marasović
Native Robustness Model Robustness Adversarial Input State of the Art Whisper Fidelity Reward Adversarial Evaluation Acoustic Echo Robustness Issue

October 5, 2023

SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
Alexander Robey, Eric Wong, Hamed Hassani, George J. Pappas
Large Language Model Jailbreak Attack Adversarial Prompt Adversarial Input

September 20, 2023

Distilling Adversarial Prompts from Safety Benchmarks: Report for the Adversarial Nibbler Challenge
Manuel Brack, Patrick Schramowski, Kristian Kersting
Adversarial Prompt Adversarial Input Unsafe Content Text Conditioned Image Generation Safety Benchmark Adversarial Challenge

September 19, 2023

Adversarial Attacks Against Uncertainty Quantification
Emanuele Ledda, Daniele Angioni, Giorgio Piras, Giorgio Fumera, Battista Biggio, Fabio Roli
Adversarial Attack Adversarial Example Uncertainty Quantification Adversarial Input Adversarial Scenario

August 16, 2023

Benchmarking Adversarial Robustness of Compressed Deep Learning Models
Brijesh Vora, Kartik Patwari, Syed Mahbub Hafiz, Zubair Shafiq, Chen-Nee Chuah
Neural Network Adversarial Attack Adversarial Training Adversarial Robustness Adversarial Input

August 3, 2023

URET: Universal Robustness Evaluation Toolkit (for Evasion)
Kevin Eykholt, Taesung Lee, Douglas Schales, Jiyong Jang, Ian Molloy, Masha Zorin
Adversarial Example Adversarial Input Robustness Evaluation Adversarial Evasion Attack Realistic Adversarial

July 3, 2023

Interpretability and Transparency-Driven Detection and Transformation of Textual Adversarial Examples (IT-DT)
Bushra Sabir, M. Ali Babar, Sharif Abuadbba
Adversarial Attack Adversarial Example Inherent Interpretability Generative Adversarial Adversarial Input Character Transformation Textual Adversarial Example

June 26, 2023

Are aligned neural networks adversarially aligned?
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, Ludwig Schmidt
Neural Network Adversarial Example Adversarial Input Adversarial Alignment

June 25, 2023

A Spectral Perspective towards Understanding and Improving Adversarial Robustness
Binxiao Huang, Rui Lin, Chaofan Tao, Ngai Wong
Native Robustness Adversarial Training Adversarial Robustness Human Understanding Adversarial Input Better Robustness Imperceptible Adversarial Perturbation Spectral Representation

May 30, 2023

A Multilingual Evaluation of NER Robustness to Adversarial Inputs
Akshay Srinivasan, Sowmya Vajjala
Language Model Entity Recognition Named Entity Recognition Adversarial Input Multilingual Evaluation Adversarial Evaluation

May 25, 2023

Don't Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Text
Ashim Gupta, Carter Wood Blum, Temma Choji, Yingjie Fei, Shalin Shah, Alakananda Vempala, Vivek Srikumar
Adversarial Attack Adversarial Robustness Text Modality Adversarial Perturbation Adversarial Input Better Adversarial Robustness

April 5, 2023

Hyper-parameter Tuning for Adversarially Robust Models
Pedro Mendes, Paolo Romano, David Garlan
Adversarial Training Hyper Parameter Adversarial Input Adversarial Scenario Fast Adversarial Training Hyper Parameter Tuning

March 24, 2023

Effective black box adversarial attack with handcrafted kernels
Petr Dvořáček, Petr Hurtik, Petra Števuliáková
Adversarial Attack Adversarial Example Adversarial Image Adversarial Input Small Kernel Black Box Attack