Robust Encoders

Robust encoders are a crucial area of research focused on developing machine learning models resistant to adversarial attacks, noisy data, and variations in input formats like user-generated content. Current efforts concentrate on improving the robustness of existing architectures like CLIP and LASER through techniques such as adversarial fine-tuning, teacher-student training, and the incorporation of watermarking for authentication. This research is vital for enhancing the reliability and security of AI systems across diverse applications, from image captioning and question answering to speech translation and other downstream tasks that rely on robust feature extraction.

Papers

September 11, 2024

Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks
Md Zarif Hossain, Ahmed Imteaj
Adversarial Attack Vision Language Model Large Vision Language Model Jailbreak Attack Adversarial Manipulation CLIP Vision Encoder Gradient Based Adversarial Robust Encoders

March 25, 2024

Making Sentence Embeddings Robust to User-Generated Content
Lydia Nishimwe, Benoît Sagot, Rachel Bawden
NLP Model Video Quality Assessment Content Generation User Generated Content Robust Encoders

August 9, 2023

SSL-Auth: An Authentication Framework by Fragile Watermarking for Pre-trained Encoders in Self-supervised Learning
Xiaobei Li, Changchun Yin, Liyue Zhu, Xiaogang Xu, Liming Fang, Run Wang, Chenhao Lin
Self Supervised Learning Self Supervised Pre Trained Supervised Learning Adversarial Threat Robust Encoders Fragile Watermarking

February 10, 2023

Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala
Diffusion Model Text to Image Diffusion Model Image Diffusion Model Conditional Control Robust Encoders

January 7, 2023

REaaS: Enabling Adversarially Robust Downstream Classifiers via Robust Encoder as a Service
Wenjie Qu, Jinyuan Jia, Neil Zhenqiang Gong
Adversarial Example Service Provider Robustness Guarantee Downstream Classification Robust Encoders

December 4, 2022

Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data
Yuhao Zhang, Chen Xu, Bojie Hu, Chunliang Zhang, Tong Xiao, Jingbo Zhu
Speech Translation Text Encoder Text Data End to End Speech Translation Source Speech Early Slavic Participle Speech Translation Model Robust Encoders

October 8, 2022

Robustness of Unsupervised Representation Learning without Labels
Aleksandar Petrov, Marta Kwiatkowska
Native Robustness Label Information Unsupervised Representation Learning Adversarial Framework Learning Benchmark Robust Unsupervised Robust Encoders

Robust Encoders

Papers

Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks

Making Sentence Embeddings Robust to User-Generated Content

SSL-Auth: An Authentication Framework by Fragile Watermarking for Pre-trained Encoders in Self-supervised Learning

Adding Conditional Control to Text-to-Image Diffusion Models

REaaS: Enabling Adversarially Robust Downstream Classifiers via Robust Encoder as a Service

Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data

Robustness of Unsupervised Representation Learning without Labels