Robust Transformer

Robust Transformer research focuses on enhancing the resilience and efficiency of Transformer models, particularly addressing challenges like computational cost for long sequences, susceptibility to adversarial attacks, and limitations in handling irregularly sampled data. Current efforts involve developing novel architectures, such as Rough Transformers utilizing continuous-time representations and path signatures for improved efficiency and robustness to data irregularities, and incorporating techniques like robust kernel density estimation and locality inductive biases to improve resistance to noise and adversarial examples. These advancements are significant for various applications, including time-series analysis in medicine, image classification, and natural language processing, where robustness and efficiency are crucial for reliable performance.

Papers

October 30, 2024

ProTransformer: Robustify Transformers via Plug-and-Play Paradigm
Zhichao Hou, Weizhi Gao, Yuchen Shen, Feiyi Wang, Xiaorui Liu
Transformer Model Plug and Play Transformer Based Architecture Robust Transformer Robust Attention

May 31, 2024

Rough Transformers: Lightweight Continuous-Time Sequence Modelling with Path Signatures
Fernando Moreno-Pino, Álvaro Arroyo, Harrison Waldon, Xiaowen Dong, Álvaro Cartea
Transformer Based Architecture Continuous Time Event Sequence Path Signature Robust Transformer

May 8, 2024

Few-Shot Class Incremental Learning via Robust Transformer Approach
Naeem Paeedeh, Mahardhika Pratama, Sunu Wibirama, Wolfgang Mayer, Zehong Cao, Ryszard Kowalczyk
Convolutional Neural Network Incremental Learning Class Incremental Learning Robust Transformer Compact Convolutional Transformer

March 15, 2024

Rough Transformers for Continuous and Efficient Time-Series Modelling
Fernando Moreno-Pino, Álvaro Arroyo, Harrison Waldon, Xiaowen Dong, Álvaro Cartea
High Efficiency Time Series Continuous Speech Robust Transformer

December 12, 2023

Adjustable Robust Transformer for High Myopia Screening in Optical Coherence Tomography
Xiao Ma, Zetian Zhang, Zexuan Ji, Kun Huang, Na Su, Songtao Yuan, Qiang Chen
Optical Coherence Tomography High Myopia Robust Transformer

January 27, 2023

Robust Transformer with Locality Inductive Bias and Feature Normalization
Omid Nejati Manzari, Hossein Kashiani, Hojat Asgarian Dehkordi, Shahriar Baradaran Shokouhi
Vision Transformer Adversarial Perturbation Feature Normalization Sign Classification Robust Transformer

October 11, 2022

Designing Robust Transformers using Robust Kernel Density Estimation
Xing Han, Tongzheng Ren, Tan Minh Nguyen, Khai Nguyen, Joydeep Ghosh, Nhat Ho
Transformer Architecture Density Estimation Kernel Density Robust Transformer

July 22, 2022

An Impartial Take to the CNN vs Transformer Robustness Contest
Francesco Pinto, Philip H. S. Torr, Puneet K. Dokania
Convolutional Neural Network State of the Art CNN Network Robust Transformer

March 1, 2022

TableFormer: Robust Transformer Modeling for Table-Text Encoding
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, Shachi Paul
Inductive Bias Text Alignment Table Understanding Structural Bias Robust Transformer Text Encoding