Regularized Transformer

Regularized Transformers aim to improve the performance and robustness of standard Transformer architectures by incorporating various regularization techniques. Current research focuses on applying these methods to diverse tasks, including offline reinforcement learning, weakly supervised semantic segmentation, and sound classification, often employing modifications to the self-attention mechanism or incorporating value functions. These improvements address issues like overconfidence, limited generalization ability, and high computational costs, leading to enhanced model accuracy, efficiency, and reliability across various domains. The resulting advancements have significant implications for improving the performance and applicability of Transformer models in numerous scientific and industrial applications.

Papers

May 27, 2024

Q-value Regularized Transformer for Offline Reinforcement Learning
Shengchao Hu, Ziqing Fan, Chaoqin Huang, Li Shen, Ya Zhang, Yanfeng Wang, Dacheng Tao
Offline Reinforcement Learning Sparse Reward Optimal Trajectory Dynamic Programming Optimal Action Regularized Transformer

August 8, 2023

All-pairs Consistency Learning for Weakly Supervised Semantic Segmentation
Weixuan Sun, Yanhao Zhang, Zhen Qin, Zheyuan Liu, Lin Cheng, Fanyi Wang, Yiran Zhong, Nick Barnes
Semantic Segmentation Object Localization Weakly Supervised Semantic Segmentation Consistency Regularization Consistency Learning Regularized Transformer

July 19, 2023

Improving Domain Generalization for Sound Classification with Sparse Frequency-Regularized Transformer
Honglin Mu, Wentian Xia, Wanxiang Che
Transformer Based Domain Generalization Transformer Model State of the Art Many Sparse Classification Model Audio Classification Long Term Recurrent Convolutional Network Regularized Transformer

June 12, 2023

Mitigating Transformer Overconfidence via Lipschitz Regularization
Wenqian Ye, Yunsheng Ma, Xu Cao, Kun Tang
Vision Task Vision Benchmark Lipschitz Bound Lipschitz Regularization Dot Product Self Attention Regularized Transformer

December 29, 2022

Robust representations of oil wells' intervals via sparse attention mechanism
Alina Ermilova, Nikita Baramiia, Valerii Kornilov, Sergey Petrakov, Alexey Zaytsev
Transformer Architecture Robust Representation Efficient Transformer Oil Well Time Interval SParse ATtention Similarity Judgment Regularized Transformer

Regularized Transformer

Papers

Q-value Regularized Transformer for Offline Reinforcement Learning

All-pairs Consistency Learning for Weakly Supervised Semantic Segmentation

Improving Domain Generalization for Sound Classification with Sparse Frequency-Regularized Transformer

Mitigating Transformer Overconfidence via Lipschitz Regularization

Robust representations of oil wells' intervals via sparse attention mechanism