Attention Bias

Attention bias, the tendency for models to disproportionately focus on certain aspects of input data, is a significant challenge across various machine learning domains. Current research focuses on mitigating this bias through techniques like attention calibration, debiasing methods (e.g., residual attention debiasing), and attention guidance, often implemented within transformer-based architectures. These efforts aim to improve model performance, robustness, and fairness by ensuring attention mechanisms accurately reflect the relevance of input features, leading to more reliable and equitable outcomes in applications ranging from natural language processing to computer vision. Addressing attention bias is crucial for building trustworthy and effective AI systems.

Papers

October 1, 2024

Replacing Paths with Connection-Biased Attention for Knowledge Graph Completion
Sharmishtha Dutta, Alex Gittens, Mohammed J. Zaki, Charu C. Aggarwal
Knowledge Graph Knowledge Graph Completion Root to Leaf Path Attention Bias Inductive Knowledge Graph Completion Relation Prediction Task Subgraph Encoding

June 23, 2024

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T. Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister
Large Language Model Low Priority Input Context LLM Bias Attention Bias Long Input Context

April 15, 2024

Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASR
Zelin Wu, Gan Song, Christopher Li, Pat Rondon, Zhong Meng, Xavier Velez, Weiran Wang, Diamantino Caseiro, Golan Pundak, Tsendsuren Munkhdalai, Angad Chandorkar, Rohit Prabhavalkar
Contextual Biasing Context Encoder Context Encoding Attention Bias Speech Recognizers Context Injection Non Streaming

March 29, 2024

Selective Attention-based Modulation for Continual Learning
Giovanni Bellitto, Federica Proietto Salanitri, Matteo Pennisi, Matteo Boschini, Angelo Porrello, Simone Calderara, Simone Palazzo, Concetto Spampinato
Continual LEArning Continual Learning Saliency Prediction Attention Bias

February 26, 2024

Beyond Self-learned Attention: Mitigating Attention Bias in Transformer-based Models Using Attention Guidance
Jiri Gesi, Iftekhar Ahmed
Self Attention Transformer Based Model Code Representation Attention Bias

October 18, 2023

From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers
Shaoxiong Duan, Yining Shi, Wei Xu
Transformer Model Interpolation Regime Extrapolation Framework Length Generalization Attention Bias Arithmetic Transformer Attention Calibration

May 28, 2023

Robust Natural Language Understanding with Residual Attention Debiasing
Fei Wang, James Y. Huang, Tianyi Yan, Wenxuan Zhou, Muhao Chen
Bias Mitigation Attention Bias Based Debiasing Robust Natural Language Understanding

May 25, 2023

Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
Huawen Feng, Zhenxi Lin, Qianli Ma
Text Classification Attention Model Attention Bias

May 9, 2023

LSAS: Lightweight Sub-attention Strategy for Alleviating Attention Bias Problem
Shanshan Zhong, Wushao Wen, Jinghui Qin, Qiangpu Chen, Zhongzhan Huang
Self Attention Module Lightweight Attention Attention Bias

January 31, 2023

Fairness-aware Vision Transformer via Debiased Self-Attention
Yao Qiang, Chengyin Li, Prashant Khanduri, Dongxiao Zhu
Vision Transformer Bias Mitigation Attention Regularization Attention Bias Exposure Fairness

September 22, 2022

A domain adaptive deep learning solution for scanpath prediction of paintings
Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani, Alessandro Bruno
Natural Image Visual Attention Internet Service Domain Artistic Painting Scanpath Prediction Adaptive Deep Attention Bias

June 9, 2022

GASP: Gated Attention For Saliency Prediction
Fares Abawi, Tom Weber, Stefan Wermter
Human Attention Human Saliency Saliency Prediction Exhaled Breath Attention Bias

February 6, 2022

Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment
Yuyang Gao, Tong Sun, Liang Zhao, Sungsoo Hong
Deep Neural Network Real Human Attention Based Character Eye Steerable Convolution Attention Bias Attention Alignment