Self Attention Matrix

The self-attention matrix, a core component of transformer-based models, is being actively investigated to improve the efficiency and accuracy of various applications, particularly in multi-modal large language models and vision transformers. Current research focuses on mitigating issues like "hallucinations" (inaccurate outputs) by analyzing and modifying the attention weights to better reflect relevant information, and on accelerating computations through techniques such as low-rank approximations and optimized sparsity patterns. These advancements are crucial for enhancing the performance and scalability of transformer models across diverse tasks, ranging from image captioning to remote sensing analysis and natural language processing.

Papers

October 14, 2024

KNN Transformer with Pyramid Prompts for Few-Shot Learning
Wenhao Li, Qiangchang Wang, Peng Zhao, Yilong Yin
LeArning Abstract Visual Prompt Transformer Based Network Effective Prompt Multi Modal PromPt Self Attention Matrix

July 21, 2024

DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer
Jinfeng Wei, Xiaofeng Zhang
Multi Modal Large Language Model Self Attention Matrix Layer Weight Image Specific Information Trust Penalization

November 29, 2023

OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, Nenghai Yu
Multi Modal Large Language Model Content Hallucination Mitigating Hallucination Capital Allocation Self Attention Matrix

November 1, 2023

CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders
Anthony Fuller, Koreen Millard, James R. Green
Multimodal Representation Self Supervised Objective Cross Modal Contrastive Learning Contrastive Masked Multispectral Model Self Attention Matrix

October 11, 2023

Accelerating Vision Transformers Based on Heterogeneous Attention Patterns
Deli Yu, Teng Xi, Jianwei Li, Baopu Li, Gang Zhang, Haocheng Feng, Junyu Han, Jingtuo Liu, Errui Ding, Jingdong Wang
Vision Transformer Self Attention Self Attention Matrix Heterogeneous Attention

May 9, 2023

The emergence of clusters in self-attention dynamics
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, Philippe Rigollet
Transformer Megatron Decepticons Self Attention Meaningful Representation Path Breaking Emergence Different Cluster Context Awareness Particle System Self Attention Matrix

November 11, 2022

Improving word mover's distance by leveraging self-attention matrix
Hiroaki Yamagiwa, Sho Yokoi, Hidetoshi Shimodaira
Word Embeddings Semantic Similarity Semantic Textual Similarity Distance Matter Paraphrase Detection Self Attention Matrix

July 21, 2022

Multi Resolution Analysis (MRA) for Approximate Self-Attention
Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, Vikas Singh
Self Attention Transformer Architecture Multi Resolution Wavelet Based Self Attention Matrix Multiresolution Analysis

July 5, 2022

Softmax-free Linear Transformers
Jiachen Lu, Junge Zhang, Xiatian Zhu, Jianfeng Feng, Tao Xiang, Li Zhang
Vision Transformer Visual Recognition Softmax Attention Self Attention Matrix

March 9, 2022

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice
Peihao Wang, Wenqing Zheng, Tianlong Chen, Zhangyang Wang
Convolutional Neural Network Vision Transformer Theoretical Understanding Deep Transformer Squashing Mitigation Approach Self Attention Block Self Attention Matrix

February 24, 2022

Self-Attention for Incomplete Utterance Rewriting
Yong Zhang, Zhitao Li, Jianzong Wang, Ning Cheng, Jing Xiao
Self Attention NLP Field Word Embeddings Coreference Information Spoken Utterance Self Attention Matrix