Transformer Attention

Transformer attention mechanisms, which allow models to weigh the importance of different input elements, are revolutionizing various fields by enabling the processing of complex, long-range dependencies within data. Current research focuses on improving efficiency (e.g., through low-rank compression and specialized attention modules), enhancing interpretability (e.g., via trainable attention mechanisms and visualization techniques), and extending applicability to diverse data types (e.g., graphs, time series, and medical images). These advancements are significantly impacting fields like natural language processing, medical image analysis, and robotics, leading to improved model performance and the development of more explainable and efficient AI systems.

Papers

December 23, 2024

December 10, 2024

Comateformer: Combined Attention Transformer for Semantic Sentence Matching
Bo Li, Di Liang, Zixin Zhang
Transformer Based Model Semantics Surfaced Matching Accuracy Hybrid Attention Transformer Attention Sentence Pair Sentence Matching Hybrid Attention Transformer

November 20, 2024

Hymba: A Hybrid-head Architecture for Small Language Models
Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, Pavlo Molchanov
Language Model Attention Mechanism Human Head Transformer Attention Knowledge Recall Lack Interpretability

October 28, 2024

SepMamba: State-space models for speaker separation using Mamba
Thor Højhus Avenstrup, Boldizsár Elek, István László Mádi, András Bence Schin, Morten Mørup, Bjørn Sand Jensen, Kenny Falkær Olsen
State Space Model Mamba in Mamba Speech Separation Transformer Based Architecture Transformer Attention Speaker Separation Single Channel Speech Separation Mamba Layer

September 10, 2024

DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
Maryam Akhavan Aghdam, Hongpeng Jin, Yanzhao Wu
Natural Language Processing Mixture of Expert Transformer Attention Expert Selection Token Importance

August 8, 2024

Efficient and Accurate Pneumonia Detection Using a Novel Multi-Scale Transformer Approach
Alireza Saber, Pouria Parhami, Alimohammad Siahkarzadeh, Mansoor Fateh, Amirreza Fateh
High Efficiency ResNet Based Lung Segmentation Transformer Attention Pneumonia Detection Multi Scale Transformer

July 29, 2024

Language-driven Grasp Detection with Mask-guided Attention
Tuan Van Vo, Minh Nhat Vu, Baoru Huang, An Vuong, Ngan Le, Thieu Vo, Anh Nguyen
Natural Language Grasp Detection Attention Mask Transformer Attention Grasp Accuracy Language Driven Grasp Detection

June 27, 2024

YZS-model: A Predictive Model for Organic Drug Solubility Based on Graph Convolutional Networks and Transformer-Attention
Chenxu Wang, Haowei Ming, Jian He, Yao Lu, Junhong Chen
Predictive Model Drug Like Transformer Attention Solubility Prediction

June 13, 2024

AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers
Emil Biju, Anirudh Sriram, Mert Pilanci
Transformer Megatron Decepticons Transformer Based Model Transformer Attention Speaker Independent Whisper Model Low Rank Compression Adaptive Compression

March 15, 2024

PASTA: Towards Flexible and Efficient HDR Imaging Via Progressively Aggregated Spatio-Temporal Alignment
Xiaoning Liu, Ao Li, Zongwei Wu, Yapeng Du, Le Zhang, Yulun Zhang, Radu Timofte, Ce Zhu
Spatio Temporal High Resolution Image High Dynamic Range Extension Study Hierarchical Representation Transformer Attention

March 7, 2024

T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers
Mariano V. Ntrougkas, Nikolaos Gkalelis, Vasileios Mezaris
Convolutional Neural Network Vision Transformer Attention Mechanism Deep Learning Architecture Transformer Based Network Transformer Attention Morphological Task

January 22, 2024

RTA-Former: Reverse Transformer Attention for Polyp Segmentation
Zhikai Li, Murong Yi, Ali Uneri, Sihan Niu, Craig Jones
Polyp Segmentation Transformer Attention Edge Segmentation Polyp Segmentation Datasets

December 1, 2023

Learning to Estimate Critical Gait Parameters from Single-View RGB Videos with Transformer-Based Attention Network
Quoc Hung T. Le, Hieu H. Pham
LeArning Abstract Computer Vision RGB Image Temporal Transformer Gait Analysis Transformer Attention Gait Parameter

November 20, 2023

DAS: A Deformable Attention to Capture Salient Information in CNNs
Farzad Salajegheh, Nader Asadi, Soroush Saryazdi, Sudhir Mudur
Convolutional Neural Network Deformable Convolution Deformable Attention Transformer Attention

October 6, 2023

How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computation
Josh Alman, Zhao Song
Attention Matrix Softmax Attention Attention Computation Transformer Attention Order Tensor Kronecker Product Higher Order Correlation Arbitrary Computation

September 22, 2023

Trading-off Mutual Information on Feature Aggregation for Face Recognition
Mohammad Akyash, Ali Zafari, Nasser M. Nasrabadi
Face Recognition Mutual Information Feature Aggregation Transformer Attention Informative Latent Representation

September 16, 2023

Extracting Interpretable Local and Global Representations from Attention on Time Series
Leonid Schwenke, Martin Atzmueller
Time Series Inherent Interpretability High Explainability Human Attention Interpretability Method Transformer Attention Global Representation Local Interpretability

September 14, 2023

Structure-Preserving Transformers for Sequences of SPD Matrices
Mathieu Seraphim, Alexis Lechervy, Florian Yger, Luc Brun, Olivier Etard
Transformer Megatron Decepticons Sequence of Sequence Riemannian Geometry Non Euclidean Transformer Attention Symmetric Positive Definite Symmetric Positive Definite Matrix Multivariate Biosignals

June 8, 2023

Sequence-to-Sequence Model with Transformer-based Attention Mechanism and Temporal Pooling for Non-Intrusive Load Monitoring
Mohammad Irani Azad, Roozbeh Rajabi, Abouzar Estebsari
Sequence to Sequence Seq2seq Model Non Intrusive Load Monitoring Transformer Attention Smart Building Temporal Pooling

Transformer Attention

Papers

Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction

HPCNeuroNet: A Neuromorphic Approach Merging SNN Temporal Dynamics with Transformer Attention for FPGA-based Particle Physics

Comateformer: Combined Attention Transformer for Semantic Sentence Matching

Hymba: A Hybrid-head Architecture for Small Language Models

SepMamba: State-space models for speaker separation using Mamba

DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models

Efficient and Accurate Pneumonia Detection Using a Novel Multi-Scale Transformer Approach

Language-driven Grasp Detection with Mask-guided Attention

YZS-model: A Predictive Model for Organic Drug Solubility Based on Graph Convolutional Networks and Transformer-Attention

AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers

PASTA: Towards Flexible and Efficient HDR Imaging Via Progressively Aggregated Spatio-Temporal Alignment

T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers

RTA-Former: Reverse Transformer Attention for Polyp Segmentation

Learning to Estimate Critical Gait Parameters from Single-View RGB Videos with Transformer-Based Attention Network

DAS: A Deformable Attention to Capture Salient Information in CNNs

How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computation

Trading-off Mutual Information on Feature Aggregation for Face Recognition

Extracting Interpretable Local and Global Representations from Attention on Time Series

Structure-Preserving Transformers for Sequences of SPD Matrices

Sequence-to-Sequence Model with Transformer-based Attention Mechanism and Temporal Pooling for Non-Intrusive Load Monitoring