Self Attention

Self-attention is a mechanism in neural networks that allows the model to weigh the importance of different parts of the input data when processing it, enabling the capture of long-range dependencies and contextual information. Current research focuses on improving the efficiency of self-attention, particularly in vision transformers and other large models, through techniques like low-rank approximations, selective attention, and grouped query attention, aiming to reduce computational costs while maintaining accuracy. These advancements are significantly impacting various fields, including computer vision, natural language processing, and time series analysis, by enabling more efficient and powerful models for tasks such as image restoration, text-to-image generation, and medical image segmentation.

Papers

June 26, 2023

PhD Thesis: Exploring the role of (self-)attention in cognitive and computer vision architecture
Mohit Vaishnav
Self Attention Integral Role Visual Reasoning Cognitive Architecture Spatial Attention Order Derivative Vision Architecture Synthetic Reasoning

June 25, 2023

A Multilingual Translator to SQL with Database Schema Pruning to Improve Self-Attention
Marcelo Archanjo Jose, Fabio Gagliardi Cozman
Self Attention Long Sequence Data Pruning Multilingual Translation Text Sequence

June 23, 2023

Knowledge-Infused Self Attention Transformers
Kaushik Roy, Yuxin Zi, Vignesh Narayanan, Manas Gaur, Amit Sheth
Language Model Self Attention Transformer Architecture Transformer Based Model Transformer Based Language Model

June 16, 2023

June 14, 2023

When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants
Anuj Diwan, Eunsol Choi, David Harwath
Self Attention Attention Mechanism Speech Analysis Global Attention Self Supervised Speech Model Vision Transformer Variant

June 9, 2023

FasterViT: Fast Vision Transformers with Hierarchical Attention
Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M. Alvarez, Jan Kautz, Pavlo Molchanov
Self Attention Hierarchical Attention Fast Vision Transformer EfficientViT SAM

June 2, 2023

June 1, 2023

Lightweight Vision Transformer with Bidirectional Interaction
Qihang Fan, Huaibo Huang, Xiaoqiang Zhou, Ran He
Vision Transformer Fine Grained Self Attention Vision Backbone

May 28, 2023

Self-attention Dual Embedding for Graphs with Heterophily
Yurui Lai, Taiyan Zhang, Rui Fan
Graph Neural Network Graph Drawing Self Attention Node Classification Homophilous Graph

May 25, 2023

Explainability Techniques for Chemical Language Models
Stefan Hödl, William Robinson, Yoram Bachrach, Wilhelm Huck, Tal Kachman
Self Attention Chemical Data Interpretable Prediction Explainability Technique Chemical Language Model Molecular String Representation

May 23, 2023

Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Ta-Chung Chi, Ting-Han Fan, Li-Wei Chen, Alexander I. Rudnicky, Peter J. Ramadge
Self Attention Transformer Language Model Positional Embeddings Frozen Language Model Latent Position

May 22, 2023

May 19, 2023

Self-Reinforcement Attention Mechanism For Tabular Learning
Kodjo Mawuena Amekoe, Mohamed Djallel Dilmi, Hanene Azzag, Mustapha Lebbah, Zaineb Chelly Dagdia, Gregoire Jaffre
Machine Learning Model Inherent Interpretability Self Attention Attention Mechanism Interpretable Model Tabular Learning

May 17, 2023

Deep Multiple Instance Learning with Distance-Aware Self-Attention
Georg Wölflein, Lucie Charlotte Magister, Pietro Liò, David J. Harrison, Ognjen Arandjelović
Self Attention Multiple Instance Learning Spatial Information Deep Multiple Instance Relative Positional

Self Attention

Papers

PhD Thesis: Exploring the role of (self-)attention in cognitive and computer vision architecture

A Multilingual Translator to SQL with Database Schema Pruning to Improve Self-Attention

Knowledge-Infused Self Attention Transformers

Systematic Architectural Design of Scale Transformed Attention Condenser DNNs via Multi-Scale Class Representational Response Similarity Analysis

Label-noise-tolerant medical image classification via self-attention and self-supervised learning

When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants

FasterViT: Fast Vision Transformers with Hierarchical Attention

The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles

A Novel Vision Transformer with Residual in Self-attention for Biomedical Image Classification

Lightweight Vision Transformer with Bidirectional Interaction

Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation

LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction

Recasting Self-Attention with Holographic Reduced Representations

Self-attention Dual Embedding for Graphs with Heterophily

Explainability Techniques for Chemical Language Models

Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings

GNCformer Enhanced Self-attention for Automatic Speech Recognition

FIT: Far-reaching Interleaved Transformers

Self-Reinforcement Attention Mechanism For Tabular Learning

Deep Multiple Instance Learning with Distance-Aware Self-Attention