Self Attention

Self-attention is a mechanism in neural networks that allows the model to weigh the importance of different parts of the input data when processing it, enabling the capture of long-range dependencies and contextual information. Current research focuses on improving the efficiency of self-attention, particularly in vision transformers and other large models, through techniques like low-rank approximations, selective attention, and grouped query attention, aiming to reduce computational costs while maintaining accuracy. These advancements are significantly impacting various fields, including computer vision, natural language processing, and time series analysis, by enabling more efficient and powerful models for tasks such as image restoration, text-to-image generation, and medical image segmentation.

Papers

December 15, 2021

EEG-Transformer: Self-attention from Transformer Architecture for Decoding EEG of Imagined Speech
Young-Eun Lee, Seo-Hyun Lee
Self Attention Transformer Architecture Encoder Decoder Self Attention Module Single Channel

December 10, 2021

December 6, 2021

December 4, 2021

Multi-scale Graph Convolutional Networks with Self-Attention
Zhilong Xiong, Jia Cai
Convolutional Neural Network Self Attention Graph Convolution Multi Scale Graph

December 1, 2021

Routing with Self-Attention for Multimodal Capsule Networks
Kevin Duarte, Brian Chen, Nina Shvetsova, Andrew Rouditchenko, Samuel Thomas, Alexander Liu, David Harwath, James Glass, Hilde Kuehne, Mubarak Shah
Self Attention Multimodal Learning Capsule Network Routing Problem

November 30, 2021

Shunted Self-Attention via Multi-Scale Token Aggregation
Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, Xinchao Wang
Vision Transformer Self Attention Self Attention Layer Spatial Token Focal Transformer

November 29, 2021

On the Integration of Self-Attention and Convolution
Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, Gao Huang
Self Attention Direct Convolution Self Attention Module Traditional Convolution

November 25, 2021

Attend to Who You Are: Supervising Self-Attention for Keypoint Detection and Instance-Aware Association
Sen Yang, Zhicheng Wang, Ze Chen, Yanjie Li, Shoukui Zhang, Zhibin Quan, Shu-Tao Xia, Yiping Bao, Erjin Zhou, Wankou Yang
Self Attention Keypoint Detection Instance Level Segmentation

November 18, 2021

November 15, 2021

Searching for TrioNet: Combining Convolution with Local and Global Self-Attention
Huaijin Pi, Huiyu Wang, Yingwei Li, Zizhang Li, Alan Yuille
Self Attention Direct Convolution

November 14, 2021

Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network
Jichao Kan, Kun Hu, Markus Hagenbuchner, Ah Chung Tsoi, Mohammed Bennamounm, Zhiyong Wang
Self Attention Graph Convolution Sign Language Temporal Graph Neural Network Sign Language Translation Temporal Graph Learning

November 13, 2021

Full-attention based Neural Architecture Search using Context Auto-regression
Yuan Zhou, Haiyang Wang, Shuwei Huo, Boyu Wang
Neural Architecture Search Self Attention Attention Based Architecture Attention Operation Self Attention Network Autoregressive Context

November 8, 2021

Mixed Transformer U-Net For Medical Image Segmentation
Hongyi Wang, Shiao Xie, Lanfen Lin, Yutaro Iwamoto, Xian-Hua Han, Yen-Wei Chen, Ruofeng Tong
Vision Transformer Medical Image Segmentation Self Attention U Net Transformer Module

November 3, 2021

Self Attention

Papers

EEG-Transformer: Self-attention from Transformer Architecture for Decoding EEG of Imagined Speech

Self-attention Does Not Need $O(n^2)$ Memory

Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences

Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention

Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production

Multi-scale Graph Convolutional Networks with Self-Attention

Routing with Self-Attention for Multimodal Capsule Networks

Shunted Self-Attention via Multi-Scale Token Aggregation

On the Integration of Self-Attention and Convolution

Attend to Who You Are: Supervising Self-Attention for Keypoint Detection and Instance-Aware Association

You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling

Self-Attending Task Generative Adversarial Network for Realistic Satellite Image Creation

Searching for TrioNet: Combining Convolution with Local and Global Self-Attention

Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network

Full-attention based Neural Architecture Search using Context Auto-regression

Mixed Transformer U-Net For Medical Image Segmentation

A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition

ProSTformer: Pre-trained Progressive Space-Time Self-attention Model for Traffic Flow Forecasting