Attention Transformer

Attention transformers are neural network architectures leveraging the attention mechanism to process sequential or spatial data, aiming to capture long-range dependencies and improve performance in various tasks. Current research focuses on enhancing efficiency through techniques like windowed attention and pooling, developing specialized architectures for specific applications (e.g., image super-resolution, speech enhancement, human pose estimation), and exploring the theoretical relationship between attention and other neural network components like Multi-Layer Perceptrons (MLPs). These advancements are significantly impacting fields like computer vision, robotics, and audio processing by enabling more accurate and efficient models for complex tasks.

Papers

July 23, 2024

Channel-Partitioned Windowed Attention And Frequency Learning for Single Image Super-Resolution
Dinh Phu Tran, Dao Duy Hung, Daeyoung Kim
Receptive Field Computer Vision Task Window Attention Frequency Learning Attention Transformer

October 4, 2023

Improving Drumming Robot Via Attention Transformer Network
Yang Yi, Zonghan Li
Vision Transformer Attention Mechanism Music Transcription Attention Transformer

September 15, 2023

Attention-Only Transformers and Implementing MLPs with Attention Heads
Robert Huben, Valerie Morris
Transformer Architecture Attention Head MLP Layer Attention Transformer

May 6, 2023

DBAT: Dynamic Backward Attention Transformer for Material Segmentation with Cross-Resolution Patches
Yuwen Heng, Srinandan Dasmahapatra, Hansung Kim
Segmentation Performance Transformer Attention Material Segmentation Attention Transformer

March 24, 2023

Few Shot Medical Image Segmentation with Cross Attention Transformer
Yi Lin, Yufan Chen, Kwang-Ting Cheng, Hao Chen
Shot Learning Cross Attention Transformer Shot Medical Image Segmentation Attention Transformer Query Image Segmentation

March 23, 2023

POTTER: Pooling Attention Transformer for Efficient Human Mesh Recovery
Ce Zheng, Xianpeng Liu, Guo-Jun Qi, Chen Chen
Attention Module Human Mesh Recovery Attention Based Pooling Pure Transformer Attention Transformer

March 16, 2023

Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers
Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Ruoyu Xue, Gregory Zelinsky, Minh Hoai, Dimitris Samaras
Transformer Megatron Decepticons Visual Attention Attention Model Attention Computation Multi Level Fusion Scanpath Prediction Cognitive Control Attention Transformer

November 30, 2022

Pattern Attention Transformer with Doughnut Kernel
WenYuan Sheng
Computer Vision Supervised ImageNet Swin Transformer Small Kernel Attention Transformer

October 4, 2022

Accurate Image Restoration with Attention Retractable Transformer
Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, Xin Yuan
Image Restoration Attention Layer Sparse Attention Attention Transformer

March 17, 2022

Cascade Transformers for End-to-End Person Search
Rui Yu, Dawei Du, Rodney LaLonde, Daniel Davila, Christopher Funk, Anthony Hoogs, Brian Clipp
Person Search Person Detection Attention Transformer Cascaded Transformer Cascade Attention

February 16, 2022

DBT-Net: Dual-branch federative magnitude and phase estimation with attention-in-attention transformer for monaural speech enhancement
Guochen Yu, Andong Li, Hui Wang, Yutian Wang, Yuxuan Ke, Chengshi Zheng
Speech Enhancement Channel Attention Dual Branch Monaural Speech Enhancement Attention Transformer Phase Difference Estimation