Sparse Transformer

Sparse transformers aim to improve the efficiency and scalability of standard transformers by reducing computational complexity, primarily by selectively attending to only a subset of input tokens. Current research focuses on developing novel sparse attention mechanisms, including various windowing strategies, hierarchical structures, and adaptive pruning techniques, often integrated into architectures like Swin Transformers and Universal Transformers. This research is significant because it enables the application of transformer models to larger datasets and more complex tasks, particularly in resource-constrained environments, with applications spanning image processing, natural language processing, and autonomous driving.

Papers

August 9, 2023

Sparse Binary Transformers for Multivariate Time Series Modeling
Matt Gorbett, Hossein Shirazi, Indrakshi Ray
Neural Network Deep Learning Time Series Multivariate Time Series Lightweight Model Sparse Transformer Precision Transformer

August 2, 2023

From Sparse to Soft Mixtures of Experts
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Neil Houlsby
Mixture Component Expert Knowledge Many Sparse Sparse Transformer Sparse Mixture Dense Transformer MoE FFD

June 26, 2023

LongCoder: A Long-Range Pre-trained Language Model for Code Completion
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, Julian McAuley
Code Completion Sparse Transformer Automatic Coding Compound Token Long Range Language

March 21, 2023

Learning A Sparse Transformer Network for Effective Image Deraining
Xiang Chen, Hao Li, Mingqiang Li, Jinshan Pan
LeArning Abstract Image Reconstruction Image Deraining Sparse Transformer Convolution Operator

March 10, 2023

Exphormer: Sparse Transformers for Graphs
Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland, Ali Kemal Sinop
Graph Drawing Graph Transformer Graph Learning Sparse Transformer Expander Graph Scalable Graph Transformer

February 28, 2023

Sampled Transformer for Point Sets
Shidi Li, Christian Walder, Alexander Soen, Lexing Xie, Miaomiao Liu
Transformer Based Self Attention Attention Layer Sparse Transformer Point Set Dense Transformer

October 21, 2022

Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences
Aosong Feng, Irene Li, Yuang Jiang, Rex Ying
Long Sequence Sparse Attention Efficient Transformer Multi Hop Sparse Transformer Multi View Diffuser

October 13, 2022

SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds
Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, Dragomir Anguelov
Point Cloud 3D Object Detection Swin Transformer Sparse Transformer Window Based Transformer Sparse Voxel

October 11, 2022

SaiT: Sparse Vision Transformers through Adaptive Token Pruning
Ling Li, David Thorsley, Joseph Hassoun
Vision Transformer Deep Convolutional Neural Network Transformer Model Sparse Transformer Token Sparsification Dynamic Token

September 14, 2022

Efficient Quantized Sparse Matrix Operations on Tensor Cores
Shigang Li, Kazuki Osawa, Torsten Hoefler
High Efficiency Sparse Matrix Sparse Transformer Tensor Core Precision Matrix Sparse Operation

August 12, 2022

An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers
Chao Fang, Aojun Zhou, Zhongfeng Wang
New Framework Efficient Transformer Sparse Transformer Efficient Hardware Anderson Acceleration Algorithm Hardware Co Design

July 13, 2022

DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation
Songhua Liu, Jingwen Ye, Sucheng Ren, Xinchao Wang
Sparse Attention Sparse Transformer Person Image Generation Exemplar Guided Image Attention Scheme

July 5, 2022

CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers
Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, Jiaqi Ma
Eye View Sparse Transformer Cooperative Behavior Bird'S Eye View Map Segmentation Multi Agent Perception V2X Perception

May 27, 2022

What Dense Graph Do You Need for Self-Attention?
Yuxin Wang, Chu-Tak Lee, Qipeng Guo, Zhangyue Yin, Yunhua Zhou, Xuanjing Huang, Xipeng Qiu
Self Attention Sparse Graph Sparse Transformer Dense Graph Hyper Transformer

May 8, 2022

SparseTT: Visual Tracking with Sparse Transformers
Zhihong Fu, Zehua Fu, Qingjie Liu, Wenrui Cai, Yunhong Wang
Visual Tracking Sparse Transformer SParse ATtention

March 23, 2022

ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention
Yang Liu, Jiaxiang Liu, Li Chen, Yuxiang Lu, Shikun Feng, Zhida Feng, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang
Self Attention Sparse Attention Hierarchical Transformer Sparse Transformer

January 22, 2022

glassoformer: a query-sparse transformer for post-fault power grid voltage prediction
Yunling Zheng, Carson Hu, Guang Lin, Meng Yue, Bao Wang, Jack Xin
Efficient Transformer Sparse Transformer Group Lasso Standard Transformer Voltage Prediction

January 6, 2022

Flow-Guided Sparse Transformer for Video Deblurring
Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, Luc Van Gool
Optical Flow Video Deblurring Multi Head Self Attention Sparse Transformer

January 4, 2022

Sparse-Dyn: Sparse Dynamic Graph Multi-representation Learning via Event-based Sparse Temporal Attention Network
Yan Pang, Chao Liu
Dynamic Graph Temporal Attention Sparse Transformer Dynamic Graph Representation Learning

December 13, 2021

Embracing Single Stride 3D Object Detector with Sparse Transformer
Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, Zhaoxiang Zhang
3D Object Detection Sparse Transformer 2 Dimensional Detection Lidar Based 3D Object Detector Frame 3D Object Detection