Token Mixing

Token mixing, a core component of many modern neural network architectures, aims to efficiently capture relationships between different parts of an input (e.g., pixels in an image, tokens in a sequence). Current research focuses on developing computationally efficient token mixers, exploring alternatives to computationally expensive self-attention mechanisms, such as those based on convolutional layers, wavelets, and Fourier transforms. These efforts are driven by the need for faster and more resource-efficient models, particularly for deployment on mobile devices and in applications involving high-resolution inputs. Improved token mixing techniques are significantly impacting the performance and scalability of various computer vision and other machine learning tasks.

Papers

August 7, 2024

CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications
Tianfang Zhang, Lei Li, Yang Zhou, Wentao Liu, Chen Qian, Xiangyang Ji
Vision Transformer Self Attention Token Mixer ViT LoS V2X Token Mixing Mobile Vision

July 1, 2023

WavePaint: Resource-efficient Token-mixer for Self-supervised Inpainting
Pranav Jeevan, Dharshan Sampath Kumar, Amit Sethi
Vision Transformer Self Supervised Convolutional Layer CNN Backbone Token Mixing

March 7, 2023

FFT-based Dynamic Token Mixer for Vision
Yuki Tatsunami, Masato Taki
Vision Paper Multi Head Self Attention Token Mixer Token Mixing

September 14, 2022

Analysis of Quantization on MLP-based Vision Models
Lingran Zhao, Zhen Dong, Kurt Keutzer
Convolutional Neural Network General Analysis Vision Model Quantization Operator Quantization Level Token Mixing

July 15, 2022

Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP
Zhicai Wang, Yanbin Hao, Xingyu Gao, Hao Zhang, Shuo Wang, Tingting Mu, Xiangnan He
Vision Transformer Multi Layer Token Level Multi Head Self Attention Relative Positional Encoding MLP Architecture Token Mixing

March 11, 2022

Active Token Mixer
Guoqiang Wei, Zhizheng Zhang, Cuiling Lan, Yan Lu, Zhibo Chen
CNN Model Token Mixer Token Mixing

March 7, 2022

WaveMix: Resource-efficient Token Mixing for Images
Pranav Jeevan, Amit Sethi
Convolutional Filter Token Mixing

November 24, 2021

Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, Bryan Catanzaro
Vision Transformer Transformer Megatron Decepticons Operator Learning Fourier Neural Operator Global Convolution Token Mixing

November 22, 2021

PointMixer: MLP-Mixer for Point Cloud Understanding
Jaesung Choe, Chunghyun Park, Francois Rameau, Jaesik Park, In So Kweon
Point Cloud Understanding MLP Mixer Token Mixing