Visual Token Merging

Follow this topic Ask about this topic

Visual token merging studies how to combine redundant image patches or video-frame tokens so vision and multimodal models use less computation, memory, and context. Common approaches pair similar tokens, rank them by attention or salience, and use spatial or temporal structure to guide merging; many methods work without retraining, while others learn adaptive budgets or combine pruning with merging. Applications include image classification, segmentation, document retrieval, video understanding, image generation, and robotic control. The central challenge is preserving small objects, boundaries, positions, and query-relevant evidence while achieving real speedups, which can vary across tasks, model architectures, and hardware.

Latest papers

36 papers9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

MPM: Mutual Pair Merging for Efficient Vision Transformers
Apr 7, 2026Simon Ravé, Pejman Rasti, David RousseauVisual Token Merging Efficient ViTs
LARIS University of Angers Angers, France · UMR INRAe-IRHS Angers, France
KiToke: Distribution-Aware Token Compression for Video Large Language Models
Apr 3, 2026Haifeng Huang, Biyin Xu, Yang LiVideo Token Compression Visual Token Compression
Department of Computer Science, Iowa State University.
Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention
Mar 23, 2026Junhao Du, Jialong Xue, Anqi Li +2Video Token Compression Large Vision-Language Models
Shanghai Jiao Tong University · Beijing University of Posts and Telecommunications
DepthCache: Depth-Guided Training-Free Visual Token Merging for Vision-Language-Action Model Inference
Mar 11, 2026Yuquan Li, Lianjie Ma, Han Ding +1Efficient VLA Model Inference Visual Token Compression
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. · School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China.
StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
Mar 7, 2026Duy M. H. Nguyen, Tuan A. Tran, Duong Nguyen +17Token Merging Image Segmentation
University of Stuttgart · DFKI · Max Planck Research School +10
Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework
Feb 23, 2026Yibo Yan, Mingdong Ou, Yi Cao +5Visual Document Retrieval Multi-Vector Retrieval
Hong Kong University of Science and Technology (Guangzhou) · Alibaba Cloud Computing · Hong Kong University of Science and Technology

Apr 7, 2026J/K move · Enter open · S save

MPM: Mutual Pair Merging for Efficient Vision Transformers

Simon Ravé, Pejman Rasti, David Rousseau

LARIS University of Angers Angers, France · UMR INRAe-IRHS Angers, France

Visual Token Merging Efficient ViTs Vision Transformer Semantic Segmentation

Open paper AskPDF arXiv

Decreasing sequence length is a common way to accelerate transformers, but prior token reduction work often targets classification and reports proxy metrics rather than end-to-end latency. For semantic segmentation, token reduction is further constrained by the need to reconstruct dense, pixel-aligned features, and on modern accelerators the overhead of computing merge maps can erase expected gains. We propose Mutual Pair Merging (MPM), a training-free token aggregation module that forms mutual nearest-neighbor pairs in cosine space, averages each pair, and records a merge map enabling a gather-based reconstruction before the decoder so that existing segmentation heads can be used unchanged. MPM introduces no learned parameters and no continuous compression knob (no keep-rate or threshold). The speed-accuracy trade-off is set by a discrete insertion schedule. We benchmark end-to-end latency on an NVIDIA H100 GPU (with and without FlashAttention-2) and a Raspberry Pi 5 across standard segmentation datasets. On ADE20K, MPM reduces per-image latency by up to 60% for ViT-Tiny on Raspberry Pi 5, and increases throughput by up to 20% on H100 with FlashAttention-2 while keeping the mIoU drop below 3%. These results suggest that simple, reconstruction-aware, training-free token merging can translate into practical wall-clock gains for segmentation when overhead is explicitly accounted for.

Latest papers

MPM: Mutual Pair Merging for Efficient Vision Transformers

KiToke: Distribution-Aware Token Compression for Video Large Language Models

Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention

DepthCache: Depth-Guided Training-Free Visual Token Merging for Vision-Language-Action Model Inference

StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models

Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework