Visual Token Merging

Follow this topic Ask about this topic

Visual token merging studies how to combine redundant image patches or video-frame tokens so vision and multimodal models use less computation, memory, and context. Common approaches pair similar tokens, rank them by attention or salience, and use spatial or temporal structure to guide merging; many methods work without retraining, while others learn adaptive budgets or combine pruning with merging. Applications include image classification, segmentation, document retrieval, video understanding, image generation, and robotic control. The central challenge is preserving small objects, boundaries, positions, and query-relevant evidence while achieving real speedups, which can vary across tasks, model architectures, and hardware.

Latest papers

36 papers9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Consommé: Conditioned Token Merging for Split Inference
Oct 23, 2026Man Ngo, Jan S. RellermeyerSplit Inference Efficient ViTs
Leibniz University Hannover Hannover, Germany
Offline Makes Online Better: Calibrated Cache Compression for Video Generation
Oct 20, 2026Zhen Wang, Ning Lai, Qingtao YuKV-Cache Compression Efficient Video Generation
Homenet Research · CarryAI · Australian National University
MaskMerge: Structure-aware Token Merging for Efficient Instance Segmentation
Oct 16, 2026Bavesh Balaji, Letian Jiang, Alexander Wong +2Token Merging Visual Token Merging
University Of Waterloo
Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation
Oct 15, 2026SeungJu Cha, Ye-Chan Kim, HyunGee Kim +2Visual Token Merging Diffusion Model Acceleration
Hanyang University, South Korea
SCMerge: Spatial-Consistency-Aware Token Pair Merging for Efficient Semantic Segmentation
Oct 15, 2026Junfeng Ren, Yi Gong, Yuan ZengVisual Token Merging Efficient ViTs
Southern University of Science and Technology · Shenzhen Technology University
Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression
Oct 13, 2026Vatsal, Soumil Malhotra, Priyanshu Chahal +1Vision Transformer Efficient ViTs
Department of Mathematics, Indian Institute of Technology Roorkee
LapMerge: Differentiable Exact-Budget Token Merging in Linear Time
Oct 9, 2026Magdalena Trędowicz, Aleksandra Nowak, Tomasz Steifer +3Token Merging Visual Token Merging
Jagiellonian University in Kraków · Institute of Fundamental Technological Research, Polish Academy of Sciences · Centre for Credible AI, Warsaw University of Technology
Investigating Spatiotemporal Redundancy in Video Transformer for Collision Anticipation
Oct 3, 2026Xiaoshan ZhouSpatiotemporal Transformers Video Token Compression
School of Project Management, Faculty of Engineering, University of Sydney, NSW, Australia
Aperture: Merge-Consistent Rotary States for Compressed Tokens
Sep 29, 2026Yuhao Du, Shunian ChenRotary Positional Embeddings Visual Token Merging
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
Refinement Symmetry in Multimodal Transformers
Sep 26, 2026Yuhao Du, Shunian ChenAttention Mechanisms Multimodal Robustness
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval
Sep 18, 2026Jianxin You, Kun NiVisual Document Retrieval Visual Token Compression
Université de Montréal Montréal, QC, Canada · Concordia University Montréal, QC, Canada
Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
Sep 16, 2026Victor Bercy, Martyna Poreba, Michal Szczepanski +1Visual Token Merging Efficient ViTs
Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France · Université Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France
How Merge-Tolerant Are Vision Transformers for Wheat Phenotyping?
Aug 24, 2026Simon Ravé, Pejman Rasti, David RousseauEfficient ViTs Visual Token Merging
LARIS, University of Angers, Angers, France · UMR INRAE-IRHS, Angers, France
HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization
Aug 23, 2026Yingying Yan, Jiaqi Tang, Wei Wei +6Visual Tokenization Remote Sensing Image Understanding
Hong Kong University of Science and Technology Hong Kong, China
MergeOver: Post-Training Token Merging for Recursive Vision Transformers
Aug 13, 2026Junseo Kim, Uraz Odyurt, Amirreza YousefzadehVision Transformer Efficient ViTs
Computer Architecture for Embedded Systems, University of Twente Enschede, The Netherlands
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
Aug 3, 2026Yu Chen, Xiaohong Li, Xiaole Wang +3Efficient VLM Inference Large Vision-Language Models
1Evermind · 2Shanda Group · 3Peking University
ORCA: ORgan-Centroid Aggregation for Training-Free 3D CT Visual Token Compression
Jul 31, 2026Renjie Liang, Zijian Xu, Jinqian Pan +6Medical VLMs Visual Token Merging
University of Florida, Gainesville, FL, USA · University of Southern California, Los Angeles, CA, USA · National University of Singapore, Singapore
Are All Tokens Necessary for Visual Place Recognition? An Empirical Study of Token Reduction for Efficient Inference
Jul 17, 2026Tong Jin, Yunpeng Liu, Shuyu Hu +4Visual Place Recognition Efficient ViTs
Shenyang Institute of Automation, Chinese Academy of Sciences. · University of Chinese Academy of Sciences. · Shenzhen University of Advanced Technology.
Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval
Jul 6, 2026Suhyeong Park, Junha Jung, Jungwoo Park +1Token Merging Multi-Vector Retrieval
The Catholic University of Korea · AIGEN Sciences Inc. · Korea University
Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers
Jul 1, 2026Aravind Pradeep, Samira Nazari, Mahdi Taheri +1Token Merging Efficient ViTs
Brandenburg University of Technology Cottbus-Senftenberg, Germany · University of Zanjan, Iran · Humboldt University, Berlin, Germany +1
Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs
Jun 28, 2026Junzhou Chen, Jindong Wang, Gang ZhouEfficient VLM Inference Vision-Language-Action Models
Department of Computer Science, William & Mary, Williamsburg, VA 23188, USA · Department of Data Science, William & Mary, Williamsburg, VA 23188, USA
Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers
Jun 18, 2026Dong Hoon Lee, Seunghoon HongToken Merging Diffusion-Based Image Generation
Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea
RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
Jun 6, 2026Kyumin Choi, Ikbeom JangVision Transformer Efficient ViTs
Hankuk University of Foreign Studies Seoul, South Korea
Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
Jun 1, 2026Hyeonwoo Cho, Donghyeon Baek, Yewon Kim +1Efficient Multimodal Inference Multimodal Large Language Models
Yonsei University · Korea Institute of Science and Technology (KIST)
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
May 29, 2026Luyuan Zhang, Siyuan Li, Zedong Wang +7Variational Autoencoders Token Merging
1Tsinghua University · 2Westlake University · 3Zhejiang University +3
AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers
May 26, 2026Semi Lee, Hyejin Go, Hyesong ChoiToken Merging Self-Attention
Electronic Engineering · Soongsil University · Seoul, South Korea
G $^2$ TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models
May 12, 2026Junxian Li, Kai Liu, Zizhong Ding +4Efficient Multimodal Inference Unified Multimodal Models
Shanghai Jiao Tong University · Huawei Technologies Ltd
DORA: Dynamic Online Reinforcement Agent for Token Merging in Vision Transformers
May 12, 2026Kaixuan He, Song Chen, Yi KangEfficient ViTs Visual Token Merging
University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
UHR-BAT: Budget-Aware Token Compression Vision-Language model for Ultra-High-Resolution Remote Sensing
Apr 15, 2026Yunkai Dang, Minxin Dai, Yuekun Yang +4Visual Token Compression Remote Sensing Image Understanding
School of Artificial Intelligence Science and Technology, Nanjing University · School of Electronic Science and Engineering, Nanjing University · School of Physics, Nanjing University
MaMe & MaRe: Matrix-Based Token Merging and Restoration for Efficient Visual Perception and Synthesis
Apr 15, 2026Simin Huo, Ning LiVisual Token Merging Visual Token Compression
Shanghai Jiao Tong University, , Shanghai, 200240, Shanghai, China

Oct 23, 2026J/K move · Enter open · S save

Consommé: Conditioned Token Merging for Split Inference

Man Ngo, Jan S. Rellermeyer

Leibniz University Hannover Hannover, Germany

Split Inference Efficient ViTs Visual Token Compression Visual Token Merging

Open paper AskPDF arXiv

Split inference partitions a Vision Transformer between a resource-constrained edge device and a server. However, transmitting intermediate tokens across the communication channel is costly. Deploying fine-tuned backbones to every endpoint becomes expensive as well. We study token reduction for split inference under a frozen backbone, where only a small adapter is shipped. We observe that the CLS token plays a dual role: it acts as the global summary the receiver needs for classification, and it serves as a signal on which patches to keep. We propose Consommé, a method utilizing both characteristics. There, an information bottleneck compresses the CLS token into a compact latent which is transmitted in place of the full CLS token and conditions a learned patch gate merging patches into exactly

k

tokens via soft assignment. It reaches any budget in a single step, thus escaping the per-layer cap of bipartite matching (BPM). On DeiT-S/T, it leads all methods at aggressive token budgets and remains operable at shallow splits where BPM cannot operate, while matching fine-tuned methods at gentle budgets at a fraction of their deployment cost.