Multimodal Token Compression

Follow this topic Ask about this topic

Multimodal token compression reduces the number or size of image, video, audio, and other input representations so models can process longer inputs with less computation and memory. Common approaches prune or merge redundant tokens, pool them into learned bottlenecks, allocate budgets using queries or modality-specific signals, and compress key-value caches. These methods are evaluated on tasks such as visual question answering, video understanding, audio comprehension, and multimodal retrieval, often by comparing accuracy with latency, FLOPs, or memory use at different compression levels. The central challenge is deciding what information can safely be removed: useful evidence varies by task, modality, and model layer, and aggressive compression can discard fine details or weaken joint understanding and generation.

Latest papers

96 papers9 papers in the last four weeks, up 80% on the four weeks before. 0.1% of all new papers.

Rethinking Sequence Compression with Test-time Regression
Oct 22, 2026Sihan Yang, Jie Huang, Hanxuan Li +4LLM Context Compression Multimodal Token Compression
The Chinese University of Hong Kong · Independent Researcher · Token Foundry, Alibaba Group
PACE: Pareto-Adaptive Compression for Efficient Native MLLMs
Oct 15, 2026Lei Tian, Chunzheng Zhu, Kai Wu +2Multimodal Token Compression Visual Token Compression
ByteDance · Hunan University · Institute of Automation, Chinese Academy of Sciences
Bottlenecked semantic compression by representation reconstruction
Oct 12, 2026Xiaoya Tang, Shilong Liu, Daniel Smolyak +3Visual Token Compression Efficient ViTs
Scientific Computing and Imaging Institute University of Utah · Princeton AI Lab Princeton University · National Institutes of Health
LapMerge: Differentiable Exact-Budget Token Merging in Linear Time
Oct 9, 2026Magdalena Trędowicz, Aleksandra Nowak, Tomasz Steifer +3Token Merging Visual Token Merging
Jagiellonian University in Kraków · Institute of Fundamental Technological Research, Polish Academy of Sciences · Centre for Credible AI, Warsaw University of Technology
Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization
Oct 8, 2026Lei Zhai, Zhihao Chang, Shuyuan Yang +1Multimodal Token Compression Multimodal Instruction Tuning
School of Artificial Intelligence, Xidian University, Xi’an, China
DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models
Oct 6, 2026Shuo Yang, Changbai Li, Linlin Yang +6Multimodal Large Language Models Token Pruning
Beihang University · Communication University of China · National University of Singapore
TRIM-ReID: Duplication-Aware Token Reduction and Modality-Aligned Interaction for Multi-Modal Object Re-Identification
Oct 3, 2026Wanke Xia, Ruiding Zhu, Xingguo Xu +6Multimodal Fusion Multimodal Token Compression
Tsinghua University · Anhui University · Dalian University of Technology +2
ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression
Sep 29, 2026Zijing Cai, Yuzhe Wang, Jingxian Zhu +2Multimodal IR Multimodal Embedding
University of Science and Technology of China · Hefei University of Technology · National University of Singapore
OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models
Sep 29, 2026Yuchen Deng, Zidang Cai, Feidiao Yang +4Efficient Multimodal Inference Audio Token Compression
Shenzhen International Graduate School, Tsinghua University, China · Pengcheng Laboratory, China
From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Models
Sep 22, 2026Shengli He, Yongchao Liang, Roumeng He +5Visual Token Pruning Multimodal Token Compression
Guizhou University, Guiyang, China. · Shanghai Ocean University, Shanghai, China.
SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Models
Sep 21, 2026Ali Kerem Bozkurt, Baris Cem Bakay, Ibrahim Kulac +3Efficient Multimodal Inference Multimodal Large Language Models
Kocı University, Department of Computer Engineering · Kocı University, Department of Pathology · Hacettepe University, Department of Artificial Intelligence and Data Engineering
EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression
Sep 10, 2026Motahare Mounesan, Irfan KhanEfficient Multimodal Inference Edge-Cloud Collaborative Inference
Texas A&M University
Layer-Wise Gate-Controlled Prompt Truncation in a Multimodal Chest X-Ray Classifier
Sep 6, 2026Jingtao Lei, Hongji Li, Dexiang ShuPrompt Tuning Multimodal Classification
Central South University, Changsha 410083, China · Hong Kong Baptist University, Hong Kong, China · University of Minnesota, Minneapolis, MN 55414, USA
SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
Sep 1, 2026Shiyu Li, Zi-Yuan Hu, Shijia Huang +3Efficient VLM Inference Multimodal Large Language Models
1The Chinese University of Hong Kong · 2Weitu AI · 3Peking University
Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling
Aug 29, 2026Wengyi Zhan, Chenqian Yan, Songwei Liu +2Efficient Multimodal Inference Unified Multimodal Models
1Xiamen University · 2ByteDance
Visual Token Coding for Video Multimodal Large Language Models
Aug 28, 2026Chenxin Fang, Tao Chen, JunChao You +3Multimodal Token Compression Video Token Compression
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
VIG: Visual Information Gain as a Reward Signal for Multimodal Chain-of-Thought Compression
Aug 22, 2026Wen Luo, Xiaohan Yi, Xiaotao Huang +1Multimodal CoT Reasoning Multimodal Token Compression
School of Software Engineering, Huazhong University of Science and Technology · Tsinghua University
Role-Conditioned Sub-Token Routing for Efficient Vision-Language-Action Policies
Aug 19, 2026Wei Jiang, Wei WangEfficient VLA Models Multimodal Token Compression
Futurewei Technologies San Jose, CA 95131, USA
GSToken: Geometry-Structured Gaussian Tokens for Compact 3D Medical Image Representation
Aug 18, 2026Xiaoduo Li, Quan GuMedical Image Representation Learning Multimodal Token Compression
Taiyuan University of Technology · independent researcher
SEER: Long-Context Reasoning via Selective Visual-Text Compression
Aug 16, 2026Jiawei Xu, Zhilin Zhai, Jinrui Fang +6Long-Context Language Model Inference VLM Reasoning
School of Information, The University of Texas at Austin · Department of Computer Science, The University of North Carolina at Chapel Hill · Department of Engineering, University of Cambridge +2
Post-Hoc Sparse Coding of Latent Communication Between Vision-Language Model Agents
Aug 10, 2026Di Wu, Xiaohui ZhuSparse Autoencoders Multimodal Token Compression
Xi’an Jiaotong-Liverpool University
Reading is not Reasoning: Bridging the Agentic Policy Gap in Vision-Text Compression
Aug 9, 2026Cheng Fan, Junyi Zhou, Tingzhang Luo +5Cross-Modal Knowledge Distillation Multimodal Token Compression
City University of Hong Kong · Huawei Technologies Ltd.
Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs
Aug 9, 2026Kyeongyoon Lee, Hongyeob Kim, Youngeun Kim +1Efficient Multimodal Inference Audio Token Compression
Sungkyunkwan University · AWS AI Labs
VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling
Aug 9, 2026Yuqi Zhang, Cheng Chen, Yuyu Guo +6Vision-Language Models Efficient VLM Inference
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University, China · Shanghai Innovation Institute, China · Ant Group, China
KVAE: Family of Tokenizers for Multimodal Generative Models
Aug 6, 2026Andrey Shutkin, Denis Parkhomenko, Ivan Kirillov +11Variational Autoencoders Audio Token Compression
Kandinsky Lab
CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
Aug 5, 2026Zhenyu Yi, Qiang Hu, Zhenhao Li +33D Medical Imaging Medical VQA
School of Biomedical Engineering, Shanghai Jiao Tong University · 2Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology
EvtGraph: Event-Adaptive Compression for Sparse Temporal Graph Learning in Multimodal Time Series
Aug 5, 2026Ziqian Wang, Tingxiong Xiao, Yuxiao Cheng +1Irregular Time-Series Modeling Efficient Multimodal Inference
Department of Automation Tsinghua University Beijing, China
OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models
Aug 4, 2026Wanshun Su, Yang Shi, Feihu Liu +10Efficient Multimodal Inference Audio Token Compression
Northwestern Polytechnical University · Peking University · Alibaba Group +1
Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression
Aug 3, 2026Tianyu Liang, Xiangxi Zheng, Yilin Wang +1VLM Adaptation Multimodal Token Compression
Southeast University Nanjing, China · Nanjing University Nanjing, China · Zhejiang University Hangzhou, China +1
Allocation Before Ranking: Decoupled Token Compression for OmniLLMs
Aug 3, 2026Zhenghui Guo, Yilin Yang, Yuanbin Man +5Audio Token Compression Unified Multimodal Models
Department of Computer Science, University of Houston · Department of Computer Science and Engineering, The University of Texas at Arlington

Oct 22, 2026J/K move · Enter open · S save

Rethinking Sequence Compression with Test-time Regression

Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo+3

The Chinese University of Hong Kong · Independent Researcher · Token Foundry, Alibaba Group

LLM Context Compression Multimodal Token Compression Visual Token Compression

Open paper AskPDF arXiv

Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.