Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models

Qingtao Pan; Zhihao Dou; Shuo Li

cs.CVMar 11, 2026

Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models

Authors: Qingtao Pan, Zhihao Dou, Shuo Li

Organizations: Case Western Reserve University

Abstract

Large Multimodal Models (LMMs) struggle to adapt varying computational budgets due to numerous visual tokens. Previous methods attempted to reduce the number of visual tokens before or within LLMs. However, these strategies inevitably result in the loss of visual semantic. To address these issues, we introduce FMVR, a plug-and-play and extremely simple Frequency-Modulated Visual Restoration strategy to boost the reasoning ability of LMMs under visual token reduction. Specifically, FMVR disentangles the visual representation of fewer visual tokens into low- and high-frequency components through AvgPool and MaxPool. The derived frequencies are subsequently modulated using lightweight learnable parameters. The high-frequency from AvgPool acts as a saliency filter to enhance saliency visual semantics, while the low-frequency from MaxPool acts as an anti-saliency filter to strengthen weak visual semantics. It enables the preservation of visual semantics dominated by few visual tokens and the restoration of diluted visual semantics. Additionally, we inject FMVR into Matryoshka Representation Learning to learn coarse-to-fine visual token sets, thus enabling to elastically adjust the number of visual tokens during inference while maintaining comparable performance. Experiments across 10 image-based and 4 video-based bench marks demonstrate that FMVR-LLaVA reduce the FLOPs of LLaVA-1.5-7B by 89%, while maintaining almost 100% of the original accuracy. The code will be open.

Figures & tables

Figure 1: Our FMVR (b) can restore the visual semantics from compressed tokens, alleviating the loss of visual contents in previous token compression methods (a).

Figure 2: Grad-CAM visualization (576 and 36 visual tokens) shows that the reduction of visual tokens leads to a noticeable degradation in visual focus.

Figure 3: Illustration of FMVR-LLaVA. The FMVR is injected into MRL to construct nested visual tokens, where FMVR is used to enhance the visual semantics of each visual token set, thus forming a set of reinforced nested visual tokens for LLM training.

Methods	#Vision Tokens	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet	Avg. (%)
BLIP-2 [ 37 ]	32	65.0	41.0	19.6	61.0	42.5	85.3	1293.8	31.2	–	22.4	–
InstructBLIP [ 19 ]	32	72.4	49.2	34.5	60.5	50.1	86.1	1391.4	36.0	23.7	26.2	50.8
IDEFICS-9B [ 28 ]	64	50.9	38.4	35.5	53.5	25.9	81.9	1177.3	48.2	–	30.0	–
IDEFICS-80B [ 28 ]	64	60.0	45.2	36.0	61.8	30.9	66.0	1518.2	54.5	–	39.7	–
Qwen-VL [ 4 ]	256	78.8	59.3	35.2	67.1	63.8	70.0	1487.6	38.2	7.4	13.0	50.7
Qwen-VL-Chat [ 4 ]	256	78.2	57.5	38.9	68.2	60.7	74.9	1487.5	60.6	–	47.3	–

Table 1: Performance comparison on LLaVA-1.5-7B across 10 image-based benchmarks. ‘#VisionTokens’ is the number of vision tokens.

\Delta

rows show the change vs. LLaVA baseline.

Method	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet	Avg.(%)
M3 (Baseline)	78.5	61.3	53.1	65.2	54.8	87.0	1451.5	66.4	56.3	30.2	62.5
FMVR (Ours)	78.6	62.3	55.1	69.7	55.5	86.4	1473.9	65.8	57.6	33.4	63.8
FMVR w/o AvgPool	77.9	62.0	54.6	68.1	54.3	85.7	1459.6	65.4	56.7	32.5	63.0
FMVR w/o MaxPool	78.3	62.6	53.3	68.4	55.0	86.9	1463.7	65.8	57.1	32.6	63.3

Table 2: Ablation study of applying different pooling unit in FMVR.

Methods	Tokens	FLOPs (TB)	Prefill Time (ms)	Total Memory (GB)	Avg (%)
LLaVA-v1.5	576	8.0	58.1	21.6	63.0
FMVR	144	2.2 (×3.6)	19.5 (×3.0)	15.0 (×1.4)	63.8
	36	0.9 (×8.9)	18.0 (×3.2)	13.8 (×1.6)	62.9
	9	0.5 (×16)	17.7 (×3.3)	13.6 (×1.6)	61.1
	1	0.4 (×20)	17.6 (×3.3)	13.5 (×1.6)	57.7

Table 3: Efficiency analysis of different numbers of vision tokens on LLaVA-1.5-7B.

Methods	Vision Encoder	Projection	FMVR	LLM	Total
LLaVA-v1.5	0.349	0.024	-	8.0	8.37
FMVR 576	0.349	0.024	6.4e-5	8.0	8.37

Table 4: FLOPs of each component in FMVR-LLaVA.

Methods	Tokens	FLOPs (TB)	Prefill Time (ms)	Avg (%)
FastV	192	2.1	34.1	55.9
PDrop	192	2.0	36.7	61.6
SparseVLM	192	2.1	36.5	61.2
FMVR	36	0.9	18.0	62.9

Table 5: Efficiency analysis of different pruning methods on LLaVA-1.5-7B.

Methods	#Vision Tokens	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet	Avg. (%)
LLaVA-NeXT-7B [ 46 ]	2880	81.3	62.5	55.2	67.5	60.3	86.8	1511.8	65.8	57.3	40.0	65.2
FastV [ 11 ]	320	61.5	49.8	51.3	66.6	52.2	49.5	1099.0	53.4	42.5	20.0	50.2
PyramidDrop [ 69 ]	320	66.8	50.4	49.7	66.7	49.0	60.8	1171.5	55.5	44.7	24.0	52.6
SparseVLM [ 83 ]	320	74.6	57.9	54.2	67.2	56.5	76.9	1386.1	63.1	56.7	32.8	60.9
Prumerge+ [ 60 ]	320	75.3	58.8	57.7	68.1	54.0	79.5	1444.3	63.0	55.6	31.4	61.6
TRIM [ 63 ]	320	74.9	59.9	53.5	66.2	50.2	86.5	1443.8	63.5	51.0	32.7	61.1

Table 6: Performance comparison on LLaVA-NeXT-7B across 10 image-based benchmarks. ‘#VisionTokens’ is the number of vision tokens.

\Delta

rows show the change vs. LLaVA-NeXT-7B baseline.

Methods	#Vision Tokens	Video-based Question-Answer				Video-based Generative Performance
Methods	#Vision Tokens	MSVD	MSRVTT	ActivityNet	Avg.	Correct.	Detail	Context.	Temp.	Consist.	Avg.
LLaMA Adapter [ 82 ]	1280	54.9	43.8	34.2	44.3	2.03	2.32	2.30	1.98	2.15	2.16
VideoChat [ 39 ]	512	56.3	45.0	26.5	42.6	2.23	2.50	2.53	1.94	2.24	2.29
Video-LLAMA [ 77 ]	1024	51.6	29.6	12.4	31.2	1.96	2.18	2.16	1.82	1.79	1.98
Video-ChatGPT [ 39 ]	$\sim$ 360	64.9	49.3	35.2	49.8	2.40	2.52	2.62	1.98	2.37	2.38
BT-Adapter [ 49 ]	$\sim$ 260	67.5	57.0	45.7	56.7	2.68	2.69	3.27	2.34	2.46	2.69

Table 7: Performance comparison on LLaVA-NeXT-7B across 4 video-based benchmarks. ‘#VisionTokens’ is the number of vision tokens.

\Delta

rows show the change vs. LLaVA-NeXT-7B baseline.

Figure 4: Comparison under different numbers of vision tokens. Our method achieves higher accuracy than M3 and MQT-LLaVA.

Figure 5: Grad-CAM visualization and response comparison between w/o FMVR and w FMVR under 36 visual tokens. From Grad-CAM visualization, the reduction of visual tokens leads to a noticeable degradation in visual semantics, which leads to hallucinations of certain objects in the response.

Figure 6: Case of image understanding on different VQA prompts. Our method yields basically correct answers under different numbers of visual tokens.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	#Vision Tokens	TextVQA	ChartQA	AI2D	MMB EN	Avg. (%)
Qwen2.5-VL	1296	84.8	86.1	80.4	82.8	83.5
FastV	128	73.8	52.2	71.4	72.9	67.6
DivPrune	128	67.0	50.4	72.1	77.8	66.8
CDPruner	128	77.8	59.2	74.0	76.2	71.8
FMVR (Ours)	81	76.3	62.2	77.5	79.6	73.9

Appendix

Table 8: Performance comparison of different pruning methods on Qwen2.5-VL-7B.

Methods	#Vision Tokens	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet	Avg. (%)
LLaVA-1.5-13B [ 48 ]	576	80.0	63.3	53.6	72.8	61.2	86.0	1531.2	68.5	63.5	36.2	66.2
FastV [ 11 ]	128	75.3	58.3	54.6	74.2	58.6	75.5	1460.6	66.1	62.3	32.8	63.1
PyramidDrop [ 69 ]	128	78.2	61.0	53.8	73.3	60.2	83.6	1489.5	67.5	62.8	32.1	64.7
SparseVLM [ 83 ]	128	77.6	59.6	51.4	74.3	59.3	85.0	1487.9	68.4	62.6	35.2	64.8
Prumerge+ [ 60 ]	128	76.2	58.3	52.8	73.3	56.1	82.7	1445.9	66.3	61.2	33.6	63.3
TRIM [ 63 ]	128	76.4	59.4	49.7	72.4	55.0	86.8	1426.9	67.1	58.4	35.1	63.2

Appendix

Table 9: Performance comparison on LLaVA-1.5-13B across 10 image-based benchmarks. ‘#VisionTokens’ is the number of vision tokens.

\Delta

rows show the change vs. LLaVA-v1.5-13B.

Num of Vis Tokens	TextVQA				MMBench
Num of Vis Tokens	Avg Pooling	Sequential	Spatial	Max Pooling	Avg Pooling	Sequential	Spatial	Max Pooling
1	49.2	46.3	45.9	47.1	60.7	57.9	58.6	59.8
9	50.8	46.7	46.4	48.2	64.2	60.4	61.5	63.9
36	55.3	51.2	49.0	52.8	65.2	61.8	63.1	64.5
144	55.5	53.7	50.6	53.3	65.8	64.2	63.7	64.9
576	57.8	55.6	52.6	54.7	65.9	64.7	64.0	65.3

Appendix

Table 10: Ablation study on Matryoshka visual token sampling, including average pooling, sequential sampling, spatial sampling, and max pooling.

Tokens	Method	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet
576	baseline	78.5	62.0	50.0	66.8	58.2	85.9	1510.7	64.3	58.3	30.5
576	Ours	79.2	63.0	56.5	68.9	57.8	87.5	1510.1	65.9	58.0	34.3
144	baseline	77.8	61.1	48.3	67.5	57.3	85.0	1480.4	65.1	57.1	30.2
144	Ours	78.6	62.3	55.1	69.7	55.5	86.4	1473.9	65.8	57.6	33.4
36	baseline	76.1	58.4	46.8	67.0	56.7	85.2	1448.3	64.5	57.7	29.4
36	Ours	76.5	60.9	52.9	69.5	55.3	85.9	1452.5	65.2	58.3	32.2

Appendix

Table 11: Baseline results of token numbers on LLaVA-1.5-7B.

Figure 7: Example demonstrating FMVR’s image understanding capability on more challenging OCR task. Response marked in red indicates errors.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Jun 1, 2026

Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference

Recent advancements in Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision-language tasks, yet the quadratic computational complexity arising from the vast number of visual tokens incurs significant memory and latency bottlenecks. While visual token reduction (VTR) strategies have been explored to mitigate this burden, existing methods overlook the positional and attentional consistency between the full and reduced sequences, resulting in a distorted representation. To this end, we propose RESTORE, a novel VTR framework that rectifies the positional and attentional distortions while maintaining efficiency. Specifically, we present a simple yet effective calibration method that restores lost visual attention by augmenting attention weights based on relative distances. We also introduce a distinctive anchor selection for token merging to mitigate information loss during feature averaging. Experimental results on multiple benchmarks demonstrate that our method consistently improves the accuracy of various reduction methods, achieving state-of-the-art performance while maintaining computational efficiency. Project page is available at https://cvlab.yonsei.ac.kr/projects/RESTORE

Hyeonwoo Cho, Donghyeon Baek, Yewon Kim +1

Yonsei University · Korea Institute of Science and Technology (KIST)

Efficient Multimodal Inference Multimodal Large Language Models Visual Token Merging Visual Token Compression

Ask PDF

Sep 28, 2026

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models~(MLLMs). Existing approaches reduce this cost by pruning redundant visual tokens, but permanently discard visual evidence that may become useful in subsequent layers. We instead ask whether all visual tokens can be preserved while reducing the cost of repeatedly evolving the representations through the Transformer. To answer this question, we perform low-rank interventions on visual-to-text information flow. We find that, after visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, suggesting the relevant visual influence is concentrated in a low-dimensional subspace. We further observe strong predictability in layer-specific visual states: lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Motivated by these findings, we propose

δ

-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval. Across image and video benchmarks,

δ

-Vision achieves higher accuracy than visual token pruning baselines at comparable or lower computation, while delivering competitive inference efficiency without discarding visual tokens.

Jingdi lei, Junxian Li, Di Zhang +3

Nanyang Technological University · Shanghai Jiao Tong University · Fudan University +2

Efficient Multimodal Inference Multimodal Large Language Models Low-Rank Adaptation

Ask PDF

Jun 30, 2026

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

Yuhao Wang, Mu Qiao, Haiwen Diao +5

School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China +1

Efficient VLM Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Ask PDF

Abstract

Efficient VLM Inference

Explore similar work

Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs