Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models

Qingtao Pan; Zhihao Dou; Shuo Li

cs.CVMar 11, 2026

Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models

Authors: Qingtao Pan, Zhihao Dou, Shuo Li

Organizations: Case Western Reserve University

Abstract

Large Multimodal Models (LMMs) struggle to adapt varying computational budgets due to numerous visual tokens. Previous methods attempted to reduce the number of visual tokens before or within LLMs. However, these strategies inevitably result in the loss of visual semantic. To address these issues, we introduce FMVR, a plug-and-play and extremely simple Frequency-Modulated Visual Restoration strategy to boost the reasoning ability of LMMs under visual token reduction. Specifically, FMVR disentangles the visual representation of fewer visual tokens into low- and high-frequency components through AvgPool and MaxPool. The derived frequencies are subsequently modulated using lightweight learnable parameters. The high-frequency from AvgPool acts as a saliency filter to enhance saliency visual semantics, while the low-frequency from MaxPool acts as an anti-saliency filter to strengthen weak visual semantics. It enables the preservation of visual semantics dominated by few visual tokens and the restoration of diluted visual semantics. Additionally, we inject FMVR into Matryoshka Representation Learning to learn coarse-to-fine visual token sets, thus enabling to elastically adjust the number of visual tokens during inference while maintaining comparable performance. Experiments across 10 image-based and 4 video-based bench marks demonstrate that FMVR-LLaVA reduce the FLOPs of LLaVA-1.5-7B by 89%, while maintaining almost 100% of the original accuracy. The code will be open.

Figures & tables

Figure 1: Our FMVR (b) can restore the visual semantics from compressed tokens, alleviating the loss of visual contents in previous token compression methods (a).

Figure 2: Grad-CAM visualization (576 and 36 visual tokens) shows that the reduction of visual tokens leads to a noticeable degradation in visual focus.

Figure 3: Illustration of FMVR-LLaVA. The FMVR is injected into MRL to construct nested visual tokens, where FMVR is used to enhance the visual semantics of each visual token set, thus forming a set of reinforced nested visual tokens for LLM training.

Methods	#Vision Tokens	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet	Avg. (%)
BLIP-2 [ 37 ]	32	65.0	41.0	19.6	61.0	42.5	85.3	1293.8	31.2	–	22.4	–
InstructBLIP [ 19 ]	32	72.4	49.2	34.5	60.5	50.1	86.1	1391.4	36.0	23.7	26.2	50.8
IDEFICS-9B [ 28 ]	64	50.9	38.4	35.5	53.5	25.9	81.9	1177.3	48.2	–	30.0	–
IDEFICS-80B [ 28 ]	64	60.0	45.2	36.0	61.8	30.9	66.0	1518.2	54.5	–	39.7	–
Qwen-VL [ 4 ]	256	78.8	59.3	35.2	67.1	63.8	70.0	1487.6	38.2	7.4	13.0	50.7
Qwen-VL-Chat [ 4 ]	256	78.2	57.5	38.9	68.2	60.7	74.9	1487.5	60.6	–	47.3	–

Table 1: Performance comparison on LLaVA-1.5-7B across 10 image-based benchmarks. ‘#VisionTokens’ is the number of vision tokens.

\Delta

rows show the change vs. LLaVA baseline.

Method	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet	Avg.(%)
M3 (Baseline)	78.5	61.3	53.1	65.2	54.8	87.0	1451.5	66.4	56.3	30.2	62.5
FMVR (Ours)	78.6	62.3	55.1	69.7	55.5	86.4	1473.9	65.8	57.6	33.4	63.8
FMVR w/o AvgPool	77.9	62.0	54.6	68.1	54.3	85.7	1459.6	65.4	56.7	32.5	63.0
FMVR w/o MaxPool	78.3	62.6	53.3	68.4	55.0	86.9	1463.7	65.8	57.1	32.6	63.3

Table 2: Ablation study of applying different pooling unit in FMVR.

Methods	Tokens	FLOPs (TB)	Prefill Time (ms)	Total Memory (GB)	Avg (%)
LLaVA-v1.5	576	8.0	58.1	21.6	63.0
FMVR	144	2.2 (×3.6)	19.5 (×3.0)	15.0 (×1.4)	63.8
	36	0.9 (×8.9)	18.0 (×3.2)	13.8 (×1.6)	62.9
	9	0.5 (×16)	17.7 (×3.3)	13.6 (×1.6)	61.1
	1	0.4 (×20)	17.6 (×3.3)	13.5 (×1.6)	57.7

Table 3: Efficiency analysis of different numbers of vision tokens on LLaVA-1.5-7B.

Methods	Vision Encoder	Projection	FMVR	LLM	Total
LLaVA-v1.5	0.349	0.024	-	8.0	8.37
FMVR 576	0.349	0.024	6.4e-5	8.0	8.37

Table 4: FLOPs of each component in FMVR-LLaVA.

Methods	Tokens	FLOPs (TB)	Prefill Time (ms)	Avg (%)
FastV	192	2.1	34.1	55.9
PDrop	192	2.0	36.7	61.6
SparseVLM	192	2.1	36.5	61.2
FMVR	36	0.9	18.0	62.9

Table 5: Efficiency analysis of different pruning methods on LLaVA-1.5-7B.

Methods	#Vision Tokens	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet	Avg. (%)
LLaVA-NeXT-7B [ 46 ]	2880	81.3	62.5	55.2	67.5	60.3	86.8	1511.8	65.8	57.3	40.0	65.2
FastV [ 11 ]	320	61.5	49.8	51.3	66.6	52.2	49.5	1099.0	53.4	42.5	20.0	50.2
PyramidDrop [ 69 ]	320	66.8	50.4	49.7	66.7	49.0	60.8	1171.5	55.5	44.7	24.0	52.6
SparseVLM [ 83 ]	320	74.6	57.9	54.2	67.2	56.5	76.9	1386.1	63.1	56.7	32.8	60.9
Prumerge+ [ 60 ]	320	75.3	58.8	57.7	68.1	54.0	79.5	1444.3	63.0	55.6	31.4	61.6
TRIM [ 63 ]	320	74.9	59.9	53.5	66.2	50.2	86.5	1443.8	63.5	51.0	32.7	61.1

Table 6: Performance comparison on LLaVA-NeXT-7B across 10 image-based benchmarks. ‘#VisionTokens’ is the number of vision tokens.

\Delta

rows show the change vs. LLaVA-NeXT-7B baseline.

Methods	#Vision Tokens	Video-based Question-Answer				Video-based Generative Performance
Methods	#Vision Tokens	MSVD	MSRVTT	ActivityNet	Avg.	Correct.	Detail	Context.	Temp.	Consist.	Avg.
LLaMA Adapter [ 82 ]	1280	54.9	43.8	34.2	44.3	2.03	2.32	2.30	1.98	2.15	2.16
VideoChat [ 39 ]	512	56.3	45.0	26.5	42.6	2.23	2.50	2.53	1.94	2.24	2.29
Video-LLAMA [ 77 ]	1024	51.6	29.6	12.4	31.2	1.96	2.18	2.16	1.82	1.79	1.98
Video-ChatGPT [ 39 ]	$\sim$ 360	64.9	49.3	35.2	49.8	2.40	2.52	2.62	1.98	2.37	2.38
BT-Adapter [ 49 ]	$\sim$ 260	67.5	57.0	45.7	56.7	2.68	2.69	3.27	2.34	2.46	2.69

Table 7: Performance comparison on LLaVA-NeXT-7B across 4 video-based benchmarks. ‘#VisionTokens’ is the number of vision tokens.

\Delta

rows show the change vs. LLaVA-NeXT-7B baseline.

Figure 4: Comparison under different numbers of vision tokens. Our method achieves higher accuracy than M3 and MQT-LLaVA.

Figure 5: Grad-CAM visualization and response comparison between w/o FMVR and w FMVR under 36 visual tokens. From Grad-CAM visualization, the reduction of visual tokens leads to a noticeable degradation in visual semantics, which leads to hallucinations of certain objects in the response.

Figure 6: Case of image understanding on different VQA prompts. Our method yields basically correct answers under different numbers of visual tokens.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	#Vision Tokens	TextVQA	ChartQA	AI2D	MMB EN	Avg. (%)
Qwen2.5-VL	1296	84.8	86.1	80.4	82.8	83.5
FastV	128	73.8	52.2	71.4	72.9	67.6
DivPrune	128	67.0	50.4	72.1	77.8	66.8
CDPruner	128	77.8	59.2	74.0	76.2	71.8
FMVR (Ours)	81	76.3	62.2	77.5	79.6	73.9

Appendix

Table 8: Performance comparison of different pruning methods on Qwen2.5-VL-7B.

Methods	#Vision Tokens	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet	Avg. (%)
LLaVA-1.5-13B [ 48 ]	576	80.0	63.3	53.6	72.8	61.2	86.0	1531.2	68.5	63.5	36.2	66.2
FastV [ 11 ]	128	75.3	58.3	54.6	74.2	58.6	75.5	1460.6	66.1	62.3	32.8	63.1
PyramidDrop [ 69 ]	128	78.2	61.0	53.8	73.3	60.2	83.6	1489.5	67.5	62.8	32.1	64.7
SparseVLM [ 83 ]	128	77.6	59.6	51.4	74.3	59.3	85.0	1487.9	68.4	62.6	35.2	64.8
Prumerge+ [ 60 ]	128	76.2	58.3	52.8	73.3	56.1	82.7	1445.9	66.3	61.2	33.6	63.3
TRIM [ 63 ]	128	76.4	59.4	49.7	72.4	55.0	86.8	1426.9	67.1	58.4	35.1	63.2

Appendix

Table 9: Performance comparison on LLaVA-1.5-13B across 10 image-based benchmarks. ‘#VisionTokens’ is the number of vision tokens.

\Delta

rows show the change vs. LLaVA-v1.5-13B.

Num of Vis Tokens	TextVQA				MMBench
Num of Vis Tokens	Avg Pooling	Sequential	Spatial	Max Pooling	Avg Pooling	Sequential	Spatial	Max Pooling
1	49.2	46.3	45.9	47.1	60.7	57.9	58.6	59.8
9	50.8	46.7	46.4	48.2	64.2	60.4	61.5	63.9
36	55.3	51.2	49.0	52.8	65.2	61.8	63.1	64.5
144	55.5	53.7	50.6	53.3	65.8	64.2	63.7	64.9
576	57.8	55.6	52.6	54.7	65.9	64.7	64.0	65.3

Appendix

Table 10: Ablation study on Matryoshka visual token sampling, including average pooling, sequential sampling, spatial sampling, and max pooling.

Tokens	Method	VQAv2	GQA	VisWiz	SQA IMG	VQA Text	POPE	MME	MMB EN	MMB CN	MMVet
576	baseline	78.5	62.0	50.0	66.8	58.2	85.9	1510.7	64.3	58.3	30.5
576	Ours	79.2	63.0	56.5	68.9	57.8	87.5	1510.1	65.9	58.0	34.3
144	baseline	77.8	61.1	48.3	67.5	57.3	85.0	1480.4	65.1	57.1	30.2
144	Ours	78.6	62.3	55.1	69.7	55.5	86.4	1473.9	65.8	57.6	33.4
36	baseline	76.1	58.4	46.8	67.0	56.7	85.2	1448.3	64.5	57.7	29.4
36	Ours	76.5	60.9	52.9	69.5	55.3	85.9	1452.5	65.2	58.3	32.2

Appendix

Table 11: Baseline results of token numbers on LLaVA-1.5-7B.

Figure 7: Example demonstrating FMVR’s image understanding capability on more challenging OCR task. Response marked in red indicates errors.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
Jun 1, 2026Hyeonwoo Cho, Donghyeon Baek, Yewon Kim +1Efficient Multimodal Inference Multimodal Large Language Models
Yonsei University · Korea Institute of Science and Technology (KIST)
Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
Sep 28, 2026Jingdi lei, Junxian Li, Di Zhang +3Efficient Multimodal Inference Multimodal Large Language Models
Nanyang Technological University · Shanghai Jiao Tong University · Fudan University +2
ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
Jun 30, 2026Yuhao Wang, Mu Qiao, Haiwen Diao +5Efficient VLM Inference Multimodal Large Language Models
School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China +1

Jun 1, 2026J/K move · Enter open · S save

Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference

Hyeonwoo Cho, Donghyeon Baek, Yewon Kim, Bumsub Ham

Yonsei University · Korea Institute of Science and Technology (KIST)

Efficient Multimodal Inference Multimodal Large Language Models Visual Token Merging Visual Token Compression

Open paper AskPDF arXiv

Recent advancements in Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision-language tasks, yet the quadratic computational complexity arising from the vast number of visual tokens incurs significant memory and latency bottlenecks. While visual token reduction (VTR) strategies have been explored to mitigate this burden, existing methods overlook the positional and attentional consistency between the full and reduced sequences, resulting in a distorted representation. To this end, we propose RESTORE, a novel VTR framework that rectifies the positional and attentional distortions while maintaining efficiency. Specifically, we present a simple yet effective calibration method that restores lost visual attention by augmenting attention weights based on relative distances. We also introduce a distinctive anchor selection for token merging to mitigate information loss during feature averaging. Experimental results on multiple benchmarks demonstrate that our method consistently improves the accuracy of various reduction methods, achieving state-of-the-art performance while maintaining computational efficiency. Project page is available at https://cvlab.yonsei.ac.kr/projects/RESTORE

Abstract

Efficient VLM Inference

Explore similar work

Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs