Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

Yuheng Shi; Xiaohuan Pei; Linfeng Wen; Minjing Dong; Chang Xu

cs.CVApr 8, 2026

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

Authors: Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, Chang Xu

Organizations: University of Sydney, Australia · Sun Yat-sen University, China · City University of Hong Kong, China

Abstract

MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global resolution scaling paradigms indiscriminately flood the quadratic self-attention mechanism with visually redundant tokens, severely bottlenecking inference throughput while ignoring spatial sparsity and query intent. To overcome this, we propose Q-Zoom, a query-aware adaptive high-resolution perception framework that operates in an efficient coarse-to-fine manner. First, a lightweight Dynamic Gating Network safely bypasses high-resolution processing when coarse global features suffice. Second, for queries demanding fine-grained perception, a Self-Distilled Region Proposal Network (SD-RPN) precisely localizes the task-relevant Region-of-Interest (RoI) directly from intermediate feature spaces. To optimize these modules efficiently, the gating network uses a consistency-aware generation strategy to derive deterministic routing labels, while the SD-RPN employs a fully self-supervised distillation paradigm. A continuous spatio-temporal alignment scheme and targeted fine-tuning then seamlessly fuse the dense local RoI with the coarse global layout. Extensive experiments demonstrate that Q-Zoom establishes a dominant Pareto frontier. Using Qwen2.5-VL-7B as a primary testbed, Q-Zoom accelerates inference by 2.52 times on Document & OCR benchmarks and 4.39 times in High-Resolution scenarios while matching the baseline's peak accuracy. Furthermore, when configured for maximum perceptual fidelity, Q-Zoom surpasses the baseline's peak performance by 1.1% and 8.1% on these respective benchmarks. These robust improvements transfer seamlessly to Qwen3-VL, LLaVA, and emerging RL-based thinking-with-image models. Project page is available at https://yuhengsss.github.io/Q-Zoom/.

Figures & tables

Fig. 1 : Comparison of adaptive high-resolution perception paradigms. Training-free methods rely on handcrafted contrastive rules, requiring multiple redundant prefilling passes. RL-based methods use the LLM to auto-regressively generate code or coordinates to find the RoI. Our Q-Zoom framework operates directly on the intermediate feature space during a single prefilling pass, yielding superior efficiency.

Max Limit	Through- put	DocVQA val	OCRBench test	InfoVQA val	V*	Ave.
512	4.6	90.9	83.3	69.1	62.8	76.5
2048	2.2	94.7	84.8	80.4	72.3	83.1

TABLE I : Performance and throughput comparison of Qwen2.5-VL 7B on Document and Vision-Centric benchmarks under different maximum visual token limits. Throughput is measured in samples/second on a single RTX A6000 GPU.

Panel of Figure 3 — Fig. 2 : Overview of the proposed Adaptive High-Resolution Perception Framework. (a) The framework derives robust supervisory signals through consistency-aware generation to train a lightweight gating module. (b) During inference, the gate dynamically evaluates the textual query. It routes simpler queries for direct, accelerated generation using coarse features, while triggering the SD-RPN for complex queries to extract targeted high-resolution regions.

Fig. 3 : Overview of the conditional Region-of-Interest extraction pipeline. When triggered by the dynamic gating module, the SD-RPN (top) leverages shared intermediate features from the frozen backbone to efficiently generate a dense spatial heatmap. During the training phase (bottom), the network is optimized through a self-distillation paradigm, utilizing denoised cross-modal attention maps from the base MLLM as supervisory pseudo-labels. Superscripts indicate network depth (layers), while subscripts denote the modality or token origin. System prompts are excluded for visual clarity.

Fig. 6 : Overview of the Spatio-Temporal Alignment and Targeted Post-SFT pipeline. The vision encoder and projector are omitted for visual brevity.

Methods	Throughput	DocVQA val	ChartQA test	OCRBench test	InfoVQA val	TextVQA val	Ave.
LLaVA-1.5-7B	1.0 $\times$	21.5	18.1	31.4	20.4	46.1	27.5
+S 2 [ECCV2024] [ 67 ]	0.70 $\times$	27.1	18.9	32.6	22.5	52.6	30.7
+ViCrop [ICLR2025] [ 23 ]	0.42 $\times$	27.0	20.0	33.2	21.4	57.2	31.8
+SD-RPN [ICLR2026] [ 37 ]	0.62 $\times$	34.2	20.6	37.3	22.3	58.8	34.6
+Q-Zoom	0.64 $\times$	34.0	20.8	38.0	22.4	58.4	34.7 $\uparrow$ 7.2
LLaVA-1.5-13B	1.0 $\times$	23.5	18.1	33.7	23.4	48.7	29.5

TABLE II : Performance on Document & OCR benchmarks. Dataset subscripts denote the evaluation split. Performance subscripts show the absolute improvement (

\uparrow

) over the baseline. Throughput is relative to the baseline, measured on a single NVIDIA A6000 GPU and our results are evaluated under a constraint of 576 maximum visual tokens.

Methods	Tp	V* Bench			MME-RW	HR-Bench 4K			HR-Bench 8K			Ave.
Methods	Tp	Attr	Spatial	Overall	Lite	FSP	FCP	Overall	FSP	FCP	Overall	Ave.
LLaVA-1.5-7B	1.00 $\times$	48.7	52.6	50.3	27.7	39.5	35.5	37.5	32.5	33.8	33.8	37.3
+S 2 [ECCV2024] [ 67 ]	0.63 $\times$	53.0	59.1	55.5	28.5	49.8	38.3	44.0	40.5	36.5	38.5	41.6
+ViCrop [ICLR2025] [ 23 ]	0.15 $\times$	53.9	50.0	52.4	27.6	60.8	34.8	47.8	38.5	33.8	36.1	41.0
+SD-RPN [ICLR2026] [ 37 ]	0.57 $\times$	70.4	71.1	70.7	27.7	59.0	35.5	47.3	48.8	34.5	41.6	46.8
+Q-Zoom	0.58 $\times$	70.4	71.1	70.7 $\uparrow$ 20.4	27.7 $\uparrow$ 0.0	58.8	35.8	47.1 $\uparrow$ 9.6	48.5	34.5	41.4 $\uparrow$ 7.6	46.7 $\uparrow$ 9.4

TABLE III : Performance on Vision-Centric and High-Resolution benchmarks. Dataset subscripts denote the specific evaluation split. Performance subscripts indicate the absolute improvement (

\uparrow

) of our latest version over the baseline. Tp denotes the relative inference throughput. Averages are computed exclusively across the four Overall metrics. Unless otherwise noted, our results are evaluated under a constraint of 4,096 maximum visual tokens. The

\dagger

symbol denotes results directly cited from the corresponding original publications.

Fig. 7 : Qualitative comparisons on challenging examples from TextVQA (left) and V* Bench (right). These examples highlight visually demanding scenarios where the target evidence is small or obscured. The baseline Qwen2.5-VL-7B suffers from resolution compression, whereas our Q-Zoom framework successfully leverages the SD-RPN to predict highly accurate RoI heatmaps, cropping the necessary fine-grained details to generate the correct answers.

Panel of Figure 10 — Fig. 8 : Accuracy vs. Efficiency Trade-offs. We evaluate the Qwen2.5-VL-7B baseline against Q-Zoom by sweeping the maximum visual token limit in a image from 256 to 4,096. Our framework establishes a dominant Pareto frontier on both (a) Document & OCR and (b) High-Resolution benchmark categories. By adaptively localizing RoIs, Q-Zoom surpasses the peak accuracy of the brute-force 4,096-token baseline while reducing the visual token cost and accelerating throughput.

Components			Document & OCR							High-Res & Vision-Centric						General QA
RPN	SFT	Gate	Tp	Doc	Chart	OCR	Info	Text	Ave.	Tp	V*	RW	HR4K	HR8K	Ave.	Tp	MME	MMS
Qwen2.5-VL-7B
-	-	-	1.0 $\times$	92.0	83.0	82.8	70.1	81.1	81.8	1.0 $\times$	64.4	35.4	57.9	52.4	52.5	1.0 $\times$	2297	62.4
✓ †	-	-	0.50 $\times$	93.6	85.5	82.9	76.9	83.5	84.5	0.47 $\times$	77.5	40.5	73.3	66.1	64.4	0.40 $\times$	2335	62.1
✓	-	-	0.59 $\times$	94.1	85.8	84.9	79.6	83.0	85.5	0.49 $\times$	80.1	44.6	75.5	66.6	66.7	0.63 $\times$	2272	62.5
✓	✓	-	0.63 $\times$	94.5	86.5	85.9	79.9	83.8	86.1	0.52 $\times$	80.1	46.0	75.8	67.4	67.3	0.66 $\times$	2322	63.1

TABLE IV : Ablation study on the core components of our proposed framework. We progressively enable the upgraded Self-Distilled Region Proposal Network ( RPN ), targeted Supervised Fine-Tuning ( SFT ), and the Dynamic Gating Network ( Gate ). The

\dagger

denotes results obtained using the configuration from our preliminary conference version. Tp denotes throughput, which is reported as relative speed (

\times

) to baseline.

Method	Document & OCR							High-Res & Vision-Centric
Method	Tp	Doc	Chart	OCR	Info	Text	Ave.	Tp	V*	RW	HR4K	HR8K	Ave.
(0) LLaVA-1.5-7B	1.0 $\times$	21.5	18.1	31.4	20.4	46.1	27.5	1.0 $\times$	50.3	27.7	37.5	33.8	37.3
(1) Response-to-Image Attention	0.39 $\times$	28.4	22.8	32.4	22.2	53.1	31.8	0.38 $\times$	57.1	27.6	41.8	35.3	40.5
(2) GroundingDINO (1 bbox)	0.37 $\times$	23.2	19.2	32.5	20.6	52.6	29.6	0.17 $\times$	56.5	27.9	48.9	44.8	44.5
(3) GroundingDINO (2 bbox)	0.32 $\times$	23.8	19.2	32.5	20.2	54.5	30.0	0.15 $\times$	62.3	27.6	48.0	46.3	46.1
(4) +SD-RPN	0.62 $\times$	34.2	20.6	37.3	22.3	58.8	34.6	0.57 $\times$	70.7	27.7	47.3	41.6	46.8

TABLE V : Ablation of training-free pseudo-label based ROI strategies. Throughput is reported as relative speed (

\times

) to each model’s baseline under the same macro-category.

	Value	Doc	Chart	OCR	Info	Text	V*	RW	Ave.
	3	92.6	85.0	82.2	70.4	81.2	61.8	36.5	72.8
	9	93.0	84.8	83.3	71.0	81.4	71.2	36.0	74.4
	15	94.0	85.6	84.4	77.6	82.7	71.7	39.6	76.5
	18	94.1	85.8	84.9	79.6	83.0	80.1	44.6	78.9
$B$	21	93.6	85.7	84.4	76.4	82.2	78.5	41.2	77.4
	1	93.6	85.2	84.6	75.5	82.5	71.7	40.4	76.2

TABLE VI : Ablation on the backbone depth (

B

) and the number of RPN layers (

R

) using Qwen2.5-VL 7B baseline. When ablating

B

R

is fixed to 3; when ablating

R

B

is fixed to 18.

Data Size	Doc	Chart	OCR	Info	Text	V*	RW	Ave.
10k	93.6	85.4	84.9	78.0	82.7	78.5	40.7	77.7
25k	93.5	85.5	85.1	78.1	83.0	80.1	42.4	78.2
50k	93.4	85.5	84.6	78.8	83.0	78.0	43.7	78.1
100k	93.6	85.3	84.9	79.2	82.9	78.0	45.1	78.4
185k	94.1	85.8	84.9	79.6	83.0	80.1	44.6	78.9
68k †	93.4	85.5	83.6	75.8	83.0	80.1	44.3	78.0

TABLE VII : Ablation on pseudo-label training data size for the SD-RPN. The

\dagger

denotes a baseline model supervised exclusively by 68K ground-truth (GT) bounding boxes from the Visual CoT dataset.

$\boldsymbol{\tau_{fg}}$	$\boldsymbol{\tau_{bg}}$	LLaVA-1.5-7B				Qwen2.5-VL-7B
$\boldsymbol{\tau_{fg}}$	$\boldsymbol{\tau_{bg}}$	OCR	Text	V*	Ave.	OCR	Text	V*	Ave.
Baseline		31.4	46.1	50.3	42.6	82.8	81.1	64.4	76.1
0.10	0.10	35.1	57.4	65.4	52.6	84.3	83.1	78.0	81.8
0.15	0.10	36.3	57.9	67.5	53.9	84.2	83.1	79.6	82.3
0.20	0.10	37.0	58.7	67.5	54.4	84.5	82.8	79.1	82.1
0.30	0.10	36.6	58.1	64.9	53.2	84.2	82.6	77.5	81.4

TABLE VIII : Ablation study on pseudo-label thresholds

\tau_{fg}

and

\tau_{bg}

across different model architectures.

Panel of Figure 16 — Fig. 9 : Ablation of the Consistency-aware Training Sample Generation upon Qwen2.5-VL 7B. (Top) Training loss curves of the dynamic gating network. (Bottom) The Pareto front illustrating the trade-off between perception accuracy and inference efficiency (No-RoI Ratio).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Config	SD-RPN	Post-SFT	Dynamic Gate
Optimizer	AdamW
Weight decay	0.0
Optimizer momentum	$(\beta_{1},\beta_{2})=(0.9,0.98)$
Batch size	128	64	128
Learning rate schedule	cosine decay
Peak learning rate	$1\times 10^{-4}$	$1\times 10^{-6}$	$1\times 10^{-4}$

Appendix

TABLE IX : Training hyperparameters for the three core components of Q-Zoom.

Component	Model	Training Source	Samples
SD-RPN	Qwen-series	GQA	72K
		OCR-VQA	80K
		VCoT-DocVQA	33K
		Total	185K
	LLaVA-series	GQA	72K
		OCR-VQA	80K

Appendix

TABLE X : Training data used by each core component of Q-Zoom.

Model	Backbone Layer $B$
LLaVA-1.5-7B	15
LLaVA-1.5-13B	15
Qwen2.5-VL-3B	24
Qwen2.5-VL-7B	18
ZwZ-Qwen2.5-VL-7B	18
Qwen3-VL-4B	24

Appendix

TABLE XI : Backbone split depth

B

for the models used in the main results. All models use the same branch depth

R=3

for both SD-RPN and dynamic gating.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAG
Jun 20, 2026Oanh N. Tran, Thanh Quoc Hung Le, Oscar Chew +2Multimodal RAG Efficient VLM Inference
VinUni-Illinois Smart Health Center, VinUniversity · Texas A&M University
LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
Feb 26, 2026Yuxiang Shen, Hailong Huang, Zhenkun Gao +6VLM Reasoning Large Vision-Language Models
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
Towards High-Resolution Visual Perception via Hierarchical Entity Exploration
Jul 1, 2026Ziyu Ma, Shidong Yang, Yuxiang Ji +5Large Vision-Language Models Fine-Grained Visual Perception
AMAP, Alibaba Group, China · Monash University, Australia

Jun 20, 2026J/K move · Enter open · S save

Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAG

Oanh N. Tran, Thanh Quoc Hung Le, Oscar Chew, Kuan-Hao Huang+1

VinUni-Illinois Smart Health Center, VinUniversity · Texas A&M University

Multimodal RAG Efficient VLM Inference Vision-Language Retrieval Fine-Grained Visual Perception

Open paper AskPDF arXiv

Vision-Language Models (VLMs) struggle as query-relevant objects become smaller. To address this, recent training-free approaches dynamically retrieve and zoom into local image regions. However, we show that indiscriminately applying retrieval ignores a critical vulnerability: the resolution-context trade-off. Patch-based zooming recovers details for small targets, but can split large objects and destroy global spatial context; attention-based retrieval better preserves large objects, but remains less reliable on tiny details; and global perception is often fastest when retrieval is unnecessary. Motivated by these failure modes, we introduce ViRGo (Visual Retrieval or Global Perception), a lightweight framework that formulates visual retrieval as an adaptive routing problem. ViRGo estimates object scale from the VLM's intrinsic localization heads during the initial forward pass and combines it with semantic token confidence to select between global perception, patch-based retrieval, and attention-based retrieval with minimal additional computation. Experiments across multiple VQA benchmarks and object-size groups show that ViRGo improves the accuracy-efficiency trade-off: it matches patch retrieval on small details, leverages attention-based retrieval for larger objects, and reduces inference time by routing to the global baseline when zooming is unnecessary.

Abstract

Efficient VLM Inference

Explore similar work

Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAG

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Towards High-Resolution Visual Perception via Hierarchical Entity Exploration