ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning

Huanxuan Liao; Zhongtao Jiang; Yupu Hao; Yuqiao Tan; Shizhu He; Ben Wang; Jun Zhao; Kun Xu

cs.CVMar 30, 2026

ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning

Authors: Huanxuan Liao, Zhongtao Jiang, Yupu Hao, Yuqiao Tan, Shizhu He, Ben Wang, Jun Zhao, Kun Xu,

Organizations: Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Abstract

Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive. We argue that the bottleneck lies not in how post-encoding representations are compressed but in the volume of pixels the encoder receives, and address it with ResAdapt, an Input-side adaptation framework that learns how much visual budget each frame should receive before encoding. ResAdapt couples a lightweight Allocator with an unchanged MLLM backbone, so the backbone retains its native visual-token interface while receiving an operator-transformed input. We formulate allocation as a contextual bandit and train the Allocator with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout feedback into a stable accuracy-cost learning signal. Across budget-controlled video QA, temporal grounding, and image reasoning tasks, ResAdapt improves low-budget operating points and often lies on or near the efficiency-accuracy frontier, with the clearest gains on reasoning-intensive benchmarks under aggressive compression. Notably, ResAdapt supports up to 16x more frames at the same visual budget while delivering over 15% performance gain. Code is available at https://github.com/Xnhyacinth/ResAdapt.

Figures & tables

Figure 1: Input-side Adaptation improves the visual-token efficiency frontier. (a) Three efficiency paradigms for video reasoning. Model-side methods compress tokens after encoding; output-side methods iteratively retrieve or zoom; ResAdapt reallocates per-frame visual budget before encoding, preserving the backbone’s native token interface and compatibility with optimized inference engines. (b) Qwen2.5-VL-7B results with 32 frames at ${\sim}$ 10% visual retention, where ResAdapt lies on or near the Pareto frontier and shows its largest gain on the reasoning-heavy benchmark.

Figure 2: ResAdapt framework. (a) At inference, a lightweight Allocator $\pi_{\theta}$ maps coarse visual features and the query to latent actions $a_{t}\sim\mathrm{Beta}(\alpha_{t},\beta_{t})$ , which parameterize per-frame input allocations. In the resize instantiation used in our experiments, these allocations are realized as scales $s_{t}\in[s_{\min},s_{\max}]$ , and the resized frames are processed by the MLLM in a single call. (b) During training, CAPO reshapes group-relative advantages with a dynamic cost pivot $\tau_{\text{dyn}}$ , while temporal-similarity regularization suppresses redundant high-budget allocation on adjacent similar frames.

Backbone	Method	Retention Ratio $R$	Reasoning	Video Perception Benchmark				Video Reasoning Benchmark
Backbone	Method	Retention Ratio $R$	Reasoning	VideoMME	LongVideoBench	MMVU	MLVU	VideoMMMU	LVBench
Qwen2.5-VL-7B	32 Frames
	Vanilla	100%	✗	62.0	58.9	52.7	63.1	49.6	38.6
	Random Drop	25.0%	✗	58.9	57.8	49.6	58.3	45.3	36.7
	ToMe ( Bolya et al., 2022 )	25.0%	✗	58.7	58.0	51.0	58.7	41.8	37.7
	VisionZip ( Yang et al., 2025c )	25.0%	✗	59.4	57.1	49.8	57.9	42.4	36.5

Table 1: Video QA results across two backbones (Qwen2.5-VL-7B, Qwen3-VL-8B) and two temporal horizons (32/128 frames). Retention ratio

R

reflects visual token count; Reasoning (✓/✗) indicates chain-of-thought use; bold marks the best result per group.

Figure 3: Efficiency–accuracy trade-offs and temporal reallocation. (a,b) VideoMMMU and VideoMME versus visual-token retention ratio

R

. ResAdapt is on or near the Pareto frontier, with the clearest advantage on reasoning-heavy settings at low retention. (c) Relative gain from trading spatial resolution for temporal coverage under a fixed 8-frame-equivalent budget.

Backbone	Method	Retention Ratio $R$	Reasoning	Temporal Grounding Benchmark
				Charades-STA				ActivityNet				NExT-GQA
				0.3	0.5	0.7	mIoU	0.3	0.5	0.7	mIoU	Acc	mIoU
Qwen2.5-VL-7B	32 Frames
	Vanilla	100%	✗	71.0	51.4	26.0	47.3	30.4	18.0	8.9	22.6	78.9	28.0
	Random Drop	25.0%	✗	39.4	23.2	11.0	25.7	15.2	8.1	3.7	11.7	77.5	16.6
	ToMe ( Bolya et al., 2022 )	25.0%	✗	39.5	23.9	11.4	26.0	16.0	8.4	4.0	12.1	77.8	16.3

Table 2: Temporal grounding results across two backbones and two temporal horizons. Notation follows Table 1 . See Sec. 4.2.2 for analysis of compression sensitivity.

Method	#Frames	Retention Ratio $R$	Scale						Inference			Total
Method	#Frames	Retention Ratio $R$	TFLOPs	Text Enc.	Visual Enc.	Scale Pred.	Scale Apply	Scale Time	TFLOPs	TTFT	Gen. Time	TFLOPs	E2E Time
Vanilla	16	100%	–	–	–	–	–	–	111.4	378.9	527.9	111.4	527.9
ResAdapt	16	76.3%	1.5	19.8	94.1	85.6	6.3	205.8	77.2 ( $\downarrow$ 30.7%)	272.5 ( $\downarrow$ 28.1%)	370.7 ( $\downarrow$ 29.8%)	80.1 ( $\downarrow$ 28.1%)	576.5 ( $\uparrow$ 9.2%)
ResAdapt	16	52.8%	1.5	19.9	102.9	94.5	8.4	225.7	51.5 ( $\downarrow$ 53.8%)	261.5 ( $\downarrow$ 31.0%)	313.1 ( $\downarrow$ 40.7%)	54.4 ( $\downarrow$ 51.2%)	538.8 ( $\uparrow$ 2.1%)
ResAdapt	16	28.9%	1.5	20.4	103.4	92.2	9.0	225.0	31.0 ( $\downarrow$ 72.2%)	227.2 ( $\downarrow$ 40.0%)	237.9 ( $\downarrow$ 54.9%)	33.9 ( $\downarrow$ 69.6%)	462.9 ( $\downarrow$ 12.3%)
Vanilla	32	100%	–	–	–	–	–	–	222.5	723.3	881.9	222.5	881.9

Table 3: Latency breakdown (ms,

\downarrow

) on Qwen2.5-VL-7B with single-GPU Allocator and 4-GPU vLLM engine. Averaged over 200 runs after 5 warm-up; E2E latency

=

Scale Time

+

Gen. Time.

Figure 4: Emergent active perception. Per-frame scale

s_{t}

over frame index for six VideoMME videos, grouped by intra-video scale diversity

\sigma

. High-diversity videos show localized scale spikes on scene changes, text overlays, and rapid motion; low-diversity videos remain near-uniform.

Figure 5: Global allocation statistics on VideoMME. (a) Aggregate predicted scale by frame position. (b) Case $\times$ frame heatmap for the first 200 videos. High-scale allocation appears as localized bursts rather than a fixed positional pattern.

Variant	$\bar{s}$	VideoMME	LongVideoBench	MMVU	VideoMMMU			LVBench
					Per.	Comp.	Adap.
$\beta$ -CAPO	0.54	60.3	58.2	51.2	65.0	54.3	28.7	37.6
$\mathcal{N}$ -CAPO	0.60	61.0	57.4	51.8	66.0	50.0	30.3	37.2

Table 4: Distribution family ablation for CAPO. The two variants follow the same training protocol.

Figure 6: Reward-design ablation. Mean predicted scale

\bar{s}

during training and validation. Direct cost penalties collapse to the minimum scale, whereas CAPO variants converge to stable intermediate operating points.

Method	VideoMME	LongVideoBench	LVBench	MMVU
Budget: 8 frames
Vanilla	54.0	53.9	33.3	48.9
Top-8 Select	52.2	51.1	32.0	49.2
Budget: 16 frames
Vanilla	58.9	56.0	36.1	50.9
Threshold Select	58.0	57.4	36.4	51.0

Table 5: Operator generalization. Zero-shot transfer of ResAdapt scores to frame selection. Combining top-

K

selection with adaptive resizing from 128 candidate frames outperforms uniform sampling baselines at lower token budgets.

Figure 7:

\mathcal{L}_{\text{sim}}

ablation: per-frame scale profiles. Without temporal-similarity regularization, the Allocator approaches near-uniform scaling; with it, the policy concentrates resolution on selected frames and suppresses redundant neighbors.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 6 — Table 6: Prompt template used for CAPO training. The template presents video frames and the task question, requires intermediate reasoning inside <think> tags, and places the final answer in \boxed{} within <answer> tags. This structure enables automatic reward extraction from MLLM outputs.

Figure 8: Per-video mean scale across benchmarks. Kernel density estimates of the per-video mean scale

\bar{s}

. Reasoning-heavy benchmarks shift toward larger

\bar{s}

than perception-heavy ones, indicating that the learned policy spends more fidelity where fine-grained evidence is more likely to matter.

Figure 9: VideoMME broken down by video duration. As clip duration grows, the policy lowers the average scale, increases within-video scale diversity, and faces lower task accuracy. Longer clips are therefore processed more aggressively and more selectively.

Figure 10: Scale allocation by VideoMME task category. Mean

\bar{s}

varies substantially across categories, with larger budgets assigned to categories that contain crowded motion or finer local evidence. Accuracy annotations show that allocation is not a trivial proxy for which category is easiest.

Figure 11: Selectivity versus prediction correctness on three representative benchmarks. Per-video Gini coefficients of the frame-level scales. Correct predictions tend to have higher Gini than incorrect ones, linking success to sharper concentration of resolution rather than merely larger average budgets.

Panel of Figure 12 — Figure 12: Sample-level robustness at 25% retention. Most originally correct predictions remain correct, but corrected and newly introduced errors are of comparable magnitude. Adaptive allocation is therefore selective rather than lossless.

Figure 13: Cross-benchmark scale diversity with and without

\mathcal{L}_{\text{sim}}

. Per-video scale standard deviation

\sigma

across five benchmarks. Without the regularizer, diversity collapses toward zero; adding

\mathcal{L}_{\text{sim}}

restores broad within-video variation on every benchmark.

Figure 14: Four diagnostics of the

\mathcal{L}_{\text{sim}}

ablation on VideoMME. With the regularizer, the frame-scale histogram becomes bimodal, the per-video range expands, adjacent-frame variation increases, and the Gini coefficient rises. The policy moves from near-uniform allocation to a genuinely selective regime.

Figure 15: Per-sample scale adaptivity under different reward designs. Scale range

s_{\max}-s_{\min}

over training on (a) training and (b) validation splits. CAPO keeps a non-trivial adaptive range, whereas direct cost collapses and cost-free training saturates.

Figure 16: Validation-time convergence under different reward designs. CAPO variants converge to stable intermediate operating points, while cost-free training saturates at the upper boundary and direct cost collapses to the lower boundary. Stability alone is not sufficient; the key is where the policy stabilizes.

Panel of Figure 17 — Figure 17: Case 1: Video-MMMU Comprehension ( Hu et al., 2025 ) (Vanilla $\times$ $\to$ ResAdapt $\checkmark$ ). The policy concentrates resolution on diagram-bearing slide frames, compresses lecturer-only frames, and suppresses the final quiz frame that the prompt explicitly marks as irrelevant.

Panel of Figure 18 — Figure 18: Case 2: Video-MMMU Adaptation ( Hu et al., 2025 ) (Vanilla $\times$ $\to$ ResAdapt $\checkmark$ ). When the answer depends on reading a numeric table and performing a $\chi^{2}$ computation, the policy keeps a much higher global budget and strongly upscales the table-bearing frames.

Panel of Figure 19 — Figure 19: Case 3: VideoMME ( Fu et al., 2025a ) (Vanilla $\times$ $\to$ ResAdapt $\checkmark$ ). Frames containing the decisive date overlays are enlarged, while the largely homogeneous sky footage is compressed. The policy spends budget on answer-bearing evidence rather than on the surrounding context.

Panel of Figure 20 — Figure 20: Case 4: VideoMME ( Fu et al., 2025a ) (Vanilla $\checkmark$ $\to$ ResAdapt $\times$ ; failure case). A nearby frame is enlarged, but the actual fork-bearing frame is compressed. The decisive fine detail is therefore lost at exactly the wrong moment.

Model	MathVista testmini	MMMU val	OCRBench	ChartQA	AI2D	TextVQA val
Qwen2.5-VL-7B	49.1 (100%)	50.9 (100%)	84.2 (100%)	83.9 (100%)	82.5 (100%)	82.9 (100%)
Random Drop	44.8 (50%)	49.0 (50%)	74.8 (50%)	71.6 (50%)	80.3 (50%)	78.1 (50%)
ToMe ( Bolya et al., 2022 )	46.2 (50%)	49.6 (50%)	79.3 (50%)	78.1 (50%)	81.9 (50%)	81.2 (50%)
VisionZip ( Yang et al., 2025c )	47.2 (50%)	48.6 (50%)	79.6 (50%)	77.9 (50%)	81.9 (50%)	81.3 (50%)
ResAdapt (Qwen2.5-VL-7B)	45.5 (42%)	51.0 (29%)	80.0 (64%)	85.9 (105%)	81.4 (41%)	69.6 (30%)
ResAdapt-RL (Qwen2.5-VL-7B)	46.7 (42%)	50.9 (29%)	80.8 (64%)	86.6 (105%)	81.1 (41%)	70.1 (30%)

Appendix

Table 7: Exploratory zero-shot transfer to image benchmarks. Parenthetical values denote per-task retention ratio

R

, and ResAdapt-RL additionally fine-tunes the MLLM via RL.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 227 papers

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs
Jul 22, 2026Pengcheng Wang, Zhiquan Wang, Jayoung Lee +5Efficient Multimodal Inference Efficient VLM Inference
Purdue University, West Lafayette, IN, USA · University of Wisconsin–Madison, Madison, WI, USA · NVIDIA, Santa Clara, CA, USA
Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models
Sep 28, 2026Zhiqiang Xia, Yang Li, Xinyuan Zhang +5Efficient Multimodal Inference Efficient VLM Inference
Xiaomi Corporation, Beijing, China
Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Oct 8, 2026Yicheng Xue, Han Wu, Jufeng Yang +4VLM Reasoning Efficient Multimodal Inference
City University of Hong Kong · Zhejiang University · Peking University +2

Jul 22, 2026J/K move · Enter open · S save

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

Pengcheng Wang, Zhiquan Wang, Jayoung Lee, Zhuoyan Xu+4

Purdue University, West Lafayette, IN, USA · University of Wisconsin–Madison, Madison, WI, USA · NVIDIA, Santa Clara, CA, USA

Efficient Multimodal Inference Efficient VLM Inference Multimodal Large Language Models Multimodal Foundation Models Visual Token Pruning Adaptive Computation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks. However, their high inference cost, arising from both the large number of input visual tokens and the heavy computation of the large language model (LLM), remains a key barrier to practical deployment. Recent work attempts to reduce the cost by adaptively optimizing individual dimensions, e.g., pruning redundant visual tokens or skipping LLM layers and heads. Nonetheless, prior approaches typically treat these dimensions independently and overlook a fundamental coupling: the available compute resources must be dynamically allocated across all dimensions based on the input content. To bridge the gap, we propose SmartVL, a unified adaptive inference framework that jointly controls vision token number and model compute capability in response to varying input contents and compute budgets. SmartVL introduces a vision-side token controller that dynamically selects informative visual tokens and an LLM-side compute controller that adaptively adjusts LLM computation. Importantly, these controllers are trained to coordinate with each other so that the overall inference cost satisfies a target budget. To allow this joint scheduling, we connect the controllers using a shared budget encoding and leverage a differentiable latency estimator for end-to-end training. This design enables SmartVL to learn cross-stage allocation strategies that adapt to both input complexity and runtime compute constraints. Experiments across multiple MLLM benchmarks demonstrate that, with joint scheduling, SmartVL consistently outperforms prior adaptive methods and achieves superior accuracy-efficiency Pareto frontiers. Project page: https://www.schaterji.io/publications/2026/jointtokencompute.

Abstract

Efficient Multimodal Inference

Explore similar work

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning