Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Aimon Rahman; Joymallya Chakraborty; Vidit Bansal; Abhishek Dan

cs.CVOct 13, 2026

Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Authors: Aimon Rahman, Joymallya Chakraborty, Vidit Bansal, Abhishek Dan

Organizations: Amazon

Abstract

Multimodal large language models have advanced rapidly, but most remain optimized for single-image inputs. Multi-image reasoning is costly because each image adds many visual tokens, increasing encoding and attention demands. Existing techniques that address the above issue, including visual token pruning, token merging, visual token compression, learned resampling, hierarchical compression, retrieval-augmented visual reasoning, and memory-based architectures, have largely been designed for single-image scenarios, where redundancy is primarily spatial and local. Multi-image reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence grounding. Moreover, existing token-reduction methods are often either (i) training-free but inference-inefficient, typically relying on accessing attention matrices to get token-importance that may be impractical to access with memory-efficient attention implementations such as FlashAttention, or (ii) training-dependent, requiring the base MLLM to be retrained or adapted to a new compressed-token paradigm. In this work, we study scalable multi-image reasoning and understanding without retraining the base model and without accessing attention matrices at inference time. To this end, we propose mmVB, a lightweight plug-and-play module that assigns query-conditioned per-image token budgets via cross-image context, adding only 1.6M trainable parameters while reducing visual input tokens by 9x. This reduction yields 5x faster time-to-first-token and 7x fewer prefill FLOPs, while maintaining 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks.

Figures & tables

Figure 1: Introduction and Motivation. Existing multi-image methods scale poorly: dense attention breaks FlashAttention and excludes API-only models, token merging disrupts positional encodings and cross-image reasoning, and video-style compression assumes frame similarity. Our pre-encoding predictor instead assigns query-conditioned token budgets using only pixels, enabling compatibility with any MLLM, including closed models.

Figure 2: mmVB architecture. Frozen EfficientNet-B0 encodes $N$ images, while frozen LLM embeddings encode the query through mean pooling and a 64-dimensional bottleneck. A two-layer, four-head Transformer processes all $N{+}1$ tokens, after which a shared MLP maps each image token to four ordinal logits.

Method	Muir	MIB	Mantis	NLVR2	VHay	MMIU	MIRB	ReMI
Samples	2,600	5,000	217	6,967	20,100	93,584	1,124	2,600
Accuracy (%) $\uparrow$
Full Res.	63.6	78.1	69.9	86.5	69.5	50.9	67.2	24.7
VisionZip (CVPR’25)	63.6	72.8	69.6	59.3	69.6	48.3	48.7	13.8
ToFu (arXiv)	42.6	71.2	56.7	60.7	14.6	11.2	42.1	18.6
VisionTrim (ICLR’26)	63.6	80.4	67.7	80.6	64.2	52.5	59.6	19.3

Table 1: Comparison of predictive performance, visual-token efficiency, and computational efficiency across eight multi-image benchmarks.

Figure 3: Budget-constrained visual-token allocation. Accuracy retention (mean across all eight benchmarks) as a function of average visual tokens per image. At matched budgets, query-conditioned greedy allocation consistently outperforms uniform allocation.

Benchmark	Acc (%)		Tok/S		Acc (%)		Tok/S		Acc (%)		Tok/S
	Qwen3.5-27B				Claude Opus 4.6				Pixtral Large
	Full	mmVB	Full	mmVB	Full	mmVB	Full	mmVB	Full	mmVB	Full	mmVB
MuirBench	72.7	68.3	3,048	256	69.6	64.5	3,074	519	45.0	42.3	3,891	905
MIBench	80.8	78.7	1,611	211	77.1	70.5	1,617	547	67.5	58.2	2,048	846
Mantis-Eval	82.0	82.0	2,119	270	86.1	78.7	2,581	298	81.0	79.2	2,488	374
NLVR2	93.8	88.8	1,649	70	91.2	86.1	1,368	250	83.7	78.6	2,048	281

Table 2: Black-box transfer: mmVB predictions (trained on Qwen3.5-4B) applied to unseen models without retraining.

Figure 4: Qualitative Examples from the benchmark.

Benchmark	N	Full	Uni	Low	Rand	SI	MI	Full	Uni	Low	Rand	SI	MI	Full	Uni	Low	Rand	SI	MI
		Accuracy (%) $\uparrow$						Vis Tok/Samp $\downarrow$						TTFT (s) $\downarrow$
Muir	2,600	63.6	63.8	52.7	58.3	61.7	62.9	3,236	1,109	5	370	388	389	3.5	0.6	0.6	0.7	0.8	0.5
MIB	5,000	78.1	77.4	64.7	72.6	74.6	77.9	1,237	1,178	5	125	231	248	1.6	1.0	0.5	0.5	0.7	0.7
Mantis	217	69.9	69.1	49.3	59.9	65.4	68.2	2,119	637	3	73	103	118	2.2	0.9	0.5	0.6	0.6	0.6
NLVR2	6,967	86.5	86.9	56.8	70.0	79.2	82.8	1,649	512	2	63	75	89	2.8	0.6	0.5	0.5	0.5	0.6
VHay	20,100	69.5	69.5	51.7	58.5	63.9	69.6	2,187	2,090	8	271	571	594	11.8	0.7	1.2	4.0	6.0	0.6

Table 3: Ablation: accuracy and efficiency across eight multi-image benchmarks. Full = native resolution (reference). Uni = all images at 512

\times

512. Low = all images at 32

\times

32 (lower bound). Rand = random level (distribution-matched). SI = mmVB single-image training. MI = mmVB single+multi-image training. Bold = best (excl. Full/Low).

Figure 5: Limitations. Scale predictor may underestimate tokens needed to answer questions. May also fail when there’s no text query.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Variant	bal_acc	$\pm$ 1 acc
Levels	3 levels	65	79
	5 levels (ours)	71	86
	7 levels	70	86
Text bottleneck	None (direct 2560 $\to$ 256)	55	71
	64-dim (ours)	71	86
	16-dim	50	71

Appendix

Table 4: Architecture ablation on the validation set. bal_acc = balanced accuracy across levels;

\pm

1 = fraction of predictions within one level of ground truth.

Figure 6: Per-task accuracy across eight benchmarks. Each subplot represents one benchmark. Solid lines show Full-res and mmVB; dashed lines show uniform resolutions; dash-dot lines show post-encoding baselines; and dotted lines show the distribution-matched random baseline. mmVB closely tracks Full-res while using 9.2 $\times$ fewer visual tokens (geometric mean across benchmarks).

Method	Score (%)	Vis tok/samp
Full Res.	59.8	1,146
mmVB	53.3	63

Appendix

Table 5: MM-Vet results (Qwen3.5-4B, Claude judge)

	MuirBench	MIBench	Mantis	NLVR2	VHs	MMIU	MIRB	ReMI
Samples	2,600	5,000	217	6,967	20,100	93,584	1,124	2,600
Images/sample	2–7	2–14	2–5	2	1–10	1–62	2–6	2–6
Tasks	12	5	6+	1	1	52	14	13
Format	MC	MC	MC+SA	T/F	Y/N	MC	Mixed	Free-form

Appendix

Table 6: Overview of the evaluation benchmarks. MC = multiple-choice, SA = short-answer, T/F = true/false, and Y/N = yes/no. Mixed includes multiple-choice, yes/no, counting, and free-form answers. VHs denotes Visual Haystacks.

Multi-Image Reasoning

See all 15 papers

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Oct 8, 2026Yicheng Xue, Han Wu, Jufeng Yang +4VLM Reasoning Efficient Multimodal Inference
City University of Hong Kong · Zhejiang University · Peking University +2
VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning
May 29, 2026Hengbo Xu, Shengjie Jin, Yanbiao Ma +1Efficient Multimodal Inference Visual Attention
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.
A More Word-like Image Tokenization for MLLMs
May 18, 2026Hyun Lee, Hyemin Jeong, Yejin Kim +4Image Tokenization Efficient VLM Inference
Seoul National University · Ewha Womans University

Oct 8, 2026J/K move · Enter open · S save

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Yicheng Xue, Han Wu, Jufeng Yang, Minjing Dong+3

City University of Hong Kong · Zhejiang University · Peking University+2

VLM Reasoning Efficient Multimodal Inference Memory-Augmented VLMs Visual Token Compression Visual Token Pruning

Open paper AskPDF arXiv

Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as reasoning unfolds, making it difficult for a fixed compressed context to retain all the details needed across stages. To address this challenge, we propose ViMoD, a lightweight framework that maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. Selected Fine tokens augment the persistent Coarse context in the frozen backbone, enabling stage-specific evidence access without continuously attending to all visual tokens. On Qwen3-VL-4B, ViMoD outperforms all evaluated baselines on all eight reasoning benchmarks at a 20% target visual token budget, improving the mean normalized score by 39.0% over the strongest evaluated one-shot baseline. These gains are achieved with only 0.0546% additional trainable parameters relative to the frozen backbone.

Abstract

Multi-Image Reasoning

Explore similar work

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning

A More Word-like Image Tokenization for MLLMs