Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Aimon Rahman; Joymallya Chakraborty; Vidit Bansal; Abhishek Dan

cs.CVOct 13, 2026

Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Authors: Aimon Rahman, Joymallya Chakraborty, Vidit Bansal, Abhishek Dan

Organizations: Amazon

Abstract

Multimodal large language models have advanced rapidly, but most remain optimized for single-image inputs. Multi-image reasoning is costly because each image adds many visual tokens, increasing encoding and attention demands. Existing techniques that address the above issue, including visual token pruning, token merging, visual token compression, learned resampling, hierarchical compression, retrieval-augmented visual reasoning, and memory-based architectures, have largely been designed for single-image scenarios, where redundancy is primarily spatial and local. Multi-image reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence grounding. Moreover, existing token-reduction methods are often either (i) training-free but inference-inefficient, typically relying on accessing attention matrices to get token-importance that may be impractical to access with memory-efficient attention implementations such as FlashAttention, or (ii) training-dependent, requiring the base MLLM to be retrained or adapted to a new compressed-token paradigm. In this work, we study scalable multi-image reasoning and understanding without retraining the base model and without accessing attention matrices at inference time. To this end, we propose mmVB, a lightweight plug-and-play module that assigns query-conditioned per-image token budgets via cross-image context, adding only 1.6M trainable parameters while reducing visual input tokens by 9x. This reduction yields 5x faster time-to-first-token and 7x fewer prefill FLOPs, while maintaining 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks.

Figures & tables

Figure 1: Introduction and Motivation. Existing multi-image methods scale poorly: dense attention breaks FlashAttention and excludes API-only models, token merging disrupts positional encodings and cross-image reasoning, and video-style compression assumes frame similarity. Our pre-encoding predictor instead assigns query-conditioned token budgets using only pixels, enabling compatibility with any MLLM, including closed models.

Figure 2: mmVB architecture. Frozen EfficientNet-B0 encodes $N$ images, while frozen LLM embeddings encode the query through mean pooling and a 64-dimensional bottleneck. A two-layer, four-head Transformer processes all $N{+}1$ tokens, after which a shared MLP maps each image token to four ordinal logits.

Method	Muir	MIB	Mantis	NLVR2	VHay	MMIU	MIRB	ReMI
Samples	2,600	5,000	217	6,967	20,100	93,584	1,124	2,600
Accuracy (%) $\uparrow$
Full Res.	63.6	78.1	69.9	86.5	69.5	50.9	67.2	24.7
VisionZip (CVPR’25)	63.6	72.8	69.6	59.3	69.6	48.3	48.7	13.8
ToFu (arXiv)	42.6	71.2	56.7	60.7	14.6	11.2	42.1	18.6
VisionTrim (ICLR’26)	63.6	80.4	67.7	80.6	64.2	52.5	59.6	19.3

Table 1: Comparison of predictive performance, visual-token efficiency, and computational efficiency across eight multi-image benchmarks.

Figure 3: Budget-constrained visual-token allocation. Accuracy retention (mean across all eight benchmarks) as a function of average visual tokens per image. At matched budgets, query-conditioned greedy allocation consistently outperforms uniform allocation.

Benchmark	Acc (%)		Tok/S		Acc (%)		Tok/S		Acc (%)		Tok/S
	Qwen3.5-27B				Claude Opus 4.6				Pixtral Large
	Full	mmVB	Full	mmVB	Full	mmVB	Full	mmVB	Full	mmVB	Full	mmVB
MuirBench	72.7	68.3	3,048	256	69.6	64.5	3,074	519	45.0	42.3	3,891	905
MIBench	80.8	78.7	1,611	211	77.1	70.5	1,617	547	67.5	58.2	2,048	846
Mantis-Eval	82.0	82.0	2,119	270	86.1	78.7	2,581	298	81.0	79.2	2,488	374
NLVR2	93.8	88.8	1,649	70	91.2	86.1	1,368	250	83.7	78.6	2,048	281

Table 2: Black-box transfer: mmVB predictions (trained on Qwen3.5-4B) applied to unseen models without retraining.

Figure 4: Qualitative Examples from the benchmark.

Benchmark	N	Full	Uni	Low	Rand	SI	MI	Full	Uni	Low	Rand	SI	MI	Full	Uni	Low	Rand	SI	MI
		Accuracy (%) $\uparrow$						Vis Tok/Samp $\downarrow$						TTFT (s) $\downarrow$
Muir	2,600	63.6	63.8	52.7	58.3	61.7	62.9	3,236	1,109	5	370	388	389	3.5	0.6	0.6	0.7	0.8	0.5
MIB	5,000	78.1	77.4	64.7	72.6	74.6	77.9	1,237	1,178	5	125	231	248	1.6	1.0	0.5	0.5	0.7	0.7
Mantis	217	69.9	69.1	49.3	59.9	65.4	68.2	2,119	637	3	73	103	118	2.2	0.9	0.5	0.6	0.6	0.6
NLVR2	6,967	86.5	86.9	56.8	70.0	79.2	82.8	1,649	512	2	63	75	89	2.8	0.6	0.5	0.5	0.5	0.6
VHay	20,100	69.5	69.5	51.7	58.5	63.9	69.6	2,187	2,090	8	271	571	594	11.8	0.7	1.2	4.0	6.0	0.6

Table 3: Ablation: accuracy and efficiency across eight multi-image benchmarks. Full = native resolution (reference). Uni = all images at 512

\times

512. Low = all images at 32

\times

32 (lower bound). Rand = random level (distribution-matched). SI = mmVB single-image training. MI = mmVB single+multi-image training. Bold = best (excl. Full/Low).

Figure 5: Limitations. Scale predictor may underestimate tokens needed to answer questions. May also fail when there’s no text query.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Variant	bal_acc	$\pm$ 1 acc
Levels	3 levels	65	79
	5 levels (ours)	71	86
	7 levels	70	86
Text bottleneck	None (direct 2560 $\to$ 256)	55	71
	64-dim (ours)	71	86
	16-dim	50	71

Appendix

Table 4: Architecture ablation on the validation set. bal_acc = balanced accuracy across levels;

\pm

1 = fraction of predictions within one level of ground truth.

Figure 6: Per-task accuracy across eight benchmarks. Each subplot represents one benchmark. Solid lines show Full-res and mmVB; dashed lines show uniform resolutions; dash-dot lines show post-encoding baselines; and dotted lines show the distribution-matched random baseline. mmVB closely tracks Full-res while using 9.2 $\times$ fewer visual tokens (geometric mean across benchmarks).

Method	Score (%)	Vis tok/samp
Full Res.	59.8	1,146
mmVB	53.3	63

Appendix

Table 5: MM-Vet results (Qwen3.5-4B, Claude judge)

	MuirBench	MIBench	Mantis	NLVR2	VHs	MMIU	MIRB	ReMI
Samples	2,600	5,000	217	6,967	20,100	93,584	1,124	2,600
Images/sample	2–7	2–14	2–5	2	1–10	1–62	2–6	2–6
Tasks	12	5	6+	1	1	52	14	13
Format	MC	MC	MC+SA	T/F	Y/N	MC	Mixed	Free-form

Appendix

Table 6: Overview of the evaluation benchmarks. MC = multiple-choice, SA = short-answer, T/F = true/false, and Y/N = yes/no. Mixed includes multiple-choice, yes/no, counting, and free-form answers. VHs denotes Visual Haystacks.

Multi-Image Reasoning

See all 15 papers

Oct 8, 2026

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as reasoning unfolds, making it difficult for a fixed compressed context to retain all the details needed across stages. To address this challenge, we propose ViMoD, a lightweight framework that maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. Selected Fine tokens augment the persistent Coarse context in the frozen backbone, enabling stage-specific evidence access without continuously attending to all visual tokens. On Qwen3-VL-4B, ViMoD outperforms all evaluated baselines on all eight reasoning benchmarks at a 20% target visual token budget, improving the mean normalized score by 39.0% over the strongest evaluated one-shot baseline. These gains are achieved with only 0.0546% additional trainable parameters relative to the frozen backbone.

Yicheng Xue, Han Wu, Jufeng Yang +4

City University of Hong Kong · Zhejiang University · Peking University +2

VLM Reasoning Efficient Multimodal Inference Memory-Augmented VLMs Visual Token Compression Visual Token Pruning

Ask PDF

May 29, 2026

VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning

With the rapid advancement of large multimodal models (LMMs), inference-time overhead has become a key bottleneck for real-world deployment. Existing methods typically prune visual tokens at prefill, assuming the required visual evidence remains static during reasoning. However, we empirically show that visual evidence is strongly step-dependent: only a sparse subset of visual tokens is critical at each decoding step, and the critical set evolves across reasoning. Furthermore, we identify a coupled bottleneck where redundant visual context can steer the model toward query-irrelevant regions, lengthening the reasoning trace. Guided by these insights, we propose VisionPulse, a step-wise visual token pruning framework during reasoning. VisionPulse computes a lightweight visual attention mass to estimate the step-wise retention budget by exploiting its strong positive correlation with LMMs' effective visual token usage and retain only the most critical tokens under this budget. By enforcing visual sparsity during reasoning, VisionPulse filters redundant visual context while preserving relevant visual evidence, shortening reasoning traces naturally. Extensive experiments show that VisionPulse only retains 5% of visual tokens per step with reasoning traces shortened by 11.2%, while keeping accuracy almost unchanged.

Hengbo Xu, Shengjie Jin, Yanbiao Ma +1

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.

Efficient Multimodal Inference Visual Attention Multimodal Reasoning VLM Reasoning Multimodal Foundation Models Visual Token Pruning+1

Ask PDF

May 18, 2026

A More Word-like Image Tokenization for MLLMs

Modern multimodal large language models (MLLMs) typically keep the language model fixed and train a visual projector that maps the pixels into a sequence of tokens in its embedding space, so that images can be presented in essentially the same form as text. However, the language model has been optimized to operate on discrete, semantically meaningful tokens, while prevailing visual projectors transform an image into a long stream of continuous and highly correlated embeddings. This causes the visual tokens to behave differently from the word-like units that LLMs are originally trained to understand. We propose a novel Disentangled Visual Tokenization (DiVT) that clusters patch embeddings into coherent semantic units, so each token corresponds to a distinct visual concept instead of a rigid grid cell. DiVT further adapts its token budget to image complexity, providing an explicit accuracy-compute trade-off modifying neither the vision encoder nor the language model. Across diverse multimodal benchmarks, DiVT matches or surpasses baselines with significantly fewer visual tokens, demonstrating robustness under limited token budgets, significantly reducing memory cost and latency while making visual inputs more compatible with LLMs. Our code is available at https://github.com/snuviplab/DiVT.

Hyun Lee, Hyemin Jeong, Yejin Kim +4

Seoul National University · Ewha Womans University

Image Tokenization Efficient VLM Inference Visual Tokenization Multimodal Large Language Models Multimodal Foundation Models Visual Token Compression+1

Ask PDF

Abstract

Multi-Image Reasoning

Explore similar work

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning

A More Word-like Image Tokenization for MLLMs