CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Models

Huijie Guo; Jingyao Wang; Lingyu Si; Jiahuan Zhou; Changwen Zheng; Wenwen Qiang

cs.LGMar 16, 2026

CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Models

Authors: Huijie Guo, Jingyao Wang, Lingyu Si, Jiahuan Zhou, Changwen Zheng, Wenwen Qiang

Organizations: Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · Wangxuan Institute of Computer Technology, Peking University

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have shown impressive reasoning capabilities across vision-language tasks, yet still face the challenge of compute-difficulty mismatch. Through empirical analyses, we identify that existing decoding methods may waste compute on easy cases while underserving hard ones, affecting both model effectiveness and efficiency. To address this issue, we first develop a theoretical framework that links sampling coverage, instance difficulty, and residual risk. Our analysis reveals that multimodal reasoning exhibits a heavy-tailed difficulty distribution; a small subset of hard or ambiguous samples dominates the residual failure probability. Based on this insight, we propose Coverage-Aware Multimodal Decoding (CAMD), an adaptive inference mechanism that dynamically allocates computation according to estimated uncertainty. CAMD integrates evidence-weighted scoring, posterior coverage estimation, and sequential Bayesian updating to balance efficiency and reliability under a limited token budget. Experiments on various benchmark datasets and baselines demonstrate the effectiveness and advantages of our approach.

Figures & tables

Figure 1: Reasoning complexity across multimodal tasks.

Panel of Figure 2 — Figure 2: Motivating experiments on MathVista. (a) Accuracy (%) vs. average generated tokens; (b) Cross-modal consistency (%) vs. tokens; (c) P95 latency (ms) vs. tokens; (d) Accuracy (%) vs. average samples.

Figure 3: The framework of the proposed CAMD (Coverage-Aware Multimodal Decoding) for MLLMs.

	Comprehensive Benchmark			General VQA		Hallucination Benchmark
Method	MMBench $\uparrow$	LLaVA W	MM-Vet $\uparrow$	VizWiz $\uparrow$	SQA $\uparrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$	POPE-R $\uparrow$	POPE-P $\uparrow$	POPE-A $\uparrow$
LLaVA-1.5	64.3	72.5	30.5	48.5	64.5	48.0	13.9	87.0	82.8	76.6
+ICD	63.1	69.7	30.4	46.9	62.8	47.7	13.6	87.9	84.0	80.2
+VCD	63.9	70.9	29.5	43.4	63.3	46.8	13.2	87.0	83.5	78.1
+CGD	64.5	71.3	30.8	49.0	64.0	46.2	13.5	87.5	83.8	79.0
+OPERA	64.4	72.0	31.4	50.0	64.9	45.2	12.7	88.8	82.8	79.2

Table 1: Comparison across image benchmarks.

\uparrow

indicates higher-is-better, while

\downarrow

indicates lower-is-better. The values in parentheses report the improvement of CAMD over the base model. The best results are highlighted in bold . More results are shown in Appendix E .

	MSVD-QA		ActivityNet-QA		Video-Based Text Generation
Method	Accuracy $\uparrow$	Score $\uparrow$	Accuracy $\uparrow$	Score $\uparrow$	Cr. $\uparrow$	Cs. $\uparrow$	De. $\uparrow$	Ct. $\uparrow$	Te. $\uparrow$
Chat-UniVi	64.6	3.6	43.1	3.2	2.84	2.93	2.55	3.16	2.43
+ FarSight	66.4	3.5	43.7	3.2	2.86	2.94	2.56	3.19	2.48
+ CAMD (Ours)	67.0 (+2.4)	3.8 (+0.2)	44.2 (+1.1)	3.3 (+0.1)	3.05 (+0.21)	3.10 (+0.17)	2.72 (+0.17)	3.35 (+0.19)	2.60 (+0.17)
Video-LLaVA	64.8	3.7	41.5	3.3	2.32	2.34	2.65	2.75	2.09
+ FarSight	66.2	3.6	42.0	3.5	2.43	2.38	2.93	2.84	2.14

Table 2: Comparison across all video benchmarks. For the Video-Based Text Generation benchmark, we evaluate five dimensions following [ 37 ] : Cr. (Correctness of Information), Cs. (Consistency), De. (Detail Orientation), Ct. (Contextual Understanding), and Te. (Temporal Understanding). Following [ 33 ] , we use GPT-3.5 Turbo to assign relative scores to model outputs on a 0-5 scale.

Figure 4: Sequence Length vs. Performance with different decoding methods on POPE-R (left) and MSRVTT-QA (right).

Figure 5: Performance of different methods on GPT-4 assisted evaluation. Note that some values were not reported in the original papers; we have replaced them with “-” in the figure.

Figure 6: Ablation study of the hyperparameters.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 227 papers

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
Jun 13, 2026David Huang, Lianlei ShanLatent World Models Multimodal Reasoning
University of Toronto · Tsinghua University
MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
Jan 29, 2026Sangyun Chung, Se Yeon Kim, Youngchae Chee +1Multimodal Robustness Hallucination in Language Models
Integrated Vision Language Lab, KAIST, South Korea
Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning
Jul 1, 2026Shijie Li, Yilin Gao, Siyuan Yang +7Multimodal Large Language Models Latent Visual Reasoning
Shanghai Jiao Tong University · Ant Group

Jun 13, 2026J/K move · Enter open · S save

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

David Huang, Lianlei Shan

University of Toronto · Tsinghua University

Latent World Models Multimodal Reasoning Latent Visual Reasoning Efficient Multimodal Inference Multimodal Understanding RL for VLMs+1

Open paper AskPDF arXiv

Reasoning capabilities of multimodal large language models (MLLMs) have improved considerably in recent years. Existing approaches typically rely on explicit chain-of-thought or continuous latent-space trajectories to enhance multi-step reasoning. However, these methods generally assume that an input admits a single latent interpretation and unfold reasoning along a fixed path or under a uniform computation budget. In real-world multimodal settings, visual observations are often subject to occlusion, blur, viewpoint variation, or semantic ambiguity, giving rise to multiple plausible interpretations. A uniform reasoning strategy not only limits the model's ability to explore multiple hypotheses but also incurs high memory usage and rollout cost. We present DLWM (Diverse Latent World Models), a multimodal reasoning framework that combines latent-space reasoning with reinforcement learning. First, we construct a set of diverse latent world hypotheses in continuous latent space, each capturing a different plausible interpretation of the visual input, and unfold latent reasoning independently on each hypothesis. An orthogonality-based diversity regularizer explicitly prevents hypothesis collapse. Second, we formulate the latent reasoning process as a resource-constrained sequential decision problem and introduce a resource-aware reinforcement learning policy that adaptively allocates computation across hypotheses, dynamically deciding whether to expand, terminate, or merge reasoning paths, thereby substantially reducing memory footprint and improving rollout efficiency. Experiments on multiple multimodal reasoning benchmarks demonstrate that DLWM outperforms existing methods by 2-5 points in accuracy while reducing memory usage by 24%.

Abstract

Efficient Multimodal Inference

Explore similar work

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning