FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

Hengjie Zhu; Dayan Wu; Zihao Zhang; Xinze Liu; Jingxuan Yu; Peng Fu; Zheng Lin; Weiping Wang

cs.CVAug 24, 2026

FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

Authors: Hengjie Zhu, Dayan Wu, Zihao Zhang, Xinze Liu, Jingxuan Yu, Peng Fu, Zheng Lin, Weiping Wang,

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · School of Cyber Science and Engineering, Southeast University

Abstract

Multimodal speculative decoding accelerates vision-language models by allowing a lightweight draft model to propose candidate tokens for parallel verification by a larger target model. Existing methods typically condition the drafter on a fixed visual interface, such as a predefined visual-token budget or a static compressed representation. However, our controlled visual-budget analysis shows that visual demand varies substantially across tasks and decoding stages, which means more visual input is not always beneficial. Actually, insufficient evidence may weaken visual grounding, while excessive context adds overhead and may disrupt drafting. We propose FOVEA (Focused On-demand Visual Evidence Adaptation), a cache-friendly approach that builds a reusable visual memory and dynamically retrieves a bounded subset for a draft state. A cumulative-mass rule determines both how many and which entries are selected. The selected entries are aggregated into a visual readout and fused with the current draft hidden state through a lightweight gated residual correction. Rather than inserting visual tokens into the autoregressive context, the correction modifies only the representation passed to the language-model head. Experiments across multiple vision-language backbones and multimodal benchmarks show that FOVEA improves draft acceptance and end-to-end decoding speed, achieving up to $2.13\times$ speedup over autoregressive decoding. These results demonstrate that state-conditioned evidence retrieval is an effective alternative to reusing a fixed visual representation throughout multimodal generation.

Figures & tables

Figure 1: Our visual-budget diagnostics in (b) show that the preferred diagnostic budget varies across decoding states and tends to be higher later in text-heavy generation. These findings expose the limitation of the fixed-budget interface in (a), which provides the same visual budget to every state. To address the content-selection aspect of this mismatch, (c) FOVEA stores visual evidence once and retrieves a dynamic number of state-conditioned entries, allowing different draft states to select text-, object-, or relation-centric information without modifying the cached token context.

Figure 2: State-dependent visual demand in multimodal speculative decoding. (a) Fixed visual budgets cause both under- and over-supply. (b) Preferred budgets vary across decoding states. (c) Later positions show higher visual demand, accept-length variation, and budget sensitivity. (d) Late-stage states exhibit stronger visual-focus shifts.

Figure 3: Overview of FOVEA. The target VLM performs multimodal prefill, while projected visual features are cached once as an image-specific key–value memory. A draft-tree node uses its hidden state and depth embedding to score the memory and attend the top relevant entries. A gated correction MLP fuses the retrieved evidence into the current scoring representation without modifying the historical KV cache. The corrected distribution expands the draft tree, which is then verified in parallel by the target VLM.

Model

Methods

MMT

TextVQA

MME

ChartQA

MathVista

OCRBench

ScienceQA

SEED

MMSpec

Avg

Speedup

\tau

LLaVA-Next-7B

AR Baseline

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

–

Training-free Methods

Lookahead

0.86\!\times

0.88\!\times

1.04\!\times

0.91\!\times

0.96\!\times

1.02\!\times

0.81\!\times

0.61\!\times

0.72\!\times

0.87\!\times

1.641

PLD

1.12\!\times

1.29\!\times

1.58\!\times

1.24\!\times

1.38\!\times

\underline{1.36\!\times}

1.03\!\times

0.85\!\times

1.13\!\times

1.22\!\times

1.281

Training-based Methods

Table 1: End-to-end speedup over autoregressive (AR) decoding and average acceptance length (

\tau

). The AR baseline is fixed at

1.00\times

, and

\tau

is not applicable. Best and second-best speculative-decoding results within each backbone are shown in bold and underlined, respectively.

Architecture Components
Variant	MMT-Bench		OCRBench
Variant	Speedup $\uparrow$	$\tau\uparrow$	Speedup $\uparrow$	$\tau\uparrow$
Full FOVEA	2.13	4.52	1.68	3.23
w/o visual correction	1.52	3.12	1.12	2.52
w/o depth embedding	2.04	4.40	1.59	3.01
w/o correction gate	1.95	4.31	1.53	2.97

Table 2: Component and objective ablations on MMT-Bench and OCRBench. Higher values are better. Best results are shown in bold.

Figure 4: Visual-retrieval budget ablation. FOVEA denotes our retrieval strategy, whereas the remaining setting uses fixed Top-

K

budgets. Positive values indicate improvements over dense retrieval.

Figure 5: State-conditioned visual retrieval along one decoding trajectory. The retrieved focus shifts from the jumpsuit to the hard hat and then to the blue hose as generation proceeds.

Interface	Time(ms) $\downarrow$	$\Delta$ Mem.(GiB) $\downarrow$	Speedup $\uparrow$	$\boldsymbol{\tau}\uparrow$
Fixed prefix	15.58	0.373	1.681	3.451
Dynamic insertion	20.86	0.361	1.532	3.719
Correction + rebuild	19.90	0.377	1.764	3.647
FOVEA correction	16.49	0.378	1.855	3.644

Table 3: Cache-efficiency comparison on MMT-Bench and OCRBench. We report draft time, additional peak memory over paired autoregressive decoding, end-to-end speedup, and average accept length (

\tau

). Best values are shown in bold.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Sensitivity to the visual-correction depth.

Dataset	Original tokens $N$	Default retained tokens
MMT-Bench	1,176–2,928	882–2,196
LLaVA/665K-Holdout	1,464–2,352	1,098–1,764
OCRBench	870–2,928	652–2,196

Appendix

Table 4: Visual-token ranges in the diagnostic data. The default condition retains

\max(1,\lfloor 0.75N\rfloor)

tokens.

Parameter	Value	Parameter	Value	Parameter	Value
$\gamma_{w}$	$1$	$\lambda_{\mathrm{hid}}$	$1.0$	$\lambda_{\mathrm{KL}}$	$0.2$
$\lambda_{\mathrm{tok}}$	$1.0$	$\beta_{h}$	$1.0$	$\beta_{\mathrm{mid}}$	$0.1$
$\beta_{\mathrm{CE}}$	$1.5$	$\beta_{\mathrm{cmp}}$	$0.5$	$m$	$0.5$

Appendix

Table 5: Loss hyperparameters used for training FOVEA.

Target backbone	Samples	Epochs	Batch size	Learning rate	Warmup	GPUs	Wall-clock time
LLaVA-v1.6-Vicuna-7B	57,671	12	$2/4/16$	$5\times 10^{-5}$	4,326 / 43,260 (linear)	2	32:58:25
Qwen2.5-VL-7B	57,671	12	$4/1/8$	$5\times 10^{-5}$	2,000 (logarithmic)	2	32:17:03

Appendix

Table 6: Draft-model training configurations. Batch size is reported as per-device batch size / gradient accumulation steps / global batch size. The target VLM is frozen in all runs.

	MMT-Bench		OCRBench		ChartQA		Macro
Variant	$\Delta S$	$\Delta\tau$	$\Delta S$	$\Delta\tau$	$\Delta S$	$\Delta\tau$	$\Delta S$	$\Delta\tau$
Random Top-16	$+0.377{\pm}0.246$	$-0.045{\pm}0.120$	$-0.209{\pm}0.319$	$+0.143{\pm}0.097$	$-0.922{\pm}0.210$	$-0.625{\pm}0.519$	$-0.176{\pm}0.128$	$-0.206{\pm}0.071$
Static Image Top-16	$-0.688{\pm}0.063$	$-0.101{\pm}0.000$	$+0.055{\pm}0.041$	$+1.074{\pm}0.000$	$-0.365{\pm}0.019$	$-1.470{\pm}0.000$	$-0.187{\pm}0.018$	$-0.225{\pm}0.000$
Query-agnostic Top-16	$-0.538{\pm}0.068$	$-0.101{\pm}0.000$	$+0.088{\pm}0.056$	$+1.076{\pm}0.000$	$-0.516{\pm}0.033$	$-1.428{\pm}0.000$	$-0.247{\pm}0.025$	$-0.213{\pm}0.000$
State-only Top-16	$-0.264{\pm}0.186$	$-0.264{\pm}0.000$	$-0.159{\pm}0.030$	$+0.037{\pm}0.000$	$+0.010{\pm}0.039$	$-0.130{\pm}0.000$	$-0.197{\pm}0.081$	$-0.204{\pm}0.000$
Full FOVEA	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$

Appendix

Table 7: Matched retrieval-control ablation. Each cell reports the absolute change from Full FOVEA in speedup (

\Delta S

) or average acceptance length (

\Delta\tau

); Full FOVEA is fixed at zero. Values are mean

\pm

standard deviation over three repeated inference runs using the same checkpoint. Higher values are better.

Backbone	$\tilde{h}_{n}$	$h_{n}$	$h_{n}^{T}$	$\tilde{h}_{n}^{T}$
LLaVA-7B	Block 1 output	Block 2 output	Final target layer	Minimum-entropy layer in $[1,10]$
LLaVA-13B	Block 1 output	Block 3 output	Final target layer	Minimum-entropy layer in $[1,10]$
Qwen2.5-VL-7B	Block 1 output	Block 2 output	Final target layer	Minimum-entropy layer in $[1,9]$

Appendix

Table 8: Sources of the draft and target states used for supervision. Draft states are measured before visual correction.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 231 papers

Apr 22, 2026

Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models

Vision-language models benefit from high-resolution images, but the increase in visual-token count incurs high compute overhead. Humans resolve this tension via foveation: a coarse view guides "where to look", while selectively acquired high-acuity evidence refines "what to think". We introduce Foveated Reasoner, an autoregressive vision-language framework that unifies foveation and reasoning within a single decoding trajectory. Starting from a low-resolution view, the model triggers foveation only when needed, retrieves high-resolution evidence from selected regions, and injects it back into the same decoding trajectory. We train the method with a two-stage pipeline: coldstart supervision to bootstrap foveation behavior, followed by reinforcement learning to jointly improve evidence acquisition and task accuracy while discouraging trivial "see-everything" solutions. Experiments show that the method learns effective foveation policies and achieves stronger accuracy under tight visual-token budgets across multiple vision-language benchmarks.

Juhong Min, Lazar Valkov, Vitali Petsiuk +2

AI Center – Mountain View, Samsung Electronics

Visual Reasoning Visual Attention Efficient VLM Inference VLM Reasoning Visually Grounded Reasoning Reasoning for VLMs+1

Ask PDF

Sep 30, 2026

HAWK: Rethinking Multimodal Drafting for Speculative Decoding

Speculative decoding has achieved substantial lossless speedups for LLMs, but remains less effective for large vision-language models (LVLMs), where lightweight drafters struggle to use rich multimodal information. A second limitation is that standard distillation supervises the drafter only along the original training trajectory, without modeling how target predictions shift after the drafter's own proposals. As drafting moves away from this trajectory, the drafter can increasingly disagree with the target, reducing acceptance in later steps. We propose HAWK to address both limitations. HAWK uses representation similarity to select informative target layers and learns how to combine their hidden states. For visual information, it directly provides the drafter with compressed visual hidden states from the target model instead of raw visual tokens, making the visual information easier for a shallow drafter to use. HAWK also trains the drafter to capture how target predictions change after its own proposals, improving its agreement with the target during multi-step drafting. On SmolVLM-256M across ten multimodal benchmarks, HAWK raises average acceptance length from 3.32 to 4.08 and speedup from 2.19x to 2.60x over EAGLE-3 under greedy decoding, and from 2.89 to 3.41 and 1.92x to 2.19x under sampling.

Wenhan Yang, Anirudh Rao, Ashwin Chandra

University of California, Los Angeles · Samsung Research America

Efficient Multimodal Inference VLM Distillation Multimodal Large Language Models

Ask PDF

Jul 13, 2026

TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding

Speculative decoding accelerates autoregressive generation by letting a lightweight drafter propose multiple tokens that are verified by a larger target model. Although effective for text-only LLMs, speculative decoding yields limited gains in VLMs because drafters often diverge on vision-critical content, while existing multimodal acceleration methods do not directly address irrelevant visual evidence or optimize the verifier-accepted prefix length that governs speedup. We propose TIGER, a Text-conditioned vIsual GatEd Routing framework for multimodal speculative decoding. TIGER dynamically selects a sparse set of context-relevant visual tokens based on the drafter's current textual state, rather than expose the full visual token set or a fixed compressed interface. To better align training with inference-time efficiency, we optimize the drafter with acceptance-aligned group-based policy training using verifier-derived rewards based on accepted prefix length, built on top of distillation warm start with KL anchoring. This encourages the drafter not only to imitate the target model, but also to produce speculative continuations that survive verification for longer prefixes. Experiments show that TIGER yields consistent gains in accepted prefix length and speculative speedup under exact verifier-side speculative decoding, while achieving favorable quality-latency trade-offs with comparable downstream accuracy in visual-routing analyses.

Quynh Vo, Cong-Duy Nguyen, Ponhvoan Srey +2

National University of Singapore · Nanyang Technological University · Center of AI Research, VinUniversity

Efficient VLM Inference Speculative Decoding

Ask PDF

Abstract

Efficient Multimodal Inference

Explore similar work

Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models

HAWK: Rethinking Multimodal Drafting for Speculative Decoding

TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding