FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

Hengjie Zhu; Dayan Wu; Zihao Zhang; Xinze Liu; Jingxuan Yu; Peng Fu; Zheng Lin; Weiping Wang

cs.CVAug 24, 2026

FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

Authors: Hengjie Zhu, Dayan Wu, Zihao Zhang, Xinze Liu, Jingxuan Yu, Peng Fu, Zheng Lin, Weiping Wang,

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · School of Cyber Science and Engineering, Southeast University

Abstract

Multimodal speculative decoding accelerates vision-language models by allowing a lightweight draft model to propose candidate tokens for parallel verification by a larger target model. Existing methods typically condition the drafter on a fixed visual interface, such as a predefined visual-token budget or a static compressed representation. However, our controlled visual-budget analysis shows that visual demand varies substantially across tasks and decoding stages, which means more visual input is not always beneficial. Actually, insufficient evidence may weaken visual grounding, while excessive context adds overhead and may disrupt drafting. We propose FOVEA (Focused On-demand Visual Evidence Adaptation), a cache-friendly approach that builds a reusable visual memory and dynamically retrieves a bounded subset for a draft state. A cumulative-mass rule determines both how many and which entries are selected. The selected entries are aggregated into a visual readout and fused with the current draft hidden state through a lightweight gated residual correction. Rather than inserting visual tokens into the autoregressive context, the correction modifies only the representation passed to the language-model head. Experiments across multiple vision-language backbones and multimodal benchmarks show that FOVEA improves draft acceptance and end-to-end decoding speed, achieving up to $2.13\times$ speedup over autoregressive decoding. These results demonstrate that state-conditioned evidence retrieval is an effective alternative to reusing a fixed visual representation throughout multimodal generation.

Figures & tables

Figure 1: Our visual-budget diagnostics in (b) show that the preferred diagnostic budget varies across decoding states and tends to be higher later in text-heavy generation. These findings expose the limitation of the fixed-budget interface in (a), which provides the same visual budget to every state. To address the content-selection aspect of this mismatch, (c) FOVEA stores visual evidence once and retrieves a dynamic number of state-conditioned entries, allowing different draft states to select text-, object-, or relation-centric information without modifying the cached token context.

Figure 2: State-dependent visual demand in multimodal speculative decoding. (a) Fixed visual budgets cause both under- and over-supply. (b) Preferred budgets vary across decoding states. (c) Later positions show higher visual demand, accept-length variation, and budget sensitivity. (d) Late-stage states exhibit stronger visual-focus shifts.

Figure 3: Overview of FOVEA. The target VLM performs multimodal prefill, while projected visual features are cached once as an image-specific key–value memory. A draft-tree node uses its hidden state and depth embedding to score the memory and attend the top relevant entries. A gated correction MLP fuses the retrieved evidence into the current scoring representation without modifying the historical KV cache. The corrected distribution expands the draft tree, which is then verified in parallel by the target VLM.

Model

Methods

MMT

TextVQA

MME

ChartQA

MathVista

OCRBench

ScienceQA

SEED

MMSpec

Avg

Speedup

\tau

LLaVA-Next-7B

AR Baseline

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

1.00\!\times

–

Training-free Methods

Lookahead

0.86\!\times

0.88\!\times

1.04\!\times

0.91\!\times

0.96\!\times

1.02\!\times

0.81\!\times

0.61\!\times

0.72\!\times

0.87\!\times

1.641

PLD

1.12\!\times

1.29\!\times

1.58\!\times

1.24\!\times

1.38\!\times

\underline{1.36\!\times}

1.03\!\times

0.85\!\times

1.13\!\times

1.22\!\times

1.281

Training-based Methods

Table 1: End-to-end speedup over autoregressive (AR) decoding and average acceptance length (

\tau

). The AR baseline is fixed at

1.00\times

, and

\tau

is not applicable. Best and second-best speculative-decoding results within each backbone are shown in bold and underlined, respectively.

Architecture Components
Variant	MMT-Bench		OCRBench
Variant	Speedup $\uparrow$	$\tau\uparrow$	Speedup $\uparrow$	$\tau\uparrow$
Full FOVEA	2.13	4.52	1.68	3.23
w/o visual correction	1.52	3.12	1.12	2.52
w/o depth embedding	2.04	4.40	1.59	3.01
w/o correction gate	1.95	4.31	1.53	2.97

Table 2: Component and objective ablations on MMT-Bench and OCRBench. Higher values are better. Best results are shown in bold.

Figure 4: Visual-retrieval budget ablation. FOVEA denotes our retrieval strategy, whereas the remaining setting uses fixed Top-

K

budgets. Positive values indicate improvements over dense retrieval.

Figure 5: State-conditioned visual retrieval along one decoding trajectory. The retrieved focus shifts from the jumpsuit to the hard hat and then to the blue hose as generation proceeds.

Interface	Time(ms) $\downarrow$	$\Delta$ Mem.(GiB) $\downarrow$	Speedup $\uparrow$	$\boldsymbol{\tau}\uparrow$
Fixed prefix	15.58	0.373	1.681	3.451
Dynamic insertion	20.86	0.361	1.532	3.719
Correction + rebuild	19.90	0.377	1.764	3.647
FOVEA correction	16.49	0.378	1.855	3.644

Table 3: Cache-efficiency comparison on MMT-Bench and OCRBench. We report draft time, additional peak memory over paired autoregressive decoding, end-to-end speedup, and average accept length (

\tau

). Best values are shown in bold.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Sensitivity to the visual-correction depth.

Dataset	Original tokens $N$	Default retained tokens
MMT-Bench	1,176–2,928	882–2,196
LLaVA/665K-Holdout	1,464–2,352	1,098–1,764
OCRBench	870–2,928	652–2,196

Appendix

Table 4: Visual-token ranges in the diagnostic data. The default condition retains

\max(1,\lfloor 0.75N\rfloor)

tokens.

Parameter	Value	Parameter	Value	Parameter	Value
$\gamma_{w}$	$1$	$\lambda_{\mathrm{hid}}$	$1.0$	$\lambda_{\mathrm{KL}}$	$0.2$
$\lambda_{\mathrm{tok}}$	$1.0$	$\beta_{h}$	$1.0$	$\beta_{\mathrm{mid}}$	$0.1$
$\beta_{\mathrm{CE}}$	$1.5$	$\beta_{\mathrm{cmp}}$	$0.5$	$m$	$0.5$

Appendix

Table 5: Loss hyperparameters used for training FOVEA.

Target backbone	Samples	Epochs	Batch size	Learning rate	Warmup	GPUs	Wall-clock time
LLaVA-v1.6-Vicuna-7B	57,671	12	$2/4/16$	$5\times 10^{-5}$	4,326 / 43,260 (linear)	2	32:58:25
Qwen2.5-VL-7B	57,671	12	$4/1/8$	$5\times 10^{-5}$	2,000 (logarithmic)	2	32:17:03

Appendix

Table 6: Draft-model training configurations. Batch size is reported as per-device batch size / gradient accumulation steps / global batch size. The target VLM is frozen in all runs.

	MMT-Bench		OCRBench		ChartQA		Macro
Variant	$\Delta S$	$\Delta\tau$	$\Delta S$	$\Delta\tau$	$\Delta S$	$\Delta\tau$	$\Delta S$	$\Delta\tau$
Random Top-16	$+0.377{\pm}0.246$	$-0.045{\pm}0.120$	$-0.209{\pm}0.319$	$+0.143{\pm}0.097$	$-0.922{\pm}0.210$	$-0.625{\pm}0.519$	$-0.176{\pm}0.128$	$-0.206{\pm}0.071$
Static Image Top-16	$-0.688{\pm}0.063$	$-0.101{\pm}0.000$	$+0.055{\pm}0.041$	$+1.074{\pm}0.000$	$-0.365{\pm}0.019$	$-1.470{\pm}0.000$	$-0.187{\pm}0.018$	$-0.225{\pm}0.000$
Query-agnostic Top-16	$-0.538{\pm}0.068$	$-0.101{\pm}0.000$	$+0.088{\pm}0.056$	$+1.076{\pm}0.000$	$-0.516{\pm}0.033$	$-1.428{\pm}0.000$	$-0.247{\pm}0.025$	$-0.213{\pm}0.000$
State-only Top-16	$-0.264{\pm}0.186$	$-0.264{\pm}0.000$	$-0.159{\pm}0.030$	$+0.037{\pm}0.000$	$+0.010{\pm}0.039$	$-0.130{\pm}0.000$	$-0.197{\pm}0.081$	$-0.204{\pm}0.000$
Full FOVEA	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$	$0.000{\pm}0.000$

Appendix

Table 7: Matched retrieval-control ablation. Each cell reports the absolute change from Full FOVEA in speedup (

\Delta S

) or average acceptance length (

\Delta\tau

); Full FOVEA is fixed at zero. Values are mean

\pm

standard deviation over three repeated inference runs using the same checkpoint. Higher values are better.

Backbone	$\tilde{h}_{n}$	$h_{n}$	$h_{n}^{T}$	$\tilde{h}_{n}^{T}$
LLaVA-7B	Block 1 output	Block 2 output	Final target layer	Minimum-entropy layer in $[1,10]$
LLaVA-13B	Block 1 output	Block 3 output	Final target layer	Minimum-entropy layer in $[1,10]$
Qwen2.5-VL-7B	Block 1 output	Block 2 output	Final target layer	Minimum-entropy layer in $[1,9]$

Appendix

Table 8: Sources of the draft and target states used for supervision. Draft states are measured before visual correction.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 231 papers

Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models
Apr 22, 2026Juhong Min, Lazar Valkov, Vitali Petsiuk +2Visual Reasoning Visual Attention
AI Center – Mountain View, Samsung Electronics
HAWK: Rethinking Multimodal Drafting for Speculative Decoding
Sep 30, 2026Wenhan Yang, Anirudh Rao, Ashwin ChandraEfficient Multimodal Inference VLM Distillation
University of California, Los Angeles · Samsung Research America
TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding
Jul 13, 2026Quynh Vo, Cong-Duy Nguyen, Ponhvoan Srey +2Efficient VLM Inference Speculative Decoding
National University of Singapore · Nanyang Technological University · Center of AI Research, VinUniversity

Apr 22, 2026J/K move · Enter open · S save

Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models

Juhong Min, Lazar Valkov, Vitali Petsiuk, Hossein Souri+1

AI Center – Mountain View, Samsung Electronics

Visual Reasoning Visual Attention Efficient VLM Inference VLM Reasoning Visually Grounded Reasoning Reasoning for VLMs+1

Open paper AskPDF arXiv

Vision-language models benefit from high-resolution images, but the increase in visual-token count incurs high compute overhead. Humans resolve this tension via foveation: a coarse view guides "where to look", while selectively acquired high-acuity evidence refines "what to think". We introduce Foveated Reasoner, an autoregressive vision-language framework that unifies foveation and reasoning within a single decoding trajectory. Starting from a low-resolution view, the model triggers foveation only when needed, retrieves high-resolution evidence from selected regions, and injects it back into the same decoding trajectory. We train the method with a two-stage pipeline: coldstart supervision to bootstrap foveation behavior, followed by reinforcement learning to jointly improve evidence acquisition and task accuracy while discouraging trivial "see-everything" solutions. Experiments show that the method learns effective foveation policies and achieves stronger accuracy under tight visual-token budgets across multiple vision-language benchmarks.

Abstract

Efficient Multimodal Inference

Explore similar work

Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models

HAWK: Rethinking Multimodal Drafting for Speculative Decoding

TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding