FOCUS: Closed-Loop Attention Feedback for Efficient Vision-Language Understanding

Akul Santhosh; Aastha Jhunjhunwala

cs.AINov 1, 2026

FOCUS: Closed-Loop Attention Feedback for Efficient Vision-Language Understanding

Authors: Akul Santhosh, Aastha Jhunjhunwala

Organizations: NVIDIA

Abstract

Processing every video frame through Vision-Language Models (VLMs) adds significant inference cost and limits their ability to comprehend real-time video. Existing approaches either sub-sample frames uniformly, ignoring content and missing important events, or prune visual tokens inside the VLM to cut per-call cost while still invoking the model on every sampled frame. We introduce FOCUS, a training-free three-layer framework that reduces the number of invocations and steers each one towards what matters. Temporal gating measures inter-frame dissimilarity in the patch-embedding space of a smaller VLM from the same family, triggering burst captures only when the scene changes in a semantically meaningful way. Patch-level dissimilarity is then accumulated across the burst and fused with a top-down attention prior into a spatial focus map that encodes where motion and semantic relevance coincide. That map calibrates the VLM's value cache during generation, progressively amplifying vision tokens in task-relevant regions and attenuating off-target ones, without removing any token from the context. The loop closes after each response: text-to-vision cross-attention is extracted from the decoder and fed back as the prior for the next burst, letting the system converge on semantically important regions over successive observations. We evaluate FOCUS on the 500-video EgoSchema subset in streaming mode, with a single Qwen3-VL-8B backbone shared by every method. It reaches 74.8% accuracy using 25.4 VLM calls per clip, against 74.6% with 55 calls for uniform sampling and 74.2% with 55 calls for BOLT. On egocentric video captioning across four domains, it matches the best baseline's quality while issuing 57% fewer VLM invocations than uniform sampling.

Figures & tables

Figure 1: Architecture of the three-layer FOCUS system.

Figure 2: Spatial prior evolution across bursts (robotic teleoperation). Columns are bursts; rows are a representative frame, the dissimilarity map, the decoder attention after generation, and the combined prior (absent at Burst 0, the orienting phase).

Method (streaming, matched cadence)	Accuracy	VLM Calls
Uniform (streaming)	74.6%	55
BOLT ( Liu et al., 2025 ) (streaming)	74.2%	55
DyCoke ( Tao et al., 2025 ) (streaming)	71.4%	12
FOCUS (gating only)	73.6%	25.4
FOCUS (w/ Feedback)	74.8%	25.4

Table 1: EgoSchema subset accuracy (500 videos, 5-way multiple choice) in the streaming setting. All methods share the Qwen3-VL-8B backbone at a matched

3.3

s cadence; accuracy is the mean over five runs, “VLM Calls” the mean invocations per three-minute clip. The two FOCUS rows issue identical calls by construction: gating alone determines how many bursts fire.

Method	Actor	Activity	Object	No Halluc.	Overall	$\bm{\rho}$	VLM Calls
Uniform Sampling	4.96	4.12	4.00	2.16	3.81	4.14	516
DyCoke	4.80	3.88	4.04	2.52	3.81	3.07	263
BOLT	4.76	4.12	4.08	2.68	3.91	0.87	263
FOCUS (gating only)	5.00	4.00	4.12	2.44	3.89	0.57	222
FOCUS (w/ Feedback)	5.00	3.96	4.12	2.60	3.92	0.59	222

Table 2: Captioning quality (GPT-5.2, 1–5) and efficiency on 25 egocentric videos.

\rho

is the VLM compute ratio (Eq. 1 ); lower is better. Same backbone throughout, but not decoding-matched: DyCoke and BOLT use sampled decoding and a 200-frame interval before their own selection, against greedy decoding and 100 for Uniform and FOCUS. A further reason to read the small quality gaps cautiously.

Figure 3: Frames from the soup-stirring clip (EPIC-Kitchens, 15s).

Figure 4: Frames from the sauce-stirring clip (EPIC-Kitchens, 25s).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$t$	index of a frame in the arriving stream
$\mathrm{ref}$	the reference frame for dissimilarity: the previously processed frame, updated every step
$g$	vision encoder used for gating (Qwen2-VL-2B ViT)
$i$	index of a spatial patch within a frame, $i\in\{1,\dots,P\}$
$P,H_{g},W_{g}$	number of patches after merging, and the patch-grid height and width, $P=H_{g}W_{g}$
$D$	patch-embedding width

Appendix

Table 3: Symbols used in Section 3 .

Symbol	Default	Swept	Role and sensitivity
$\tau$	$0.4$	yes	Gating threshold; trades compute against quality, flat over a wide band
$B$	$32$	yes	Burst size; quality within $0.07$ across $8$ – $64$ , $\rho$ minimised at $32$
$N_{\max}$	$300$	no	Forced-capture timeout; liveness guard, inactive on all evaluated clips
$\alpha_{f}$	$1.05$	partly	Amplification strength; ablated on/off in Table 6
$\beta$	$0.04$	partly	Suppression strength; ablated on/off in Table 6
$[r_{\mathrm{lo}},r_{\mathrm{hi}}]$	$[0.75,1.0]$	no	Focus range; what fraction of tokens is amplified vs. suppressed

Appendix

Table 4: Complete hyperparameter inventory. “Swept” indicates whether the paper reports a sweep for that parameter.

Table 9

Domain	$\bm{n}$	$\bm{\tau{=}0.30}$	$\bm{\tau{=}0.40}$	$\bm{\tau{=}0.50}$	Best	@0.40
Cooking & food prep	14	79%	86%	79%	86%	86%
Crafts, art & textiles	11	64%	64%	55%	73%	64%
Construction & workshop	9	89%	89%	89%	100%	89%
Housework & cleaning	6	83%	100%	83%	100%	100%
Leisure, games & social	10	60%	60%	50%	70%	60%
All (pooled)	50	74%	78%	70%	82%	78%

Appendix

Table 7: Accuracy by domain across the gating threshold sweep (50 EgoSchema videos). “Best” is the highest accuracy that domain reaches at any

\tau

in the full sweep, and “@0.40” repeats the accuracy at our default.

Figure 5: Quality and efficiency as a function of (a) gating threshold $\tau$ ( $B{=}32$ ) and (b) burst size $B$ ( $\tau{=}0.4$ ). Circled markers denote the default configuration. The dotted grey line marks $\rho{=}1$ (real-time).

$\bm{\tau}$	Actor	Activity	Object	No Halluc.	Overall	$\bm{\rho}$	Avg Calls	Capture
0.2	4.76	3.68	4.04	2.56	3.76	2.31	20.4	95%
0.3	4.80	3.92	4.40	2.60	3.93	2.11	17.0	65%
0.4 $\dagger$	5.00	3.96	4.12	2.60	3.92	0.59	8.9	27%
0.5	4.76	3.80	4.04	2.52	3.78	0.63	5.2	12%
0.6	4.84	3.88	4.08	2.56	3.84	0.81	5.2	12%

Appendix

Table 8: Gating threshold ablation (

B=32

, spatial + feedback enabled) on the 25-video captioning benchmark. Capture ratio is the fraction of input frames included in bursts.

\dagger

denotes the default configuration used in Table 2 .

$\bm{B}$	Actor	Activity	Object	No Halluc.	Overall	$\bm{\rho}$	Avg Calls	Capture
8	4.96	3.88	4.12	2.52	3.87	2.26	17.6	15%
16	4.96	4.00	4.04	2.64	3.91	1.64	12.3	20%
32 $\dagger$	5.00	3.96	4.12	2.60	3.92	0.59	8.9	27%
64	5.00	3.92	4.12	2.72	3.94	1.18	7.2	36%

Appendix

Table 9: Burst size ablation (

\tau=0.4

, spatial + feedback enabled) on the 25-video captioning benchmark.

\dagger

denotes the default configuration used in Table 2 .

	A	B	C	D	E	Overall
GT count	101	108	91	83	117	500
Predicted	99	108	94	93	106	500
Correct	72	79	70	67	78	366
Accuracy	71.3%	73.1%	76.9%	80.7%	66.7%	73.2%

Appendix

Table 10: EgoSchema subset per-answer breakdown (500 videos, 5-way multiple choice), single run of FOCUS with feedback in streaming mode,

\tau=0.4

B=32

, 6 fps.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 500 papers

Jun 3, 2026

Frames2LoRA: Parametric Video Internalization for Vision-Language Models

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Frames2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Frames2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Frames2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Frames2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.

Manan Suri, Sarvesh Baskar, Dinesh Manocha

University of Maryland, College Park

VLM Adaptation Efficient VLM Inference Low-Rank Adaptation Multimodal Foundation Models

Ask PDF

Mar 3, 2026

Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models

Large Vision Language Models (LVLMs) exhibit strong Chain-of-Thought (CoT) capabilities, yet most existing paradigms assume full-video availability before inference, a batch-style process misaligned with real-world video streams where information arrives sequentially. Motivated by the streaming nature of video data, we investigate two streaming reasoning paradigms for LVLMs. The first, an interleaved paradigm, alternates between receiving frames and producing partial reasoning but remains constrained by strictly ordered cache updates. To better match streaming inputs, we propose Think-as-You-See (TaYS), a unified framework enabling true concurrent reasoning. TaYS integrates parallelized CoT generation, stream-constrained training, and stream-parallel inference. It further employs temporally aligned reasoning units, streaming attention masks and positional encodings, and a dual KV-cache that decouples visual encoding from textual reasoning. We evaluate all paradigms on the Qwen2.5-VL family across representative video CoT tasks, including event dynamics analysis, causal reasoning, and thematic understanding. Experiments show that TaYS consistently outperforms both batch and interleaved baselines, improving reasoning performance while substantially reducing time-to-first-token (TTFT) and overall reasoning delay. These results demonstrate the effectiveness of data-aligned streaming reasoning in enabling efficient and responsive video understanding for LVLMs. We release our code at https://github.com/EIT-NLP/StreamingLLM/tree/main/TaYS

Jialiang Zhang, Junlong Tong, Junyan Lin +4

Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ocean University of China · Shanghai Jiao Tong University +3

VLM Reasoning Streaming Inference Streaming Video Understanding Large Vision-Language Models Efficient VLM Inference

Ask PDF

May 5, 2026

VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Models

Video vision-language models (VLMs) keep paying for visual state the stream already told us was stable. The factory wall did not move, but most VLM pipelines still hand the model dense RGB frames or a fresh prefix again. We study that waste as training-free anti-recomputation: reuse state when validation says it survives, and buy fresh evidence when the scene, query, or cache topology requires it. The largest measured win is after ingest. On frozen Qwen2.5-VL-7B-Instruct-4bit, adaptive same-video follow-up reuse preserves paired choices and correctness on a 93-query VideoMME breadth setting while reducing follow-up latency by 14.90-35.92x. The first query is still cold; the win starts when later questions reuse the same video state. Stress tests bound the result: repeated-question schedules hold through 50 turns, while dense-answer-anchored prompt variation separates conservative fixed K=1 repair from faster aggressive policies that drift. Fresh-video pruning is smaller but real. C-VISION skips timed vision-tower work before the first answer is generated. On Gemma 4-E4B-4bit, the clean 32f short cell reaches 1.316x first-query speedup with no paired drift or parse failures on 20 items; Qwen shows the fidelity/speed boundary. Stage-share ceiling (C-CEILING) is the accounting guardrail: a component speedup becomes an end-to-end speedup only in proportion to the wall-clock share it accelerates, so C-VISION and after-ingest follow-up reuse do not multiply. Candidate C-STREAM remains a native-rate target, not a headline result here. The broader direction is VLM-native media that expose change, motion, uncertainty, object state, sensor time, and active tiles directly, so models do not have to rediscover the world from dense RGB every frame.

JF Bastien, Sam D'Amico

Look, they said attention was all you needed, well we got your attention with this title.

Efficient VLM Inference Video-Language Models

Ask PDF

Abstract

Efficient VLM Inference

Explore similar work

Frames2LoRA: Parametric Video Internalization for Vision-Language Models

Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models

VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Models