FOCUS: Closed-Loop Attention Feedback for Efficient Vision-Language Understanding

Akul Santhosh; Aastha Jhunjhunwala

cs.AINov 1, 2026

FOCUS: Closed-Loop Attention Feedback for Efficient Vision-Language Understanding

Authors: Akul Santhosh, Aastha Jhunjhunwala

Organizations: NVIDIA

Abstract

Processing every video frame through Vision-Language Models (VLMs) adds significant inference cost and limits their ability to comprehend real-time video. Existing approaches either sub-sample frames uniformly, ignoring content and missing important events, or prune visual tokens inside the VLM to cut per-call cost while still invoking the model on every sampled frame. We introduce FOCUS, a training-free three-layer framework that reduces the number of invocations and steers each one towards what matters. Temporal gating measures inter-frame dissimilarity in the patch-embedding space of a smaller VLM from the same family, triggering burst captures only when the scene changes in a semantically meaningful way. Patch-level dissimilarity is then accumulated across the burst and fused with a top-down attention prior into a spatial focus map that encodes where motion and semantic relevance coincide. That map calibrates the VLM's value cache during generation, progressively amplifying vision tokens in task-relevant regions and attenuating off-target ones, without removing any token from the context. The loop closes after each response: text-to-vision cross-attention is extracted from the decoder and fed back as the prior for the next burst, letting the system converge on semantically important regions over successive observations. We evaluate FOCUS on the 500-video EgoSchema subset in streaming mode, with a single Qwen3-VL-8B backbone shared by every method. It reaches 74.8% accuracy using 25.4 VLM calls per clip, against 74.6% with 55 calls for uniform sampling and 74.2% with 55 calls for BOLT. On egocentric video captioning across four domains, it matches the best baseline's quality while issuing 57% fewer VLM invocations than uniform sampling.

Figures & tables

Figure 1: Architecture of the three-layer FOCUS system.

Figure 2: Spatial prior evolution across bursts (robotic teleoperation). Columns are bursts; rows are a representative frame, the dissimilarity map, the decoder attention after generation, and the combined prior (absent at Burst 0, the orienting phase).

Method (streaming, matched cadence)	Accuracy	VLM Calls
Uniform (streaming)	74.6%	55
BOLT ( Liu et al., 2025 ) (streaming)	74.2%	55
DyCoke ( Tao et al., 2025 ) (streaming)	71.4%	12
FOCUS (gating only)	73.6%	25.4
FOCUS (w/ Feedback)	74.8%	25.4

Table 1: EgoSchema subset accuracy (500 videos, 5-way multiple choice) in the streaming setting. All methods share the Qwen3-VL-8B backbone at a matched

3.3

s cadence; accuracy is the mean over five runs, “VLM Calls” the mean invocations per three-minute clip. The two FOCUS rows issue identical calls by construction: gating alone determines how many bursts fire.

Method	Actor	Activity	Object	No Halluc.	Overall	$\bm{\rho}$	VLM Calls
Uniform Sampling	4.96	4.12	4.00	2.16	3.81	4.14	516
DyCoke	4.80	3.88	4.04	2.52	3.81	3.07	263
BOLT	4.76	4.12	4.08	2.68	3.91	0.87	263
FOCUS (gating only)	5.00	4.00	4.12	2.44	3.89	0.57	222
FOCUS (w/ Feedback)	5.00	3.96	4.12	2.60	3.92	0.59	222

Table 2: Captioning quality (GPT-5.2, 1–5) and efficiency on 25 egocentric videos.

\rho

is the VLM compute ratio (Eq. 1 ); lower is better. Same backbone throughout, but not decoding-matched: DyCoke and BOLT use sampled decoding and a 200-frame interval before their own selection, against greedy decoding and 100 for Uniform and FOCUS. A further reason to read the small quality gaps cautiously.

Figure 3: Frames from the soup-stirring clip (EPIC-Kitchens, 15s).

Figure 4: Frames from the sauce-stirring clip (EPIC-Kitchens, 25s).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$t$	index of a frame in the arriving stream
$\mathrm{ref}$	the reference frame for dissimilarity: the previously processed frame, updated every step
$g$	vision encoder used for gating (Qwen2-VL-2B ViT)
$i$	index of a spatial patch within a frame, $i\in\{1,\dots,P\}$
$P,H_{g},W_{g}$	number of patches after merging, and the patch-grid height and width, $P=H_{g}W_{g}$
$D$	patch-embedding width

Appendix

Table 3: Symbols used in Section 3 .

Symbol	Default	Swept	Role and sensitivity
$\tau$	$0.4$	yes	Gating threshold; trades compute against quality, flat over a wide band
$B$	$32$	yes	Burst size; quality within $0.07$ across $8$ – $64$ , $\rho$ minimised at $32$
$N_{\max}$	$300$	no	Forced-capture timeout; liveness guard, inactive on all evaluated clips
$\alpha_{f}$	$1.05$	partly	Amplification strength; ablated on/off in Table 6
$\beta$	$0.04$	partly	Suppression strength; ablated on/off in Table 6
$[r_{\mathrm{lo}},r_{\mathrm{hi}}]$	$[0.75,1.0]$	no	Focus range; what fraction of tokens is amplified vs. suppressed

Appendix

Table 4: Complete hyperparameter inventory. “Swept” indicates whether the paper reports a sweep for that parameter.

Table 9

Domain	$\bm{n}$	$\bm{\tau{=}0.30}$	$\bm{\tau{=}0.40}$	$\bm{\tau{=}0.50}$	Best	@0.40
Cooking & food prep	14	79%	86%	79%	86%	86%
Crafts, art & textiles	11	64%	64%	55%	73%	64%
Construction & workshop	9	89%	89%	89%	100%	89%
Housework & cleaning	6	83%	100%	83%	100%	100%
Leisure, games & social	10	60%	60%	50%	70%	60%
All (pooled)	50	74%	78%	70%	82%	78%

Appendix

Table 7: Accuracy by domain across the gating threshold sweep (50 EgoSchema videos). “Best” is the highest accuracy that domain reaches at any

\tau

in the full sweep, and “@0.40” repeats the accuracy at our default.

Figure 5: Quality and efficiency as a function of (a) gating threshold $\tau$ ( $B{=}32$ ) and (b) burst size $B$ ( $\tau{=}0.4$ ). Circled markers denote the default configuration. The dotted grey line marks $\rho{=}1$ (real-time).

$\bm{\tau}$	Actor	Activity	Object	No Halluc.	Overall	$\bm{\rho}$	Avg Calls	Capture
0.2	4.76	3.68	4.04	2.56	3.76	2.31	20.4	95%
0.3	4.80	3.92	4.40	2.60	3.93	2.11	17.0	65%
0.4 $\dagger$	5.00	3.96	4.12	2.60	3.92	0.59	8.9	27%
0.5	4.76	3.80	4.04	2.52	3.78	0.63	5.2	12%
0.6	4.84	3.88	4.08	2.56	3.84	0.81	5.2	12%

Appendix

Table 8: Gating threshold ablation (

B=32

, spatial + feedback enabled) on the 25-video captioning benchmark. Capture ratio is the fraction of input frames included in bursts.

\dagger

denotes the default configuration used in Table 2 .

$\bm{B}$	Actor	Activity	Object	No Halluc.	Overall	$\bm{\rho}$	Avg Calls	Capture
8	4.96	3.88	4.12	2.52	3.87	2.26	17.6	15%
16	4.96	4.00	4.04	2.64	3.91	1.64	12.3	20%
32 $\dagger$	5.00	3.96	4.12	2.60	3.92	0.59	8.9	27%
64	5.00	3.92	4.12	2.72	3.94	1.18	7.2	36%

Appendix

Table 9: Burst size ablation (

\tau=0.4

, spatial + feedback enabled) on the 25-video captioning benchmark.

\dagger

denotes the default configuration used in Table 2 .

	A	B	C	D	E	Overall
GT count	101	108	91	83	117	500
Predicted	99	108	94	93	106	500
Correct	72	79	70	67	78	366
Accuracy	71.3%	73.1%	76.9%	80.7%	66.7%	73.2%

Appendix

Table 10: EgoSchema subset per-answer breakdown (500 videos, 5-way multiple choice), single run of FOCUS with feedback in streaming mode,

\tau=0.4

B=32

, 6 fps.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 500 papers

Frames2LoRA: Parametric Video Internalization for Vision-Language Models
Jun 3, 2026Manan Suri, Sarvesh Baskar, Dinesh ManochaVLM Adaptation Efficient VLM Inference
University of Maryland, College Park
Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models
Mar 3, 2026Jialiang Zhang, Junlong Tong, Junyan Lin +4VLM Reasoning Streaming Inference
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ocean University of China · Shanghai Jiao Tong University +3
VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Models
May 5, 2026JF Bastien, Sam D'AmicoEfficient VLM Inference Video-Language Models
Look, they said attention was all you needed, well we got your attention with this title.

Jun 3, 2026J/K move · Enter open · S save

Frames2LoRA: Parametric Video Internalization for Vision-Language Models

Manan Suri, Sarvesh Baskar, Dinesh Manocha

University of Maryland, College Park

VLM Adaptation Efficient VLM Inference Low-Rank Adaptation Multimodal Foundation Models

Open paper AskPDF arXiv

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Frames2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Frames2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Frames2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Frames2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.

Abstract

Efficient VLM Inference

Explore similar work

Frames2LoRA: Parametric Video Internalization for Vision-Language Models

Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models

VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Models