SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understanding

Senbin Xu; Zijun Zhang; Xinhua Wu; Li He; Jubo Feng; Li Qiao; Zhen Gao; Yan Zhang

cs.CVOct 10, 2026

SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understanding

Authors: Senbin Xu, Zijun Zhang, Xinhua Wu, Li He, Jubo Feng, Li Qiao, Zhen Gao, Yan Zhang,

Organizations: Xiamen University · Beijing Institute of Technology · Fudan University · Shanghai Jiao Tong University ·

Abstract

Streaming video understanding requires answering questions from the observed video, with evidence drawn from recent observations or earlier events. Historical context can improve recall but may interfere with judgments based on recent frames. Its usefulness depends on both when it is introduced and which clips are selected, while relevance alone does not guarantee sufficient evidence for an answer. We propose SPARE, a framework that combines adaptive historical retrieval with visual encoder sharing. A lightweight gate predicts whether recent observations provide sufficient evidence to answer the question. When additional evidence is predicted to be necessary, a multimodal embedding model retrieves relevant historical clips to supplement the current context. LoRA instruction tuning trains the understanding model to use recent observations and retrieved history jointly and abstain when the evidence remains insufficient. To reduce redundant visual parameters, the understanding and retrieval branches share a visual encoder while retaining separate language backbones. Four linear adapters align the shared features with the retrieval model through distillation at the visual token and final retrieval embedding levels. With visual-only inputs, SPARE achieves 77.78% on the evaluated Real-time and Backward task groups of OVO-Bench and 73.31% on StreamingBench. Compared with Qwen3-VL-8B-Instruct evaluated at 2 fps, SPARE improves accuracy by 20.68 percentage points on OVO-Bench and 14.15 percentage points on StreamingBench.

Figures & tables

Figure 1: Effect of historical context on OVO-Bench ( Li et al., 2025 ) . We report question-level accuracy within (a) Real-time (837 questions) and (b) Backward (631 questions). All configurations use Qwen3-VL-8B-Instruct with identical resolution and prompts, varying only input frame selection and count. Recent-4 uses up to four recent frames; +Uniform-4 and +Retrieval add up to four historical frames via uniform sampling or retrieval with CLIP ViT-L/14 or Qwen3-VL-Embedding-2B. Both retrieval variants retrieve history for every question without gating. Global-32 uniformly samples up to 32 frames from the observed video prefix. Blue dashed lines indicate Recent-4; bold entries highlight Qwen-based retrieval.

Figure 2: Overview of SPARE . Memory construction: the two branches share visual-encoder weights and index historical clips by embeddings and timestamps. Adaptive answering: a gate reads the current-input prefill state and triggers historical retrieval when evidence is insufficient. The retrieved clips and current window form a fresh second-pass input; otherwise, the first-pass answer is retained. LoRA tuning supports answering or abstention from the supplied evidence.

Figure 3: Shared visual encoding for understanding and retrieval. Left: separate visual encoders serve the two language models. Right: SPARE reuses the understanding model’s visual encoder and adapts its three DeepStack merger features and final merger features through four linear adapters for the retrieval branch. Each branch retains its own language backbone.

Offline Video LLMs
Model	Frames / sampling	StreamingBench				OVO-Bench
		RTVU	OSU	CU	All	Real-Time Visual Perception							Backward Tracing				All
		RTVU	OSU	CU	All	OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	All
LLaVA-OneVision2-8B	16	74.20	37.90	44.80	59.60	72.48	61.47	71.55	63.48	65.35	60.87	65.87	56.57	64.86	11.29	44.24	55.05
InternVL2-8B	32 / $\leq$ 64	67.60	35.20	36.90	53.58	69.80	59.63	67.24	50.56	71.29	57.61	62.69	52.53	58.11	25.81	45.48	54.08
Dispider-7B (offline)	16/clip	68.72	42.20	29.90	54.20	57.05	49.54	62.07	44.94	60.40	51.63	54.27	48.82	55.41	4.30	36.18	45.22

Table 1: Measured results on OVO-Bench and StreamingBench (%). For SPARE,

a+b

denotes the frame counts in the first and retrieval-triggered second answering passes. Native and shared ViT refer to the retrieval branch’s visual encoder. Bold denotes the best result and underlining the second-best result in each column, including ties.

Policy	Real-time	Backward	All	History use
Recent-only	81.06	68.00	74.53	0.00
Uniform history	81.05	70.70	75.87	99.46
Always retrieve	81.16	73.22	77.19	99.46
Gated retrieve	81.93	73.62	77.78	40.74

Table 2: Retrieval policies with the same shared ViT, LoRA adapter, and retrieval gate on OVO-Bench (%). All is the mean of the Real-time and Backward task-macro accuracies. History use is the fraction of questions whose final responses use historical frames.

Current frames	Frames/ clip	Retrieved clips	Score (%)
2	–	–	72.47
4	–	–	74.53
8	–	–	74.87
4	2	1	76.70
4	4	1	77.78
4	8	1	77.02

Table 3: The impact of frames on OVO-Bench.

Panel of Figure 4 — Figure 4: Efficiency on OVO-Bench. (a) Mean per-question peak answering-context length and accuracy on all Real-time and Backward questions. (b) Mean time to first token on OVO-Bench’s Real-time and Backward tasks at 16, 64, and 256 observed frames.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Task family	CLIP	SigLIP2	Native Embedding-2B
Video classification	44.98	38.11	64.21
Video retrieval	24.67	32.78	47.48
Moment retrieval	25.12	27.50	49.84
Video QA	35.47	34.76	60.47
Overall	33.38	33.93	56.13

Appendix

Table 4: Top-1 accuracy on MMEB-V2 video tasks (%). Each task-family score averages its datasets equally; Overall averages all 18 datasets. Qwen uses its native visual encoder. Bold marks the highest accuracy in each row.

	All 18 datasets		Text-to-video (5 datasets)
Metric	Native teacher	Shared student	Native teacher	Shared student
Top-1 accuracy	56.13	51.90	47.48	43.40
Top-5 success rate	87.25	84.83	72.09	68.09
Top-10 success rate	93.25	91.84	80.08	76.55
MRR $\times 100$	69.49	66.06	58.74	54.77

Appendix

Table 5: Direct retrieval with native teacher and shared student on MMEB-V2. Accuracy and success rates are percentages; MRR is multiplied by 100. Each column is an unweighted mean over the indicated datasets. The five text-to-video datasets are DiDeMo, MSR-VTT, MSVD, VATEX, and YouCook2.

Task	Recent-only	Uniform history	Always retrieve	Gated retrieve
OCR	93.96	93.96	93.96	95.97
ACR	84.40	83.49	84.40	85.32
ATR	81.90	79.31	83.62	83.62
STU	70.79	70.79	68.54	70.79
FPD	73.27	77.23	73.27	73.27
OJR	82.07	81.52	83.15	82.61

Appendix

Table 6: Task-level accuracy for the shared-ViT, LoRA-enabled OVO-Bench policies (%). The first six tasks form Real-time and the final three form Backward. All weights the two task-group macro averages equally. Bold marks the highest score in each row, including ties.

Policy	Real-time	Backward	All	History use
Recent-only	81.11	50.39	65.75	0.00
Always retrieve	82.34	56.95	69.64	99.46
Gated retrieve	82.37	56.28	69.32	39.51

Appendix

Table 7: Retrieval-policy control with native visual encoders and no LoRA on the same 1,468 OVO-Bench questions (%). History use counts questions whose final responses use historical frames.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

What Should a Streaming Video Model Remember?
Jun 15, 2026Haonan Ge, Yiwei Wang, Hang Wu +1Streaming Video Understanding Video-Language Models
University of California, Santa Barbara · University of California, Merced · The University of Queensland
When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs
Sep 29, 2026Xiang Hu, Jiazuo Yu, Lu Zhang +2Streaming Video Understanding Evidence Selection
IIAU Lab, Dalian University of Technology
StreamScout: Learning When to Look Deeper for Streaming Video Understanding
Aug 31, 2026Ce Zhang, Jing Bi, Jinxi He +9Adaptive Inference Streaming Video Understanding
Carnegie Mellon University · TikTok · University of Rochester +2

Jun 15, 2026J/K move · Enter open · S save

What Should a Streaming Video Model Remember?

Haonan Ge, Yiwei Wang, Hang Wu, Yujun Cai

University of California, Santa Barbara · University of California, Merced · The University of Queensland

Streaming Video Understanding Video-Language Models Memory-Augmented Video Understanding

Open paper AskPDF arXiv

Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets. Existing methods address this by adding memory banks, retrieval modules, or visual token compression to preserve long-range history. However, strong recent-window baselines show that indiscriminate history injection can dilute current-scene perception, suggesting that the key challenge is not whether to use memory, but how to allocate it selectively. We formulate this as budgeted online latent evidence allocation and propose SelectStream, a selective latent-memory framework that keeps the current observation directly visible to a frozen VLM while exposing historical information only through a compact, query-conditioned evidence budget. Three coordinated mechanisms govern when to write, what to preserve, and how to retrieve: surprise-driven adaptive windowing, priority-preserving consolidation, and query-conditioned graph reasoning over a fixed-capacity latent memory graph. Retrieved evidence is calibrated and injected as latent tokens for answer generation, without replaying frames or growing the context with stream length. Experimental results show that SelectStream achieves strong online streaming performance and preserves general video understanding, reaching 82.67% on StreamingBench, 67.03% on OVO-Bench, and 74.4% average accuracy on offline video benchmarks, while outperforming strong recent-window baselines and prior streaming memory methods.

Abstract

Streaming Video Understanding

Explore similar work

What Should a Streaming Video Model Remember?

When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs

StreamScout: Learning When to Look Deeper for Streaming Video Understanding