WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs

Yulin Zhang; Cheng Shi; Sibei Yang

cs.CVFeb 25, 2026

WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs

Authors: Yulin Zhang, Cheng Shi, Sibei Yang

Organizations: ShanghaiTech University · School of Computing and Data Science, The University of Hong Kong · School of Computer Science and Engineering, Sun Yat-sen University

Abstract

Recent advances in Multimodal Large Language Models have greatly improved visual understanding and reasoning, yet their quadratic attention and offline training protocols make them ill-suited for streaming settings where frames arrive sequentially and future observations are inaccessible. We diagnose a core limitation of current Video-LLMs, namely Time-Agnosticism, in which videos are treated as an unordered bag of evidence rather than a causally ordered sequence, yielding two failures in streams: temporal order ambiguity, in which the model cannot follow or reason over the correct chronological order, and past-current focus blindness where it fails to distinguish present observations from accumulated history. We present WeaveTime, a simple, efficient, and model agnostic framework that first teaches order and then uses order. We introduce a lightweight Temporal Reconstruction objective-our Streaming Order Perception enhancement-that instills order aware representations with minimal finetuning and no specialized streaming data. At inference, a Past-Current Dynamic Focus Cache performs uncertainty triggered, coarse-to-fine retrieval, expanding history only when needed. Plugged into exsiting Video-LLM without architectural changes, WeaveTime delivers consistent gains on representative streaming benchmarks, improving accuracy while reducing latency. These results establish WeaveTime as a practical path toward time aware stream Video-LLMs under strict online, time causal constraints. Code and weights will be made publicly available. Project Page: https://zhangyl4.github.io/publications/weavetime/

Figures & tables

Figure 1: Illustrative examples of two coupled challenges in streaming Video-LLMs stemming from Time-Agnosticism. Top: Temporal Order Ambiguity. The model struggles to correctly interpret the temporal sequence of events (e.g., entering vs. leaving a room), leading to erroneous spatial inferences (e.g., mislocating the orange flowers). Bottom: Past–Current Focus Blindness. The model fails to dynamically allocate attention between the immediate observation and relevant past memories. For the "What color is the flower in the painting now?" query, the answer is in the current frame, but the model recalls an irrelevant past moment. Conversely, for the "Where is the full body mirror?" query, which requires historical context, the model fixates on the current frame, leading to an incorrect answer.

Figure 2: Impact of ground-truth answer-window shift on Video-QA accuracy. This heatmap shows model accuracy when the ground-truth (GT) answer window is shifted to different relative positions (0–100%) along the video, evaluated over videos grouped by binned length (e.g., $345$ s, $480$ s). Systematic changes in accuracy across shift positions and lengths reveal a temporal positional bias, whereby the model prefers specific time locations rather than consistently locating evidence based on the query.

Task	Base	Model Shuffle		Human Shuffle
		w/ TS	w/o TS	w/ TS	w/o TS
Information Synopsis	0.80	0.80	0.80	1.00	0.40
Attribute Perception	0.40	0.40	0.20	1.00	0.60
Action Reasoning	0.60	0.80	0.60	1.00	0.00
Spatial Reasoning	1.00	1.00	0.80	1.00	0.60
Object Recognition	0.80	0.60	0.80	1.00	0.40

Table 1: Model vs. human performance under shuffled-frame inputs. Base is accuracy on ordered videos; other columns show accuracy with shuffled frames, with or without timestamps. Humans remain reliable on non-temporal tasks but collapse on temporal/action tasks without timestamps and recover once timestamps are provided, indicating genuine use of temporal order. The model, however, stays strong under shuffling and barely benefits from timestamps, revealing time-agnostic, bag-of-evidence behavior. Cells where shuffling improves accuracy are highlighted in red.

Figure 3: Overview of WeaveTime. (Left) The VideoLLM-SOPE enhances temporal perception by reconstructing correct frame order from shuffled inputs during training. (Right) The Past–Current Dynamic Focus (PCDF) Cache adaptively controls memory retrieval at inference, balancing between immediate observations and recalled past content. Together, these components jointly mitigate temporal ambiguity and inefficient memory access, enabling robust and temporally coherent streaming reasoning.

Figure 4: Overview of the Past–Current Dynamic Focus Cache (PCDF-Cache). (a) Given streaming inputs, the PCDF-Cache monitors prediction entropy: when uncertainty is low, the model answers directly from the current observation; when high, it triggers a recall from long-term memory via Coarse-to-Fine Recall . (b) The recall module performs hierarchical selection—first pooling coarse candidates, then applying a max-similarity criterion for fine retrieval—balancing retrieval cost and contextual accuracy.

Method	# of Frames	OVO-Bench Real-Time							Streaming-Bench Real-Time
		OCR	ACR	ATR	STU	FPD	OJR	AVG	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	AVG
Human
Human	-	93.96	92.57	94.83	92.70	91.09	93.20	91.30	89.47	92.00	93.60	91.47	95.65	92.52	88.80	88.75	89.74	91.30	91.46
Proprietary Models (Offline), Single-Turn Evaluation
Gemini 1.5 pro [ 39 ]	1 FPS	85.91	66.97	79.31	58.43	63.37	61.96	69.32	79.02	80.47	83.54	79.67	80.00	84.74	77.78	64.23	71.95	48.70	75.69
GPT-4o [ 19 ]	64	69.80	64.22	71.55	51.12	70.30	59.78	64.46	77.11	80.47	83.91	76.47	70.19	83.80	66.67	62.19	69.12	49.22	73.28

Table 2: Comparison of various Video LLMs on OVO-Bench Real-Time and Streaming-Bench Real-Time . † indicates experimental results derived from integrating the respective dataset into the ReKV [ 10 ] codebase.

LLaVA-OV-7B + ReKV
SOPE Training		PCDF	OVO-Bench		Streaming-Bench
w/ TP	w/ TR	Cache	Overall	$\Delta$	Real-Time	$\Delta$
			53.56	–	66.15	–
✔			49.88	-3.68	65.91	-0.54
✔	✔		55.70	+5.82	68.49	+2.58
✔	✔	✔	57.57	+1.87	72.13	+3.64

Table 3: Ablation on SOPE Training and PCDF-Cache. Performance is reported across OVO-Bench and the real-time subset of Streaming-Bench. TP refers to training that incorporates a small amount of offline video data augmented with timestamp tokens. TR denotes the inclusion of a temporal re-ordering auxiliary task.

Method	QAEGO4D		MLVU	EVENTHALL
Method	Recall $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$
LLaVA-OV	14.0	52.8	64.7	60.1
+ ReKV	23.9	54.3	68.5	60.6
+ C2F (Ours)	25.2	55.2	68.9	61.4
+ Fine	OOM

Table 4: Retrieval strategy comparison across datasets. Each dataset reports Recall and/or Accuracy; “OOM” indicates out-of-memory.

Figure 5: Ablation on the entropy threshold in PCDF-Cache. On OvO-Bench [ 26 ] , the threshold controls when memory recall is triggered. Accuracy (blue) peaks at 0.6, balancing current observation and past memory, while response latency (green) decreases with larger thresholds, yielding the best accuracy–efficiency trade-off.

Figure 6: Example of Temporal Re-Ordering . Visualization of the model’s ability to restore chronological order. The top rows show a temporally-scrambled input sequence, which the model re-orders to produce the coherent output sequence in the bottom rows.

Figure 7: Distribution of Temporal Re-order Overlap. Histogram illustrating the model’s performance distribution on the temporal re-ordering task.

Method	OVO-Bench	GPUs	stream-tailored data
LLaVA-OV + ReKV	53.6	–	–
(+SOPE)	55.7	8	0
StreamForest Base	53.9	32	–
(+Online IT)	55.6	32	121K

Table 5: Data and Compute Efficiency Comparison.

Video-Language Models

Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.

See all 161 papers

Aug 11, 2026

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.

Muxin Fu, Yifan Zhang, Wentao Zhang +5

Tongji University · Nanyang Technological University · University of Michigan +2

Memory-Augmented VLMs Streaming Video Understanding Efficient VLM Inference Memory-Augmented Video Understanding

Ask PDF

Mar 3, 2026

Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models

Large Vision Language Models (LVLMs) exhibit strong Chain-of-Thought (CoT) capabilities, yet most existing paradigms assume full-video availability before inference, a batch-style process misaligned with real-world video streams where information arrives sequentially. Motivated by the streaming nature of video data, we investigate two streaming reasoning paradigms for LVLMs. The first, an interleaved paradigm, alternates between receiving frames and producing partial reasoning but remains constrained by strictly ordered cache updates. To better match streaming inputs, we propose Think-as-You-See (TaYS), a unified framework enabling true concurrent reasoning. TaYS integrates parallelized CoT generation, stream-constrained training, and stream-parallel inference. It further employs temporally aligned reasoning units, streaming attention masks and positional encodings, and a dual KV-cache that decouples visual encoding from textual reasoning. We evaluate all paradigms on the Qwen2.5-VL family across representative video CoT tasks, including event dynamics analysis, causal reasoning, and thematic understanding. Experiments show that TaYS consistently outperforms both batch and interleaved baselines, improving reasoning performance while substantially reducing time-to-first-token (TTFT) and overall reasoning delay. These results demonstrate the effectiveness of data-aligned streaming reasoning in enabling efficient and responsive video understanding for LVLMs. We release our code at https://github.com/EIT-NLP/StreamingLLM/tree/main/TaYS

Jialiang Zhang, Junlong Tong, Junyan Lin +4

Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ocean University of China · Shanghai Jiao Tong University +3

VLM Reasoning Streaming Inference Streaming Video Understanding Large Vision-Language Models Efficient VLM Inference

Ask PDF

Sep 29, 2026

When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs

Streaming video understanding requires Video Large Language Models (Video-LLMs) to reason over continuous visual streams under causal constraints. As the visual history grows, a bounded visual?processing budget requires evidence selection that balances temporal recency with query relevance. Recent-only selection excludes potentially relevant historical evidence, whereas Semantic-only retrieval can displace useful recent context when relevance scores are ambiguous. We introduce WRWS (When to Retrieve, When to Stay), a training-free framework for uncertainty-adaptive evidence allocation. A lightweight external vision-language encoder scores query relevance across the observed history, while an adaptive allocation module uses the normalized entropy of the similarity distribution as a proxy for retrieval uncertainty. WRWS favors semantic retrieval when relevance cues are reliable and strengthens the recency prior under uncertainty. Following a retrieve-first, encode-later pipeline, WRWS selects evidence before target-model visual encoding, such that only the selected observations are processed by the costly target Video-LLM. Experiments across four Video-LLM families and multiple model scales demonstrate competitive accuracy on StreamingBench and OVO-Bench. In our efficiency evaluation, WRWS reduces average vision-to-answer time to 47.93% of the state-of-the-art method. Code will be released.

Xiang Hu, Jiazuo Yu, Lu Zhang +2

IIAU Lab, Dalian University of Technology

Streaming Video Understanding Evidence Selection Query-Aware Video Frame Selection VLM Inference

Ask PDF

Abstract

Video-Language Models

Explore similar work

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models

When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs