WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs
Authors: Yulin Zhang, Cheng Shi, Sibei Yang
Organizations: ShanghaiTech University · School of Computing and Data Science, The University of Hong Kong · School of Computer Science and Engineering, Sun Yat-sen University
Recent advances in Multimodal Large Language Models have greatly improved visual understanding and reasoning, yet their quadratic attention and offline training protocols make them ill-suited for streaming settings where frames arrive sequentially and future observations are inaccessible. We diagnose a core limitation of current Video-LLMs, namely Time-Agnosticism, in which videos are treated as an unordered bag of evidence rather than a causally ordered sequence, yielding two failures in streams: temporal order ambiguity, in which the model cannot follow or reason over the correct chronological order, and past-current focus blindness where it fails to distinguish present observations from accumulated history. We present WeaveTime, a simple, efficient, and model agnostic framework that first teaches order and then uses order. We introduce a lightweight Temporal Reconstruction objective-our Streaming Order Perception enhancement-that instills order aware representations with minimal finetuning and no specialized streaming data. At inference, a Past-Current Dynamic Focus Cache performs uncertainty triggered, coarse-to-fine retrieval, expanding history only when needed. Plugged into exsiting Video-LLM without architectural changes, WeaveTime delivers consistent gains on representative streaming benchmarks, improving accuracy while reducing latency. These results establish WeaveTime as a practical path toward time aware stream Video-LLMs under strict online, time causal constraints. Code and weights will be made publicly available. Project Page: https://zhangyl4.github.io/publications/weavetime/
Figures & tables
Figure 1: Illustrative examples of two coupled challenges in streaming Video-LLMs stemming from Time-Agnosticism. Top: Temporal Order Ambiguity. The model struggles to correctly interpret the temporal sequence of events (e.g., entering vs. leaving a room), leading to erroneous spatial inferences (e.g., mislocating the orange flowers). Bottom: Past–Current Focus Blindness. The model fails to dynamically allocate attention between the immediate observation and relevant past memories. For the "What color is the flower in the painting now?" query, the answer is in the current frame, but the model recalls an irrelevant past moment. Conversely, for the "Where is the full body mirror?" query, which requires historical context, the model fixates on the current frame, leading to an incorrect answer.
Figure 2: Impact of ground-truth answer-window shift on Video-QA accuracy. This heatmap shows model accuracy when the ground-truth (GT) answer window is shifted to different relative positions (0–100%) along the video, evaluated over videos grouped by binned length (e.g., 345 s, 480 s). Systematic changes in accuracy across shift positions and lengths reveal a temporal positional bias, whereby the model prefers specific time locations rather than consistently locating evidence based on the query.
Task
Base
Model Shuffle
Human Shuffle
w/ TS
w/o TS
w/ TS
w/o TS
Information Synopsis
0.80
0.80
0.80
1.00
0.40
Attribute Perception
0.40
0.40
0.20
1.00
0.60
Action Reasoning
0.60
0.80
0.60
1.00
0.00
Spatial Reasoning
1.00
1.00
0.80
1.00
0.60
Object Recognition
0.80
0.60
0.80
1.00
0.40
Table 1: Model vs. human performance under shuffled-frame inputs. Base is accuracy on ordered videos; other columns show accuracy with shuffled frames, with or without timestamps. Humans remain reliable on non-temporal tasks but collapse on temporal/action tasks without timestamps and recover once timestamps are provided, indicating genuine use of temporal order. The model, however, stays strong under shuffling and barely benefits from timestamps, revealing time-agnostic, bag-of-evidence behavior. Cells where shuffling improves accuracy are highlighted in red.
Figure 3: Overview of WeaveTime. (Left) The VideoLLM-SOPE enhances temporal perception by reconstructing correct frame order from shuffled inputs during training. (Right) The Past–Current Dynamic Focus (PCDF) Cache adaptively controls memory retrieval at inference, balancing between immediate observations and recalled past content. Together, these components jointly mitigate temporal ambiguity and inefficient memory access, enabling robust and temporally coherent streaming reasoning.
Figure 4: Overview of the Past–Current Dynamic Focus Cache (PCDF-Cache). (a) Given streaming inputs, the PCDF-Cache monitors prediction entropy: when uncertainty is low, the model answers directly from the current observation; when high, it triggers a recall from long-term memory via Coarse-to-Fine Recall . (b) The recall module performs hierarchical selection—first pooling coarse candidates, then applying a max-similarity criterion for fine retrieval—balancing retrieval cost and contextual accuracy.
Table 2: Comparison of various Video LLMs on OVO-Bench Real-Time and Streaming-Bench Real-Time . † indicates experimental results derived from integrating the respective dataset into the ReKV [ 10 ] codebase.
SOPE Training
PCDF
OVO-Bench
Streaming-Bench
w/ TP
w/ TR
Cache
Overall
Δ
Real-Time
Δ
LLaVA-OV-7B + ReKV
53.56
–
66.15
–
✔
49.88
-3.68
65.91
-0.54
✔
✔
55.70
+5.82
68.49
+2.58
✔
✔
✔
57.57
+1.87
72.13
+3.64
Table 3: Ablation on SOPE Training and PCDF-Cache. Performance is reported across OVO-Bench and the real-time subset of Streaming-Bench. TP refers to training that incorporates a small amount of offline video data augmented with timestamp tokens. TR denotes the inclusion of a temporal re-ordering auxiliary task.
Method
QAEGO4D
MLVU
EVENTHALL
Recall ↑
Acc ↑
Acc ↑
Acc ↑
LLaVA-OV
14.0
52.8
64.7
60.1
+ ReKV
23.9
54.3
68.5
60.6
+ C2F (Ours)
25.2
55.2
68.9
61.4
+ Fine
OOM
Table 4: Retrieval strategy comparison across datasets. Each dataset reports Recall and/or Accuracy; “OOM” indicates out-of-memory.
Figure 5: Ablation on the entropy threshold in PCDF-Cache. On OvO-Bench [ 26 ] , the threshold controls when memory recall is triggered. Accuracy (blue) peaks at 0.6, balancing current observation and past memory, while response latency (green) decreases with larger thresholds, yielding the best accuracy–efficiency trade-off.
Figure 6: Example of Temporal Re-Ordering . Visualization of the model’s ability to restore chronological order. The top rows show a temporally-scrambled input sequence, which the model re-orders to produce the coherent output sequence in the bottom rows.
Figure 7: Distribution of Temporal Re-order Overlap. Histogram illustrating the model’s performance distribution on the temporal re-ordering task.