Human cognition of new concepts is inherently a streaming process: we continuously recognize new objects or identities and update our memories over time. However, current multimodal personalization methods are largely limited to static images or offline videos. This disconnects continuous visual input from instant real-world feedback, limiting their ability to provide the real-time, interactive personalized responses essential for future AI assistants. To bridge this gap, we first propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark designed specifically to evaluate this challenging setting. It evaluates a model's ability to respond to personalized concepts at exact timestamps under two modes: (1) Frame-level, focusing on a specific person or object in discrete frames, and (2) a novel Video-level, focusing on personalized actions unfolding across continuous frames. PEARL-Bench comprises 132 unique videos and 2,173 fine-grained annotations with precise timestamps. Concept diversity and annotation quality are strictly ensured through a combined pipeline of automated generation and human verification. To tackle this challenging new setting, we further propose PEARL, a plug-and-play, training-free strategy that serves as a strong baseline. Extensive evaluations across 8 offline and online models demonstrate that PEARL achieves state-of-the-art performance. Notably, it brings consistent PSVU improvements when applied to 3 distinct architectures, proving to be a highly effective and robust strategy. We hope this work advances vision-language model (VLM) personalization and inspires further research into streaming personalized AI assistants. Code is available at https://github.com/Yuanhong-Zheng/PEARL.
Figures & tables
Figure 1: Comparison between our proposed P ersonalized S treaming V ideo U nderstanding (PSVU) task and traditional Personalized Image/Video Understanding tasks. Unlike traditional settings, PSVU better aligns with real-world scenarios by featuring continuous streaming video inputs, dynamically defined concepts, and multi-turn conversations regarding these customized concepts.
Table 2
Figure 2: Overview of PEARL-Bench. (Left) Frame-level split: Concept-Definition QA registers a person at early timestamps; Real-Time QA queries the concept’s current state in the scene; Past-Time QA requires retrieving a historical evidence clip to answer questions about prior states. (Right) Video-level split: Concept-Definition QA registers a personalized action observed in a clip; Real-Time QA asks whether the defined action is being performed at the current timestamp.
Figure 3: PEARL framework. (a) Dual-grained Memory System: Concept Memory stores user-defined concepts with visual evidence and textual descriptions; Streaming Memory archives segmented clips with multimodal embeddings. (b) Concept-aware Retrieval Algorithm: Upon a user query, PEARL retrieves relevant concepts and top- K historical clips via concept-rewritten query embeddings, then feeds them together with the current clip into a VLM for real-time personalized response.
Method
#Frames
Frame-level
Video-level
Real-Time
Past-Time
Avg
Real-Time
Human Score
Human
-
97.61
96.45
97.03
97.49
Text-only
Qwen3-VL-8B [ 2 ]
-
11.06
17.45
14.26
7.04
Proprietary Offline Model
Table 3: Results on PEARL-Bench. We report Frame-level Real-Time/Past-Time (with Avg) and Video-level Real-Time metrics. Bold and underline denote the best and second-best results among open-source models and PEARL, respectively.
Text
Current
Concept
Streaming
Rewrite
Real-Time
Past-Time
Avg
✓
✗
✗
✗
✗
11.06
17.45
14.26
✓
✓
✗
✗
✗
15.84
20.30
18.07
✓
✓
✓
✗
✗
51.41
25.43
38.42
✓
✓
✓
✓
✗
50.22
45.69
47.96
✓
✓
✓
✓
✓
54.99
49.49
52.24
Table 4: Ablation of PEARL components on the frame-level split with Qwen3-VL-8B, showing progressive gains from Text-only to the full PEARL pipeline.
Method
#Frames
F-Avg
Latency (ms)
LLaVA-OV-7B
64
29.48
670
Qwen3-VL-8B
64
28.77
1,594
ReKV (LLaVA-OV-7B)
0.5fps
31.83
1,818
StreamForest-7B
1fps
35.02
1,164
TimeChat-Online-7B
1fps
33.59
4,769
LLaVA-OV-7B+PEARL
1fps
38.03
775
Table 5: Comparison of end-to-end inference latency across models. F-Avg denotes the frame-level average accuracy.
Figure 8
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Distribution of fine-grained sub-categories in PEARL-Bench. “(Frame)” denotes sub-categories belonging to the frame-level split, while “(Video)” denotes those in the video-level split.
QA Type
Sub-category
Question Templates
Concept-Definition
Direct
• This {EntityPronoun} is called {ConceptName}. Please remember this name. {DefinitionSuffix} • This {EntityPronoun} is my friend, named {ConceptName}. Please remember this name. {DefinitionSuffix} • The sequence of movements shown in this clip is defined as {ConceptName}. Please remember this name. {DefinitionSuffix}
Contextual
• The {EntityPronoun} in the {Location} is named {ConceptName}. Please remember this name. {DefinitionSuffix} • The {EntityPronoun} with {ConceptFeature} in the scene is my friend, named {ConceptName}. Please remember this name. {DefinitionSuffix} • The {EntityPronoun} who is {Event} is called {ConceptName}. Please remember this name. {DefinitionSuffix}
Real-Time
Presence
• Is this {EntityPronoun} {ConceptName} at this moment? • Are {ConceptName} and {OtherConcept} here now? • Is {ConceptName} currently among this group of people?
Behavior
• What is {ConceptName} doing now? • Is {ConceptName} {Event} right now? • What is {ConceptName}’s expression now?
Appearance
• What color is {ConceptName} wearing now? • What is the shape of the {Object} {ConceptName} is wearing? • What color is {ConceptName}’s {Object} now?
Location
• Where is {ConceptName} located in the frame? • Is {ConceptName} on the {Location} in the picture? • Where is {ConceptName} now?
Appendix
Table 6: Representative question templates across different QA types and sub-categories in PEARL-Bench.
Figure 7: Visualization example of frame-level multi-turn interactions in PEARL-Bench. The model successfully registers multiple user-defined concepts ( e.g . , <Nuriya> , <Kavery> , <Truz> ) and accurately answers subsequent Real-Time and Past-Time queries by retrieving corresponding historical evidence.
Figure 8: Visualization example of video-level multi-turn interactions in PEARL-Bench. The model successfully registers multiple user-defined action sequences ( e.g . , <Action A> , <Action B> , <Action C> ) and accurately recognizes these customized actions when they are performed by different characters later in the video stream.
Figure 9: Real-Time accuracy under different top- K ( K ) and expansion sizes ( N ).
Model
Real-Time
Past-Time
Avg
Qwen2-VL-2B [ 46 ]
31.24
22.84
27.04
Qwen2-VL-7B [ 46 ]
23.21
35.79
29.50
Qwen2-VL-2B+PEARL
29.93 ↓ 1.31
32.49 ↑ 9.65
31.21 ↑ 4.17
Qwen2-VL-7B+PEARL
33.30 ↑ 10.09
44.42 ↑ 8.63
38.86 ↑ 9.36
Qwen3-VL-4B [ 2 ]
24.08
30.96
27.52
Qwen3-VL-8B [ 2 ]
27.33
30.20
28.77
Appendix
Table 7: Performance comparison across different model scales on the frame-level split of PEARL-Bench. Bold and underline denote the best and second-best results within each model family (Qwen2-VL and Qwen3-VL), respectively.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
Streaming video understanding demands more than watching longer videos: assistants must decide when to speak in real time, balancing responsiveness against verbosity. Yet most video-language models (VideoLLMs) are trained for offline inference, and existing streaming benchmarks externalize this timing decision to the evaluator. We address this gap with RealStreamEval, a frame-level multi-turn evaluation protocol that exposes models to sequential observations and penalizes unnecessary responses. Under this protocol, we observed that strong offline VideoLLMs retain useful visual understanding but lack an interaction policy for deciding when to respond. Motivated by this observation, we propose EvoStreaming, a self-evolved streaming adaptation framework in which the base model itself acts as data generator, relevance annotator, and roll-out policy to synthesize streaming trajectories without external supervision. With only 1,000 self-generated samples (139× less than the leading streaming instruction-tuning approach) and no architectural changes, EvoStreaming consistently improves the overall RealStreamEval score by up to 10.8 points across five open VideoLLM backbones (Qwen2/2.5/3-VL, InternVL-3.5, MiniCPM-V4.5) while largely preserving offline video performance. These results suggest that data-efficient interaction tuning is a practical path for adapting existing VideoLLMs to streaming assistants.
Zichen Wen, Boxue Yang, Junlong Ke +5
EPIC Lab, Shanghai Jiao Tong University · Tsinghua University · The Hong Kong University of Science and Technology (Guangzhou) +1
Streaming video models should respond the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming predictions untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception and assistive capabilities required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming predictions by their temporal precision and balanced coverage across the entire video. Our benchmark reveals: (i) offline models detect events reliably but spam predictions unprompted; (ii) post-training for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency. These findings motivate AsynKV, a training-free streaming adaptation of offline models, that retains their event perception while improving their streaming behavior. AsynKV features a long-short term memory, utilized efficiently by scaling compute during dead-time. It serves as a strong baseline on SPOT-Bench, outperforming existing streaming models, and achieves state-of-the-art on retrospective benchmarks.
Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core mechanism for real-time visual assistants. Existing VLM frameworks predominantly assess models in offline settings. In contrast, the performance of a streaming VLM depends on additional metrics beyond pure video understanding, including proactiveness, which reflects the timeliness of the model's responses, and consistency, which captures the robustness of its responses over time. To address this limitation, we propose VSAS-Bench, a new framework and benchmark for Visual Streaming Assistants. In contrast to prior benchmarks that primarily employ single-turn question answering on video inputs, VSAS-Bench features temporally dense annotations with over 18,000 annotations across diverse input domains and task types. We introduce standardized synchronous and asynchronous evaluation protocols, along with metrics that isolate and measure distinct capabilities of streaming VLMs. Using this framework, we conduct large-scale evaluations of recent video and streaming VLMs, analyzing the accuracy-latency trade-off under key design factors such as memory buffer length, memory access policy, and input resolution, yielding several practical insights. Finally, we show empirically that conventional VLMs can be adapted to streaming settings without additional training, and demonstrate that these adapted models outperform recent streaming VLMs. For example, Qwen3-VL-4B surpasses Dispider, the best streaming VLM on our benchmark, by 3% under the asynchronous protocol. The benchmark and code will be available at https://github.com/apple/ml-vsas-bench.
Pavan Kumar Anasosalu Vasu, Cem Koc, Fartash Faghri +6