Human cognition of new concepts is inherently a streaming process: we continuously recognize new objects or identities and update our memories over time. However, current multimodal personalization methods are largely limited to static images or offline videos. This disconnects continuous visual input from instant real-world feedback, limiting their ability to provide the real-time, interactive personalized responses essential for future AI assistants. To bridge this gap, we first propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark designed specifically to evaluate this challenging setting. It evaluates a model's ability to respond to personalized concepts at exact timestamps under two modes: (1) Frame-level, focusing on a specific person or object in discrete frames, and (2) a novel Video-level, focusing on personalized actions unfolding across continuous frames. PEARL-Bench comprises 132 unique videos and 2,173 fine-grained annotations with precise timestamps. Concept diversity and annotation quality are strictly ensured through a combined pipeline of automated generation and human verification. To tackle this challenging new setting, we further propose PEARL, a plug-and-play, training-free strategy that serves as a strong baseline. Extensive evaluations across 8 offline and online models demonstrate that PEARL achieves state-of-the-art performance. Notably, it brings consistent PSVU improvements when applied to 3 distinct architectures, proving to be a highly effective and robust strategy. We hope this work advances vision-language model (VLM) personalization and inspires further research into streaming personalized AI assistants. Code is available at https://github.com/Yuanhong-Zheng/PEARL.
Figures & tables
Figure 1: Comparison between our proposed P ersonalized S treaming V ideo U nderstanding (PSVU) task and traditional Personalized Image/Video Understanding tasks. Unlike traditional settings, PSVU better aligns with real-world scenarios by featuring continuous streaming video inputs, dynamically defined concepts, and multi-turn conversations regarding these customized concepts.
Table 2
Figure 2: Overview of PEARL-Bench. (Left) Frame-level split: Concept-Definition QA registers a person at early timestamps; Real-Time QA queries the concept’s current state in the scene; Past-Time QA requires retrieving a historical evidence clip to answer questions about prior states. (Right) Video-level split: Concept-Definition QA registers a personalized action observed in a clip; Real-Time QA asks whether the defined action is being performed at the current timestamp.
Figure 3: PEARL framework. (a) Dual-grained Memory System: Concept Memory stores user-defined concepts with visual evidence and textual descriptions; Streaming Memory archives segmented clips with multimodal embeddings. (b) Concept-aware Retrieval Algorithm: Upon a user query, PEARL retrieves relevant concepts and top- K historical clips via concept-rewritten query embeddings, then feeds them together with the current clip into a VLM for real-time personalized response.
Method
#Frames
Frame-level
Video-level
Real-Time
Past-Time
Avg
Real-Time
Human Score
Human
-
97.61
96.45
97.03
97.49
Text-only
Qwen3-VL-8B [ 2 ]
-
11.06
17.45
14.26
7.04
Proprietary Offline Model
Table 3: Results on PEARL-Bench. We report Frame-level Real-Time/Past-Time (with Avg) and Video-level Real-Time metrics. Bold and underline denote the best and second-best results among open-source models and PEARL, respectively.
Text
Current
Concept
Streaming
Rewrite
Real-Time
Past-Time
Avg
✓
✗
✗
✗
✗
11.06
17.45
14.26
✓
✓
✗
✗
✗
15.84
20.30
18.07
✓
✓
✓
✗
✗
51.41
25.43
38.42
✓
✓
✓
✓
✗
50.22
45.69
47.96
✓
✓
✓
✓
✓
54.99
49.49
52.24
Table 4: Ablation of PEARL components on the frame-level split with Qwen3-VL-8B, showing progressive gains from Text-only to the full PEARL pipeline.
Method
#Frames
F-Avg
Latency (ms)
LLaVA-OV-7B
64
29.48
670
Qwen3-VL-8B
64
28.77
1,594
ReKV (LLaVA-OV-7B)
0.5fps
31.83
1,818
StreamForest-7B
1fps
35.02
1,164
TimeChat-Online-7B
1fps
33.59
4,769
LLaVA-OV-7B+PEARL
1fps
38.03
775
Table 5: Comparison of end-to-end inference latency across models. F-Avg denotes the frame-level average accuracy.
Figure 8
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Distribution of fine-grained sub-categories in PEARL-Bench. “(Frame)” denotes sub-categories belonging to the frame-level split, while “(Video)” denotes those in the video-level split.
QA Type
Sub-category
Question Templates
Concept-Definition
Direct
• This {EntityPronoun} is called {ConceptName}. Please remember this name. {DefinitionSuffix} • This {EntityPronoun} is my friend, named {ConceptName}. Please remember this name. {DefinitionSuffix} • The sequence of movements shown in this clip is defined as {ConceptName}. Please remember this name. {DefinitionSuffix}
Contextual
• The {EntityPronoun} in the {Location} is named {ConceptName}. Please remember this name. {DefinitionSuffix} • The {EntityPronoun} with {ConceptFeature} in the scene is my friend, named {ConceptName}. Please remember this name. {DefinitionSuffix} • The {EntityPronoun} who is {Event} is called {ConceptName}. Please remember this name. {DefinitionSuffix}
Real-Time
Presence
• Is this {EntityPronoun} {ConceptName} at this moment? • Are {ConceptName} and {OtherConcept} here now? • Is {ConceptName} currently among this group of people?
Behavior
• What is {ConceptName} doing now? • Is {ConceptName} {Event} right now? • What is {ConceptName}’s expression now?
Appearance
• What color is {ConceptName} wearing now? • What is the shape of the {Object} {ConceptName} is wearing? • What color is {ConceptName}’s {Object} now?
Location
• Where is {ConceptName} located in the frame? • Is {ConceptName} on the {Location} in the picture? • Where is {ConceptName} now?
Appendix
Table 6: Representative question templates across different QA types and sub-categories in PEARL-Bench.
Figure 7: Visualization example of frame-level multi-turn interactions in PEARL-Bench. The model successfully registers multiple user-defined concepts ( e.g . , <Nuriya> , <Kavery> , <Truz> ) and accurately answers subsequent Real-Time and Past-Time queries by retrieving corresponding historical evidence.
Figure 8: Visualization example of video-level multi-turn interactions in PEARL-Bench. The model successfully registers multiple user-defined action sequences ( e.g . , <Action A> , <Action B> , <Action C> ) and accurately recognizes these customized actions when they are performed by different characters later in the video stream.
Figure 9: Real-Time accuracy under different top- K ( K ) and expansion sizes ( N ).
Model
Real-Time
Past-Time
Avg
Qwen2-VL-2B [ 46 ]
31.24
22.84
27.04
Qwen2-VL-7B [ 46 ]
23.21
35.79
29.50
Qwen2-VL-2B+PEARL
29.93 ↓ 1.31
32.49 ↑ 9.65
31.21 ↑ 4.17
Qwen2-VL-7B+PEARL
33.30 ↑ 10.09
44.42 ↑ 8.63
38.86 ↑ 9.36
Qwen3-VL-4B [ 2 ]
24.08
30.96
27.52
Qwen3-VL-8B [ 2 ]
27.33
30.20
28.77
Appendix
Table 7: Performance comparison across different model scales on the frame-level split of PEARL-Bench. Bold and underline denote the best and second-best results within each model family (Qwen2-VL and Qwen3-VL), respectively.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.