Proact-VL: A Proactive VideoLLM for Real-Time AI Companions

Weicai Yan; Yuhong Dai; Qi Ran; Haodong Li; Wang Lin; Tao Jin; Xing Xie; Hao Liao

cs.CVMar 3, 2026

Proact-VL: A Proactive VideoLLM for Real-Time AI Companions

Authors: Weicai Yan, Yuhong Dai, Qi Ran, Haodong Li, Wang Lin, Tao Jin, Xing Xie, Hao Liao, +1 more

Organizations: Zhejiang University · Shenzhen University · South China University of Technology · Zhejiang University School of Software (Ningbo) Innovation and Management Center · Microsoft Research Asia

Abstract

Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous streaming inputs, (2) autonomously deciding when to respond, and (3) controlling both quality and quantity of generated content to meet real-time constraints. In this work, we instantiate AI companions through two gaming scenarios, commentator and guide, selected for their suitability for automatic evaluation. We introduce the Live Gaming Benchmark, a large-scale dataset with three representative scenarios: solo commentary, co-commentary, and user guidance, and present Proact-VL, a general framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction. Extensive experiments show Proact-VL achieves superior response latency and quality while maintaining strong video understanding capabilities, demonstrating its practicality for real-time interactive applications.

Figures & tables

Figure 2: Overview of the Live Gaming Dataset. The inner, middle, and outer rings represent the three data categories, 12 specific game titles, and their corresponding genres, respectively.

Figure 4: Illustration of the Proact-VL. At each second, Proact-VL consumes multi-source tokens (video, query, and context) and decides whether to speak by feeding the FLAG hidden state into a response head to obtain a score, then thresholding with $\tau$ . If triggered, it appends the assistant prefix and generates a short clip-level text; otherwise, it appends the prefix with a Silence token to output silence.

Model	Solo Commentary			Co-Commentary			Guidance			Overall
Model	CC $\uparrow$	LiveU $\uparrow$	FinalQ $\uparrow$	CC $\uparrow$	LiveU $\uparrow$	FinalQ $\uparrow$	CC $\uparrow$	LiveU $\uparrow$	FinalQ $\uparrow$	CC $\uparrow$	LiveU $\uparrow$	FinalQ $\uparrow$
Offline Models
GPT-4o	21.54	4.56	4.74	46.72	3.35	2.99	50.00	5.95	6.66	39.42	4.62	4.80
Gemini 2.5 Pro	-	4.87	5.19	-	3.49	3.59	-	5.73	5.67	-	4.70	4.82
Proactive Models
VideoLLM-online	16.71	4.26	2.07	17.55	2.57	1.39	7.08	3.85	1.75	13.78	3.56	1.74

Table 1: Main results of text quality on Live Gaming Benchmark. Bold indicates the best results, and underline the second best.

Model	Solo Commentary			Co-Commentary			Guidance			Overall
Model	TimeDiff $\downarrow$	PAUC $\uparrow$	F1 $\uparrow$	TimeDiff $\downarrow$	PAUC $\uparrow$	F1 $\uparrow$	TimeDiff $\downarrow$	PAUC $\uparrow$	F1 $\uparrow$	TimeDiff $\downarrow$	PAUC $\uparrow$	F1 $\uparrow$
Offline Models
GPT-4o	1.16	25.14	62.53	3.42	3.31	58.80	4.62	42.26	43.30	3.07	23.57	54.88
Gemini 2.5 Pro	1.03	22.72	59.02	2.77	5.38	56.86	3.96	25.96	31.82	2.59	18.02	49.23
Proactive Models
VideoLLM-online	10.86	0.04	7.02	8.95	0.00	8.16	17.97	0.00	4.44	12.59	0.01	6.54

Table 2: Main results of response quality on Live Gaming Benchmark. Bold indicates the best results, and underline the second best.

Model	Ego4D						Black Myth Wukong
Model	CC $\uparrow$	LiveU $\uparrow$	FinalQ $\uparrow$	TimeDiff $\downarrow$	PAUC $\uparrow$	F1 $\uparrow$	CC $\uparrow$	LiveU $\uparrow$	FinalQ $\uparrow$	TimeDiff $\downarrow$	PAUC $\uparrow$	F1 $\uparrow$
Offline Models
GPT-4o	50.00	5.02	4.35	15.23	14.93	16.36	23.33	4.66	4.40	1.51	9.35	58.24
Gemini 2.5 Pro	-	3.32	3.55	46.21	15.64	14.55	-	4.97	5.10	1.06	8.30	56.00
Proactive Models
VideoLLM-online	13.06	3.73	2.28	16.64	0.80	17.90	42.50	4.59	2.92	11.62	0.00	6.62

Table 3: Full results on text and response quality for common and general commentary. Bold indicates the best results, while Underline denotes the second-best results.

Metrics	10	20	30	40	50	Overall
SC	73.75	75.00	76.25	78.27	79.48	82.03
LiveU	5.45	5.31	5.50	5.52	5.53	5.71
FinalQ	4.13	4.04	4.20	4.24	4.23	4.12

Table 4: Text quality over time.

Metrics	10	20	30	40	50	Overall
TimeDiff	0.70	0.76	0.92	0.88	0.82	0.81
PAUC	9.08	7.10	7.58	8.17	8.07	7.63
F1	74.42	71.15	69.29	69.85	70.10	69.23

Table 5: Proactivity quality over time.

$\mathcal{L}_{cls}$	$\mathcal{L}_{reg}$	CC	TimeDiff	P	R	F1
$\checkmark$	-	45.54	18.50	12.13	14.00	11.03
-	$\checkmark$	47.53	8.28	45.20	67.02	47.39
$\checkmark$	$\checkmark$	50.91	3.41	65.72	62.41	60.08

Table 6: Effectiveness of training loss.

Panel of Figure 5 — Figure 5: Score curve visualization. Green: labeled response; Red: labeled silence; Dashed line: threshold; Above-threshold scores: model triggers responses.

Frame	WS	Mem	Cache	Forward	Chunk	Token
299	8192	16.07	0.0794	0.2704	0.3545	0.0433
299	16384	16.45	0.0850	0.2775	0.3709	0.0451
299	24576	16.83	0.0902	0.2728	0.3677	0.0441
299	32768	17.20	0.0967	0.3242	0.4313	0.0431
510	16384	16.57	0.1232	0.2446	0.3759	0.0454
364	16384	16.50	0.0975	0.2614	0.3642	0.0446

Table 7: Inference efficiency of Proact-VL under streaming inference. Frame : tokens per frame; WS : streaming window size (in tokens); Mem (GB): peak GPU memory; Cache (s): time to ingest the current chunk and update the KV cache; Forward (s): average text generation time per chunk; Chunk (s): end-to-end time per chunk; Token (s): average time per generated token.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Illustration of ChatML template.

Source	Train	Test
Baldur’s Gate 3	10,286	240
CS:GO	10,347	240
Cyberpunk 2077	7,986	240
Elden Ring	7,398	240
League of Legends	5,506	240
Minecraft	6,543	480

Appendix

Table 8: Data distribution across splits.

Model	Solo Commentary		Co-Commentary		Guidance		Ego4d		Black Myth Wukong		Overall
Model	CC $\uparrow$	F1 $\uparrow$	CC $\uparrow$	F1 $\uparrow$	CC $\uparrow$	F1 $\uparrow$	CC $\uparrow$	F1 $\uparrow$	CC $\uparrow$	F1 $\uparrow$	CC $\uparrow$	F1 $\uparrow$
Qwen2-VL	28.67	22.41	38.44	38.00	27.08	24.62	6.75	16.36	55.62	17.47	31.31	23.77
Proact-VL Qwen2-VL ( $\tau=0.5$ )	53.79	56.21	54.90	70.09	40.94	50.68	61.94	46.72	64.38	46.30	55.19	54.00
Qwen2.5-VL	12.75	62.31	20.26	61.53	23.44	45.21	3.17	16.36	21.67	57.53	16.26	48.59
Proact-VL Qwen2.5-VL ( $\tau=0.3$ )	49.62	63.66	50.94	75.84	42.71	52.94	58.58	44.72	56.25	59.68	51.62	59.37
Qwen3-VL	15.04	51.85	40.94	24.34	35.62	36.91	19.05	16.36	14.79	49.51	25.09	35.79

Appendix

Table 9: Comparison of downstream task performance of Proact-VL models trained with different base models.

Figure 7: Game-Wise Analysis.

Caption Model	Judge Model	Model	Time	Rate	TextU	Fidelity	Continuity	Substance	LiveU	FinalQ	Win Rate
Gemini 2.5 Pro	GPT-5.1	LiveCC-7B-Base	4.65	4.42	5.49	4.33	4.11	3.63	4.85	4.02	41.17
Gemini 2.5 Pro	GPT-5.1	LiveCC-7B-Instruct	5.41	5.31	6.79	4.91	4.72	4.47	5.84	4.70	34.33
Gemini 2.5 Pro	GPT-5.1	Proact-VL	6.70	6.34	7.64	5.88	5.69	4.87	6.89	5.48	53.62
GPT-4o	Gemini 2.5 Flash	LiveCC-7B-Base	6.71	6.77	6.99	5.63	5.23	5.42	6.82	5.43	64.83
GPT-4o	Gemini 2.5 Flash	LiveCC-7B-Instruct	7.21	7.26	7.56	5.66	5.06	5.50	7.34	5.41	68.75
GPT-4o	Gemini 2.5 Flash	Proact-VL	7.41	7.47	7.71	6.30	5.74	5.91	7.53	5.98	76.79

Appendix

Table 10: Robustness of LLM-as-a-Judge on SOLO commentary scenario.

Caption Model	Judge Model	Model	Time	Rate	TextU	Fidelity	Continuity	Substance	LiveU	FinalQ	Win Rate
Gemini 2.5 Pro	GPT-5.1	LiveCC-7B-Base	3.29	3.98	4.08	3.66	2.99	2.55	3.78	3.07	45.89
Gemini 2.5 Pro	GPT-5.1	LiveCC-7B-Instruct	3.53	4.50	4.83	3.44	3.22	3.19	4.29	3.28	37.40
Gemini 2.5 Pro	GPT-5.1	Proact-VL	4.54	5.34	5.58	4.11	3.53	3.13	5.15	3.59	51.46
GPT-4o	Gemini 2.5 Flash	LiveCC-7B-Base	7.02	7.09	7.26	5.39	4.99	4.99	7.12	5.12	51.93
GPT-4o	Gemini 2.5 Flash	LiveCC-7B-Instruct	7.40	7.37	7.65	5.17	4.93	5.03	7.47	5.04	54.11
GPT-4o	Gemini 2.5 Flash	Proact-VL	7.45	7.50	7.68	5.59	5.28	5.28	7.54	5.38	61.72

Appendix

Table 11: Robustness of LLM-as-a-Judge on co-commentary scenario.

Caption Model	Judge Model	Model	Time	Rate	TextU	Fidelity	Continuity	Substance	LiveU	FinalQ	Win Rate
Gemini 2.5 Pro	GPT-5.1	LiveCC-7B-Base	2.83	2.75	3.17	3.52	3.15	2.57	2.92	3.08	29.58
Gemini 2.5 Pro	GPT-5.1	LiveCC-7B-Instruct	4.27	3.97	5.45	4.23	3.71	3.72	4.56	3.89	13.33
Gemini 2.5 Pro	GPT-5.1	Proact-VL	6.93	7.71	7.90	6.34	6.36	5.37	7.51	6.02	42.60
GPT-4o	Gemini 2.5 Flash	LiveCC-7B-Base	6.52	6.79	7.03	4.75	4.35	4.45	6.78	4.52	22.92
GPT-4o	Gemini 2.5 Flash	LiveCC-7B-Instruct	4.31	4.46	5.06	3.56	2.74	3.25	4.61	3.18	33.75
GPT-4o	Gemini 2.5 Flash	Proact-VL	7.15	7.43	7.57	6.17	5.81	5.94	7.38	5.97	51.46

Appendix

Table 12: Robustness of LLM-as-a-Judge on guidance scenario.

Table 19

Model	Black Myth Wukong	CSGO	Minecraft
Gemini 2.5 Pro	20.0	20.0	0.0
Livestar	10.0	10.0	0.0
LiveCC-7B-Instruct	30.0	10.0	10.0
Proact-VL	80.0	86.7	96.7

Appendix

Table 15: Performance comparison across different games.

Figure 8: Interface for Pairwise Model Comparison.

Model	Ego4D		Gaming		Livesports
Model	CC	F1	CC	F1	CC
LiveCC-7B-Base	12.69	14.94	38.88	36.10	70.04
Proact-VL(w/o game)	56.34	47.13	37.83	50.93	78.88
Proact-VL(w/o ego4d)	36.19	33.08	51.84	61.21	74.82
Proact-VL(w/o live-sft)	57.46	45.77	51.56	60.50	64.78
Proact-VL	58.58	47.95	50.91	60.08	72.47

Appendix

Table 16: Effectiveness of Training Data.

LiveCC-Base
Prompt		Solo		Co		Guidance
system	user	CC	F1	CC	F1	CC	F1
-	-	41.46	47.05	54.79	29.71	30.73	12.08
$\checkmark$	$\checkmark$	44.71	16.40	45.78	30.62	35.52	16.77
-	$\checkmark$	41.17	47.05	45.89	43.25	29.58	18.00
LiveCC-Instruct

Appendix

Table 17: Prompt injection ablation on LiveCC-Base and LiveCC-Instruct. “system” indicates injecting task/persona instructions into the system prompt; “user” indicates injecting history/query context into the user template.

Action
Task	Base	Proact-VL	$\Delta$
Action Sequence	73.0	78.0	+5.0
Action Prediction	64.0	64.5	+0.5
Action Antonym	82.5	83.0	+0.5
Fine-grained Action	46.0	47.0	+1.0
Unexpected Action	80.0	80.0	+0.0

Appendix

Table 18: Per-task accuracy (%) comparison with a Qwen3-VL baseline on MVBench.

\Delta

denotes Proact-VL

-

Base.

Model	Subs	Short	Medium	Long	Overall
Qwen3-VL	–	66.4	55.4	49.9	57.3
Proact-VL ${}_{\text{Qwen3-VL}}$	–	67.6	58.2	51.4	59.1
Qwen3-VL	✓	68.0	56.7	50.3	58.3
Proact-VL ${}_{\text{Qwen3-VL}}$	✓	68.9	57.4	53.1	59.8
Qwen2.5-VL	–	61.9	51.2	46.2	53.1
Proact-VL ${}_{\text{Qwen2.5-VL}}$	–	64.8	56.8	49.7	57.1

Appendix

Table 19: Results on Video-MME using 8 frames. “Subs” indicates whether subtitles are used.

Model	15	60	600	3600	Overall
Qwen3-VL	70.4	64.0	55.1	45.7	54.5
Proact-VL ${}_{\text{Qwen3-VL}}$	66.7	68.6	51.0	44.7	52.8

Appendix

Table 20: Results on LongVideoBench using 8 frames. Columns denote different video duration ranges in seconds.

Panel of Figure 9 — Figure 9: Threshold ablation on CC and F1 across SOLO, Co-Commentary, and Guidance.

Panel of Figure 10 — Figure 10: Window-size (context window) hyperparameter analysis on CC and F1 across SOLO, Co-Commentary, and Guidance.

Figure 11: Solo Commentary Scenario Case 1.

Figure 12: Solo Commentary Scenario Case 2.

Figure 16: Failure Case 2 for Baldur’s Gate 3.

Figure 17: Score distributions of human ratings, LiveU scores, and FinalQ scores across 100 sampled instances.

LLM Score	Method Pair	Strict Match	Strict Acc.
LiveU	Livecc-7B-Base vs. LiveStar	90/96	0.938
LiveU	Livecc-7B-Base vs. Proact-VL	72/85	0.847
LiveU	LiveStar vs. Proact-VL	93/93	1.000
FinalQ	Livecc-7B-Base vs. LiveStar	71/96	0.740
FinalQ	Livecc-7B-Base vs. Proact-VL	68/85	0.800
FinalQ	LiveStar vs. Proact-VL	90/93	0.968

Appendix

Table 21: Pairwise preference alignment between LLM-based scores and human judgments. Strict accuracy excludes cases where human ratings are tied.

Method	Score	Bias	MAE	RMSE	KS	Wass.
Livecc-7B-Base	LiveU	0.070	0.910	1.370	0.090	0.277
Livecc-7B-Base	FinalQ	-0.130	1.543	2.324	0.150	0.417
Livecc-7B-Base	Average	-0.030	1.107	1.534	0.160	0.410
LiveStar	LiveU	-0.283	0.843	1.098	0.320	0.403
LiveStar	FinalQ	-0.843	1.157	1.445	0.580	0.877
LiveStar	Average	-0.563	0.940	1.166	0.500	0.597

Appendix

Table 22: Distribution and calibration difference between LLM-based scores and human ratings. Bias is computed as LLM score minus human score.

Explore similar work

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
Jun 10, 2026Dingyu Yao, Junhao Zhou, Chenxu Yang +17Human-AI Interaction Vision-Language Agents
LiveProBench: Can Streaming Video Models Really Interact Like Humans?
Sep 11, 2026Kaixuan Du, Xin Wan, Hang Zhang +4Streaming Video Understanding Video-Language Model Evaluation
School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba
RIVER: A Real-Time Interaction Benchmark for Video LLMs
Mar 4, 2026Yansong Shi, Qingsong Zhao, Tianxiang Jiang +3Video-Language Model Evaluation Streaming Video Understanding
School of Information Science And Technology, University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · College of Computer Science and Artificial Intelligence, Fudan University +1

Jun 10, 2026J/K move · Enter open · S save

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin+16

Human-AI Interaction Vision-Language Agents

Open paper AskPDF arXiv

Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.

Abstract

Explore similar work

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

LiveProBench: Can Streaming Video Models Really Interact Like Humans?

RIVER: A Real-Time Interaction Benchmark for Video LLMs