Streaming video LLMs can now commentate live gameplay, but they are trained and tested on the same games. We study cross-game streaming commentary, in which a model trained on one game commentates games it has never seen, and find that current commentators transfer their timing but not their knowledge of the game world. We propose StreamDuo, which pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played. When the commentator sees something it cannot name, it sends the current frames, and the research agent returns a few wiki entries two seconds later while the stream continues. We train only the commentator, on a single game, in two stages. Information-gain labels teach it to ask only where an answer would help it say what the human caster said, and on-policy self-distillation from a copy of itself that sees the ideal answer teaches it to use noisy answers. Trained on Baldur's Gate 3 and tested on ten unseen games, StreamDuo raises entity recall from 0.082 to 0.116 over Proact-VL trained on the same data, cuts source-game leakage from 9.9% to 1.0%, and achieves the best timing and the highest judged liveliness and overall quality among streaming models.
Figures & tables
Figure 2: Overview of StreamDuo. Top: on an unseen game, the commentator asks the frozen research agent with the current frames, and the wiki entry it returns two seconds later lets the commentator name what is on screen. Bottom left: Stage 1 labels an ask only where an answer would make the human line more likely. Bottom right: Stage 2 distills a frozen copy of the commentator that sees the ideal answer into the student that sees the noisy one.
Timing
Knowledge
LLM judge
Method
Venue
TimeDiff ↓
F1 ↑
Ent-R ↑
Ent-F1 ↑
K-strict ↑
PAUC ↑
CC ↑
LiveU ↑
FinalQ ↑
Zero-shot
VideoLLM-online [ 6 ]
CVPR’24
11.63
0.063
0.002
0.003
0.001
0.000
0.134
3.49
2.39
LiveCC-7B-Instruct [ 7 ]
CVPR’25
3.03
0.470
0.099
0.139
0.115
0.031
0.127
3.56
2.59
Dispider [ 29 ]
CVPR’25
6.10
0.156
0.043
0.042
0.055
0.006
0.083
3.61
2.78
MMDuet [ 43 ]
EMNLP’25
4.01
0.297
0.017
0.023
0.020
0.002
0.049
3.55
2.50
Table 1: Cross-game streaming commentary. Models are trained on Baldur’s Gate 3 only (or not trained) and evaluated on 2,470 clips from ten unseen games. Knowledge metrics use the first 126 words of each clip (twice the human average), so a model cannot gain recall by talking more. Judge metrics are scored by Qwen3-30B. ‡ Supervised stage of the official recipe only, without RL. Bold : best; underline : second best.
Components
Timing
Knowledge
LLM judge
Rename & dossier
Asking
OPSD
TimeDiff ↓
F1 ↑
Ent-R ↑
Ent-F1 ↑
K-strict ↑
PAUC ↑
CC ↑
LiveU ↑
FinalQ ↑
×
×
×
1.84
0.536
0.082
0.118
0.085
0.050
0.095
4.91
3.32
✓
×
×
1.88
0.523
0.083
0.116
0.090
0.045
0.093
5.03
3.38
✓
✓
×
1.78
0.544
0.092
0.123
0.094
0.052
0.104
5.10
3.48
✓
✓
✓
1.76
0.552
0.116
0.141
0.127
0.055
0.120
5.14
3.53
Table 2: Ablation of StreamDuo components. Rename & dossier: random entity renaming during training plus the game dossier in context. Asking: Stage 1. OPSD: Stage 2. The first row corresponds to the Proact-VL baseline trained on Baldur’s Gate 3. Best results are in bold.
Timing
Knowledge
LLM judge
Setting
TimeDiff ↓
F1 ↑
Ent-R ↑
Ent-F1 ↑
K-strict ↑
PAUC ↑
CC ↑
LiveU ↑
FinalQ ↑
Stage 1 ask labels
Naive
1.903
0.534
0.079
0.107
0.087
0.046
0.098
5.01
3.38
Info. gain
1.784
0.544
0.092
0.123
0.094
0.052
0.104
5.10
3.48
Test-time asking
Never
1.764
0.553
0.104
0.127
0.114
0.051
0.117
5.14
3.54
Table 3: When to ask. Top: Stage 1 ask labels, placed at every human entity mention (naive) or only where the answer yields an information gain of γ=1 nat (info. gain). Bottom: test-time asking policy of the full model. Shaded rows are the settings used by StreamDuo; best results within each block are in bold.
Figure 3: Entity naming around questions (orange) and random seconds (blue). The answer arrives at 2 s.
Figure 4: Stage 2 variants. Dashed line: Stage 1.
Knowledge
LLM judge
Research agent
Answers
Ent-R ↑
Ent-F1 ↑
K-strict ↑
LiveU ↑
FinalQ ↑
Qwen3-VL-2B
18,103
0.101
0.125
0.107
5.14
3.53
Qwen3-VL-4B
50,445
0.105
0.129
0.111
5.13
3.52
Qwen3-VL-8B
52,548
0.116
0.141
0.127
5.14
3.53
Qwen3-VL-30B-A3B
54,888
0.107
0.129
0.117
5.15
3.55
Table 4: Swapping the research agent at test time without retraining the commentator. All agents answer the same moments, every second of the 2,470 test clips (80,529 in total). Answers: number of these moments for which the agent returns an answer. The shaded row is the default agent used by StreamDuo. Best results are in bold.
Figure 5: Examples on unseen games, with the frame at the second the research agent’s answer arrives. Orange entries are used by the commentator, and gray ones are not. Proact-VL is retrained on the same data and shown at the same seconds, and names from Baldur’s Gate 3 are in red.
Method
Mentions ↑
Leaked ↓
Leakage ↓
VideoLLM-online [ 6 ]
372
14
3.6%
+ agent dossier
676
1
0.1%
LiveCC-7B-Instruct [ 7 ]
217
2
0.9%
+ agent dossier
867
4
0.5%
MMDuet [ 43 ]
298
0
0.0%
+ agent dossier
99
1
1.0%
Table 5: Source-game leakage on all 2,470 test clips, without a word cap. Mentions: names of the test game. Leaked: Baldur’s Gate 3 names that exist neither in the test game nor in its wiki. Leakage: share of leaked names among all names said.
Stage
Mean
Median
P90
Max
Research agent (s per answer)
Propose search keys
0.322
0.272
0.594
0.628
BM25 wiki search
0.002
0.001
0.005
0.011
Pick entries
0.082
0.091
0.114
0.188
Total
0.405
0.363
0.643
0.713
Commentator (s per video second)
Table 6: Latency on one NVIDIA H100 GPU, one request at a time. Research agent: seconds per answer over 200 test moments. Commentator: seconds of compute per second of video over 20 test clips, including frame decoding.
Figure 6: More examples from the ten unseen games, with StreamDuo and Proact-VL at the same seconds (1/2). Chips: the entries in the research agent’s answer, in orange those the commentator names. Red: names from Baldur’s Gate 3.
Figure 7: More examples from the ten unseen games (2/2).
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
A live esports broadcast needs a director who chooses which camera goes on air and a caster who talks over that choice, every second and without knowing what comes next. Streaming video language models have learned to commentate, but only over a broadcast that a human director has already produced. We study live broadcast production, in which a model receives the raw camera of every player and must both direct and commentate from the past alone. We present DirCast, a single streaming video language model that does both. Every second it scans all cameras at low resolution, decides with a hold-or-switch director whether and where to cut, and reads the chosen camera at high resolution before it speaks. Its context keeps what went on air and what was said and lets the other views expire, which bounds the cost of a stream of any length. We build a dataset of 27 professional Dota 2 matches with a camera for every hero, the human director's choices and the casters' commentary. DirCast directs better than rule-based and zero-shot directors including GPT-5, and its commentary wins 29.2 percent of comparisons against GPT-4o under the Proact-VL protocol, against 19.4 percent for the strongest of eleven streaming commentators that watch the human broadcast.
Zixian Gao, Lixin Xiu, Penghao Luo +5
Alaya Lab · The University of Tokyo · Georgia Institute of Technology +2
Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.
Anum Afzal, Yuki Saito, Hiroya Takamura +5
School of CIT, Technical University of Munich · National Institute of Advanced Industrial Science and Technology (AIST) · The University of Tokyo +3
Streaming video understanding demands more than watching longer videos: assistants must decide when to speak in real time, balancing responsiveness against verbosity. Yet most video-language models (VideoLLMs) are trained for offline inference, and existing streaming benchmarks externalize this timing decision to the evaluator. We address this gap with RealStreamEval, a frame-level multi-turn evaluation protocol that exposes models to sequential observations and penalizes unnecessary responses. Under this protocol, we observed that strong offline VideoLLMs retain useful visual understanding but lack an interaction policy for deciding when to respond. Motivated by this observation, we propose EvoStreaming, a self-evolved streaming adaptation framework in which the base model itself acts as data generator, relevance annotator, and roll-out policy to synthesize streaming trajectories without external supervision. With only 1,000 self-generated samples (139× less than the leading streaming instruction-tuning approach) and no architectural changes, EvoStreaming consistently improves the overall RealStreamEval score by up to 10.8 points across five open VideoLLM backbones (Qwen2/2.5/3-VL, InternVL-3.5, MiniCPM-V4.5) while largely preserving offline video performance. These results suggest that data-efficient interaction tuning is a practical path for adapting existing VideoLLMs to streaming assistants.
Zichen Wen, Boxue Yang, Junlong Ke +5
EPIC Lab, Shanghai Jiao Tong University · Tsinghua University · The Hong Kong University of Science and Technology (Guangzhou) +1