Streaming video LLMs can now commentate live gameplay, but they are trained and tested on the same games. We study cross-game streaming commentary, in which a model trained on one game commentates games it has never seen, and find that current commentators transfer their timing but not their knowledge of the game world. We propose StreamDuo, which pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played. When the commentator sees something it cannot name, it sends the current frames, and the research agent returns a few wiki entries two seconds later while the stream continues. We train only the commentator, on a single game, in two stages. Information-gain labels teach it to ask only where an answer would help it say what the human caster said, and on-policy self-distillation from a copy of itself that sees the ideal answer teaches it to use noisy answers. Trained on Baldur's Gate 3 and tested on ten unseen games, StreamDuo raises entity recall from 0.082 to 0.116 over Proact-VL trained on the same data, cuts source-game leakage from 9.9% to 1.0%, and achieves the best timing and the highest judged liveliness and overall quality among streaming models.
Figures & tables
Figure 2: Overview of StreamDuo. Top: on an unseen game, the commentator asks the frozen research agent with the current frames, and the wiki entry it returns two seconds later lets the commentator name what is on screen. Bottom left: Stage 1 labels an ask only where an answer would make the human line more likely. Bottom right: Stage 2 distills a frozen copy of the commentator that sees the ideal answer into the student that sees the noisy one.
Timing
Knowledge
LLM judge
Method
Venue
TimeDiff ↓
F1 ↑
Ent-R ↑
Ent-F1 ↑
K-strict ↑
PAUC ↑
CC ↑
LiveU ↑
FinalQ ↑
Zero-shot
VideoLLM-online [ 6 ]
CVPR’24
11.63
0.063
0.002
0.003
0.001
0.000
0.134
3.49
2.39
LiveCC-7B-Instruct [ 7 ]
CVPR’25
3.03
0.470
0.099
0.139
0.115
0.031
0.127
3.56
2.59
Dispider [ 29 ]
CVPR’25
6.10
0.156
0.043
0.042
0.055
0.006
0.083
3.61
2.78
MMDuet [ 43 ]
EMNLP’25
4.01
0.297
0.017
0.023
0.020
0.002
0.049
3.55
2.50
Table 1: Cross-game streaming commentary. Models are trained on Baldur’s Gate 3 only (or not trained) and evaluated on 2,470 clips from ten unseen games. Knowledge metrics use the first 126 words of each clip (twice the human average), so a model cannot gain recall by talking more. Judge metrics are scored by Qwen3-30B. ‡ Supervised stage of the official recipe only, without RL. Bold : best; underline : second best.
Components
Timing
Knowledge
LLM judge
Rename & dossier
Asking
OPSD
TimeDiff ↓
F1 ↑
Ent-R ↑
Ent-F1 ↑
K-strict ↑
PAUC ↑
CC ↑
LiveU ↑
FinalQ ↑
×
×
×
1.84
0.536
0.082
0.118
0.085
0.050
0.095
4.91
3.32
✓
×
×
1.88
0.523
0.083
0.116
0.090
0.045
0.093
5.03
3.38
✓
✓
×
1.78
0.544
0.092
0.123
0.094
0.052
0.104
5.10
3.48
✓
✓
✓
1.76
0.552
0.116
0.141
0.127
0.055
0.120
5.14
3.53
Table 2: Ablation of StreamDuo components. Rename & dossier: random entity renaming during training plus the game dossier in context. Asking: Stage 1. OPSD: Stage 2. The first row corresponds to the Proact-VL baseline trained on Baldur’s Gate 3. Best results are in bold.
Timing
Knowledge
LLM judge
Setting
TimeDiff ↓
F1 ↑
Ent-R ↑
Ent-F1 ↑
K-strict ↑
PAUC ↑
CC ↑
LiveU ↑
FinalQ ↑
Stage 1 ask labels
Naive
1.903
0.534
0.079
0.107
0.087
0.046
0.098
5.01
3.38
Info. gain
1.784
0.544
0.092
0.123
0.094
0.052
0.104
5.10
3.48
Test-time asking
Never
1.764
0.553
0.104
0.127
0.114
0.051
0.117
5.14
3.54
Table 3: When to ask. Top: Stage 1 ask labels, placed at every human entity mention (naive) or only where the answer yields an information gain of γ=1 nat (info. gain). Bottom: test-time asking policy of the full model. Shaded rows are the settings used by StreamDuo; best results within each block are in bold.
Figure 3: Entity naming around questions (orange) and random seconds (blue). The answer arrives at 2 s.
Figure 4: Stage 2 variants. Dashed line: Stage 1.
Knowledge
LLM judge
Research agent
Answers
Ent-R ↑
Ent-F1 ↑
K-strict ↑
LiveU ↑
FinalQ ↑
Qwen3-VL-2B
18,103
0.101
0.125
0.107
5.14
3.53
Qwen3-VL-4B
50,445
0.105
0.129
0.111
5.13
3.52
Qwen3-VL-8B
52,548
0.116
0.141
0.127
5.14
3.53
Qwen3-VL-30B-A3B
54,888
0.107
0.129
0.117
5.15
3.55
Table 4: Swapping the research agent at test time without retraining the commentator. All agents answer the same moments, every second of the 2,470 test clips (80,529 in total). Answers: number of these moments for which the agent returns an answer. The shaded row is the default agent used by StreamDuo. Best results are in bold.
Figure 5: Examples on unseen games, with the frame at the second the research agent’s answer arrives. Orange entries are used by the commentator, and gray ones are not. Proact-VL is retrained on the same data and shown at the same seconds, and names from Baldur’s Gate 3 are in red.
Method
Mentions ↑
Leaked ↓
Leakage ↓
VideoLLM-online [ 6 ]
372
14
3.6%
+ agent dossier
676
1
0.1%
LiveCC-7B-Instruct [ 7 ]
217
2
0.9%
+ agent dossier
867
4
0.5%
MMDuet [ 43 ]
298
0
0.0%
+ agent dossier
99
1
1.0%
Table 5: Source-game leakage on all 2,470 test clips, without a word cap. Mentions: names of the test game. Leaked: Baldur’s Gate 3 names that exist neither in the test game nor in its wiki. Leakage: share of leaked names among all names said.
Stage
Mean
Median
P90
Max
Research agent (s per answer)
Propose search keys
0.322
0.272
0.594
0.628
BM25 wiki search
0.002
0.001
0.005
0.011
Pick entries
0.082
0.091
0.114
0.188
Total
0.405
0.363
0.643
0.713
Commentator (s per video second)
Table 6: Latency on one NVIDIA H100 GPU, one request at a time. Research agent: seconds per answer over 200 test moments. Commentator: seconds of compute per second of video over 20 test clips, including frame decoding.
Figure 6: More examples from the ten unseen games, with StreamDuo and Proact-VL at the same seconds (1/2). Chips: the entries in the research agent’s answer, in orange those the commentator names. Red: names from Baldur’s Gate 3.
Figure 7: More examples from the ten unseen games (2/2).
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.