cs.CVOct 31, 2026

StreamDuo: A Live Commentator and a Research Agent for Unseen Games

Authors: Zixian Gao, Penghao Luo, Xiangcheng Cao, Lixin Xiu, Ziqi Cai, Zhen Li, Erwin Wu, Kaipeng Zhang

Organizations: Alaya Lab · The University of Tokyo · Georgia Institute of Technology · Peking University · 

Abstract

Streaming video LLMs can now commentate live gameplay, but they are trained and tested on the same games. We study cross-game streaming commentary, in which a model trained on one game commentates games it has never seen, and find that current commentators transfer their timing but not their knowledge of the game world. We propose StreamDuo, which pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played. When the commentator sees something it cannot name, it sends the current frames, and the research agent returns a few wiki entries two seconds later while the stream continues. We train only the commentator, on a single game, in two stages. Information-gain labels teach it to ask only where an answer would help it say what the human caster said, and on-policy self-distillation from a copy of itself that sees the ideal answer teaches it to use noisy answers. Trained on Baldur's Gate 3 and tested on ten unseen games, StreamDuo raises entity recall from 0.082 to 0.116 over Proact-VL trained on the same data, cuts source-game leakage from 9.9% to 1.0%, and achieves the best timing and the highest judged liveliness and overall quality among streaming models.

Figures & tables

Explore similar work

CardsList
  1. DirCast: Live Esports Directing and Commentary with a Single Streaming Video-Language Model

    Oct 22, 2026Zixian Gao, Lixin Xiu, Penghao Luo +5Video-Language ModelsStreaming Video Understanding

  2. Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

    Mar 3, 2026Anum Afzal, Yuki Saito, Hiroya Takamura +5Language Model DecodingMultimodal Large Language Models

  3. EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistant

    May 11, 2026Zichen Wen, Boxue Yang, Junlong Ke +5Streaming Video UnderstandingSelf-Training