LOLGORITHM: Funny Comment Generation Agent For Short Videos
Authors: Xuan Ouyang, Bouzhou Wang, Senan Wang, Siyuan Xiahou, Jinrong Zhou, Yuekang Li
Organizations: University of New South Wales Sydney, Australia · University of Sydney Sydney, Australia · The University of Hong Kong Hong Kong, China · University of Southern California Los Angeles, USA
Short-form video platforms have become central to multimedia information dissemination, where comments play a critical role in driving engagement, propagation, and algorithmic feedback. However, existing approaches -- including video summarization and live-streaming danmaku generation -- fail to produce authentic comments that conform to platform-specific cultural and linguistic norms. In this paper, we present LOLGORITHM, a novel modular multi-agent framework for stylized short-form video comment generation. LOLGORITHM supports six controllable comment styles and comprises three core modules: video content summarization, video classification, and comment generation with semantic retrieval and hot meme augmentation. We further construct a bilingual dataset of 3,267 videos and 16,335 comments spanning five high-engagement categories across YouTube and Douyin. Evaluation combining automatic scoring and large-scale human preference analysis demonstrates that LOLGORITHM consistently outperforms baseline methods, achieving human preference selection rates of 80.46% on YouTube and 84.29% on Douyin across 107 respondents. Ablation studies confirm that these gains are attributable to the framework architecture rather than the choice of backbone LLM, underscoring the robustness and generalizability of our approach.
Figures & tables
Figure 1. Workflow of the LOLGORITHM framework …
Figure 2. A Single Composite Image Example …
Platform
TS
HC
FA
DLJ
CS
Other
YouTube
3
3
3
4
3
4
Douyin
3
4
2
4
3
4
Table 1. Details of the videos used for benchmarking. TS: Talk Show; HC: Humorous Commentary; FA: Funny Animal; DLJ: Daily Life Jokes; CS: Comedy Skits.
Model
Orig.
Rel.
Style
Total
V2Xum-LLM
1.38
8.24
1.10
3.57
LiveChat
1.21
7.49
5.44
4.71
Qwen3.5-9b (Direct)
1.75
7.66
2.42
3.94
LOLGORITHM
1.75
7.98
6.59
5.44
Table 2. Scoring results on Douyin dataset.
Model
Orig.
Rel.
Style
Total
V2Xum-LLM
2.73
8.03
1.75
4.17
LiveChat
2.37
5.14
6.77
4.76
Qwen3.5-9b (Direct)
2.21
5.40
4.09
3.90
LOLGORITHM
3.26
7.70
4.76
4.90
Table 3. Scoring results on YouTube dataset.
Model
Orig.
Rel.
Style
Total
Qwen3.5-9b (Direct)
1.75
7.66
2.42
3.94
DeepSeek-R1
1.22
8.33
4.37
4.64
LLaMa4
1.29
8.26
5.43
4.99
LOLGORITHM (Qwen3.5-9b)
1.75
7.98
6.59
5.44
GPT-5.4
1.43
8.25
7.45
5.71
Table 4. Impact of backbone LLM on Douyin dataset.
Model
Orig.
Rel.
Style
Total
Qwen3.5-9b (Direct)
2.21
5.40
4.09
3.90
LOLGORITHM (Qwen3.5-9b)
2.25
7.70
4.76
4.90
GPT-5.4
2.27
7.78
4.73
4.93
LLaMa4
2.09
7.73
5.49
5.11
DeepSeek-R1
2.32
8.04
5.10
5.15
Table 5. Impact of backbone LLM on YouTube dataset.
Figure 3. Human evaluation results for comment generation.
Figure 4. Example comments generated by different techniques.
Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.
Anum Afzal, Yuki Saito, Hiroya Takamura +5
School of CIT, Technical University of Munich · National Institute of Advanced Industrial Science and Technology (AIST) · The University of Tokyo +3
Online comments play a crucial role in shaping public sentiment and opinion dynamics on social media. However, evaluating their popularity remains challenging, not only because it depends on linguistic quality, originality, and emotional resonance, but also because stylistic preferences vary widely across platforms and user groups, causing the same comment to resonate differently in different communities. In this work, we present HotComment, a multimodal benchmark integrating video and text modalities that comprehensively quantifies popularity from three enhanced aspects: (1) Content Quality, which evaluates semantic similarity with ground-truth human comments and extends quality assessment through four interpretable dimensions; (2) Popularity Prediction, based on trends from models trained on real-world interaction data; and (3) User Behavior Simulation, which models the distribution of platform users and approximates engagement scores through an agent-based framework. Furthermore, we propose StyleCmt, inspired by social ripple effects, where multiple stylistic dimensions align to amplify socially resonant expressions and suppress incongruent ones.
Yafeng Wu, Yunyao Zhang, Liliang Ye +4
Huazhong University of Science and Technology Wuhan, China · Beijing Institute of Computer Technology and Applications Beijing, China
We propose a fully automated AI system that produces short comedic videos similar to sketch shows such as Saturday Night Live. Starting from character references, the system employs a population of agents loosely modeled on roles in real production studios and structured to optimize the quality and diversity of ideas and outputs through iterative competition, evaluation, and refinement. A key contribution is the introduction of LLM-based critics aligned with real viewer preferences through the analysis of a corpus of comedy videos on YouTube, enabling automatic evaluation of humor. We further propose multi-island relativistic evolution for script improvement, along with script-conditioned rendering critics that perform shot- and video-level tournaments. In both human and automated evaluations, COMIC receives higher scores than agentic and video generation baselines.
Susung Hong, Brian Curless, Ira Kemelmacher-Shlizerman +1