Video generation has progressed from isolated clips to coherent short-form audiovisual sequences, and the bottleneck is shifting from rendering to semantic control. This report summarizes the observations and insights on multimodal understanding gained from successive generations of Kling. We argue that generation quality is the projection of multimodal understanding through the entire data--conditioning--alignment--evaluation chain, and we examine this chain module by module. Kling-MLLM, a shared multimodal foundation model that unifies image, video, audio, synchronized audio--video, and interleaved inputs, is specialized for each downstream module. Because a DiT is trained on far fewer training samples than a language model, we treat captioning as a semantic contract that maximizes description density, and extend it to reference-grounded captioning that makes reference bindings explicit. Prompt enhancement translates user intent into this caption distribution: basic PE is derived from captions of compelling videos and refined with text-level and generator-in-the-loop reinforcement learning, while agentic PE refines the story and retrieves compelling examples. We show that a reliable text encoder comes from perceptual training rather than from stronger language ability, and that candidates can be screened from text alone before end-to-end training. Reward modeling pairs a general reward model for explainable, localized critiques with a decision reward model for fast multidimensional scoring, treating perceptual bandwidth as a first-class design parameter. Finally, we review task-specific automatic evaluation, focusing on protocols that expose scoring shortcuts and help trace failures to their upstream source. We hope these lessons offer practical guidance for building the understanding stack behind controllable video generation.
Figures & tables
Figure 2: Kling-MLLM architecture. Multimodal inputs are encoded by visual and audio paths into aligned continuous tokens, delimited by modality boundary markers, anchored with physical timestamps, and interleaved with text before entering the LLM backbone. The audio path illustrates the dual-branch Mel–VAE tower used in larger variants. ‘Sync A-V’ denotes synchronized audio–video input.
Modality
Conversion-time normalization and validation
Representative supervision
Text
Encoding & language checks, document deduplication, conversational turn normalization
Table 2: Audio benchmarks. Paired values normally list our evaluation result followed by the model developer’s reported result; for LATBench, values separated by “ | ” denote Chinese and English results, respectively.
Model
OmniBench ↑
video-SALMONN
UGC-VideoCap
OmniCap-IF
Multimodal interleaving
Miss ↓
Hallucination ↓
Total error ↓
CSR ↑
ISR ↑
IV-Bench ↑
MVU-Eval ↑
Qwen3-Omni-30B-A3B-Instruct
60.35
36.32
16.87
53.20
80.01
65.87
9.06
35.02
56.41
MiniCPM-o-4.5
52.04
33.37
15.35
48.72
78.45
70.84
11.69
24.70
47.81
Qwen3.5-Omni-Plus (DashScope)
67.61
17.47
21.67
39.13
87.78
75.90
18.95
53.31
60.64
Doubao2.0-Lite (non-thinking)
57.79
28.20
17.26
45.46
84.22
76.56
20.29
43.48
60.42
Doubao2.0-Lite (thinking, high)
70.18
19.47
19.97
39.45
91.05
78.21
21.62
61.41
63.71
Table 3: Audio–video and multimodal-interleaving benchmarks.
Model
MMBench
OCRBench ↑
TextPecker
PerceptionBench ↑
EN (dev) ↑
CN (dev) ↑
EN + ↑
EN - ↓
CN + ↑
CN - ↓
Qwen3-Omni-30B-A3B-Instruct
87.29
83.68
86.80
0.032
0.078
0.047
0.041
32.17
MiniCPM-o-4.5
85.22
82.99
61.00 / 87.6
0.094
0.099
0.024
0.085
29.37
Gemma-4-31B-IT
87.11
87.11
84.70
0.252
0.054
0.187
0.072
47.23
Qwen3-VL-32B
87.63
86.17
84.80 / 89.5
0.159
0.166
0.171
0.069
33.97
Qwen3.5-Omni-Plus (DashScope)
91.24
88.75
85.20
0.106
0.060
0.072
0.038
37.50
Table 4: Image benchmarks.
Model
LongVideo Bench ↑
MVBench ↑
MME- VideoOCR ↑
VSI- Bench ↑
ShortVid- Bench ↑
DSI- Bench ↑
Event- Bench ↑
Motion- Bench ↑
TimeLens
CameraMotion VQA ↑
VidCapBench ↑
Overall ↑
Charades ↑
QVHighlights ↑
ActivityNet ↑
Qwen3-Omni-30B-A3B-Instruct
64.70
70.00
62.35
61.12
63.90
51.95
69.91
61.57
12.64
14.90
8.77
14.26
31.68
17.78|84.24
MiniCPM-o-4.5
64.85 / 66.0
64.00
62.20
50.38
64.60
26.23
66.48
60.75 / 61.4
10.01
9.21
9.62
11.19
23.53
15.22|79.16
Gemma-4-31B-IT
61.78
64.40
68.25
48.10
68.50
55.23
65.11
61.05
51.98
46.66
61.80
47.49
28.09
16.64|90.17
Qwen3-VL-32B
67.61
73.65 / 72.8
75.25
64.16 / 61.5
67.30
53.36
74.89
67.32
52.16
52.79
55.71
47.99
26.71
20.01|87.49
Qwen3.5-Omni-Plus (DashScope)
69.26
80.37
79.20
60.67
70.10
59.36
76.62
71.55
33.31
43.14
25.92
30.88
37.70
18.29|92.60
Table 5: Video benchmarks. For VidCapBench, values separated by “ | ” denote Accuracy and Coverage.
Figure 3: Two captions of a 17.7 s clip from Meridian . Circled numbers link seven facts across the frames, timeline, and captions. The fluent caption reads well but omits or misrepresents key details. The generation-oriented caption preserves shot timing, audiovisual alignment, and visible text, including mirror-reversed lettering.
Information family
Required caption elements
Downstream failure if omitted or corrupted
Subject & action
Distinct entities, action phases, interaction objects, resulting states
Hallucinated entities, actions, relations, or text
Beneficial diversity
Use uncontrolled synonyms or poetic terms
Fragmented TE distribution → unstable control
Internal consistency
Reuse rigid templates; ignore local nuances
Lost dynamic state changes → robotic generation
Table 7: Shortcuts from optimizing each caption criterion in isolation.
Model
VidCapBench
UGC-VideoCap
Motion (Internal)
Camera (Internal)
Sound (Internal)
Acc.
Pre.
Cov.
Con.
Audio
Visual
Detail
Avg.
Pre.
Rec.
Pre.
Rec.
Pre.
Rec.
Gemini-4-Argon
16.20
51.66
85.63
10.69
79.80
87.69
85.67
84.38
64.62
45.78
40.83
76.22
79.74
29.38
Doubao2.1-Lite (non-thinking)
17.20
51.78
87.47
8.32
82.88
90.67
88.81
87.45
62.87
56.31
28.37
61.46
73.18
26.79
Doubao2.1-Lite (thinking, high)
20.23
56.44
94.15
3.36
90.37
96.10
94.97
93.81
64.15
59.02
32.89
64.61
77.56
31.69
Kling-Captioner-35B-A3B
21.84
59.28
94.81
3.56
92.83
96.46
95.90
95.06
67.28
64.75
52.18
80.93
78.52
55.83
Kling-Captioner-397B-A17B
21.70
60.51
96.05
3.19
93.41
96.70
96.38
95.50
68.11
67.42
54.82
83.43
80.21
58.11
Table 8: Captioning benchmarks.
Figure 4: Reference-grounded captioning on a 22.5 s clip from Meridian , given two portraits (IMG_1, IMG_2) and a voice track (AUD_1). The caption states each binding with its scope and writes out everything the references cannot supply. The timeline shows who is on screen and when PERSON_1 speaks. Circled numbers mark where each property of the semantic contract applies, contrasting typical shortcuts with correct formulations.
Product task
Typical references
Binding question
Residual target facts
Video editing
Source video, instruction, optional local ref.
Edited entity/region and protected properties
Requested state change, time span, secondary effects
Multi-reference generation
Character, object, scene, style, voice, or motion refs.
Reference governing each role, attribute, and modality
Bind speaker/event IDs to visible actions; no unseen visual hallucination
Text
Edit instruction, script, names, constraints
Conflicts with sources or over-specifies details
Treat as typed change spec.; resolve authority at attribute level
Mixed set
Joint composition of the above
Overlapping authority, duplication, contradiction
Build attribute-level authority map before composing final caption
Table 11: Authority and ambiguity of each reference modality.
Model
ViDiC
MRVBench
Avg.
Diff.
Sim.
KP-Cov
VQA
Ref-Tag-R
Ref-Bind
Subj-F1
MRVScore
Gemini-4-Argon
77.18
75.51
81.86
93.31
74.41
97.23
93.56
90.37
91.22
Doubao2.1-Lite (non-thinking)
69.24
65.27
80.33
89.45
68.09
75.43
93.35
49.45
78.10
Doubao2.1-Lite (thinking, high)
75.63
74.96
77.49
90.75
71.55
81.41
93.98
56.86
81.32
Kling-RGC-35B-A3B
77.57
75.59
83.06
94.12
76.01
94.80
92.88
84.51
88.57
Kling-RGC-397B-A17B
78.89
76.29
86.12
95.18
76.34
95.28
94.85
85.24
90.31
Table 12: RGC benchmarks.
Figure 5: Overview of PE. Top: three training stages of basic PE. Stage 1 learns the mapping from real-format user prompts to captions of compelling videos, using only pairs that pass faithfulness, reasonableness, and vividness (F/R/V) verification; Stage 2 optimizes the same three properties with text-only rewards; Stage 3 optimizes basic PE through the frozen text encoder and DiT with video-level rewards. Bottom: at inference, agentic PE makes the plot coherent and retrieves five captions of compelling videos; basic PE writes the final condition, using the retrieved captions as optional evidence.
Figure 6: Three levels of caption alignment contrasting generic LLM rewrites with caption-aligned PE.
Setting
Text metrics
Video metrics
Faithfulness
Reasonableness
Vividness
StoryEval
T2AV- Compass
AVE- Compass
MultiRef- Compass
Avg.
≥ 4 (%)
Avg.
≥ 4 (%)
Avg.
≥ 4 (%)
Basic PE
SFT, random-source captions
4.02
75.42
3.95
69.17
3.72
62.33
65.42
70.15
56.2
4.6137
+ compelling-video sources
4.00
74.92
3.92
68.08
4.03
72.58
66.87
70.60
56.7
4.6224
+ F/R/V data verification
4.27
80.83
4.18
74.33
4.11
75.08
71.93
73.07
57.9
4.6591
Table 13: Cumulative contribution of PE components.
Kling-MLLM TE configuration
Overall
Instruction Following
Aesthetics
Multimodal adaptation
1.10
1.04
1.11
+ Multi-granular conditioning
1.22
1.11
1.22
+ Conditioning stability
1.27
1.14
1.25
Table 14: Human preference evaluation ( (G+S)/(S+B) ) relative to the Qwen baseline.
Figure 7: Three-stage training pipeline for our RMs, starting from Kling-MLLM-27B. Stage 1 adapts the vision encoder and aligner with the LLM frozen for low-level artifact perception; Stage 2 trains the General Reward Model (GRM) with cold-start SFT and RL with verifiable rewards for explainable evaluation; Stage 3 initializes the Decision Reward Model (DRM) from the GRM and performs decision-head warm-up, unified pointwise–pairwise SFT with TA grounding, and symmetry calibration with gated inference for fast multi-dimensional scoring.
Despite remarkable progress toward general-purpose video models, a critical question remains unanswered: how far are these models from achieving true multimodal reasoning? Existing benchmarks fail to address this question rigorously, as they remain constrained by straightforward task designs and fragmented evaluation metrics that neglect complex multimodal reasoning. To bridge this gap, we introduce CLVG-Bench, an evaluation framework designed to probe video models' zero-shot reasoning capabilities via Context Learning in Video Generation. CLVG-Bench comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories, covering complex scenarios including physical simulation, logical reasoning, and interactive contexts. To enable rigorous and scalable assessment, we further propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations, delivering interpretable textual feedback across diverse video context tasks. Extensive experiments reveal a striking answer to our central question: while state-of-the-art (SOTA) video models, such as Seedance 2.0, demonstrate competence on certain understanding and reasoning subtasks, they fall substantially short with logically grounded and interactive generation tasks (achieving success rates <25% and ~0%, respectively), exposing multimodal reasoning and physical grounding as critical bottlenecks. By systematically quantifying these limitations, the proposed method provides actionable feedbacks and a clear roadmap toward truly robust, general-purpose video models. CLVG-Bench and code are released here.
Unified multimodal models integrating visual understanding and generation face a fundamental challenge: visual generation incurs substantially higher computational costs than understanding, particularly for video. This imbalance motivates us to invert the conventional paradigm: rather than extending understanding-centric MLLMs to support generation, we propose Uni-ViGU, a framework that unifies video generation and understanding by extending a video generator as the foundation. We introduce a unified flow method that performs continuous flow matching for video and discrete flow matching for text within a single process, enabling coherent multimodal generation. We further propose a modality-driven MoE-based framework that augments Transformer blocks with lightweight layers for text generation while preserving generative priors. To repurpose generation knowledge for understanding, we design a bidirectional training mechanism with two stages: Knowledge Recall reconstructs input prompts to leverage learned text-video correspondences, while Capability Refinement fine-tunes on detailed captions to establish discriminative shared representations. Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation and understanding, validating generation-centric architectures as a scalable path toward unified multimodal intelligence. Project Page and Code: https://fr0zencrane.github.io/uni-vigu-page/.
Luozheng Qin, Jia Gong, Qian Qiao +6
Shanghai Academy of AI for Science · Independent Researcher · Singapore University of Technology and Design +1
A recent cutting-edge topic in multimodal modeling is to unify visual comprehension and generation within a single model. However, the two tasks demand mismatched decoding regimes and visual representations, making it non-trivial to jointly optimize within a shared feature space. In this work, we present Cheers, a unified multimodal model that decouples patch-level details from semantic representations, thereby stabilizing semantics for multimodal understanding and improving fidelity for image generation via gated detail residuals. Cheers includes three key components: (i) a unified vision tokenizer that encodes and compresses image latent states into semantic tokens for efficient LLM conditioning, (ii) an LLM-based Transformer that unifies autoregressive decoding for text generation and diffusion decoding for image generation, and (iii) a cascaded flow matching head that decodes visual semantics first and then injects semantically gated detail residuals from the vision tokenizer to refine high-frequency content. Experiments on popular benchmarks demonstrate that Cheers matches or surpasses advanced UMMs in both visual understanding and generation. Cheers also achieves 4x token compression, enabling more efficient high-resolution image encoding and generation. Notably, Cheers outperforms the Tar-1.5B on the popular benchmarks GenEval and MMBench, while requiring only 20% of the training cost, indicating effective and efficient (i.e., 4x token compression) unified multimodal modeling. We will release all code and data for future research.
Yichen Zhang, Da Peng, Zonghao Guo +19
Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences