Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling

Kling Team; Dong Chen; Jialu Chen; Xinlong Chen; Yifan Dai; Haotian Dong; Binzong Geng; Yushuo Guan

cs.CVNov 3, 2026

Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling

Authors: Kling Team, Dong Chen, Jialu Chen, Xinlong Chen, Yifan Dai, Haotian Dong, Binzong Geng, Yushuo Guan, Minghui Guo, Guanlin He, Guanxiao He, Jingyun Hua, Yanwei Huang, Yiyan Ji, Xueping Jia, Yizhuo Jia, Bozhou Li, Jiaze Li, Min Li, Shihao Li, Shujie Li, Shengyi Liao, Weihong Lin, Shiyuan Liu, Xinyu Liu, Siyang Mao, Xiaoru Qu, Dianmo Sheng, Yang Shi, Jiafu Tang, Longxiang Tang, Yuqi Tang, Chengzhuo Tong, Kuo Wang, Qixun Wang, Rui Wang, Tianduo Wang, Yuran Wang, Zekun Wang, Zihao Wang, Boyang Xia, Yifan Xu, Liu Yang, Zhenlong Yuan, Bohan Zeng, Mujin Zhang, Yuanxing Zhang, Zhicheng Zhang, Zhuoran Zhang, Yuanhong Zheng, Kun Gai, Chengru Song, Pengfei Wan

Abstract

Video generation has progressed from isolated clips to coherent short-form audiovisual sequences, and the bottleneck is shifting from rendering to semantic control. This report summarizes the observations and insights on multimodal understanding gained from successive generations of Kling. We argue that generation quality is the projection of multimodal understanding through the entire data--conditioning--alignment--evaluation chain, and we examine this chain module by module. Kling-MLLM, a shared multimodal foundation model that unifies image, video, audio, synchronized audio--video, and interleaved inputs, is specialized for each downstream module. Because a DiT is trained on far fewer training samples than a language model, we treat captioning as a semantic contract that maximizes description density, and extend it to reference-grounded captioning that makes reference bindings explicit. Prompt enhancement translates user intent into this caption distribution: basic PE is derived from captions of compelling videos and refined with text-level and generator-in-the-loop reinforcement learning, while agentic PE refines the story and retrieves compelling examples. We show that a reliable text encoder comes from perceptual training rather than from stronger language ability, and that candidates can be screened from text alone before end-to-end training. Reward modeling pairs a general reward model for explainable, localized critiques with a decision reward model for fast multidimensional scoring, treating perceptual bandwidth as a first-class design parameter. Finally, we review task-specific automatic evaluation, focusing on protocols that expose scoring shortcuts and help trace failures to their upstream source. We hope these lessons offer practical guidance for building the understanding stack behind controllable video generation.

Figures & tables

Figure 2: Kling-MLLM architecture. Multimodal inputs are encoded by visual and audio paths into aligned continuous tokens, delimited by modality boundary markers, anchored with physical timestamps, and interleaved with text before entering the LLM backbone. The audio path illustrates the dual-branch Mel–VAE tower used in larger variants. ‘Sync A-V’ denotes synchronized audio–video input.

Modality	Conversion-time normalization and validation	Representative supervision
Text	Encoding & language checks, document deduplication, conversational turn normalization	Next-token prediction, instruction following, reasoning
Image	Decode & resolution checks, perceptual deduplication, aspect-ratio handling, OCR/metadata extraction	Short & dense captions, VQA, OCR, grounding, counting, pose
Video	Decode check, shot segmentation, timestamped frame selection, optional OCR; audio routed to sync branch	Temporal captioning, event & motion QA, camera understanding, multi-shot reasoning
Audio	Waveform/channel normalization, segmentation, VAD & signal-quality checks, transcript alignment	ASR, audio captioning & QA, acoustic-event reasoning, temporal grounding
Audio–video	Synchronization validation, shared timestamped token timeline construction	Joint transcription, cross-modal QA, grounding, captioning
Interleaved media	Document/dialogue order preservation, media-reference checking, cross-record deduplication	Contextual QA & reasoning over multiple heterogeneous media items

Table 1: High-level normalization and supervision pipelines by modality.

Model	LibriSpeech		WorldSense $\uparrow$	TREA $\uparrow$	LATBench
	clean $\downarrow$	other $\downarrow$			mIoU $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	FENSE $\uparrow$
Qwen3-Omni-30B-A3B-Instruct	1.32 / 1.22	2.66 / 2.48	46.03	71.0	14.8\|15.8	21.4\|26.4	12.4\|16.0	7.0\|7.0	28.4\|31.0
MiniCPM-o-4.5	1.54 / 1.4	3.79 / 2.8	44.99	54.8	3.0\|5.2	4.0\|7.4	2.1\|3.7	0.8\|2.0	41.1\|41.9
Qwen3.5-Omni-Plus (DashScope)	1.32	3.14	63.90	87.7	60.2\|51.2	81.1\|71.1	63.2\|54.5	40.5\|33.8	64.2\|53.5
Doubao2.0-Lite (non-thinking)	1.28	2.41	53.94	71.5	53.1\|53.2	73.9\|73.2	58.9\|57.9	38.2\|37.2	57.4\|48.7
Doubao2.0-Lite (thinking, high)	1.35	2.55	66.77	79.2	52.6\|55.1	72.4\|77.5	59.1\|61.9	39.0\|37.8	58.7\|42.1

Table 2: Audio benchmarks. Paired values normally list our evaluation result followed by the model developer’s reported result; for LATBench, values separated by “ | ” denote Chinese and English results, respectively.

Model	OmniBench $\uparrow$	video-SALMONN			UGC-VideoCap	OmniCap-IF		Multimodal interleaving
		Miss $\downarrow$	Hallucination $\downarrow$	Total error $\downarrow$		CSR $\uparrow$	ISR $\uparrow$	IV-Bench $\uparrow$	MVU-Eval $\uparrow$
Qwen3-Omni-30B-A3B-Instruct	60.35	36.32	16.87	53.20	80.01	65.87	9.06	35.02	56.41
MiniCPM-o-4.5	52.04	33.37	15.35	48.72	78.45	70.84	11.69	24.70	47.81
Qwen3.5-Omni-Plus (DashScope)	67.61	17.47	21.67	39.13	87.78	75.90	18.95	53.31	60.64
Doubao2.0-Lite (non-thinking)	57.79	28.20	17.26	45.46	84.22	76.56	20.29	43.48	60.42
Doubao2.0-Lite (thinking, high)	70.18	19.47	19.97	39.45	91.05	78.21	21.62	61.41	63.71

Table 3: Audio–video and multimodal-interleaving benchmarks.

Model	MMBench		OCRBench $\uparrow$	TextPecker				PerceptionBench $\uparrow$
	EN (dev) $\uparrow$	CN (dev) $\uparrow$		EN + $\uparrow$	EN - $\downarrow$	CN + $\uparrow$	CN - $\downarrow$
Qwen3-Omni-30B-A3B-Instruct	87.29	83.68	86.80	0.032	0.078	0.047	0.041	32.17
MiniCPM-o-4.5	85.22	82.99	61.00 / 87.6	0.094	0.099	0.024	0.085	29.37
Gemma-4-31B-IT	87.11	87.11	84.70	0.252	0.054	0.187	0.072	47.23
Qwen3-VL-32B	87.63	86.17	84.80 / 89.5	0.159	0.166	0.171	0.069	33.97
Qwen3.5-Omni-Plus (DashScope)	91.24	88.75	85.20	0.106	0.060	0.072	0.038	37.50

Table 4: Image benchmarks.

Model	LongVideo Bench $\uparrow$	MVBench $\uparrow$	MME- VideoOCR $\uparrow$	VSI- Bench $\uparrow$	ShortVid- Bench $\uparrow$	DSI- Bench $\uparrow$	Event- Bench $\uparrow$	Motion- Bench $\uparrow$	TimeLens				CameraMotion VQA $\uparrow$	VidCapBench $\uparrow$
									Overall $\uparrow$	Charades $\uparrow$	QVHighlights $\uparrow$	ActivityNet $\uparrow$
Qwen3-Omni-30B-A3B-Instruct	64.70	70.00	62.35	61.12	63.90	51.95	69.91	61.57	12.64	14.90	8.77	14.26	31.68	17.78\|84.24
MiniCPM-o-4.5	64.85 / 66.0	64.00	62.20	50.38	64.60	26.23	66.48	60.75 / 61.4	10.01	9.21	9.62	11.19	23.53	15.22\|79.16
Gemma-4-31B-IT	61.78	64.40	68.25	48.10	68.50	55.23	65.11	61.05	51.98	46.66	61.80	47.49	28.09	16.64\|90.17
Qwen3-VL-32B	67.61	73.65 / 72.8	75.25	64.16 / 61.5	67.30	53.36	74.89	67.32	52.16	52.79	55.71	47.99	26.71	20.01\|87.49
Qwen3.5-Omni-Plus (DashScope)	69.26	80.37	79.20	60.67	70.10	59.36	76.62	71.55	33.31	43.14	25.92	30.88	37.70	18.29\|92.60

Table 5: Video benchmarks. For VidCapBench, values separated by “ | ” denote Accuracy and Coverage.

Figure 3: Two captions of a 17.7 s clip from Meridian . Circled numbers link seven facts across the frames, timeline, and captions. The fluent caption reads well but omits or misrepresents key details. The generation-oriented caption preserves shot timing, audiovisual alignment, and visible text, including mirror-reversed lettering.

Information family	Required caption elements	Downstream failure if omitted or corrupted
Subject & action	Distinct entities, action phases, interaction objects, resulting states	Static posing, role swaps, unresolved actions
Space & composition	Left–right/front–back relations, entry/exit directions, framing, depth	Reversed motion, misplaced subjects, uncontrolled composition
Cinematography & light	Shot scale, viewpoint, camera motion, focal shift, light direction/quality, contrast, color temp.	Generic camera behavior, appealing but irreproducible style
Time & shots	Event order, state changes, shot boundaries, timestamps, recurring subjects	Frame-list descriptions, discontinuity, unintended cuts or freezes
Visible & audible text	Faithful OCR, dialogue/narration, env. & action sounds, music, time anchors	Misspelled text, hallucinated audio, misaligned A-V events
Aesthetics	Concrete texture, palette, medium, style, and finish (beyond composition/light)	Vague non-executable descriptors (e.g., “cinematic”)

Table 6: Generation-oriented caption specification.

Criterion	Characteristic shortcut	Downstream consequence
Factual accuracy	Omit uncertain or complex attributes	Underspecified condition $\rightarrow$ dataset-prior fallback
Semantic coverage	Add plausible but unverified details	Hallucinated entities, actions, relations, or text
Beneficial diversity	Use uncontrolled synonyms or poetic terms	Fragmented TE distribution $\rightarrow$ unstable control
Internal consistency	Reuse rigid templates; ignore local nuances	Lost dynamic state changes $\rightarrow$ robotic generation

Table 7: Shortcuts from optimizing each caption criterion in isolation.

Model	VidCapBench				UGC-VideoCap				Motion (Internal)		Camera (Internal)		Sound (Internal)
	Acc.	Pre.	Cov.	Con.	Audio	Visual	Detail	Avg.	Pre.	Rec.	Pre.	Rec.	Pre.	Rec.
Gemini-4-Argon	16.20	51.66	85.63	10.69	79.80	87.69	85.67	84.38	64.62	45.78	40.83	76.22	79.74	29.38
Doubao2.1-Lite (non-thinking)	17.20	51.78	87.47	8.32	82.88	90.67	88.81	87.45	62.87	56.31	28.37	61.46	73.18	26.79
Doubao2.1-Lite (thinking, high)	20.23	56.44	94.15	3.36	90.37	96.10	94.97	93.81	64.15	59.02	32.89	64.61	77.56	31.69
Kling-Captioner-35B-A3B	21.84	59.28	94.81	3.56	92.83	96.46	95.90	95.06	67.28	64.75	52.18	80.93	78.52	55.83
Kling-Captioner-397B-A17B	21.70	60.51	96.05	3.19	93.41	96.70	96.38	95.50	68.11	67.42	54.82	83.43	80.21	58.11

Table 8: Captioning benchmarks.

Figure 4: Reference-grounded captioning on a 22.5 s clip from Meridian , given two portraits (IMG_1, IMG_2) and a voice track (AUD_1). The caption states each binding with its scope and writes out everything the references cannot supply. The timeline shows who is on screen and when PERSON_1 speaks. Circled numbers mark where each property of the semantic contract applies, contrasting typical shortcuts with correct formulations.

Product task	Typical references	Binding question	Residual target facts
Video editing	Source video, instruction, optional local ref.	Edited entity/region and protected properties	Requested state change, time span, secondary effects
Multi-reference generation	Character, object, scene, style, voice, or motion refs.	Reference governing each role, attribute, and modality	Target interaction, action, camera, timing, composition
Video continuation	Prefix video + optional entity or style refs.	Subjects, states, and scene geometry persisting across cut	Post-boundary events and causal link to prefix
Performance transfer	Driving video, pose, trajectory, or audio	Identity source and performance source	Target retargeting, camera behavior, synchronization

Table 9: Product tasks covered by RGC.

Requirement	What it guarantees	Characteristic shortcut and downstream failure
Scoped binding	Pointer binds to target entity, attribute, event, & interval	Property bound to wrong target $\rightarrow$ spurious correlations
Residual sufficiency	All unreferenced target facts stated explicitly	Vague “same as ref” omits action/camera/time/audio $\rightarrow$ underspecified
Authority consistency	Reference-governed traits kept unless scoped override	Verbose text contradicts ref. $\rightarrow$ competing conditions

Table 10: The semantic contract of RGC.

Reference	Typical authority	Main ambiguity	Caption-side response
Image	Identity, appearance, object detail, scene/style	Which entity and image subset should transfer	Bind explicit ID & scope; state target action, placement, camera, changes
Video	Motion, state, performance, continuity, camera	Time alignment; transfer vs. keep; cut boundary	Declare source interval & role; timestamp new events & preserved content
Audio	Voice, dialogue, rhythm, sound effects	Speaker/source assignment & A-V sync	Bind speaker/event IDs to visible actions; no unseen visual hallucination
Text	Edit instruction, script, names, constraints	Conflicts with sources or over-specifies details	Treat as typed change spec.; resolve authority at attribute level
Mixed set	Joint composition of the above	Overlapping authority, duplication, contradiction	Build attribute-level authority map before composing final caption

Table 11: Authority and ambiguity of each reference modality.

Model	ViDiC			MRVBench
	Avg.	Diff.	Sim.	KP-Cov	VQA	Ref-Tag-R	Ref-Bind	Subj-F1	MRVScore
Gemini-4-Argon	77.18	75.51	81.86	93.31	74.41	97.23	93.56	90.37	91.22
Doubao2.1-Lite (non-thinking)	69.24	65.27	80.33	89.45	68.09	75.43	93.35	49.45	78.10
Doubao2.1-Lite (thinking, high)	75.63	74.96	77.49	90.75	71.55	81.41	93.98	56.86	81.32
Kling-RGC-35B-A3B	77.57	75.59	83.06	94.12	76.01	94.80	92.88	84.51	88.57
Kling-RGC-397B-A17B	78.89	76.29	86.12	95.18	76.34	95.28	94.85	85.24	90.31

Table 12: RGC benchmarks.

Panel of Figure 5 — Figure 5: Overview of PE. Top: three training stages of basic PE. Stage 1 learns the mapping from real-format user prompts to captions of compelling videos, using only pairs that pass faithfulness, reasonableness, and vividness (F/R/V) verification; Stage 2 optimizes the same three properties with text-only rewards; Stage 3 optimizes basic PE through the frozen text encoder and DiT with video-level rewards. Bottom: at inference, agentic PE makes the plot coherent and retrieves five captions of compelling videos; basic PE writes the final condition, using the retrieved captions as optional evidence.

Panel of Figure 6 — Figure 6: Three levels of caption alignment contrasting generic LLM rewrites with caption-aligned PE.

Setting	Text metrics						Video metrics
	Faithfulness		Reasonableness		Vividness		StoryEval	T2AV- Compass	AVE- Compass	MultiRef- Compass
	Avg.	$\geq$ 4 (%)	Avg.	$\geq$ 4 (%)	Avg.	$\geq$ 4 (%)	StoryEval	T2AV- Compass	AVE- Compass	MultiRef- Compass
Basic PE
SFT, random-source captions	4.02	75.42	3.95	69.17	3.72	62.33	65.42	70.15	56.2	4.6137
+ compelling-video sources	4.00	74.92	3.92	68.08	4.03	72.58	66.87	70.60	56.7	4.6224
+ F/R/V data verification	4.27	80.83	4.18	74.33	4.11	75.08	71.93	73.07	57.9	4.6591

Table 13: Cumulative contribution of PE components.

Kling-MLLM TE configuration	Overall	Instruction Following	Aesthetics
Multimodal adaptation	$1.10$	$1.04$	$1.11$
+ Multi-granular conditioning	$1.22$	$1.11$	$1.22$
+ Conditioning stability	$\mathbf{1.27}$	$\mathbf{1.14}$	$\mathbf{1.25}$

Table 14: Human preference evaluation (

(G+S)/(S+B)

) relative to the Qwen baseline.

Scenario	Model	Regime	Input	Output	Key constraint
DiT RL	DRM	Online	Rollouts of one prompt	Per-dimension rewards (GRPO); preference pairs (DPO)	Latency
PE RL	DRM	Online	Videos rendered from candidate rewrites	Per-rewrite reward on VQ, MQ, and TA	TA judged against the user prompt
Auxiliary machine evaluation	GRM	Offline	Single video with prompt or reference	Defect tags, intervals, boxes, and rationale	Verifiable evidence
Checkpoint selection	DRM & GRM	Offline	Videos from competing checkpoints	Per-dimension win rates; defect profiles	Low variance
Data filtering	DRM	Offline	Single clip	Calibrated defect probabilities for keep/drop	Throughput; low false positives

Table 15: Application scenarios of our RMs.

Figure 7: Three-stage training pipeline for our RMs, starting from Kling-MLLM-27B. Stage 1 adapts the vision encoder and aligner with the LLM frozen for low-level artifact perception; Stage 2 trains the General Reward Model (GRM) with cold-start SFT and RL with verifiable rewards for explainable evaluation; Stage 3 initializes the Decision Reward Model (DRM) from the GRM and performs decision-head warm-up, unified pointwise–pairwise SFT with TA grounding, and symmetry calibration with gated inference for fast multi-dimensional scoring.

L1 Category	L2 Evaluation Dimension	Input	Representative L3/L4 Leaf Defects
1. Visual Quality	1.1 Image Clarity & Compression	Video	Low clarity/defocus blur, codec blockiness & mosquito noise
	1.2 Visual Content Integrity	Video	AI plastic/greasy skin, texture smearing, edge halo, text distortion, watermark, exposure
	1.3 Visual Aesthetics	Video	Color/lighting harmony, spatial composition, material texture, subject styling, set design
2. Motion Quality	2.1 Basic Motion Quality	Video	Anatomy, deformation/liquefaction, temporal drift, collision, physics, flicker
	2.2 Motion Aesthetics & Narrative	Video	Weak pacing, acting, editing, framing, camera path, or creativity
	2.3 Motion Magnitude	Video	Insufficient motion amplitude, abnormal freezing

Table 16: Unified evaluation taxonomy of our RMs.

	Avg		VQ		MQ			Consistency		TA
Model	Macro	Micro	Visual Content	Visual	Basic Motion	Motion Aesth.	Motion	Input Image	Multi-Shot	Basic Prompt	Multi-Shot
	Macro	Micro	Integrity	Aesthetics	Quality	& Narrative	Magnitude	Consistency	Consistency	Following	Prompt Fol.
Prompting MLLM
Gemini-3.8-Flash †	$67.59$	$67.13$	$66.27$	$67.03$	$61.99$	$69.81$	$88.89$	$\underline{75.00}$	$58.93$	$47.10$	$\underline{73.33}$
Gemini-4-Argon †	$73.54$	$72.59$	$72.78$	$76.92$	$65.42$	$72.96$	$85.71$	$\underline{75.00}$	$66.07$	$60.30$	$\mathbf{86.67}$
Ours

Table 17: Pairwise preference accuracy (%) on the internal benchmark.

Model	VideoScore-Bench-v2	VideoPhy2-test	VideoGen-RewardBench
Prompting MLLM
Gemini-3.8-Flash	$36.90$	$\mathbf{47.80}$	$54.37$
Gemini-4-Argon	$33.60$	$43.70$	$56.00$
Reward/Scoring Models for Video
VideoReward [ 28 ]	–	$31.33$	$\mathbf{59.69}$
UnifiedReward [ 55 ]	$25.07$	$22.02$	$53.31$

Table 18: Accuracy (%) on open-source video reward and scoring benchmarks.

Task	Core Objective	Main Failure Modes	Representative Tools
Video captioning	Accurate & consistent coverage of salient facts	Omissions, hallucinations, vague references, lost temporal structure	Keypoint QA, entailment judges, coreference metrics, downstream generation tests
T2V / T2AV	Prompt adherence & perceptual/physical plausibility	Missing attributes/events, implausible motion, visual artifacts, wrong/desynced audio	Checklist judges, DOVER++, aesthetic/audio quality models, CLIP/CLAP, Synchformer, lip-sync
Reference-conditioned	Identity/detail preservation, binding, & natural integration	Identity drift, entity swaps, attribute leakage, source mismatch, copy–paste	Detectors/segmenters, ID embeddings, face/speaker models, binding checks, anti-paste judges
Keyframe / storyboard	Prescribed frames in correct order, timing, & duration	Missed/reordered frames, hard cuts, flashes, freezes, repeats, static storyboard	DINO/SSIM/PSNR matching, Kendall rank correlation, window matching, camera/tracking
Video / A-V editing	Execute requested changes while preserving unedited content	Non-response, incomplete edit, total regeneration, collateral changes, A-V mismatch	Atomic edit checklists, response detection, source–target tracking/seg., SSIM/DINO

Table 19: Evaluation taxonomy by task.

Video Reward Modeling

See all 9 papers

How Far Are Video Models from True Multimodal Reasoning?
Apr 21, 2026Xiaotian Zhang, Jianhui Wei, Yuan Wang +9Multimodal Reasoning Video Reasoning
Zhejiang University · ByteDance
Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator
Apr 9, 2026Luozheng Qin, Jia Gong, Qian Qiao +6Unified Multimodal Models Video Generation
Shanghai Academy of AI for Science · Independent Researcher · Singapore University of Technology and Design +1
Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation
Mar 13, 2026Yichen Zhang, Da Peng, Zonghao Guo +19Unified Multimodal Models Multimodal Generation
Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences

Apr 21, 2026J/K move · Enter open · S save

How Far Are Video Models from True Multimodal Reasoning?

Xiaotian Zhang, Jianhui Wei, Yuan Wang, Jie Tan+8

Zhejiang University · ByteDance

Multimodal Reasoning Video Reasoning Physical Reasoning in Video Multimodal Understanding Video Generation Evaluation

Open paper AskPDF arXiv

Despite remarkable progress toward general-purpose video models, a critical question remains unanswered: how far are these models from achieving true multimodal reasoning? Existing benchmarks fail to address this question rigorously, as they remain constrained by straightforward task designs and fragmented evaluation metrics that neglect complex multimodal reasoning. To bridge this gap, we introduce CLVG-Bench, an evaluation framework designed to probe video models' zero-shot reasoning capabilities via Context Learning in Video Generation. CLVG-Bench comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories, covering complex scenarios including physical simulation, logical reasoning, and interactive contexts. To enable rigorous and scalable assessment, we further propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations, delivering interpretable textual feedback across diverse video context tasks. Extensive experiments reveal a striking answer to our central question: while state-of-the-art (SOTA) video models, such as Seedance 2.0, demonstrate competence on certain understanding and reasoning subtasks, they fall substantially short with logically grounded and interactive generation tasks (achieving success rates <25% and ~0%, respectively), exposing multimodal reasoning and physical grounding as critical bottlenecks. By systematically quantifying these limitations, the proposed method provides actionable feedbacks and a clear roadmap toward truly robust, general-purpose video models. CLVG-Bench and code are released here.

Abstract

Video Reward Modeling

Explore similar work

How Far Are Video Models from True Multimodal Reasoning?

Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator

Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation