cs.CVOct 22, 2026

Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

Authors: Songlin Yang, Zhe Wang, Xiaotong Zhao, Xi Wang, Ran Zhang, Shuai Li, Taiyi Wu, Songlian Li, 

Organizations: Tencent Video · LIX, École Polytechnique

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Cinematic video generation requires both temporal inference from limited inputs (i.e., sparse rather than frame-by-frame guidance) and factor-aware control of character, scene, action, and camera. Multimodal large language models (MLLMs) offer useful priors for both capabilities, but their representations are not automatically actionable for video generation: a Video Diffusion Transformer (DiT) may bypass them through native conditioning pathways, and using them does not ensure either factor-aware control or generation-compatible temporal prediction. We propose Factor-Aware Predictive Learning (FAPL), which establishes effective MLLM latent use as a shared prerequisite and then targets the two capabilities with complementary learning signals. First, latent-only generator grounding trains the DiT with a frozen MLLM and connector, strengthening the DiT's use of MLLM latents for generation. Second, for factor-aware control, paired videos specify a target-factor change and non-target preservation, and factor-span replacement trains the DiT to realize this relationship in its output. Third, for temporal inference, text-only and text-plus-complete-video observations share a video target, and the trained DiT is then frozen while the MLLM and connector are optimized through it. This final stage trains limited-input representations for generation while retaining complete-observation reconstruction as a complementary training mode. Extensive experiments demonstrate our consistently better performance on three cinematic video generation tasks (i.e., sparse-to-video, factor-aware editing, and multi-shot generation) over baselines and validate FAPL's contributions to MLLM feature use, temporal control, and factor coordination.

Figures & tables

Figure 1: Cinematic video generation requires inference from limited inputs and factor-aware control. (i) The man turns toward the box, places his right hand on top, and steadies the bundle with his left hand. Text-only conditioning misses this sequence, while observing the complete video enables the same model to reproduce it. (ii) Keeping the descriptions of all other factors fixed and replacing only the character-related prompt, the non-character factors change as well. These cases illustrate failures in temporal inference and factor control, with editing providing a probe of what the generator changes and preserves.
Figure 2: MLLM latents are easily bypassed in existing MLLM–DiT models. (i) Under MLLM-only conditioning, Omni-Video 2 (1.3B DiT) produces noise, while Omni-Video 2 (14B DiT) and UniVideo produce artifacts and depict one person’s successive actions as two people acting differently. MLLM latents alone therefore fail to provide reliable generation guidance in these examples. (ii) Under conflicting conditions, VAE references dominate the output.
Figure 3: Overview of Factor-Aware Predictive Learning (FAPL). Stage I freezes the MLLM and its original connector and trains the DiT using MLLM latents. Stage II retains this configuration and swaps factor-specific output spans between paired examples to supervise factor-aware changes. Stage III freezes the trained DiT and jointly updates the MLLM and connector through video reconstruction, and text-only and text-plus-complete-video inputs share the same generation target.
 Sparse-to-VideoFactor-Aware EditingMulti-Shot Consistency
MethodMotion Order (%) ↑\uparrowTrans. Error ↓\downarrowRot. Error (rad) ↓\downarrow∩\cap -Acc (%) ↑\uparrowSubject ↑\uparrowScene ↑\uparrowFaceSim-Arc ↑\uparrow
Omni-Video 2 (1.3B)330.10841.87280.4350.3730.412
Omni-Video 2 (14B)430.08171.49370.4760.4230.468
UniVideo (13B)480.07631.54420.4620.4270.451
FAPL (Ours)610.05861.02580.5160.4710.523
Table 1: Quantitative comparison across three evaluation tasks.
Figure 4: Under limited inputs, baseline models fail to predict temporal visual details. With only text and no visual references, we compare UniVideo with FAPL (Ours), highlighting representative baseline limitations across four cinematic factors. (i) Character. Long hair is generated as short hair, failing appearance details specified only by text. (ii) Scene. The background is not preserved and incorrect content appears. (iii) Action. The requested sequence, e.g. , gesturing outward with the left hand and extended fingers, is missing. (iv) Camera. UniVideo ignores the instruction and keeps the camera static, while Ours orbits around the character. FAPL better realizes prompt-specified appearance, scene, and motion from text alone.
Figure 5: FAPL improves factor-aware control. For each factor, we modify only the corresponding description in the structured prompt and generate twice, comparing UniVideo with FAPL (Ours). UniVideo does not confine the change to the requested factor and alters unrelated content, whereas Ours modifies the selected factor while preserving the others. This indicates that factor-level latent augmentation enables selective control.
Figure 7
Exp IndexTraining ConfigurationMotion Order (%) ↑\uparrowTrans. Error ↓\downarrowRot. Error (rad) ↓\downarrow∩\cap -Acc (%) ↑\uparrow
0Pretrained UniVideo480.07631.5442
1Stage I only540.07081.3844
2Stage II: training without factor swapping (based on Exp. 1)540.06541.3048
3Stage II: training with factor swapping (based on Exp. 1)530.06711.3257
4Stage III: text-only training (based on Exp. 3)570.06291.2656
5Stage III: (Full FAPL) mixed-observation training (based on Exp. 3)610.05861.0258
Table 2: Ablation of FAPL training stages on UniVideo (13B).
More in this topic

Multimodal Video Generation

See all 29 papers

Explore similar work

CardsList
Aug 14, 2026

Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation

Diffusion Transformers (DiTs) have become the dominant paradigm for high-fidelity video generation, yet their ability to perform high-level semantic planning remains limited. While hybrid architectures integrating MLLMs with diffusion backbones have shown strong advantages in image synthesis, such designs remain underexplored in video generation, where existing approaches often treat MLLMs primarily as frozen feature encoders rather than semantic generators. To fill this gap, we systematically study how an MLLM should be integrated with a DiT for video generation by answering three questions: what intermediate representation should bridge the MLLM and DiT, how the MLLM should generate it, and how the DiT should incorporate it during diffusion rendering. Our analysis reveals three key findings: (1) discrete semantic visual tokens produced by an EMA-based tokenizer provide a stable and expressive interface, (2) autoregressive causal modeling is effective for generating these tokens, and (3) explicit visual-token conditioning is more effective than prompt refinement or latent bridging. Based on these findings, we propose BiVidGen, a hybrid framework where an MLLM first generates semantic visual tokens and a DiT renders videos conditioned on both text and these tokens via multi-layer cross-attention. Extensive experiments show that BiVidGen improves semantic alignment and temporal coherence over a fine-tuned DiT baseline, achieving stronger performance on VBench-Long. These results demonstrate that explicit MLLM-based visual planning provides an effective intermediate interface for text-to-video generation beyond text-only conditioning.

Yanbo Ding, Yijia Fan, Caihua Shan +8

Chinese Academy of Sciences · Microsoft Research · Sun Yat-sen University +3

Multimodal Video GenerationT2V GenerationVideo Diffusion ModelsMultimodal GenerationDiffusion TransformerVisual Autoregressive Models
AskPDF
May 21, 2026

Bernini: Latent Semantic Planning for Video Diffusion

Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM's pretrained understanding translating into strong generalization on challenging editing tasks.

Bernini Team, Chenchen Liu, Junyi Chen +9

Bernini Team, Bytedance · Bytedance

Video Diffusion ModelsUnified Multimodal ModelsVideo EditingVideo GenerationMultimodal Foundation ModelsMultimodal Video Generation
AskPDF
Jul 29, 2026

CineWeaver: Training-Free Reference-Controllable Multi-Shot Long Video Generation for Cinematic Storytelling

Cinematic video generation is challenging for text-to-video diffusion models due to concurrent requirements on multi-shot generation, fine-grained controllability over characters and scenes, and long-form generation across extended temporal horizons. Existing methods rely on customization and retraining to separately address specific requirements, and cannot simultaneously fulfill all the requirements with a unified framework. In this paper, we shed light on the training-free paradigm with the key insight that the difficulty of multi-shot generation arises from a structural bias toward temporal continuity in pretrained video diffusion models, and consequently, propose a unified framework named CineWeaver to achieve reference-controllable multi-shot long-video generation without retraining. We manipulate positional encoding and attention patterns to break temporal continuity during inference to enable clear shot transitions using pretrained video diffusion models. Furthermore, we extend the proposed framework with a shot-routed reference conditioning mechanism for per-shot fine-grained controllability, and develop an anchor memory mechanism to allow long-form generation with consistent global appearance cues. To our best knowledge, CineWeaver is the first unified framework to simultaneously enable long-form, reference-controllable, and multi-shot video generation in a training-free fashion. Experimental results demonstrate that CineWeaver produces high-quality cinematic videos of long durations with consistent identities, stable global appearance, and clear shot transitions. The project page is available at: https://cineweaver.github.io.

Yuyang Huang, Yabo Chen, Wenrui Dai +6

Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China. · Institute of Artificial Intelligence (TeleAI), China Telecom, China. · Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China.

Video Diffusion ModelsLong Video GenerationMulti-Shot Video GenerationControllable Video Generation
AskPDF