Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation

Yanbo Ding; Yijia Fan; Caihua Shan; Yifan Yang; Yifei Shen; Weijie Wang; Xirui Hu; Dongsheng Li

cs.CVAug 14, 2026

Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation

Authors: Yanbo Ding, Yijia Fan, Caihua Shan, Yifan Yang, Yifei Shen, Weijie Wang, Xirui Hu, Dongsheng Li,

Organizations: Chinese Academy of Sciences · Microsoft Research · Sun Yat-sen University · Zhejiang University ·

Abstract

Diffusion Transformers (DiTs) have become the dominant paradigm for high-fidelity video generation, yet their ability to perform high-level semantic planning remains limited. While hybrid architectures integrating MLLMs with diffusion backbones have shown strong advantages in image synthesis, such designs remain underexplored in video generation, where existing approaches often treat MLLMs primarily as frozen feature encoders rather than semantic generators. To fill this gap, we systematically study how an MLLM should be integrated with a DiT for video generation by answering three questions: what intermediate representation should bridge the MLLM and DiT, how the MLLM should generate it, and how the DiT should incorporate it during diffusion rendering. Our analysis reveals three key findings: (1) discrete semantic visual tokens produced by an EMA-based tokenizer provide a stable and expressive interface, (2) autoregressive causal modeling is effective for generating these tokens, and (3) explicit visual-token conditioning is more effective than prompt refinement or latent bridging. Based on these findings, we propose BiVidGen, a hybrid framework where an MLLM first generates semantic visual tokens and a DiT renders videos conditioned on both text and these tokens via multi-layer cross-attention. Extensive experiments show that BiVidGen improves semantic alignment and temporal coherence over a fine-tuned DiT baseline, achieving stronger performance on VBench-Long. These results demonstrate that explicit MLLM-based visual planning provides an effective intermediate interface for text-to-video generation beyond text-only conditioning.

Figures & tables

Figure 1: Quantization Design Variants. Codebooks include (a) EMA-based updates, (b) gradient-based embedding updates, (c) embedding updates with MLLM understanding supervision, (d) embedding updates with pixel reconstruction supervision, and (e) hierarchical EMA-based updates.

Figure 2: MLLM-Based Generation Variants. We comprehensively and fairly compare mainstream modeling strategies for visual generation using MLLMs, including causal versus bidirectional attention mechanisms, and continuous versus discrete input representations.

Figure 3: DiT-Based Generation Variants. We categorize MLLM-DiT connection strategies into (a-d) explicit visual-token conditioning, including self-attention, cross-attention, and multi-layer injection; (e) prompt refinement, which transfers MLLM information through text only; and (f) latent bridging, which uses learnable queries and a connector to extract implicit MLLM tokens.

Figure 4: Overview. BiVidGen consists of three main components. First, an EMA-based vision tokenizer provides discrete visual representations. Then, an MLLM predicts semantic visual tokens. Finally, the DiT renders high-fidelity videos conditioned on these planned tokens.

Method	MLLM Signal	Fusion Strategy	Quality $\uparrow$	Semantic $\uparrow$	Total Score $\uparrow$
DiT-only (Wan2.2-FT)	Text	Text-condition	79.81	55.16	74.88
+ Prompt Refinement	Text	Text-condition	80.27	60.74	76.36
+ Latent Bridging	w/ Learnable Token	Connector	76.14	32.58	67.43
+ Latent Bridging	w/o Learnable Token	Connector	77.04	35.23	68.68
+ Visual Tokens	Discrete Token	Cross-attn	81.74	63.95	78.18
+ Visual Tokens	Discrete Token	Multi-cross-attn	81.90	76.25	80.77

Table 1: Effectiveness of MLLM-DiT for Generation on VBench-Long Benchmark.

Codebook	Entropy↑	Active Ratio↑	Top-10 Ratio↓	Quantize Error↓
Embedding	0.321	0.052	0.704	0.551
Embedding + Decoder	0.641	0.458	0.243	0.586
Embedding + LLM	0.797	0.974	0.186	0.938
EMA	0.812	0.841	0.146	0.130

Table 2: Evaluation of Codebook Intrinsic Properties.

Input	Attention	Loss	Quality↑	Semantic↑	Total Score↑
Continuous Token	Causal Attn	MSE	78.33	37.65	70.19
Continuous Token	Bidirectional Attn	MSE	77.24	36.77	69.14
Discrete Token	Bidirectional Attn	CE	79.63	46.26	72.95
Discrete Token	Causal Attn	CE	78.81	48.74	72.79

Table 3: Ablation of MLLM-based Generation on VBench-Long Benchmark.

Codebook	Injection Module	PSNR↑	SSIM↑	LPIPS↓
Embedding	Self Attention	12.50	0.338	0.628
Embedding + Decoder	Self Attention	13.07	0.347	0.574
Embedding + LLM	Self Attention	11.94	0.290	0.715
EMA	Self Attention	13.71	0.358	0.542
EMA	Cross Attention	14.05	0.388	0.528

Table 4: Ablation of VQ and DiT for Reconstruction on DAVIS Dataset.

Codebook	Injection Module	Quality↑	Semantic↑	Total Score↑
Embedding	Self Attention	79.07	45.80	72.41
Embedding + Decoder	Self Attention	79.20	40.82	71.52
Embedding + LLM	Self Attention	75.72	34.68	67.51
EMA	Self Attention	78.81	48.74	72.79
EMA	Cross Attention	78.72	49.43	72.86

Table 5: Ablation of VQ and DiT for Generation on VBench-Long Benchmark.

Model	Denoising Steps	Latency (s) ↓	Memory (GB) ↓	Total Score on VBench-Long ↑
DiT-only	50	37	29 (Peak)	74.88
MLLM + DiT	50	45	29 (Peak)	78.18
MLLM + DiT	30	31	29 (Peak)	77.27

Table 6: Comparison of Latency and Memory Usage.

Figure 5: Qualitative Comparisons. MLLM+DiT yields better causal transitions, temporal consistency, and semantic alignment than the DiT-only baseline in these examples.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Input	Layers	Quality↑	Semantic↑	Total Score↑
VBench-Long	Discrete	0	78.81	48.74	72.79
	Discrete Hierarchical	0, 0, 0, 0	79.68	42.93	72.33
	Discrete Hierarchical	0, 8, 16, 24	79.04	41.45	71.52
	Discrete Hierarchical	0, 1, 2, 3	79.48	44.78	72.54

Appendix

Table 7: Comparison With Hierarchical Discrete Tokens.

Figure 6: Image Generation Results. Our method generates visually coherent images across diverse scenes with fine-grained details, including accurate text content. Prompts are provided in Table 8 .

Row	Col	Caption
1	1	A bright green background featuring the white text "Hello BiVidGen".
1	2	A sweeping aerial landscape of a massive valley glacier.
1	3	A dramatic photo of a lightning strike illuminating dark storm clouds.
1	4	A dried brown leaf floating on water, creating concentric ripples.
2	1	A photo of the words ’SEA SALT’ formed by coarse white salt crystals on a dark slate board.
2	2	A warm, bright fire burning within a dark brick fireplace.

Appendix

Table 8: Detailed Prompts for the Images in Figure 6 .

Figure 9: Video Comparisons With Existing Methods. With the help of MLLM, our BiVidGen demonstrates clear advantages in prompt adherence and superior video quality.

Method	Overall Scores			Technical Quality					Aesthetic Quality
Method	Total Score↑	Quality↑	Semantic↑	Subject↑	Background↑	Temporal↑	Motion↑	Dynamic↑	Aesthetic↑	Imaging↑
Sora [ Liu et al., 2024 ]	84.28	85.51	79.35	96.23	96.35	98.87	98.74	79.91	63.46	68.28
OpenSora [ Zheng et al., 2024 ]	79.76	81.35	73.39	96.75	97.61	99.53	98.50	42.39	56.85	63.34
HunyuanVideo [ Kong et al., 2024 ]	83.24	85.86	75.82	97.32	97.93	99.49	98.99	70.83	60.36	67.56
Veo-3 [ Wiedemer et al., 2025 ]	85.06	85.70	82.49	97.36	96.89	99.30	99.16	72.43	63.81	68.23
Gen-3 [ Runway, 2024 ]	82.32	84.11	75.17	97.01	96.62	99.61	99.23	60.14	63.34	66.82

Appendix

Table 9: T2V Performance on VBench-Long [ Huang et al., 2024 ] Benchmark. Bold indicates the best results, underline indicates the second best, and italic indicates the third best.

Figure 10: Video Comparisons With the Baseline. Our method integrates an MLLM with a DiT for video generation. The semantic planning of the MLLM enables the DiT to generate better videos, especially in terms of temporal consistency, causal transition, and semantic alignment.

Table 19

T2V Generation

Text-to-video generation creates moving visual scenes from natural-language prompts, with growing control over motion, identity, events, and style.

See all 110 papers

May 21, 2026

Bernini: Latent Semantic Planning for Video Diffusion

Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM's pretrained understanding translating into strong generalization on challenging editing tasks.

Bernini Team, Chenchen Liu, Junyi Chen +9

Bernini Team, Bytedance · Bytedance

Video Diffusion Models Unified Multimodal Models Video Editing Video Generation Multimodal Foundation Models

Ask PDF

May 29, 2026

TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generation

Text-to-video (T2V) generation faces challenging questions when generating videos with long horizons containing multiple events. Inspired by the intrinsics of the diffusion process, we probe video diffusion transformers (DiTs) and uncover intrinsic turning points in the DiT denoising trajectory where conditioning text affects generation from global layout to fine-grained details. Building on this finding, we present TunerDiT, a simple yet effective progressive steering method that requires no additional training for multi-event generation. TunerDiT comprises two steering handles: (1) Event-Partitioned Masking that enforces event boundaries while allowing cross-event transition bands; (2) Cross-Event Prompt Fusion that injects neighboring event semantics for late-stage refinement. We contribute a self-curated prompt suite for benchmarking multi-event generation, i.e., Meve. TunerDiT achieves state-of-the-art performance across 8 metrics and offers a tunable trade-off between video consistency and event separation, compared with other training-free methods. The improvement in text alignment increases with the event count, indicating a scaling possibility with increasing event count.

Ruotong Liao, Guowen Huang, Qing Cheng +6

Ludwig Maximilian University of Munich · MCML · Technical University of Munich +2

Diffusion Transformer T2V Generation Long-Horizon Video Generation Video Generation Diffusion Models

Ask PDF

Aug 17, 2026

MLLM-Guided Semantic Correction for Text-to-Video Generation

Recent advances in diffusion models and Transformer architectures have led to significant progress in text-to-video generation. However, these models often suffer from semantic errors such as missing objects, incorrect attributes, or mismatched actions. Although some semantic correction methods perform optimization before sampling or refinement after sampling, how to detect and correct semantic deviations during the video generation process remains underexplored. In this paper, we introduce a training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop. Our framework achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection. We propose two key modules: a Semantic Assessment Supervisor that generates intermediate preview frames for semantic evaluations and deviation diagnostics, and a Semantic Modification Assistant that corrects semantic drift during inference via a controllable latent trajectory intervention. Our method improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters. We validate the effectiveness of our approach through extensive experiments across multiple benchmarks.

Junhao Chen, Zheqi Lv, Keting Yin +6

School of Software Technology, Zhejiang University, Hangzhou 310027, China · College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China · AI System Innovation Lab, Huawei Cloud Computing Technology Co., Ltd., Hangzhou 310005, China

T2V Generation Diffusion Model Guidance Controllable Video Generation

Ask PDF

Abstract

T2V Generation

Explore similar work

Bernini: Latent Semantic Planning for Video Diffusion

TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generation

MLLM-Guided Semantic Correction for Text-to-Video Generation