MLLM-Guided Semantic Correction for Text-to-Video Generation

Junhao Chen; Zheqi Lv; Keting Yin; Shengyu Zhang; Zhou Zhao; Feiyang Chen; Xinyu Duan; Baoxing Huai

cs.CVAug 17, 2026

MLLM-Guided Semantic Correction for Text-to-Video Generation

Authors: Junhao Chen, Zheqi Lv, Keting Yin, Shengyu Zhang, Zhou Zhao, Feiyang Chen, Xinyu Duan, Baoxing Huai,

Organizations: School of Software Technology, Zhejiang University, Hangzhou 310027, China · College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China · AI System Innovation Lab, Huawei Cloud Computing Technology Co., Ltd., Hangzhou 310005, China

Abstract

Recent advances in diffusion models and Transformer architectures have led to significant progress in text-to-video generation. However, these models often suffer from semantic errors such as missing objects, incorrect attributes, or mismatched actions. Although some semantic correction methods perform optimization before sampling or refinement after sampling, how to detect and correct semantic deviations during the video generation process remains underexplored. In this paper, we introduce a training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop. Our framework achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection. We propose two key modules: a Semantic Assessment Supervisor that generates intermediate preview frames for semantic evaluations and deviation diagnostics, and a Semantic Modification Assistant that corrects semantic drift during inference via a controllable latent trajectory intervention. Our method improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters. We validate the effectiveness of our approach through extensive experiments across multiple benchmarks.

Figures & tables

Fig. 1: Overview of semantic correction strategies for text-to-video generation. (a) Non–self-correcting : fixed-scale guidance tends to amplify artifacts and motion errors. (b) Starting-point correction : prompt/noise optimization before sampling cannot prevent drift during generation. (c) End-point correction : post-generation refinement reacts to errors but cannot prevent them. (d) Intermediate correction(Ours) : a training-free, MLLM-guided mid-generation correction that injects semantic feedback into the diffusion loop to maintain coherence during synthesis.

Fig. 2: Overview of the proposed mid-generation semantic correction framework (SASMA). The framework, termed SASMA (Semantic Assessment Supervisor and Modification Assistant), integrates MLLM feedback into the diffusion sampling loop for dynamic semantic correction during text-to-video generation. During sampling, intermediate latents are periodically decoded into preview frames for semantic evaluation (left and middle). The MLLM provides diagnostic feedback and corrective prompts, which are encoded into semantic deltas $\Delta c_{t}^{\pm}$ and injected back into the diffusion process (right) to guide subsequent denoising steps toward semantically consistent video synthesis.

Model	Method	Subject Cons. $\uparrow$	Aesthetic $\uparrow$	Imaging $\uparrow$	Human Act. $\uparrow$	Spatial Rel. $\uparrow$	Scene $\uparrow$	Overall Cons. $\uparrow$
CogVideoX1.5	Standard	0.9088	0.5435	0.5624	0.8200	0.4328	0.3343	0.2483
	Ours (SASMA)	0.9410	0.5633	0.5857	0.8440	0.4738	0.3894	0.2536
HunyuanVideo	Standard	0.9625	0.6012	0.6179	0.8840	0.5307	0.3581	0.2619
	Ours (SASMA)	0.9604	0.6038	0.6323	0.9020	0.5803	0.3728	0.2643
AnimateDiff	Standard	0.9524	0.5938	0.6161	0.9240	0.5196	0.4987	0.2720
	Ours (SASMA)	0.9569	0.6066	0.6463	0.9460	0.5355	0.5259	0.2713

TABLE I: Quantitative comparison per dimension on VBench.

Model	Method	Quality $\uparrow$	Semantic $\uparrow$	Total $\uparrow$
CogVideoX1.5	Standard	0.7717	0.6419	0.7457
	Ours (SASMA)	0.7982	0.6689	0.7723
HunyuanVideo	Standard	0.8171	0.6975	0.7932
	Ours (SASMA)	0.8201	0.7048	0.7970
AnimateDiff	Standard	0.8087	0.7195	0.7909
	Ours (SASMA)	0.8136	0.7277	0.7963

TABLE II: Quantitative comparison of overall performance on VBench.

Fig. 3: Qualitative comparison on text-to-video generation. Our method introduces an MLLM-guided feedback mechanism into the diffusion process, allowing mid-generation semantic correction without retraining. This integration enhances both temporal consistency and visual-semantic alignment in challenging cases such as multi-object composition, spatial relations, actions, and scenes.

Fig. 4: Case of interpretable evaluation and correction. Along the denoising trajectory from $x_{T}$ to $x_{0}$ , our framework performs semantic assessment at discrete intervals on intermediate latents. For each evaluated timestep, the MLLM produces three interpretable signals: a structured diagnostic signal $f$ identifying semantic deviations, a corrective prompt $p^{+}$ providing refined positive guidance, and a constraint prompt $p^{-}$ specifying exclusion constraints.

Method	UMT-FVD $\downarrow$	UMTScore $\uparrow$	MTScore $\uparrow$	CHScore $\uparrow$
Standard	216.68	2.8485	0.3420	45.566
+ Semantic Injection	213.26	2.8619	0.3476	59.187
+ Evaluation Module	212.50	2.8771	0.3452	60.578
+ Preview Module	213.18	2.8653	0.3486	61.784

TABLE III: Ablation study of SASMA modules on Chronomagic-Bench-150 with CogVideoX1.5.

Fig. 5: Effect of evaluation interval and stage scheduling. We compare three stage configurations (Early: 1-24, Late: 25-49, Full: 1-49) with three evaluation intervals (3, 5, 10) on CogVideoX1.5. Early-stage high-frequency evaluation improves subject consistency and motion smoothness, while late-stage performance remains stable across frequencies. Full-process mid-frequency scheduling (interval=5) achieves optimal balance. The gray dashed line indicates baseline performance without SASMA.

Method / Configuration	Motion Smooth. $\uparrow$	Dynamic Deg. $\downarrow$	Subject Cons. $\uparrow$	Aesthetic $\uparrow$	Imaging $\uparrow$	Overall Cons. $\uparrow$
Standard	0.9561	0.6639	0.9088	0.5435	0.5624	0.2483
Ours (1 round)	0.9812	0.4944	0.9396	0.5530	0.5745	0.2511
Ours (2 rounds)	0.9817	0.4167	0.9485	0.5668	0.6004	0.2531
Ours (3 rounds)	0.9812	0.5111	0.9410	0.5633	0.5857	0.2536

TABLE IV: Ablation on different numbers of evaluation–feedback rounds within the SASMA framework on CogVideoX1.5.

MLLM	Motion Smooth.	Dynamic Deg.	Subject Cons.	Aesthetic	Imaging	Overall Cons.	VRAM (GB)
Qwen2.5-VL-3B	0.9587	0.5018	0.9266	0.5345	0.5684	0.2467	9,290
Qwen2.5-VL-7B	0.9808	0.4831	0.9424	0.5669	0.5945	0.2528	17,142
Qwen2.5-VL-72B	0.9815	0.4228	0.9478	0.5656	0.5912	0.2526	155,354

TABLE V: Effect of MLLM scale on semantic correction quality.

Fig. 6: Effect of Intermediate Previewing in Mid-Generation Evaluation. From timestep $t_{s}$ to $t_{e}$ , we insert preview operations at an interval of $\Delta$ steps along the diffusion trajectory. For each selected timestep $t$ , we compare the noisy intermediate decoding $D(x_{t})$ with the preview video obtained from $D({x}_{t}^{pvw})$ . The prompt is ”A teddy bear sitting between a football and a violin.”

Fig. 7: Quantitative analysis of mid-generation semantic evaluation. (a) Agreement rate between Qwen2.5-VL-7B and Qwen2.5-VL-72B. (b) MATCH ratio under raw decoding. (c) MATCH ratio with preview. Raw decoding produces near-zero MATCH predictions before

t/T\approx 0.5

, whereas preview enables earlier semantic recognition.

Fig. 8: Evaluation of textual feedback quality across three models (Qwen2.5-VL-7B, VideoLLaMA3-7B, and InternVL2.5-8B). Each subplot shows the agreement rates between the model-generated feedback and the Qwen2.5-VL-72B verifier for (a) Diagnostic Signal, (b) Corrective Prompts, and (c) Constraint Prompt. The results are based on 165 test prompts from VBench. Darker bars indicate agreement (Yes), while lighter bars indicate disagreement (No).

Configuration	Overall Cons.	Time (s)	MLLM Calls	VRAM (GB)
Standard	0.2483	254.43	0	13.7
Ours (1-round)	0.2511	579.28	4.631	30.9
Ours (2-rounds)	0.2531	588.95	3.289	30.9
Ours (3-rounds)	0.2536	590.04	3.168	30.9

TABLE VI: Inference cost comparison under different configurations on CogVideoX1.5-5B with a single RTX 3090 GPU.

Dataset	Stop $\leq$ 1	Stop $\leq$ 2	Stop $\leq$ 3	Avg. Calls
VBench	34.96%	66.95%	78.79%	3.168

TABLE VII: Early stopping statistics on VBench using CogVideoX1.5-5B.

Component	Avg. Time (s)	Total Time (s)	Percentage
MLLM Inference	7.53	75.28	19.68%
Intermediate Preview	14.76	147.59	38.59%
Semantic Injection	15.96	159.58	41.73%
Total	—	382.45	100%

TABLE VIII: Component-wise efficiency breakdown of SASMA on CogVideoX1.5-5B.

T2V Generation

Text-to-video generation creates moving visual scenes from natural-language prompts, with growing control over motion, identity, events, and style.

See all 110 papers

KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing
May 28, 2026Mingshu Cai, Miao Zhang, Chenghe Yang +3Video Diffusion Models T2V Generation
Waseda University, Japan · College of Computer Engineering, Jimei University, Xiamen, Fujian, China · Department of Language Science and Technology, The Hong Kong Polytechnic University, Hong Kong, China +1
Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation
Aug 14, 2026Yanbo Ding, Yijia Fan, Caihua Shan +8T2V Generation Video Diffusion Models
Chinese Academy of Sciences · Microsoft Research · Sun Yat-sen University +3
Training-Free Semantic Correction for Autoregressive Visual Models
Jun 21, 2026Junhao Chen, Chanyu Zhu, Zheqi Lv +2Visual Autoregressive Models Autoregressive Image Generation
Zhejiang University · Shandong University

May 28, 2026J/K move · Enter open · S save

KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing

Mingshu Cai, Miao Zhang, Chenghe Yang, Yixuan Li+2

Waseda University, Japan · College of Computer Engineering, Jimei University, Xiamen, Fujian, China · Department of Language Science and Technology, The Hong Kong Polytechnic University, Hong Kong, China+1

Video Diffusion Models T2V Generation Video Editing Video Generation Controllable Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

In recent years, training-free video generation has progressed remarkably. However, when handling complex textual instructions, existing methods still suffer from semantic ambiguity, incorrect concept binding, and cross-frame inconsistency. To address these issues, we propose KGEdit, a structured semantic control framework for text-to-video (T2V) diffusion models. Specifically, we first construct an ambiguity-aware knowledge graph (AAKG) to disentangle and disambiguate the input prompt, converting it into four types of structured semantics: identity, relation, attribute, and negative constraints. We then design a structured semantic injection module (SSIM) to inject these semantic signals into key layers of the diffusion Transformer, enabling fine-grained semantic control. In addition, we introduce a temporal-aware semantic control (TASC) module that dynamically schedules semantic objectives according to the stage-wise characteristics of the denoising process, further improving semantic alignment and temporal consistency. Experiments show that KGEdit outperforms existing methods in editing precision and temporal stability, while offering higher efficiency and controllability in text-driven interaction scenarios.

Abstract

T2V Generation

Explore similar work

KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing

Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation

Training-Free Semantic Correction for Autoregressive Visual Models