While large-scale datasets have driven significant progress in Text-to-Video (T2V) generative models, these models remain highly sensitive to input prompts, demonstrating that prompt design is critical to generation quality. Current methods for improving video output often fall short: they either depend on complex, post-editing models, risking the introduction of artifacts, or require expensive fine-tuning of the core generator, which severely limits both scalability and accessibility. In this work, we introduce 3R, a novel RAG based prompt optimization framework. 3R utilizes the power of current state-of-the-art T2V diffusion model and vision language model. It can be used with any T2V model without any kind of model training. The framework leverages three key strategies: RAG-based modifiers extraction for enriched contextual grounding, diffusion-based Preference Optimization for aligning outputs with human preferences, and temporal frame interpolation for producing temporally consistent visual contents. Together, these components enable more accurate, efficient, and contextually aligned text-to-video generation. Experimental results demonstrate the efficacy of 3R in enhancing the static fidelity and dynamic coherence of generated videos, underscoring the importance of optimizing user prompts.
Figures & tables
Figure 1: Overview of 3R pipeline. A short user prompt I is used to extract a few relevant subject, scene, actions modifiers from a relation database D . Then MLLM is used to merge those modifiers iteratively to the original user prompt to get detaild prompt Pm , and RLLM checks Pm for any contradictory or missing information from the original prompt I , and generate N refined prompts. The refined prompts are fed to a T2V base model G to generate initial videos for each prompt. Next, a video selection model selects the best candidate based on a question answering test, and a temporal interpolation network enhances temporal consistency of the final video.
Model
Total Score
Motion Quality
Text-Video Alignment
Visual Quality
Temporal Consistency
Show-1
229
53.74
62.07
52.19
60.83
LaVie
234
52.83
68.49
57.99
54.23
IPO
234
53.39
54.62
62.56
63.40
Videocrafter2
243
54.82
63.16
63.98
61.46
3R
245
54.72
68.73
58.79
62.65
Table 1: Results on EvalCrafter benchmark. The first and second best results in each column are highlighted in the corresponding colors. 3R achieves the best total result, and either best or second best results in most of the individual metrics.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Rank
Weight
Question
1
1.1418
Does the video not completely fail to meet the requirements stated in the text “[prompt]”?
2
0.9544
Does the video meet all the requirements stated in the text “[prompt]”?
3
0.4390
Is the object’s movement completely realistic?
4
0.4293
Are the details very refined?
5
0.3942
Is the video content part of the physical world?
Appendix
Table 2: Top 5 weighted questions used in VisionReward-Video scoring out of 29 questions. Higher scores are assigned to text-video alignment questions.
Category
Content
User Prompt
A tranquil tableau of barn.
Retrieved Modifiers
a lively and energetic performance, walking inside, eating hay, being kept in the barn for some purpose, appears to be a farmer, a lively and joyful scene, wearing pink outfits, in a barn, preparing for a shooting competition, dancing in a barn, laying on the ground, a barn, opening the door of a barn, inside a barn
RAPO
A lively and energetic performance is taking place inside a barn. A farmer, wearing pink outfits, is walking inside, eating hay, and appears to be preparing for a shooting competition. The scene is lively and joyful, and the animals seem to be laying on the ground. The farmer opens the door of the barn and invites the audience to witness the performance, which includes dancing in the barn.
3R
In a peaceful countryside setting, a barn sits quietly surrounded by lush green fields. The wooden structure, weathered yet sturdy, is bathed in the warm glow of the afternoon sun.
Appendix
Table 3: Comparison of Prompt Optimization Outputs. RAPO contains information such as ‘eating hay’ or ‘shooting competition’ that has no relevance to the user intent ‘a tranquil tableau of barn’ while 3R filters out irrelevant modifiers and generates relevant ones like ‘wooden structure’ and ‘peaceful countryside setting’.
Figure 2: Qualitative comparison of Lavie, IPO, and 3R in two common video generation failure modes. The left side shows prompts and video frames representing challenges in semantic alignment such as mushroom growing out of human head or zoom-in and the right side shows prompts and video frames representing challenges in addressing fictional references such as Darth Vedar or Pikachu Jedi.
Figure 3: Qualitative comparison of approaches in the common failure mode of generating videos containing text. We compare the first frame of the videos generated by Lavie (left), IPO (middle), and 3R (right) in the common failure mode of text generation in videos, as observed from prompts provided by the EvalCrafter benchmark. All three approaches show strong limitations in generating correct text, but 3R manages to generate qualitatively more legible text where the intended text in the prompt (”keep off the grass” or ”keep off”) can still be partially inferred despite typos. The prompts and respective video frames show how our approach can address prompts requiring multiple semantic conditions while producing less distorted outputs.
Model
Total Score
Motion Quality
Text-Video Alignment
Visual Quality
Temporal Consistency
LaVie (Baseline)
234
52.83
68.49
57.99
54.23
One Prompt
241
54.73
67.54
59.75
58.75
N Prompts + Video Selection
243
54.64
69.44
59.86
58.94
Video Selection + VLM Critique
241
54.89
66.35
60.20
59.74
Video Selection + Temporal Inter.
245
54.72
68.73
58.79
62.65
Appendix
Table 4: Ablation results. The first and second best results in each column are highlighted in the corresponding colors. Initial prompt refinement, video selection model, and temporal interpolation model each contribute to the total score while vision-language model critique degrades text-video alignment severely due to it’s over-correction nature.