State-of-the-art text-to-video (T2V) generators frequently violate physical laws despite high visual quality. We show this stems from insufficient physical constraints in prompts rather than model limitations: manually adding physics details reliably produces physically plausible videos, but requires expertise and does not scale. We present PhyPrompt, a two-stage reinforcement learning framework that automatically refines prompts for physically realistic generation. First, we fine-tune a large language model on a physics-focused Chain-of-Thought dataset to integrate principles like object motion and force interactions while preserving user intent. Second, we apply Group Relative Policy Optimization with a dynamic reward curriculum that initially prioritizes semantic fidelity, then progressively shifts toward physical commonsense. This curriculum achieves synergistic optimization: PhyPrompt-7B reaches 40.8% joint success on VideoPhy2 (8.6pp gain), improving physical commonsense by 11pp (55.8% to 66.8%) while simultaneously increasing semantic adherence by 4.4pp (43.4% to 47.8%). Remarkably, our curriculum exceeds single-objective training on both metrics, demonstrating compositional prompt discovery beyond conventional multi-objective trade-offs. PhyPrompt outperforms GPT-4o (+3.8% joint) and DeepSeek-V3 (+2.2%, 100× larger) using only 7B parameters. The approach transfers zero-shot across diverse T2V architectures (Lavie, VideoCrafter2, CogVideoX-5B) with up to 16.8% improvement, establishing that domain-specialized reinforcement learning with compositional curricula surpasses general-purpose scaling for physics-aware generation.
Figures & tables
Figure 1: Video Generated by CogVideoX-5B Using Different Prompts. We compare the original prompt A wine is poured from a bottle in to a glass with three enhanced versions: manual rewrite, GPT-4o, and PhyPrompt. Semantic Adherence (SA) and Physical Commonsense (PC) scores are shown for each. The original prompt fails to depict rising liquid levels (PC:3). Manual rewriting with ”rises steadily” achieves perfect scores (SA:5, PC:5). GPT-4o emphasizes the ”visible stream of wine” but slightly reduces semantic alignment (SA:4, PC:5). PhyPrompt automatically generates ”slowly and smoothly,” matching manual rewrite quality (SA:5, PC:5) without human intervention.
Figure 2: PhyPrompt GRPO Pipeline. For each input prompt, the LLM (Qwen2.5 [ 51 ] ) generates multiple enhanced prompts and for each enhanced prompt, T2V generator (CogVideoX-2B [ 52 ] ) generates one video. Video evaluator (VideoPhy2-Auto [ 5 ] ) scores each video. We utilize GRPO to optimize the LLM.
Enhanced Model
SA (%) ↑
PC (%) ↑
SA & PC (%) ↑
Avg SA ↑
Avg PC ↑
N/A (Original Prompt)
43.4
55.8
32.2
3.842
4.410
Promptist [ 15 ]
41.2
58.2
30.2
3.782
4.452
PhyT2V [ 50 ]
45.0
62.0
36.2
3.854
4.522
Qwen2.5-1.5B
37.6
64.0
31.8
3.616
4.562
PhyPrompt-1.5B
42.8
65.4
37.4
3.800
4.580
Qwen2.5-3B
40.0
64.2
32.0
3.754
4.582
Table 1: VideoPhy2 quality results with CogVideoX-2B. For each prompt-rewriting strategy we report the percentage of videos scoring ≥4 for semantic adherence (SA) and physical commonsense (PC), their joint percentage (SA & PC), and the average SA/PC scores (Avg SA, Avg PC).
Figure 3: Cross-Generator Transfer of PhyPrompt. Performance comparison across different prompt enhancement methods on three text-to-video generation models (Lavie, CogVideoX-5B, and VideoCrafter2) measured by joint Physical Commonsense and Semantic Adherence (PC & SA) metrics. Our method consistently outperforms baseline approaches (Original, Promptist, PhyT2V) and similarly-sized Qwen2.5 models across all three video generation backbends. The highest performance (42.0%) is achieved by our 7B model on CogVideoX-5B. Gold stars indicate the best-performing method for each generator.
Figure 4: Training reward curves under the dynamic (orange) and static (purple) reward formulations. Each line is smoothed with a 10-step moving average; lightly shaded traces show the unsmoothed per-episode rewards. The dynamic curriculum converges more rapidly and plateaus higher (4.2) than the static baseline (3.5), indicating more effective optimization.
Figure 5: Frame‐by‐frame comparison of CogVideoX-2B outputs for the “hammer and nail” scenario. Top row shows results using the static‐reward prompt “ Show a heavy hammer striking a wooden plank with precision, ensuring that the impact creates a realistic sound effect and deformation of the wood surface. ” The hammer impacts the plank but no nail is visible or driven. Bottom row shows results using the dynamic‐reward prompt “ Heavy hammer hits the nail into the wood plank. The hammer’s force is concentrated, causing high pressure that drives the nail deep into the plank. ” Here the nail appears in frame 1, is pressed gradually into the wood across frames 2–3, and achieves full penetration by frame 4, with realistic wood deformation and depth of entry.
A can of soda is slowly poured out in the space station, releasing the liquid into the surrounding area.
Physical Law
Lack of gravity.
Explicit Caption
A can of soda is slowly poured out in the space station, releasing the liquid into the surrounding area. The liquid forms spherical droplets and floats freely in the air.
Chain of Thought
In microgravity (or “zero-g” conditions), there is no dominant downward force pulling the liquid. Instead of flowing downward as it would on Earth, the soda drifts away once it exits the can. Surface tension becomes the main force acting on the soda droplets, pulling the liquid into the smallest possible shape, a sphere. With no significant gravitational force to break them apart, these droplets remain spherical and continue floating until they contact another surface or droplet. Hence, in the absence of gravity, soda poured in a spacecraft forms and floats as spherical droplets.
Table 3: Example from our curated physics‐focused Chain‐of‐Thought dataset. The original caption sets the scene in microgravity, the annotated law (‘Lack of gravity’) defines the physical constraint, the explicit caption specifies the outcome (spherical droplets floating), and the chain of thought.
Enhanced Model
SA (%)
PC (%)
PC & SA (%)
N/A (Original Prompt)
41.0
57.0
29.2
Promptist
37.8
58.4
27.0
PhyT2V
41.2
60.8
30.0
Qwen2.5-1.5B
34.6
59.0
24.8
PhyPrompt-1.5B
40.2
61.2
30.6
Qwen2.5-3B
35.2
60.4
28.0
Table 4: Semantic Adherence (SA), Physical Commonsense (PC), and joint SA&PC scores for various prompt enhancement methods evaluated on the Lavie generator.
Enhanced Model
SA (%)
PC (%)
PC & SA (%)
N/A (Original Prompt)
45.6
48.0
29.8
Promptist
43.8
51.4
27.2
PhyT2V
45.0
56.8
30.6
Qwen2.5-1.5B
35.0
53.6
27.8
PhyPrompt-1.5B
41.4
57.4
30.2
Qwen2.5-3B
40.0
55.8
28.0
Table 5: Semantic Adherence (SA), Physical Commonsense (PC), and joint SA&PC scores for various prompt enhancement methods evaluated on the VideoCrafter2 generator.
Enhanced Model
SA (%)
PC (%)
PC&SA (%)
N/A (Original Prompt)
50.2
63.4
39.4
Promptist
46.4
63.0
37.4
PhyT2V
50.6
65.0
40.8
Qwen2.5-1.5B
39.2
66.4
31.0
PhyPrompt-1.5B
47.4
66.6
39.0
Qwen2.5-3B
45.2
64.6
35.8
Table 6: Semantic Adherence (SA), Physical Commonsense (PC), and joint SA&PC scores for various prompt enhancement methods evaluated on CogVideoX-5B .
Model
Core architecture
Size
Hidden Dim
Pos Enc
Max Prompt
CogVideoX-5B
3D causal VAE + Transformer
5B
4096
RoPE
226
CogVideoX-2B
3D causal VAE + Transformer
2B
4096
sinusoidal
226
Lavie
Cascaded Video Latent Diffusion
860M
768
RoPE
77
VideoCrafter 2
UNet + temporal convolution
1.2B
768
Fourier
77
VideoPhy2
7B Transformer with LoRA
7B
768
sinusoidal
77
Table 7: Model details reported across the four papers.