State-of-the-art text-to-video (T2V) generators frequently violate physical laws despite high visual quality. We show this stems from insufficient physical constraints in prompts rather than model limitations: manually adding physics details reliably produces physically plausible videos, but requires expertise and does not scale. We present PhyPrompt, a two-stage reinforcement learning framework that automatically refines prompts for physically realistic generation. First, we fine-tune a large language model on a physics-focused Chain-of-Thought dataset to integrate principles like object motion and force interactions while preserving user intent. Second, we apply Group Relative Policy Optimization with a dynamic reward curriculum that initially prioritizes semantic fidelity, then progressively shifts toward physical commonsense. This curriculum achieves synergistic optimization: PhyPrompt-7B reaches 40.8% joint success on VideoPhy2 (8.6pp gain), improving physical commonsense by 11pp (55.8% to 66.8%) while simultaneously increasing semantic adherence by 4.4pp (43.4% to 47.8%). Remarkably, our curriculum exceeds single-objective training on both metrics, demonstrating compositional prompt discovery beyond conventional multi-objective trade-offs. PhyPrompt outperforms GPT-4o (+3.8% joint) and DeepSeek-V3 (+2.2%, 100× larger) using only 7B parameters. The approach transfers zero-shot across diverse T2V architectures (Lavie, VideoCrafter2, CogVideoX-5B) with up to 16.8% improvement, establishing that domain-specialized reinforcement learning with compositional curricula surpasses general-purpose scaling for physics-aware generation.
Figures & tables
Figure 1: Video Generated by CogVideoX-5B Using Different Prompts. We compare the original prompt A wine is poured from a bottle in to a glass with three enhanced versions: manual rewrite, GPT-4o, and PhyPrompt. Semantic Adherence (SA) and Physical Commonsense (PC) scores are shown for each. The original prompt fails to depict rising liquid levels (PC:3). Manual rewriting with ”rises steadily” achieves perfect scores (SA:5, PC:5). GPT-4o emphasizes the ”visible stream of wine” but slightly reduces semantic alignment (SA:4, PC:5). PhyPrompt automatically generates ”slowly and smoothly,” matching manual rewrite quality (SA:5, PC:5) without human intervention.
Figure 2: PhyPrompt GRPO Pipeline. For each input prompt, the LLM (Qwen2.5 [ 51 ] ) generates multiple enhanced prompts and for each enhanced prompt, T2V generator (CogVideoX-2B [ 52 ] ) generates one video. Video evaluator (VideoPhy2-Auto [ 5 ] ) scores each video. We utilize GRPO to optimize the LLM.
Enhanced Model
SA (%) ↑
PC (%) ↑
SA & PC (%) ↑
Avg SA ↑
Avg PC ↑
N/A (Original Prompt)
43.4
55.8
32.2
3.842
4.410
Promptist [ 15 ]
41.2
58.2
30.2
3.782
4.452
PhyT2V [ 50 ]
45.0
62.0
36.2
3.854
4.522
Qwen2.5-1.5B
37.6
64.0
31.8
3.616
4.562
PhyPrompt-1.5B
42.8
65.4
37.4
3.800
4.580
Qwen2.5-3B
40.0
64.2
32.0
3.754
4.582
Table 1: VideoPhy2 quality results with CogVideoX-2B. For each prompt-rewriting strategy we report the percentage of videos scoring ≥4 for semantic adherence (SA) and physical commonsense (PC), their joint percentage (SA & PC), and the average SA/PC scores (Avg SA, Avg PC).
Figure 3: Cross-Generator Transfer of PhyPrompt. Performance comparison across different prompt enhancement methods on three text-to-video generation models (Lavie, CogVideoX-5B, and VideoCrafter2) measured by joint Physical Commonsense and Semantic Adherence (PC & SA) metrics. Our method consistently outperforms baseline approaches (Original, Promptist, PhyT2V) and similarly-sized Qwen2.5 models across all three video generation backbends. The highest performance (42.0%) is achieved by our 7B model on CogVideoX-5B. Gold stars indicate the best-performing method for each generator.
Figure 4: Training reward curves under the dynamic (orange) and static (purple) reward formulations. Each line is smoothed with a 10-step moving average; lightly shaded traces show the unsmoothed per-episode rewards. The dynamic curriculum converges more rapidly and plateaus higher (4.2) than the static baseline (3.5), indicating more effective optimization.
Figure 5: Frame‐by‐frame comparison of CogVideoX-2B outputs for the “hammer and nail” scenario. Top row shows results using the static‐reward prompt “ Show a heavy hammer striking a wooden plank with precision, ensuring that the impact creates a realistic sound effect and deformation of the wood surface. ” The hammer impacts the plank but no nail is visible or driven. Bottom row shows results using the dynamic‐reward prompt “ Heavy hammer hits the nail into the wood plank. The hammer’s force is concentrated, causing high pressure that drives the nail deep into the plank. ” Here the nail appears in frame 1, is pressed gradually into the wood across frames 2–3, and achieves full penetration by frame 4, with realistic wood deformation and depth of entry.
A can of soda is slowly poured out in the space station, releasing the liquid into the surrounding area.
Physical Law
Lack of gravity.
Explicit Caption
A can of soda is slowly poured out in the space station, releasing the liquid into the surrounding area. The liquid forms spherical droplets and floats freely in the air.
Chain of Thought
In microgravity (or “zero-g” conditions), there is no dominant downward force pulling the liquid. Instead of flowing downward as it would on Earth, the soda drifts away once it exits the can. Surface tension becomes the main force acting on the soda droplets, pulling the liquid into the smallest possible shape, a sphere. With no significant gravitational force to break them apart, these droplets remain spherical and continue floating until they contact another surface or droplet. Hence, in the absence of gravity, soda poured in a spacecraft forms and floats as spherical droplets.
Table 3: Example from our curated physics‐focused Chain‐of‐Thought dataset. The original caption sets the scene in microgravity, the annotated law (‘Lack of gravity’) defines the physical constraint, the explicit caption specifies the outcome (spherical droplets floating), and the chain of thought.
Enhanced Model
SA (%)
PC (%)
PC & SA (%)
N/A (Original Prompt)
41.0
57.0
29.2
Promptist
37.8
58.4
27.0
PhyT2V
41.2
60.8
30.0
Qwen2.5-1.5B
34.6
59.0
24.8
PhyPrompt-1.5B
40.2
61.2
30.6
Qwen2.5-3B
35.2
60.4
28.0
Table 4: Semantic Adherence (SA), Physical Commonsense (PC), and joint SA&PC scores for various prompt enhancement methods evaluated on the Lavie generator.
Enhanced Model
SA (%)
PC (%)
PC & SA (%)
N/A (Original Prompt)
45.6
48.0
29.8
Promptist
43.8
51.4
27.2
PhyT2V
45.0
56.8
30.6
Qwen2.5-1.5B
35.0
53.6
27.8
PhyPrompt-1.5B
41.4
57.4
30.2
Qwen2.5-3B
40.0
55.8
28.0
Table 5: Semantic Adherence (SA), Physical Commonsense (PC), and joint SA&PC scores for various prompt enhancement methods evaluated on the VideoCrafter2 generator.
Enhanced Model
SA (%)
PC (%)
PC&SA (%)
N/A (Original Prompt)
50.2
63.4
39.4
Promptist
46.4
63.0
37.4
PhyT2V
50.6
65.0
40.8
Qwen2.5-1.5B
39.2
66.4
31.0
PhyPrompt-1.5B
47.4
66.6
39.0
Qwen2.5-3B
45.2
64.6
35.8
Table 6: Semantic Adherence (SA), Physical Commonsense (PC), and joint SA&PC scores for various prompt enhancement methods evaluated on CogVideoX-5B .
Model
Core architecture
Size
Hidden Dim
Pos Enc
Max Prompt
CogVideoX-5B
3D causal VAE + Transformer
5B
4096
RoPE
226
CogVideoX-2B
3D causal VAE + Transformer
2B
4096
sinusoidal
226
Lavie
Cascaded Video Latent Diffusion
860M
768
RoPE
77
VideoCrafter 2
UNet + temporal convolution
1.2B
768
Fourier
77
VideoPhy2
7B Transformer with LoRA
7B
768
sinusoidal
77
Table 7: Model details reported across the four papers.
Generative video models achieve high visual fidelity but often violate basic physical principles, limiting reliability in real-world settings. Prior attempts to inject physics rely on conditioning: frame-level signals are domain-specific and short-horizon, while global text prompts are coarse and noisy, missing fine-grained dynamics. We present PhysVid, a physics-aware local conditioning scheme that operates over temporally contiguous chunks of frames. Each chunk is annotated with physics-grounded descriptions of states, interactions, and constraints, which are fused with the global prompt via chunk-aware cross-attention during training. At inference, we introduce negative physics prompts (descriptions of locally relevant law violations) to steer generation away from implausible trajectories. On VideoPhy, PhysVid improves physical commonsense scores by ≈33% over baseline video generators, and by up to ≈8% on VideoPhy2. These results show that local, physics-aware guidance substantially increases physical plausibility in generative video and marks a step toward physics-grounded video models.
While diffusion-based text-to-video (T2V) models have demonstrated impressive capability in generating realistic and temporally coherent videos, they often fail to respect fundamental physical dynamics. Although recent physics-constrained methods incorporate explicit dynamics priors to improve physical plausibility, they remain limited to simple single-type motions, depend on manually specified parameters, and struggle to generalize to unseen physical laws. In this work, we propose CompAdapt, a physics-consistent T2V framework for adaptable generation across complex real-world scenarios. It extends neural dynamics modeling beyond single-type motions to encompass composite physical behaviors, including coupled motions, multi-stage transitions, and multi-object collisions. Furthermore, CompAdapt translates natural language prompts into structured physical semantics, enabling end-to-end specification of motion types, temporal relations, and initial physical parameters. To generalize to novel physical environments, CompAdapt introduces dynamics-aware prior matching, achieving one-shot adaptation without retraining the core dynamics module. In addition, a physics-aware latent feature fusion module improves visual fidelity under fast and complex motion. Experiments on physics-focused T2V benchmarks demonstrate that CompAdapt improves physical consistency over both general T2V models and physics-constrained baselines, while preserving high visual quality and adaptability to unseen dynamics. The project page is available at https://makapic.github.io/CompAdapt/ .
Haoran Qin, Renlong Wu, Tianyu Huang +3
Harbin Institute of Technology, China · Taobao, Alibaba Group, China
Text-to-video (T2V) models are highly sensitive to prompt specificity, while iterative prompt optimization typically improves generation quality only through repeated video generation and vision-language model evaluation at inference time. We propose TIDE, a framework that distills this costly search process into an interpretable and reusable prompt-rewriting Skill. TIDE formulates prompt rewriting as a set of source-conditioned rule modules, each targeting a distinct semantic dimension. Once distilled, the frozen Skill activates the relevant rules and generates both positive and negative prompts in a single LLM call, followed by only one T2V generation, without test-time visual search or backbone fine-tuning. Evolved using only 100 prompts over 14 hours, TIDE achieves prompt-optimization performance comparable to computationally expensive search-based methods. Moreover, when transferred to previously unseen video-generation models, TIDE outperforms existing approaches specifically designed or trained for those models. Code is available at https://github.com/xiningin/TIDE_T2V.
Chengxing Zhou, Yuliang Huang, Jingwei Song
Sun Yat-sen University, China · The University of Hong Kong, China