PhyRubric: Rubric-Grounded Diagnostic Evaluation of Physical Commonsense in Text-to-Video Generation
Organizations: Department of Computer Science, University of Illinois Chicago
Abstract
As text-to-video (T2V) models are increasingly positioned as world simulators, it is essential to evaluate whether their videos conform to physical commonsense. However, existing benchmarks rely on annotators who rate each video with a single score based on intuition, yielding ratings that are subjective and do not tell which physical principle is violated. We introduce PhyRubric, a rubric-grounded diagnostic framework for evaluating physical commonsense in T2V generation. For each caption, PhyRubric constructs a caption-specific rubric before any video is seen, decomposing the caption into semantic and physical items, and derives scores from item-level verdicts grounded in timestamped evidence. We instantiate this framework as PhyRubric-Bench, which spans 34 categories across five physical domains, and scale it up to PhyRubric-Train with 3,788 videos, on which we fine-tune the open-source PhyRubric-Judge. Empirical results across ten T2V models show that physical commonsense remains far from solved, and that PhyRubric-Judge correlates more closely with human judgments than existing open-source judges while matching the frontier proprietary Gemini 4 Argon. Further analysis provides actionable insights for T2V models.
Figures & tables
| Mechanics | Optics | Heat | Electricity | Acoustics | Avg | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | SA | PC | JP | SA | PC | JP | SA | PC | JP | SA | PC | JP | SA | PC | JP | SA | PC | JP |
| Proprietary Models | ||||||||||||||||||
| Seedance 2.5 [ 11 ] | 3.95 | 4.24 | 48.00 | 3.96 | 3.96 | 48.98 | 3.42 | 4.14 | 38.00 | 3.94 | 4.13 | 56.00 | 3.60 | 3.47 | 30.00 | 3.77 | 3.99 | 44.20 |
| Veo 3.1 [ 17 ] | 3.88 | 4.06 | 44.00 | 3.78 | 3.89 | 48.00 | 3.48 | 3.92 | 36.00 | 3.80 | 3.35 | 38.00 | 3.45 | 2.65 | 15.00 | 3.68 | 3.57 | 36.20 |
| Open-Source Models | ||||||||||||||||||
| LTX-Video-2B [ 22 ] | 2.28 | 2.59 | 5.33 | 2.56 | 2.94 | 6.00 | 2.20 | 3.73 | 2.00 | 2.30 | 3.42 | 2.00 | 2.00 | 2.50 | 5.00 | 2.27 | 3.04 | 4.07 |
| SA | PC | |||
| Judge | AUC | AUC | ||
| Zero-Shot VLMs | ||||
| Qwen3-VL-8B [ 5 ] | 0.46 | 0.78 | 0.02 | 0.53 |
| Qwen3-VL-32B [ 5 ] | 0.49 | 0.81 | 0.14 | 0.58 |
| Qwen3-VL-235B [ 5 ] | 0.46 | 0.81 | 0.14 | 0.57 |
| General Video Evaluators | ||||
| (a) Seen vs. unseen generators | ||||||||
|---|---|---|---|---|---|---|---|---|
| Seen | Unseen | |||||||
| SA | PC | SA | PC | |||||
| Judge | AUC | AUC | AUC | AUC | ||||
| Qwen3-VL-32B | 0.48 | 0.82 | 0.18 | 0.60 | 0.49 | 0.80 | 0.11 | 0.57 |
| VideoScore2 | 0.32 | 0.69 | 0.12 | 0.55 | 0.30 | 0.67 | 0.10 | 0.56 |
| VideoPhy-2-AutoEval | 0.25 | 0.66 | 0.24 | 0.64 | 0.18 | 0.62 | 0.23 | 0.63 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Data | VLM judge | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Benchmark | Mechanics | Optics | Heat | Electricity | Acoustics | Commonsense only | Rubric | Train set | Explainable | Open-source |
| VideoPhy [ 7 ] | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ |
| VideoPhy-2 [ 8 ] | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ |
| PhyWorldBench [ 19 ] | ✓ | ✓ | ◐ | ◐ | ✗ | ✗ | ✗ | ✗ | ◐ | ✗ |
| PhyGenBench [ 47 ] | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| T2VPhysBench [ 20 ] | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | – | – |
| Domain | Categories | #Train | #Eval |
|---|---|---|---|
| Mechanics | gravity and fall, inertia, collision, friction and rolling, projectile, balance and toppling, tension, deformation, cutting and tearing, fluid, granular, gas and wind, biomechanics, soft material, occlusion and containment | 242 | 75 |
| Heat | melting and freezing, boiling and steam, condensation, burning and smoke, heat response | 200 | 50 |
| Optics | shadow, reflection, refraction and transparency, illumination, color mixing | 227 | 50 |
| Electricity | switch and light, plug and power, lightning, spark, magnet, static | 202 | 50 |
| Acoustics | struck vibration, sound-driven vibration, source motion | 76 | 20 |
| Total | 34 categories | 947 | 245 |
| Type | Level | Definition | #Items |
|---|---|---|---|
| Permanence & count | L1 | Objects appear from nothing, vanish, teleport, duplicate, or merge. | 1,099 |
| Shape & identity | L1 | Rigid objects morph or melt, change identity, or bodies grow or lose parts. | 492 |
| Solidity | L1 | Matter passes through matter. | 198 |
| Causeless motion | L1 | Inert objects start, stop, or change direction with nothing acting on them. | 752 |
| Gravity direction | L2 | Unsupported objects hover, rise, or fall the wrong way. | 735 |
| Trajectory & momentum | L2 | Paths, speeds, or rebounds are inconsistent with how the object was set in motion. | 317 |
| Model | Resolution | Frames | FPS | Steps | CFG |
|---|---|---|---|---|---|
| CogVideoX1.5-5B | 81 | 16 | 50 | 6.0 | |
| HunyuanVideo-1.5-8B | 121 | 24 | 50 | 6.0 | |
| LTX-Video-2B | 121 | 24 | 8+4 | – | |
| Wan2.2-T2V-A14B | 81 | 16 | 40 | 4.0 / 3.0 | |
| Wan2.2-TI2V-5B | 121 | 24 | 50 | 5.0 | |
| LTX-2.3-22B | 121 | 24 | 8 | – |
| Family | Metric | Score | VideoCon-Physics | VideoPhy-2-AutoEval | VideoScore | VideoScore2 | Ours |
|---|---|---|---|---|---|---|---|
| A. Correlation | Pearson | S / H | ✓ | ✓ | ✓ | ||
| Spearman | S / H | ✓ | ✓ | ||||
| B. Ranking | ROC-AUC | S | ✓ | ✓ | |||
| C. Agreement | Binary Accuracy | pass line | ✓ | ✓ | |||
| F1 (binary) | pass line | ✓ | ✓ | ||||
| Exact accuracy | H | ✓ | ✓ |
| Judge | Dimension used (SA / PC) | Native output | Range | Pass line |
| Zero-shot VLMs | SA / PC (our prompt) | generated integer | 1–5 | 4 |
| VideoScore ‡ [ 26 ] | text-to-video alignment / factual consistency | regression output | 4 after mapping | |
| VideoScore2 [ 25 ] | T2V alignment / physical & common-sense consistency | generated integer | 1–5 | 4 |
| VideoCon-Physics | SA / PC prompts | 0.5 (official) | ||
| VideoPhy-2-AutoEval | SA / PC | generated integer | 1–5 | 4 (official) |
| Gemini 4 Argon § [ 18 ] | SA / PC (our prompt) | generated integer | 1–5 | 4 |
| SA | PC | ||||
| Judge | Score | ||||
| Zero-shot VLMs | |||||
| Qwen3-VL-8B | hard | 0.451 | 0.472 | 0.002 | 0.021 |
| soft | 0.459 | 0.495 | -0.001 | 0.043 | |
| Qwen3-VL-32B | hard | 0.467 | 0.502 | 0.143 | 0.146 |
| soft | 0.487 | 0.546 | 0.143 | 0.143 | |
| Judge | SA | PC |
|---|---|---|
| Zero-shot VLMs | ||
| Qwen3-VL-8B | 0.778 | 0.525 |
| Qwen3-VL-32B | 0.811 | 0.582 |
| Qwen3-VL-235B | 0.808 | 0.569 |
| General video evaluators | ||
| VideoScore | 0.477 | 0.589 |
| SA | PC | Joint | ||||||||
| (1) Binary | (2) Score-level | (1) Binary | (2) Score-level | (1) Binary | ||||||
| Judge | Acc | F1 | Exact | W-1 | Acc | F1 | Exact | W-1 | Acc | F1 |
| Zero-shot VLMs | ||||||||||
| Qwen3-VL-8B | 0.576 | 0.443 | 0.238 | 0.632 | 0.585 | 0.731 | 0.435 | 0.596 | 0.546 | 0.354 |
| Qwen3-VL-32B | 0.533 | 0.440 | 0.287 | 0.587 | 0.601 | 0.696 | 0.390 | 0.559 | 0.592 | 0.392 |
| Qwen3-VL-235B | 0.522 | 0.431 | 0.264 | 0.565 | 0.604 | 0.751 | 0.569 | 0.605 | 0.481 | 0.341 |
| Generator | Human | Ours | Argon |
|---|---|---|---|
| Seedance 2.5 | 44.20 | 32.30 | 12.98 |
| Veo 3.1 | 36.20 | 37.53 | 6.53 |
| HunyuanVideo-1.5-8B | 14.93 | 14.60 | 1.60 |
| Wan2.2-T2V-A14B | 14.87 | 20.13 | 1.60 |
| Cosmos-Predict2.5-14B | 12.22 | 19.34 | 2.93 |
| LTX-2.3-22B | 11.47 | 10.93 | 1.47 |
| SA | PC | |||
| Setting | AUC | AUC | ||
| Default (caption only) | 0.608 | 0.845 | 0.317 | 0.680 |
| (a) Rubric input | ||||
| + rubric input, scores only | 0.668 | 0.869 | 0.368 | 0.711 |
| + rubric input, item verdicts | 0.568 | 0.821 | 0.254 | 0.629 |
| Zero-shot, caption only | 0.463 | 0.779 | 0.023 | 0.534 |
| Mean | Pass rate | ||||
|---|---|---|---|---|---|
| Condition | SA | PC | SA | PC | Joint |
| Original video | 2.55 | 3.37 | 11 | 51 | 6 |
| New seed | 2.76 ∗ | 3.29 | 18 | 50 | 10 |
| Rewrite + diagnosis | 3.19 ∗† | 3.37 | 31 | 51 | 18 |
| Violation type | Mech. | Opt. | Heat | Elec. | Acou. | All |
| Missing consequence | 9.1 | 16.1 | 13.5 | 24.6 | 44.4 | 17.3 |
| Permanence & count | 18.5 | 14.2 | 8.3 | 15.8 | 15.5 | 14.7 |
| Causeless motion | 10.9 | 5.4 | 4.2 | 11.0 | 21.8 | 9.2 |
| Process direction | 1.8 | 4.1 | 11.2 | 13.0 | 1.4 | 6.4 |
| Attribute drift | 4.5 | 9.3 | 3.9 | 5.6 | 2.8 | 5.5 |
| Material response | 8.6 | 1.6 | 4.9 | 2.0 | 3.5 | 4.7 |
| Seedance | Veo | LTX-Video | Cosmos- | Wan2.2- | CogVideoX | Hunyuan | Wan2.2- | Cosmos- | LTX-2.3 | |
| Violation type | 2.5 | 3.1 | 2B | Predict2.5-2B | TI2V-5B | 1.5-5B | Video-1.5-8B | T2V-A14B | Predict2.5-14B | 22B |
| Missing consequence | 11.3 | 14.7 | 22.1 | 14.4 | 22.2 | 17.6 | 20.0 | 16.3 | 18.8 | 18.7 |
| Permanence & count | 5.2 | 12.9 | 22.8 | 13.8 | 22.2 | 33.3 | 8.6 | 10.7 | 8.6 | 16.3 |
| Causeless motion | 5.7 | 6.7 | 10.3 | 10.0 | 10.8 | 12.1 | 7.6 | 12.8 | 9.1 | 9.0 |
| Process direction | 3.0 | 7.1 | 2.2 | 9.4 | 9.7 | 6.1 | 5.4 | 5.6 | 9.7 | 6.0 |
| Attribute drift | 2.6 | 2.7 | 8.8 | 3.8 | 9.7 | 12.7 | 3.8 | 4.1 | 4.3 | 5.4 |
Physical Consistency in Video Generation
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
See all 205 papers