PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA

Siwei Meng; Xuerou Cai; Rui Hu; Ping Liu

cs.CVOct 14, 2026

PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA

Authors: Siwei Meng, Xuerou Cai, Rui Hu, Ping Liu

Organizations: Department of Computer Science & Engineering University of Nevada, Reno Reno, NV 89512

Abstract

Text-to-video (T2V) generation models can produce visually realistic videos that nevertheless violate the physical consequences implied by the prompt and scene. Existing benchmarks largely rely on holistic plausibility scores or broad quality dimensions, leaving it unclear which physical behavior is being tested and why a model fails. We introduce PhyMetric, a scene-level diagnostic benchmark that recasts physical plausibility evaluation as verifiable, scene-grounded question answering. PhyMetric contains 1{,}115 scenes spanning 50 physical categories across six domains, paired with 8{,}920 answer-balanced binary questions over 12 evaluation dimensions. Each scene is associated with eight targeted yes/no questions and a structured physical prior specifying the governing law, expected temporal sequence, observable signatures, and common failure modes. At evaluation time, a vision-language model (VLM) judge answers each question from question-aware sampled frames conditioned on the prior, and correctness is hierarchically aggregated into scene, dimension, physical category, and model scores. Experiments on four open-source T2V models and three VLM judges show that scene-specific binary QA is a critical component of the protocol: when each targeted question is replaced by a single generic plausibility query, evaluator accuracy drops to the chance level. Structured physical priors offer a complementary signal that improves human alignment of the resulting scores, with effects that vary across VLM judges of different capacity. These findings establish PhyMetric as a benchmark that turns physical plausibility evaluation from opaque holistic scoring into interpretable, scene-level inspection. Our project page is available at: https://mengsiwei.github.io/PhyMetric/.

Figures & tables

Figure 1: Why holistic plausibility is insufficient for physical evaluation. Generic plausibility questions tend to yield high-confidence but unreliable judgments by overlooking scene-specific physical consequences. PhyMetric instead decomposes evaluation into structured diagnostic QA with question-aware frame sampling, enabling more robust and interpretable physical assessment. In the right subfigure, as confidence thresholds increase, generic-question accuracy degrades sharply and only achieves around 50% accuracy, whereas structured scene-specific questions remain substantially more stable.

Benchmark	Main target	Scenes / Prompts	Questions	QA Format	Physics / Prior	Balanced Labels	Year
General and fine-grained video generation evaluation
VBench Huang et al. (2024)	General video quality	–	–	✗	$\sim$	✗	2024
EvalCrafter Liu et al. (2024)	General video quality	700	–	✗	✗	✗	2024
FETV Liu et al. (2023)	Fine-grained T2V eval.	619	–	✗	✗	✗	2025
T2V-CompBench Sun et al. (2025)	Compositional T2V	700	–	$\sim$	✗	✗	2025
Video-Bench Han et al. (2025)	Human-aligned gen. eval.	419	–	$\sim$	$\sim$	✗	2025

Table 1: Comparison with existing video generation benchmarks. We group prior benchmarks into two main aspects. ✓denotes explicit support, ✗denotes no support, and

\sim

denotes partial support. Balanced Labels indicates whether positive/negative answers are balanced for an interpretable chance baseline. Compared with broad video-generation and physics-oriented benchmarks, PhyMetric combines large-scale scene coverage, scene-specific yes/no QA, structured physical priors, and balanced labels for physical plausibility evaluation.

Figure 2: PhyMetric benchmark construction. PhyMetric organizes physical video generation into a structured taxonomy with 50 categories across six domains. Each category is defined by physical principles, object pools, environment pools, and diagnostic dimensions. These metadata are expanded into diverse scenes and balanced scene-specific binary questions. The bottom examples show representative physical events.

Figure 3: PhyMetric evaluation pipeline. Given a physical scene $s$ , a text-to-video model generates a video $V_{s}$ from the scene caption $c_{s}$ . For each scene-specific binary question, PhyMetric samples question-aware frames and provides a structured physical prior $K_{s}$ describing the expected law, outcome, and common failure modes. A VLM then answers each question conditioned on the sampled frames and $K_{s}$ . Question correctness is averaged into the scene score $S(s,V_{s})$ , and model performance is obtained by averaging scene scores over all benchmark scenes. This pipeline turns holistic video evaluation into grounded, interpretable physical diagnosis.

Table 5

Figure 4: Per-dimension physical performance of current T2V models. Accuracy across 12 evaluation dimensions for four T2V models, with diamonds denoting mean accuracy. Radar plots showing each model’s dimension profile and the dimensions where it achieves the best performance.

Table 7

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Categories	Scenes	Representative phenomena
Mechanics	12	280	gravity, collisions, friction, projectile, pendulum
Fluid Dynamics	10	230	buoyancy, viscosity, splash, vortex, granular flow
Material	8	180	fracture, slicing, cloth, smoke, explosion
Thermal	8	165	melting, boiling, freezing, convection, condensation
Chemistry	7	145	diffusion, adsorption, dissolution, precipitation
Electromagnetism	5	115	reflection, refraction, shadow, electrostatics, magnetism

Appendix

Table 6: Six physics domains in PhyMetric with category and scene counts.

Dimension	What it asks about
existence	whether a specific object, region, or event is present in the video
size	absolute or relative size of an object or region
appearance/texture	color, surface pattern, transparency, sheen, smoothness
shape	geometric form and how it changes during the event
motion	whether/how something moves, including speed and acceleration
number	object count, including stages or instances of a repeating event

Appendix

Table 7: The 12 question dimensions used to tag every binary question. Each question carries 1–2 dimension tags; each scene covers

\geq

3 distinct dimensions across its 8 questions.

Model	Role	Notes
Qwen3.5-9B (thinking)	Default backbone	Thinking suppressed via chat-template flag
Qwen2.5-VL-7B-Instruct	Ablation	Text-only mode; enable_thinking=False
DeepSeek-R1-Distill-Llama-8B	Ablation	Strips <think>...</think> ; budget 16,384 tokens
Mistral-7B-Instruct-v0.3	Ablation	Drop-in via TextLLM wrapper

Appendix

Table 8: LLM backbones used for structured physical prior generation. All models are run text-only and the default backbone for the released benchmark is Qwen3.5-9B in text mode.

Judge	Family	Source	Params	Role
Qwen3.5-9B	Qwen3.5 VL	Open-source	9 B	Default judge
Qwen2.5-VL-7B-Instruct	Qwen2.5 VL	Open-source	7 B	Same family, earlier generation
GPT-5.4	GPT (OpenAI)	Closed-source	—	Cross-family closed-source judge

Appendix

Table 9: VLM judges used in PhyMetric . “Default” marks the judge whose scores are reported in Table 3 and Table 3 , and the others are used for cross-evaluator robustness checks in subsection 4.4 ).

Generator	Architecture	Params	Resolution	Frames	FPS	Dur. (s)	Scenes
CogVideoX-5B	DiT (T2V)	5 B	720 $\times$ 480	81	16	5.06	1,115
Wan2.1-1.3B	Wan T2V	1.3 B	832 $\times$ 480	81	16	5.06	1,115
Wan2.2-14B	Wan MoE T2V	14 B	832 $\times$ 480	81	16	5.06	1,115
Cosmos-7B	Cosmos Diffusion T2W	7 B	1280 $\times$ 704	121	30	4.03	1,115

Appendix

Table 10: Text-to-video generators evaluated by PhyMetric .

Condition	Prior?	Specific Q?	Tests
A1	✓	✓	Full system (reference)
A2		✓	Contribution of the structured prior
A3			Raw VLM vision baseline
A4	✓		Prior without scene-specific QA

Appendix

Table 11: Part A —

2\times 2

component ablation: structured physical prior crossed with question specificity. A1 is the full system; A2–A4 each disable one component. All conditions are evaluated on every video model with all three VLM judges.

Figure 5: The interface of PhyMetric human study.

Video Generation Evaluation

Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.

See all 191 papers

PhysVidBench: Language-Grounded Evaluation of Physical Commonsense in Text-to-Video Models
Jul 21, 2025Enes Sanli, Baris Sarper Tezcan, Aykut Erdem +1Physical Consistency in Video Generation Physical Reasoning in Video
Koç University, Istanbul, Turkey · KUIS AI Center, Istanbul, Turkey · Purdue University, Indiana, USA +1
Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation
Jun 24, 2026Atin Pothiraj, Jaemin Cho, Yue Zhang +2VLM Evaluation Physical Consistency in Video Generation
University of North Carolina at Chapel Hill, USA · AI2, USA · Johns Hopkins University, USA +1
PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements
Oct 19, 2026Kaiwei Chen, Moran Li, Zhenye Gan +5Physical Consistency in Video Generation Video Generation Evaluation
APRIL Lab, Zhejiang University · Huazhong University of Science and Technology · Tencent Youtu Lab

Jul 21, 2025J/K move · Enter open · S save

PhysVidBench: Language-Grounded Evaluation of Physical Commonsense in Text-to-Video Models

Enes Sanli, Baris Sarper Tezcan, Aykut Erdem, Erkut Erdem

Koç University, Istanbul, Turkey · KUIS AI Center, Istanbul, Turkey · Purdue University, Indiana, USA+1

Physical Consistency in Video Generation Physical Reasoning in Video Video Generation Evaluation

Open paper AskPDF arXiv

Text-to-video (T2V) models now produce striking visuals, yet they routinely violate everyday physics; objects float, tools are misused, and causal sequences break down. Existing benchmarks mostly probe isolated physical laws and rely on vision-language models to score videos directly, which entangles perception and reasoning in one judgment and correlates poorly with humans. We introduce PhysVidBench, a benchmark of 383 base prompts, expanded to 766 prompts with enriched variants and 4,123 manually reviewed QA items, together with a language-grounded evaluation framework that takes a different route: instead of asking a VLM "Is this video physically correct?", we caption the video, then ask a language model to answer prompt-derived yes/no physics questions using only the captions. This split between seeing and reasoning makes each score traceable to the captions that justify it and aligns more closely with human judgment than direct VLM scoring (Pearson r=0.45-0.69, compared with 0.21-0.47 for the strongest direct VLM evaluator). To test the framework, we carefully curate a set of human-validated prompts spanning seven physical dimensions, with a focus on tool use and affordances - areas largely absent from prior physics-focused benchmarks. Across 12 open and proprietary T2V systems, the best model reaches only 36.2% average accuracy, and no model consistently handles everyday physical reasoning. The same pipeline can also guide iterative error-guided prompt refinement, improving CogVideoX-2B from 21.6 to 32.7 and CogVideoX-5B from 17.8 to 29.7 without retraining.

Abstract

Video Generation Evaluation

Explore similar work

PhysVidBench: Language-Grounded Evaluation of Physical Commonsense in Text-to-Video Models

Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation

PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements