cs.CVOct 14, 2026

PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA

Authors: Siwei Meng, Xuerou Cai, Rui Hu, Ping Liu

Organizations: Department of Computer Science & Engineering University of Nevada, Reno Reno, NV 89512

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Text-to-video (T2V) generation models can produce visually realistic videos that nevertheless violate the physical consequences implied by the prompt and scene. Existing benchmarks largely rely on holistic plausibility scores or broad quality dimensions, leaving it unclear which physical behavior is being tested and why a model fails. We introduce PhyMetric, a scene-level diagnostic benchmark that recasts physical plausibility evaluation as verifiable, scene-grounded question answering. PhyMetric contains 1{,}115 scenes spanning 50 physical categories across six domains, paired with 8{,}920 answer-balanced binary questions over 12 evaluation dimensions. Each scene is associated with eight targeted yes/no questions and a structured physical prior specifying the governing law, expected temporal sequence, observable signatures, and common failure modes. At evaluation time, a vision-language model (VLM) judge answers each question from question-aware sampled frames conditioned on the prior, and correctness is hierarchically aggregated into scene, dimension, physical category, and model scores. Experiments on four open-source T2V models and three VLM judges show that scene-specific binary QA is a critical component of the protocol: when each targeted question is replaced by a single generic plausibility query, evaluator accuracy drops to the chance level. Structured physical priors offer a complementary signal that improves human alignment of the resulting scores, with effects that vary across VLM judges of different capacity. These findings establish PhyMetric as a benchmark that turns physical plausibility evaluation from opaque holistic scoring into interpretable, scene-level inspection. Our project page is available at: https://mengsiwei.github.io/PhyMetric/.

Figures & tables

Figure 1: Why holistic plausibility is insufficient for physical evaluation. Generic plausibility questions tend to yield high-confidence but unreliable judgments by overlooking scene-specific physical consequences. PhyMetric instead decomposes evaluation into structured diagnostic QA with question-aware frame sampling, enabling more robust and interpretable physical assessment. In the right subfigure, as confidence thresholds increase, generic-question accuracy degrades sharply and only achieves around 50% accuracy, whereas structured scene-specific questions remain substantially more stable.
BenchmarkMain targetScenes / PromptsQuestionsQA FormatPhysics / PriorBalanced LabelsYear
General and fine-grained video generation evaluation
VBench Huang et al. (2024)General video quality––✗∼\sim✗2024
EvalCrafter Liu et al. (2024)General video quality700–✗✗✗2024
FETV Liu et al. (2023)Fine-grained T2V eval.619–✗✗✗2025
T2V-CompBench Sun et al. (2025)Compositional T2V700–∼\sim✗✗2025
Video-Bench Han et al. (2025)Human-aligned gen. eval.419–∼\sim∼\sim✗2025
Table 1: Comparison with existing video generation benchmarks. We group prior benchmarks into two main aspects. ✓denotes explicit support, ✗denotes no support, and ∼\sim denotes partial support. Balanced Labels indicates whether positive/negative answers are balanced for an interpretable chance baseline. Compared with broad video-generation and physics-oriented benchmarks, PhyMetric combines large-scale scene coverage, scene-specific yes/no QA, structured physical priors, and balanced labels for physical plausibility evaluation.
Figure 2: PhyMetric benchmark construction. PhyMetric organizes physical video generation into a structured taxonomy with 50 categories across six domains. Each category is defined by physical principles, object pools, environment pools, and diagnostic dimensions. These metadata are expanded into diverse scenes and balanced scene-specific binary questions. The bottom examples show representative physical events.
Figure 3: PhyMetric evaluation pipeline. Given a physical scene ss , a text-to-video model generates a video VsV_{s} from the scene caption csc_{s} . For each scene-specific binary question, PhyMetric samples question-aware frames and provides a structured physical prior KsK_{s} describing the expected law, outcome, and common failure modes. A VLM then answers each question conditioned on the sampled frames and KsK_{s} . Question correctness is averaged into the scene score S(s,Vs)S(s,V_{s}) , and model performance is obtained by averaging scene scores over all benchmark scenes. This pipeline turns holistic video evaluation into grounded, interpretable physical diagnosis.
Table 5
Figure 4: Per-dimension physical performance of current T2V models. Accuracy across 12 evaluation dimensions for four T2V models, with diamonds denoting mean accuracy. Radar plots showing each model’s dimension profile and the dimensions where it achieves the best performance.
Table 7
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

DomainCategoriesScenesRepresentative phenomena
Mechanics12280gravity, collisions, friction, projectile, pendulum
Fluid Dynamics10230buoyancy, viscosity, splash, vortex, granular flow
Material8180fracture, slicing, cloth, smoke, explosion
Thermal8165melting, boiling, freezing, convection, condensation
Chemistry7145diffusion, adsorption, dissolution, precipitation
Electromagnetism5115reflection, refraction, shadow, electrostatics, magnetism
Appendix
Table 6: Six physics domains in PhyMetric with category and scene counts.
DimensionWhat it asks about
existencewhether a specific object, region, or event is present in the video
sizeabsolute or relative size of an object or region
appearance/texturecolor, surface pattern, transparency, sheen, smoothness
shapegeometric form and how it changes during the event
motionwhether/how something moves, including speed and acceleration
numberobject count, including stages or instances of a repeating event
Appendix
Table 7: The 12 question dimensions used to tag every binary question. Each question carries 1–2 dimension tags; each scene covers ≥\geq 3 distinct dimensions across its 8 questions.
ModelRoleNotes
Qwen3.5-9B (thinking)Default backboneThinking suppressed via chat-template flag
Qwen2.5-VL-7B-InstructAblationText-only mode; enable_thinking=False
DeepSeek-R1-Distill-Llama-8BAblationStrips <think>...</think> ; budget 16,384 tokens
Mistral-7B-Instruct-v0.3AblationDrop-in via TextLLM wrapper
Appendix
Table 8: LLM backbones used for structured physical prior generation. All models are run text-only and the default backbone for the released benchmark is Qwen3.5-9B in text mode.
JudgeFamilySourceParamsRole
Qwen3.5-9BQwen3.5 VLOpen-source9 BDefault judge
Qwen2.5-VL-7B-InstructQwen2.5 VLOpen-source7 BSame family, earlier generation
GPT-5.4GPT (OpenAI)Closed-source—Cross-family closed-source judge
Appendix
Table 9: VLM judges used in PhyMetric . “Default” marks the judge whose scores are reported in Table 3 and Table 3 , and the others are used for cross-evaluator robustness checks in subsection 4.4 ).
GeneratorArchitectureParamsResolutionFramesFPSDur. (s)Scenes
CogVideoX-5BDiT (T2V)5 B720 ×\times 48081165.061,115
Wan2.1-1.3BWan T2V1.3 B832 ×\times 48081165.061,115
Wan2.2-14BWan MoE T2V14 B832 ×\times 48081165.061,115
Cosmos-7BCosmos Diffusion T2W7 B1280 ×\times 704121304.031,115
Appendix
Table 10: Text-to-video generators evaluated by PhyMetric .
ConditionPrior?Specific Q?Tests
A1✓✓Full system (reference)
A2 ✓Contribution of the structured prior
A3  Raw VLM vision baseline
A4✓ Prior without scene-specific QA
Appendix
Table 11: Part A — 2×22\times 2 component ablation: structured physical prior crossed with question specificity. A1 is the full system; A2–A4 each disable one component. All conditions are evaluated on every video model with all three VLM judges.
Figure 5: The interface of PhyMetric human study.
More in this topic

Video Generation Evaluation

Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.

See all 191 papers

Explore similar work

CardsList
Jul 21, 2025

PhysVidBench: Language-Grounded Evaluation of Physical Commonsense in Text-to-Video Models

Text-to-video (T2V) models now produce striking visuals, yet they routinely violate everyday physics; objects float, tools are misused, and causal sequences break down. Existing benchmarks mostly probe isolated physical laws and rely on vision-language models to score videos directly, which entangles perception and reasoning in one judgment and correlates poorly with humans. We introduce PhysVidBench, a benchmark of 383 base prompts, expanded to 766 prompts with enriched variants and 4,123 manually reviewed QA items, together with a language-grounded evaluation framework that takes a different route: instead of asking a VLM "Is this video physically correct?", we caption the video, then ask a language model to answer prompt-derived yes/no physics questions using only the captions. This split between seeing and reasoning makes each score traceable to the captions that justify it and aligns more closely with human judgment than direct VLM scoring (Pearson r=0.45-0.69, compared with 0.21-0.47 for the strongest direct VLM evaluator). To test the framework, we carefully curate a set of human-validated prompts spanning seven physical dimensions, with a focus on tool use and affordances - areas largely absent from prior physics-focused benchmarks. Across 12 open and proprietary T2V systems, the best model reaches only 36.2% average accuracy, and no model consistently handles everyday physical reasoning. The same pipeline can also guide iterative error-guided prompt refinement, improving CogVideoX-2B from 21.6 to 32.7 and CogVideoX-5B from 17.8 to 29.7 without retraining.

Enes Sanli, Baris Sarper Tezcan, Aykut Erdem +1

Koç University, Istanbul, Turkey · KUIS AI Center, Istanbul, Turkey · Purdue University, Indiana, USA +1

Physical Consistency in Video GenerationPhysical Reasoning in VideoVideo Generation Evaluation
AskPDF
Jun 24, 2026

Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation

Video generation models are increasingly capable of producing realistic videos, but they still struggle to generate videos that follow basic physical laws. Compounding this is a lack of reliable granular evaluation methods for localizing and specifying physical law violations in videos. We address this by introducing Physics Question Scene Graph (PQSG), a hierarchical question-based evaluation pipeline. PQSG evaluates generated videos by checking their faithfulness to a prompt across objects, actions, and adherence to physical laws using a graph-based hierarchy of questions generated by a vision-language model (VLM), guided by high-quality in-context examples. By representing questions as a graph, PQSG introduces logical dependencies within questions, ensuring that each query is contextually valid. Moreover, PQSG provides granular assessments of which qualities of the video violate physical plausibility constraints. We validate PQSG by creating FinePhyEval, a dataset with physics-based prompts and corresponding generated videos from diverse state-of-the-art video generation models (Sora 2, Veo 3, and Wan 2.1), with each video annotated across multiple categories by humans. Using FinePhyEval, we measure the correlation between PQSG's fine-grained scores and human judgments, showing higher overall correlations than prior work. We also find that PQSG ranks closed-source models higher than Wan 2.1 on physical realism. Lastly, we show that the annotations we provide in FinePhyEval can also be used for subtask evaluation: we benchmark two strong VLMs on generating and answering questions, finding that while models can create human-like questions, they still fall short of human performance in answering them.

Atin Pothiraj, Jaemin Cho, Yue Zhang +2

University of North Carolina at Chapel Hill, USA · AI2, USA · Johns Hopkins University, USA +1

VLM EvaluationPhysical Consistency in Video GenerationVideo Generation Evaluation
AskPDF
Oct 19, 2026

PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements

Recent video generation models produce visually compelling outputs, yet their physical plausibility remains poorly measured. Physical plausibility asks whether generated objects obey geometry, kinematics, dynamics, and conservation laws. Existing benchmarks rely on coarse visual-language model judgments or target embodied world models rather than video generation itself. We introduce PVBench, a physical video benchmark that quantitatively measures the plausibility of generated videos across 46 dimensions organized into seven domains, spanning: i) instruction and semantic fidelity, ii) behavioral and causal consistency, iii) perceptual and temporal quality, iv) geometric and spatial consistency, v) material properties and field states, vi) kinematic and spatiotemporal persistence, vii) and mechanical plausibility with conservation laws. PVBench comprises 600 carefully crafted prompts, of which 30% involve physical embodied agents such as robot arms, humanoids, quadrupeds, and multi-agent systems. For challenging physically tractable dimensions, it pairs each with specialized vision models that derive verifiable, physically grounded measurements, such as object-level 6D pose, metric scale, surface normals, kinematics, and numerical conservation-law residuals. It thus avoids relying solely on subjective scoring, while supporting both text-to-video and image-to-video generation. Finally, we validate the agreement between automatic metrics and human judgments through human evaluation. Meanwhile, we evaluate mainstream open-source and closed-source models and present several insights. By releasing the benchmark, prompts, and calibration protocol, we aim to foster physically grounded video generation.

Kaiwei Chen, Moran Li, Zhenye Gan +5

APRIL Lab, Zhejiang University · Huazhong University of Science and Technology · Tencent Youtu Lab

Physical Consistency in Video GenerationVideo Generation EvaluationGeometric Consistency in Video GenerationTemporal Consistency in Video Generation
AskPDF