PhysicsLENS: Diagnosing Physical Property Blindness in Video Generation Models
Organizations: School of Computing and Augmented Intelligence Arizona State University Tempe, AZ, USA
Abstract
Reliable video world models could provide scalable predictive environments for robot learning, planning, and evaluation. However, generated robot videos can violate physical principles and complete tasks through physically implausible behavior, limiting their reliability for robot learning and planning. Current video-generation benchmarks exclude physics that are inherently hidden by visuals (e.g., weight, viscosity, friction). Due to this, video models are evaluated on the fidelity of physics, not the underlying accuracy of physics. We introduce PhysicsLENS, a dataset and benchmark for evaluating plausibility of physical properties grounded in robotics. PhysicsLENS uses matched scenario pairs that hold the same conditioning frame and task, while varying underlying physics in the scene description. Scenarios are curated from public robot video sources and annotated across seven physical domains: collision, gravity, momentum, friction, deformation, fluid, and causality. We evaluate across four video generation models, producing over 400 human-annotated labels. Results show that plausible-looking videos often ignore the stated property (34 of 47), and that stating the property lowers plausibility only slightly and not significantly.
Figures & tables
| Category | Method | Physics | Robotics | Human | Ref.-free | Localized | Obs. axis |
| Physics benchmarks | VideoPhy-2 ( Bansal et al., 2026 ) | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ |
| PhyGenBench ( Meng et al., 2025 ) | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | |
| Morpheus ( Zhang et al., 2025 ) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| Physics-IQ ( Motamed et al., 2026 ) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| NewtonRewards ( Le et al., 2025 ) | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | |
| GeoPhys ( Internò et al., 2026 ) | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ |
| Physical plausibility | Task | Violation | Physics | Hidden | |||
| System | Obs. | Unobs. | All | completion | detection | family | property |
| Generator identity | 0.63 0.03 | 0.72 0.05 | 0.65 0.03 | 0.72 0.02 | 0.73 0.03 | 0.51 0.02 | 0.68 0.05 |
| Signals only | 0.61 0.03 | 0.65 0.05 | 0.62 0.03 | 0.70 0.03 | 0.70 0.03 | 0.51 0.02 | 0.69 0.05 |
| VLM † | 0.53 0.06 | 0.53 0.06 | 0.53 0.05 | 0.71 0.04 | 0.58 0.08 | 0.59 0.02 | 0.57 0.03 |
| VLM + signals † | 0.60 0.02 | 0.66 0.01 | 0.61 0.01 | 0.76 0.02 | 0.72 0.01 | 0.58 0.02 | 0.70 0.02 |
| Debiased VLM † | 0.64 0.07 | 0.63 0.04 | 0.63 0.06 | 0.71 0.04 | 0.58 0.08 | 0.59 0.02 | 0.57 0.03 |
| Standard | Physics-error | + signals | Hidden | ||||
|---|---|---|---|---|---|---|---|
| VLM | Obs. | Unobs. | Obs. | Unobs. | Obs. | Unobs. | property |
| Qwen3-VL-32B | 0.55 0.00 | 0.57 0.02 | 0.75 0.00 | 0.76 0.05 | 0.74 0.00 | 0.76 0.05 | 0.57 0.01 |
| InternVL3-14B | 0.51 0.01 | 0.45 0.02 | 0.71 0.01 | 0.65 0.00 | 0.71 0.02 | 0.68 0.01 | 0.51 0.01 |
| Qwen3-VL-8B | 0.54 0.01 | 0.58 0.01 | 0.68 0.00 | 0.70 0.01 | 0.68 0.01 | 0.71 0.01 | 0.56 0.01 |
| Qwen2.5-VL-32B | 0.45 0.01 | 0.44 0.01 | 0.66 0.02 | 0.64 0.02 | 0.67 0.01 | 0.71 0.01 | 0.57 0.01 |
| InternVL3-8B | 0.63 0.00 | 0.62 0.02 | 0.65 0.01 | 0.59 0.01 | 0.65 0.01 | 0.62 0.01 | 0.60 0.01 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Robot (type) | License | Source |
|---|---|---|---|
| Humanoid Everyday ( Zhao et al., 2026 ) | Unitree G1, H1 (humanoid) | Apache-2.0 | USC-GVL/humanoid-everyday |
| RoboCOIN ( Wu et al., 2025b ) | AlphaBot 2, Galbot G1, Unitree G1edu (dual-arm, humanoid) | Apache-2.0, with use terms | RoboCOIN/* |
| Unitree G1 Dex1 ( Unitree Robotics, 2025a ) | Unitree G1 with Dex1 grippers (humanoid) | Apache-2.0; none stated for UniBot-V1 challenge sets | unitreerobotics/G1_Dex1_* |
| Unitree G1 Dex3 ( Unitree Robotics, 2025b ) | Unitree G1 with Dex3 hands (humanoid) | Apache-2.0 | unitreerobotics/G1_Dex3_* |
| Unitree Z1 dual-arm ( Unitree Robotics, 2025c ) | Two Unitree Z1 arms (dual-arm) | Apache-2.0 | unitreerobotics/Z1_Dual_Dex1_PourCoffee_Dataset |
| GR00T-Teleop-GR1 ( Gao et al., 2026 ) | Fourier GR-1 (humanoid) | CC BY-NC 4.0 | nvidia/PhysicalAI-Robotics-GR00T-Teleop-GR1 |
| Model | Checkpoint | Resolution a | Frames (fps) | Steps | Guidance | Seed |
|---|---|---|---|---|---|---|
| Wan 2.2 | Wan2.2-TI2V-5B | 960 704 | 121 (24) | 40 | 5.0 | 0 |
| Cosmos 3 Nano | Cosmos3-Nano | 960 720 | 121 (24) | 35 | 6.0 | 0 |
| HunyuanVideo 1.5 | HunyuanVideo-1.5-480p-I2V | 720 544 | 121 (24) | 50 | 6.0 | 0 |
| MAGI 4.5B Distill | MAGI-1 4.5B distill | 960 720 | 120 (24) | 16 b | – c | 0 |
| Field | Response | Collected for | Instructions |
|---|---|---|---|
| Physical plausibility ( ) physical_plausibility | 1–4 | All clips | 1 : clearly breaks physics, including clipping, floating, objects appearing or disappearing, or impossible motion. 2 : mostly implausible, with one clear issue or several smaller issues. 3 : mostly plausible, with minor oddities only. 4 : physically realistic. |
| Issue description description_of_issue | Free text, 1–2 sentences | Clips with | Briefly describe what is physically incorrect. Leave blank when . |
| Violation labels physics_category | Multi-select, 8 labels | Clips with | Select at least one applicable violation category, choosing the most obvious issue first. Leave blank when . |
| Action completion ( ) action_completed | Yes/no | All clips | Judge whether the requested task is completed by the end of the clip, even if the process is physically unrealistic. |
| Hidden-property adherence ( ) property_followed | 1–4 | Unobservable clips | 1 : property completely ignored; the outcome resembles a generic response. 2 : mostly ignored, with only a faint or inconsistent indication. 3 : mostly followed, but with incorrect magnitude, timing, or degree. 4 : fully and correctly reflected. |
| Label | Definition |
|---|---|
| Causality | An effect occurs without a visible cause, or events occur in an impossible order. |
| Collision | Objects pass through one another or fail to exhibit an expected response after impact. |
| Contact | An object floats without support, rests at an incorrect contact point, or fails to maintain visible contact. |
| Deformation | A rigid object warps or melts, or a soft object fails to deform under an interaction that should alter its shape. |
| Fluid | Pouring, splashing, containment, or other liquid behavior is physically inconsistent. |
| Friction | Sliding, sticking, or gripping behavior is inconsistent with the visible interaction or stated surface condition. |
| Question | Prompt |
|---|---|
| Header | Look at these frames, sampled in order from a video of a robot. The task: “ task ”. |
| Physics error (plausibility) | Ignore whether the action succeeds and ignore how much motion there is. Count only PHYSICS ERRORS: objects passing through each other, appearing or vanishing, floating without support, changing shape impossibly, or moving with nothing touching them. 1 = many clear physics errors, 2 = some errors, 3 = minor oddities, 4 = no physics errors. Answer with exactly one digit, 1 to 4. |
| Standard plausibility (baseline) | Rate how PHYSICALLY PLAUSIBLE this video is — do objects, contact, motion, liquids and deformation behave as they would in the real world? 1 = clearly impossible physics, 2 = noticeable physics errors, 3 = minor oddities, 4 = fully plausible. Answer with exactly one digit, 1 to 4. |
| Task completion | Did the robot actually COMPLETE the task shown? Answer with exactly one digit: 1 = no, 2 = yes. |
| Hidden property (unobservable only) | Hidden property of this scene (not visible in the frames): property — value . If the video respects it, this should happen: expected outcome Rate how well the video FOLLOWS that hidden property. 1 = contradicts it, 2 = mostly contradicts, 3 = mostly follows, 4 = clearly follows. Answer with exactly one digit, 1 to 4. |
| Violation type | (Header: Look at these frames, sampled in order from a video of: “ task ”.) Which ONE of these best describes the main physics problem in this video? options Answer with exactly one letter. |