cs.CVOct 1, 2026
SavePhysicsLENS: Diagnosing Physical Property Blindness in Video Generation Models
Organizations: School of Computing and Augmented Intelligence Arizona State University Tempe, AZ, USA
Abstract
Reliable video world models could provide scalable predictive environments for robot learning, planning, and evaluation. However, generated robot videos can violate physical principles and complete tasks through physically implausible behavior, limiting their reliability for robot learning and planning. Current video-generation benchmarks exclude physics that are inherently hidden by visuals (e.g., weight, viscosity, friction). Due to this, video models are evaluated on the fidelity of physics, not the underlying accuracy of physics. We introduce PhysicsLENS, a dataset and benchmark for evaluating plausibility of physical properties grounded in robotics. PhysicsLENS uses matched scenario pairs that hold the same conditioning frame and task, while varying underlying physics in the scene description. Scenarios are curated from public robot video sources and annotated across seven physical domains: collision, gravity, momentum, friction, deformation, fluid, and causality. We evaluate across four video generation models, producing over 400 human-annotated labels. Results show that plausible-looking videos often ignore the stated property (34 of 47), and that stating the property lowers plausibility only slightly and not significantly.
Figures & tables
Figure 1: Overview of PhysicsLENS: robotics video generation, evaluation, and core findings.
| Category | Method | Physics | Robotics | Human | Ref.-free | Localized | Obs. axis |
| Physics benchmarks | VideoPhy-2 ( Bansal et al., 2026 ) | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ |
| PhyGenBench ( Meng et al., 2025 ) | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | |
| Morpheus ( Zhang et al., 2025 ) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| Physics-IQ ( Motamed et al., 2026 ) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| NewtonRewards ( Le et al., 2025 ) | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | |
| GeoPhys ( Internò et al., 2026 ) | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ |
Table 1: Comparison of physics benchmarks and video critics. Ref.-free requires no paired real-world recording; Obs. axis tests prompt-specified properties not inferable from the initial frame.
Figure 2: Example matched prompt pairs. Each pair shares the initial frame and task. The observable description (blue) states what the frame shows; the unobservable description (orange) replaces it with a latent property that departs from what the frame suggests.
Figure 3: Evidence flow through the PhysicsLENS diagnostic pipeline.
Figure 6: Three annotated failure cases (six frames each). : physical plausibility; : hidden-property adherence (both 1–4). (a) A clean, realistic pour that ignores the stated viscosity. (b) A video rated as following the stated property that is nevertheless physically implausible. (c) A nearly static video rated fully plausible.
| Physical plausibility | Task | Violation | Physics | Hidden | |||
| System | Obs. | Unobs. | All | completion | detection | family | property |
| Generator identity | 0.63 0.03 | 0.72 0.05 | 0.65 0.03 | 0.72 0.02 | 0.73 0.03 | 0.51 0.02 | 0.68 0.05 |
| Signals only | 0.61 0.03 | 0.65 0.05 | 0.62 0.03 | 0.70 0.03 | 0.70 0.03 | 0.51 0.02 | 0.69 0.05 |
| VLM † | 0.53 0.06 | 0.53 0.06 | 0.53 0.05 | 0.71 0.04 | 0.58 0.08 | 0.59 0.02 | 0.57 0.03 |
| VLM + signals † | 0.60 0.02 | 0.66 0.01 | 0.61 0.01 | 0.76 0.02 | 0.72 0.01 | 0.58 0.02 | 0.70 0.02 |
| Debiased VLM † | 0.64 0.07 | 0.63 0.04 | 0.63 0.06 | 0.71 0.04 | 0.58 0.08 | 0.59 0.02 | 0.57 0.03 |
Table 4: PhysicsLENS against human judgment (AUC on 439 videos; 0.5 is chance). Generator identity scores each video by its generator’s average rating; signals are the PhysicsLENS Stage-1/2 measurements; debiased is the PhysicsLENS physics-error question; the ensemble averages all ten VLMs. Values are mean standard deviation across the three runs, except for † rows, which report mean standard deviation across the ten VLMs. Bold indicates the best result in each column.
| Standard | Physics-error | + signals | Hidden | ||||
|---|---|---|---|---|---|---|---|
| VLM | Obs. | Unobs. | Obs. | Unobs. | Obs. | Unobs. | property |
| Qwen3-VL-32B | 0.55 0.00 | 0.57 0.02 | 0.75 0.00 | 0.76 0.05 | 0.74 0.00 | 0.76 0.05 | 0.57 0.01 |
| InternVL3-14B | 0.51 0.01 | 0.45 0.02 | 0.71 0.01 | 0.65 0.00 | 0.71 0.02 | 0.68 0.01 | 0.51 0.01 |
| Qwen3-VL-8B | 0.54 0.01 | 0.58 0.01 | 0.68 0.00 | 0.70 0.01 | 0.68 0.01 | 0.71 0.01 | 0.56 0.01 |
| Qwen2.5-VL-32B | 0.45 0.01 | 0.44 0.01 | 0.66 0.02 | 0.64 0.02 | 0.67 0.01 | 0.71 0.01 | 0.57 0.01 |
| InternVL3-8B | 0.63 0.00 | 0.62 0.02 | 0.65 0.01 | 0.59 0.01 | 0.65 0.01 | 0.62 0.01 | 0.60 0.01 |
Table 5: Plausibility AUC per VLM on observable ( ) and unobservable ( ) videos, with a standard question, the PhysicsLENS physics-error question, and that question plus Stage-1/2 signals. Hidden property : whether the stated property was followed. Mean sd over three runs that show the VLM different frames of each video. Bold: best per column.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Robot (type) | License | Source |
|---|---|---|---|
| Humanoid Everyday ( Zhao et al., 2026 ) | Unitree G1, H1 (humanoid) | Apache-2.0 | USC-GVL/humanoid-everyday |
| RoboCOIN ( Wu et al., 2025b ) | AlphaBot 2, Galbot G1, Unitree G1edu (dual-arm, humanoid) | Apache-2.0, with use terms | RoboCOIN/* |
| Unitree G1 Dex1 ( Unitree Robotics, 2025a ) | Unitree G1 with Dex1 grippers (humanoid) | Apache-2.0; none stated for UniBot-V1 challenge sets | unitreerobotics/G1_Dex1_* |
| Unitree G1 Dex3 ( Unitree Robotics, 2025b ) | Unitree G1 with Dex3 hands (humanoid) | Apache-2.0 | unitreerobotics/G1_Dex3_* |
| Unitree Z1 dual-arm ( Unitree Robotics, 2025c ) | Two Unitree Z1 arms (dual-arm) | Apache-2.0 | unitreerobotics/Z1_Dual_Dex1_PourCoffee_Dataset |
| GR00T-Teleop-GR1 ( Gao et al., 2026 ) | Fourier GR-1 (humanoid) | CC BY-NC 4.0 | nvidia/PhysicalAI-Robotics-GR00T-Teleop-GR1 |
Table 6: Public datasets that supply the starting frames. Source is the Hugging Face repository we used; † marks an unofficial copy, with the official release in parentheses.
| Model | Checkpoint | Resolution a | Frames (fps) | Steps | Guidance | Seed |
|---|---|---|---|---|---|---|
| Wan 2.2 | Wan2.2-TI2V-5B | 960 704 | 121 (24) | 40 | 5.0 | 0 |
| Cosmos 3 Nano | Cosmos3-Nano | 960 720 | 121 (24) | 35 | 6.0 | 0 |
| HunyuanVideo 1.5 | HunyuanVideo-1.5-480p-I2V | 720 544 | 121 (24) | 50 | 6.0 | 0 |
| MAGI 4.5B Distill | MAGI-1 4.5B distill | 960 720 | 120 (24) | 16 b | – c | 0 |
Table 7: Video generation settings. All clips are image-to-video, conditioned on a start frame from the source clip plus a text prompt, and last about 5 s.
Figure 9: The human-annotation interface. Annotators assess physical plausibility and action completion for every generated clip, identify visible violations when present, and evaluate property adherence for unobservable scenarios.
| Field | Response | Collected for | Instructions |
|---|---|---|---|
| Physical plausibility ( ) physical_plausibility | 1–4 | All clips | 1 : clearly breaks physics, including clipping, floating, objects appearing or disappearing, or impossible motion. 2 : mostly implausible, with one clear issue or several smaller issues. 3 : mostly plausible, with minor oddities only. 4 : physically realistic. |
| Issue description description_of_issue | Free text, 1–2 sentences | Clips with | Briefly describe what is physically incorrect. Leave blank when . |
| Violation labels physics_category | Multi-select, 8 labels | Clips with | Select at least one applicable violation category, choosing the most obvious issue first. Leave blank when . |
| Action completion ( ) action_completed | Yes/no | All clips | Judge whether the requested task is completed by the end of the clip, even if the process is physically unrealistic. |
| Hidden-property adherence ( ) property_followed | 1–4 | Unobservable clips | 1 : property completely ignored; the outcome resembles a generic response. 2 : mostly ignored, with only a faint or inconsistent indication. 3 : mostly followed, but with incorrect magnitude, timing, or degree. 4 : fully and correctly reflected. |
Table 8: Fields in the human-annotation form. Monospace text gives each field’s identifier. : physical plausibility; : action completion; : hidden-property adherence.
Figure 10: Dataset composition. (a) Robot type in the 80 starting frames. Mobile dual-arm robots are wheeled platforms (AlphaBot 2, Galbot G1, AgiBot G1, and Mobile ALOHA). (b) Property stated in each unobservable scenario and the task it is paired with. (c) Share of the 439 videos with each violation label. As in Appendix C , Contact is merged into Collision. A video can have several labels (66% of videos with a violation have more than one), so the violation bars sum to over 100%.
| Label | Definition |
|---|---|
| Causality | An effect occurs without a visible cause, or events occur in an impossible order. |
| Collision | Objects pass through one another or fail to exhibit an expected response after impact. |
| Contact | An object floats without support, rests at an incorrect contact point, or fails to maintain visible contact. |
| Deformation | A rigid object warps or melts, or a soft object fails to deform under an interaction that should alter its shape. |
| Fluid | Pouring, splashing, containment, or other liquid behavior is physically inconsistent. |
| Friction | Sliding, sticking, or gripping behavior is inconsistent with the visible interaction or stated surface condition. |
Table 9: Definitions of the eight violation labels used in the human annotations.
| Question | Prompt |
|---|---|
| Header | Look at these frames, sampled in order from a video of a robot. The task: “ task ”. |
| Physics error (plausibility) | Ignore whether the action succeeds and ignore how much motion there is. Count only PHYSICS ERRORS: objects passing through each other, appearing or vanishing, floating without support, changing shape impossibly, or moving with nothing touching them. 1 = many clear physics errors, 2 = some errors, 3 = minor oddities, 4 = no physics errors. Answer with exactly one digit, 1 to 4. |
| Standard plausibility (baseline) | Rate how PHYSICALLY PLAUSIBLE this video is — do objects, contact, motion, liquids and deformation behave as they would in the real world? 1 = clearly impossible physics, 2 = noticeable physics errors, 3 = minor oddities, 4 = fully plausible. Answer with exactly one digit, 1 to 4. |
| Task completion | Did the robot actually COMPLETE the task shown? Answer with exactly one digit: 1 = no, 2 = yes. |
| Hidden property (unobservable only) | Hidden property of this scene (not visible in the frames): property — value . If the video respects it, this should happen: expected outcome Rate how well the video FOLLOWS that hidden property. 1 = contradicts it, 2 = mostly contradicts, 3 = mostly follows, 4 = clearly follows. Answer with exactly one digit, 1 to 4. |
| Violation type | (Header: Look at these frames, sampled in order from a video of: “ task ”.) Which ONE of these best describes the main physics problem in this video? options Answer with exactly one letter. |
Table 10: VLM questions, verbatim (line breaks omitted). Each prompt follows the frames and the header; marks fields filled per video. The violation options appear as A)–I) in a per-video shuffled order; family names in parentheses are not shown to the model.