TriPhysBench: Evaluating Physical Intelligence for Embodied Agents
Authors: Zhongxiang Sun, Chenghao Yang, Haojie Ding, Boheng Zhang, Fan Yang, Qingsong Wen, Kun Wang
Organizations: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Information Engineering, Zhongnan University of Economics and Law, Wuhan, China · Kuaishou Technology, Beijing, China · Squirrel AI Learning · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China
Embodied agents need physical intelligence to understand their surroundings through physical perception, physical knowledge retrieval, and physical reasoning. These capabilities must work together to interpret events and make quantitative judgments. Existing benchmarks advance physical understanding and general-purpose tool use, yet the coordination of these capabilities in physical scenarios remains insufficiently characterized. Inspired by Popper's Three Worlds, we introduce TriPhysBench, a benchmark of 410 questions from 200 videos that brings physical perception, physical knowledge retrieval, and physical reasoning into a shared evaluation setting. Its measurement-grounded and knowledge-grounded subtasks impose complementary demands on obtaining observations and acquiring external specifications, then applying them to quantitative problems. Among six representative vision-language models, the best achieves an answer accuracy of only 50.38%, revealing limited performance on tasks that require these capabilities to work together. We further introduce TriPhys, a framework that learns a unified policy for physical perception, knowledge retrieval, and quantitative reasoning from complete tool-integrated trajectories. The policy uses intermediate reasoning to guide retrieval and computation, then incorporates tool feedback into subsequent physical inference. On 110 questions from videos excluded from training, TriPhys improves answer and solution scores across three Qwen configurations, with a 17.7% relative gain in answer accuracy for TriPhys-27B over its Qwen3.6-27B backbone. This work provides a benchmark and a learning pathway toward embodied agents that move beyond recognizing events to understanding and reasoning about the physical processes behind them.
Figures & tables
Figure 1: Overview of TriPhysBench and TriPhys. Blue, yellow, and pink denote physical perception (W1), knowledge retrieval (W3), and reasoning (W2), respectively. Scenes and workflow are schematic.
Table 1: Comparison with representative benchmarks.
Figure 2: TriPhysBench construction: (a) video processing for MPR (EgoExo4D) and KPR (Physics-aware-videos); (b) physical event graphs; (c) question generation and review. Solid arrows indicate data flow; dashed arrows indicate external inputs. Scenes are schematic.
Figure 3: TriPhysBench dataset statistics. A–D denote point estimation, interval estimation, physical judgment, and scenario comparison. Panel (d) groups all questions into eight physics topics. Boxplots show medians and interquartile ranges.
Category
Statistic
MPR
KPR
Total
Videos
90
110
200
Data statistics
Questions
228
182
410
Subcategories
32
70
94
Easy
82
90
172
Difficulty
Medium
77
57
134
Hard
69
35
104
Table 2: Dataset composition by subtask. Subcategory totals count distinct labels in the union.
Figure 4: Four answer forms illustrated by reference cases. Type A is an MPR case from EgoExo4D; Type B, C, and D are KPR cases from Physics-aware-videos. Highlighted frames mark key observations; the hand mass, friction coefficients, curve radius, and thermal conductivities follow the reference assumptions.
Multimodal Large Language Models (MLLMs) show promising results as decision-making engines for embodied agents operating in complex, physical environments. However, existing benchmarks often prioritize high-level planning or spatial reasoning, leaving the fine-grained action intelligence required for embodied physical interaction underexplored. To address this gap, we introduce CFG-Bench, a new benchmark designed to systematically evaluate this crucial capability. CFG-Bench consists of 1,368 curated videos paired with 19,562 question-answer pairs spanning three evaluation paradigms targeting four cognitive abilities: 1) Physical Interaction, 2) Temporal-Causal Relation, 3) Intentional Understanding, and 4) Evaluative Judgment. Together, these dimensions provide a systematic framework for assessing a model's ability to translate visual observations into actionable knowledge, moving beyond mere surface-level recognition. Our comprehensive evaluation on CFG-Bench reveals that leading MLLMs struggle to produce detailed instructions for physical interactions and exhibit profound limitations in the higher-order reasoning of intention and evaluation. Moreover, supervised fine-tuning (SFT) on our data demonstrates that teaching an MLLMs to articulate fine-grained actions directly translates to significant performance gains on established embodied benchmarks. Our analysis highlights these limitations and offers insights for developing more capable and grounded embodied agents. Project page: https://cfg-bench.github.io/
Dayong Liu, Chao Xu, Weihong Chen +5
Zhejiang University · IROOTECH TECHNOLOGY · Wolf 1069 b Lab, Sany Group +2
Reliable physical reasoning from video requires understanding how objects move, interact, and respond to interventions. Existing vision-language models (VLMs) often struggle to interpret these dynamics and reason reliably about future and counterfactual outcomes. We introduce PhysMind, a training-free agentic framework that constructs one reusable, question-agnostic executable world per video. PhysMind recovers a temporally consistent dynamic scene through object segmentation, mesh reconstruction, and 6D pose tracking, then fits analytic continuous-time dynamics and latent physical parameters without unrolling a time-stepped simulator. Given a question, it inspects, continues, or edits the world and answers from the resulting trajectories and interactions. Relative to direct chain-of-thought (CoT) reasoning with the same VLM, PhysMind improves accuracy by 38.23 points on CLEVRER and 8.08 points on Physion++. On counterfactual questions, it exceeds the strongest evaluated VLM baseline, GPT-5.5, by 19.25 points.
Chen Yang, Shenxiang Zeng, Haoyang Zhao +6
1Tsinghua University · 2The University of Hong Kong
We introduce HumanPhys, a human-centric benchmark for real-world physical perception, and a training corpus emitted by the same pipeline. Each question targets a physically grounded quantity related to people, including depth, inter-person distance, gaze angle, displacement, velocity, heading-step trajectory, blink count, and contact-deformation strength. The rendered probes additionally test illumination inference from body and face cues and camera ego-motion, both with and without a person in the frame. All labels are derived programmatically from sensor calibration, source metadata, or renderer state, rather than written by annotators or produced by estimators on the benchmark frames. HumanPhys systematically probes six dimensions of human-centric physical perception: spatial state, kinematics, blink events, deformation, illumination, and camera pose. To the best of our knowledge, no existing MLLM benchmark-and-training corpus jointly supervises this set of human-centered physical quantities under direct metadata- or renderer-derived labels. The six dimensions appear only partially dissociable across the tested MLLMs. We benchmark a range of open- and closed-source MLLMs on HumanPhys and find that all six capability dimensions remain challenging for current models. We hope HumanPhys will support future research on multimodal systems that perceive and reason about people in the physical world.
Yiming Zhang, Enzhi Zhou, Junan Lin +4
Nanyang Technological University, Singapore · Ant Group, Hangzhou, China