HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning

Yiming Zhang; Enzhi Zhou; Junan Lin; Dulan Zhou; Weiqiang Wang; Yuge Huang; Jian Liu

cs.CVOct 10, 2026

HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning

Authors: Yiming Zhang, Enzhi Zhou, Junan Lin, Dulan Zhou, Weiqiang Wang, Yuge Huang, Jian Liu

Organizations: Nanyang Technological University, Singapore · Ant Group, Hangzhou, China

Abstract

We introduce HumanPhys, a human-centric benchmark for real-world physical perception, and a training corpus emitted by the same pipeline. Each question targets a physically grounded quantity related to people, including depth, inter-person distance, gaze angle, displacement, velocity, heading-step trajectory, blink count, and contact-deformation strength. The rendered probes additionally test illumination inference from body and face cues and camera ego-motion, both with and without a person in the frame. All labels are derived programmatically from sensor calibration, source metadata, or renderer state, rather than written by annotators or produced by estimators on the benchmark frames. HumanPhys systematically probes six dimensions of human-centric physical perception: spatial state, kinematics, blink events, deformation, illumination, and camera pose. To the best of our knowledge, no existing MLLM benchmark-and-training corpus jointly supervises this set of human-centered physical quantities under direct metadata- or renderer-derived labels. The six dimensions appear only partially dissociable across the tested MLLMs. We benchmark a range of open- and closed-source MLLMs on HumanPhys and find that all six capability dimensions remain challenging for current models. We hope HumanPhys will support future research on multimodal systems that perceive and reason about people in the physical world.

Figures & tables

Figure 1: Left : radar of the eight MLLMs on HumanPhys, with one axis for each sub-task, grouped into the six taxonomy categories of Figure 2 ; the two proprietary models are dashed (numerical scores in Table 3 ). Right : checkpoint ablation of fine-tuning on HumanPhys and complementary spatial data across ten benchmarks, with Qwen3-VL-4B-Instruct dashed as reference; this data-mixture ablation is tabulated in Table 5 and analysed in § 4.3 (Preserving General Spatial Abilities).

Figure 2: Overview of the HumanPhys taxonomy.

Benchmark	Human-Centric	Input Type		Source		Task Taxonomy		Train Data Ava.
Benchmark	Human-Centric	Image	Video	Real	Simulation	Static	Dynamic	Train Data Ava.
VSI-Bench [ 60 ]	✗	✗	✓	✓	✗	✓	✗	✗
STI-Bench [ 32 ]	✗	✗	✓	✓	✗	✓	✓	✗
3DSRBench [ 41 ]	✗	✓	✗	✓	✓	✓	✗	✗
BLINK [ 20 ]	✗	✓	✗	✓	✓	✓	✗	✗
ViewSpatial [ 27 ]	✗	✓	✗	✓	✗	✓	✗	✗

Table 1: Comparison of representative spatial reasoning benchmarks. Human-Centric : humans are the targets of perception; Input Type : accepted modalities, ”Video” means temporally ordered sampled frames derived from source clips; Source : provenance of the visual data, real or simulated; Task Taxonomy : static or dynamic queried target; Train Data Ava. : whether a training corpus is released. For HumanPhys, Static and Dynamic correspond to its frame-level and temporal questions (§ 3.1 ). For HumanMoveVQA all questions are video-level multiple-choice questions about human motion, so Static is marked ✗, and Train Data Ava. denotes the authors’ own training split rather than a separately released corpus; it is compared with HumanPhys axis by axis in § 2.2 .

Figure 3: The construction pipeline of HumanPhys: (1) data collection from open-source datasets and Blender-based simulation; (2) preprocessing with metadata extraction and scene construction; (3) multi-engine QA generation with random sampling; (4) manual review and iterative refinement.

Closed-source models
	Spatial state				Kinematics			Event	Deform.	Illumination		Camera pose
Model	Depth	Dist	Count	Gaze	Disp	Vel	Traj	Blink	Deform	LS-Head	LS-Body	CP-Room	CP-Human	Overall
Random	7.20	8.30	23.40	28.20	5.80	12.20	37.80	22.00	26.20	13.40	48.10	6.10	2.80	18.60
Text Only	20.60	24.00	0.00	32.55	0.00	14.80	54.03	0.00	28.00	24.58	54.19	4.62	2.32	18.03
Human	25.20	52.00	79.20	60.40	36.50	30.30	63.50	87.40	36.10	50.60	70.10	24.90	12.20	48.30
GPT-5.6-Sol	37.15	29.90	70.05	79.35	20.35	40.50	64.72	87.95	46.00	27.51	57.48	30.55	18.15	46.94

Table 3: Main results on HumanPhys. Scores of eight MLLMs on the 13 sub-tasks (best per column among the eight models in bold), grouped by the six taxonomy dimensions of Figure 2 . Column keys: Dist = Distance, Disp = Displacement, Traj = Trajectory, LS = light-shadow, CP = camera-pose. Row keys: SSRL = Spatial-SSRL-7B; SpaceR/SpatialEvo/SSRL/ViLaSR are Qwen2.5-VL-7B-Instruct fine-tunes; the shaded group rows separate the closed-source (blue) from the open-source (green) checkpoints. “Overall” (our primary metric) is the question-weighted composite

\sum_{k}n_{k}\bar{s}_{k}/2500

over the full item bank (

n_{k}{=}200

;

100

for Deform ), scoring an answer that cannot be parsed into the required type

0

and keeping it in the denominator, as lmms-eval does; the 13 sub-task columns instead average only the answers a model produced in parseable form, so they are diagnostic rather than additive, exceeding “Overall” by up to

4.06

points where truncated generations are frequent. The Human row is indicative only: ten colleagues each answered an availability-selected, non-uniform

\approx

40% subset (mean times in Table 6 , Appendix A ); the Random row is a uniform random baseline over the full question bank; the Text Only row answers from the question text alone, without any visual input.

Figure 5: One scored CP-Room QA. Step 1 shares the gold axis (yaw) but not the magnitude ( $-30^{\circ}$ vs. $-27^{\circ}$ ), passing 8 of the 10 MRA thresholds for $8/10=0.8$ ; Step 2 matches exactly for $1.0$ ; the sequence score averages over gold steps: $(0.8+1.0)/2=0.9$ .

Benchmark	Subtask	Baseline	SFT
SPAR-Bench	distance_prediction_oc	11.20	56.36 $\uparrow$
	distance_prediction_oc_mv	18.05	59.62 $\uparrow$
	depth_prediction_oc	33.47	59.39 $\uparrow$
	depth_prediction_oc_mv	37.66	58.53 $\uparrow$
MMSI-Bench	Motion(Cam.)	22.97	32.43 $\uparrow$

Table 4: Transfer to existing benchmarks for related sub-tasks. Scores of Qwen3-VL-4B-Instruct before (Baseline) and after training with HumanPhys only (SFT ① in Table 5 ) on selected sub-tasks of SPAR-Bench and MMSI-Bench;

\uparrow

marks the gain over the baseline. The consistent gains on distance and depth sub-tasks indicate that HumanPhys supervision transfers effectively to related tasks in existing benchmarks, while the change on Motion(Cam.) (22.97 vs. 32.43, i.e. 17/74 vs. 24/74 correct; CLT

1\sigma

\pm 4.89

and

\pm 5.44

pp,

95\%

[-4.9,+23.8]

, so this

n{=}74

category is not individually significant) is suggestive only. _mv marks the multi-view variant of the sub-task.

	Data			HumanPhys	Spatial intelligence							General
Model	HP	SpR	SpC	Overall	3DSRB.	BLINK	ViewSp.	SPAR	MMSI	VSI	WMA	MMStar	MMB.
Qwen3-VL-4B	✗	✗	✗	27.27	60.91	64.01	39.29	35.81	27.30	54.74	48.37	61.18	83.51
SFT ①	✓	✗	✗	84.92	53.31	45.78	29.59	35.71	27.20	40.77	44.06	57.00	82.56
SFT ②	✓	✓	✗	84.89	58.79	56.11	38.73	39.09	27.50	54.65	48.59	65.13	84.45
SFT ③	✓	✗	✓	86.58	57.03	58.73	42.96	42.37	26.60	43.30	47.48	59.72	82.39
SFT ④	✓	✓	✓	86.40	57.28	59.48	39.76	43.05	27.40	54.90	49.46	64.64	84.36

Table 5: Data-mixture ablation. Scores (best per column in bold) of the Qwen3-VL-4B-Instruct base model and four SFT checkpoints on HumanPhys, seven spatial-intelligence benchmarks and two general multimodal benchmarks. The Data columns mark which training corpora each run mixes (HP = HumanPhys corpus, SpR = SpaceR-151K, SpC = SpatialCorpus-331K); SFT ④ is the final model.

Figure 6: Diagnosis of CP-Room errors (Gemini-3.5-Flash, $n=200$ ). Left: per-step gold-vs-predicted axis confusion. Right: score-loss waterfall from a perfect mean step-MRA of $1.0$ to the actual $0.274$ .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	Depth	Dist	Count	Gaze	Disp	Vel	Traj	Blink	Deform	LS-Head	LS-Body	CP-Room	CP-Human
Random	7.2	8.3	23.4	28.2	5.8	12.2	37.8	22.0	26.2	13.4	48.1	6.1	2.8
Text Only	20.60	24.00	0.00	32.55	0.00	14.80	54.03	0.00	28.00	24.58	54.19	4.62	2.32
Time (s)	40.4	25.2	96.5	35.4	44.9	180.3	131.5	82.3	136.2	1141.8	533.7	303.4	823.8
Human	25.2	52.0	79.2	60.4	36.5	30.3	63.5	87.4	36.1	50.6	70.1	24.9	12.2

Appendix

Table 6: Human study per sub-task. From top: uniform random baseline over the full bank; text-only baseline; mean human answering time per question in seconds, with cell shading proportional to time (darker = slower); and human scores on the same availability-selected, non-uniform

\approx

40% subset as the Human row of Table 3 , reported as an indicative reference only.

Human Motion Understanding

See all 16 papers

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
Aug 5, 2026Chen Yang, Shenxiang Zeng, Haoyang Zhao +6Counterfactual Reasoning Physical Reasoning in Video
1Tsinghua University · 2The University of Hong Kong
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Sep 9, 2026Yizhan Li, Jianxin You, Mengyang Xiong +5Physical Reasoning Robotics Simulation
Université de Montréal · Mila – Quebec AI Institute · McGill University +2
BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
May 29, 2026Ben Wang, Xiaogang Li, Ruochen Gao +6Physical Reasoning Multimodal Large Language Models
Alibaba Group Beijing, China

Aug 5, 2026J/K move · Enter open · S save

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning

Chen Yang, Shenxiang Zeng, Haoyang Zhao, Zhouyuan Xu+5

1Tsinghua University · 2The University of Hong Kong

Counterfactual Reasoning Physical Reasoning in Video Embodied World Models Temporal Reasoning in VLMs

Open paper AskPDF arXiv

Reliable physical reasoning from video requires understanding how objects move, interact, and respond to interventions. Existing vision-language models (VLMs) often struggle to interpret these dynamics and reason reliably about future and counterfactual outcomes. We introduce PhysMind, a training-free agentic framework that constructs one reusable, question-agnostic executable world per video. PhysMind recovers a temporally consistent dynamic scene through object segmentation, mesh reconstruction, and 6D pose tracking, then fits analytic continuous-time dynamics and latent physical parameters without unrolling a time-stepped simulator. Given a question, it inspects, continues, or edits the world and answers from the resulting trajectories and interactions. Relative to direct chain-of-thought (CoT) reasoning with the same VLM, PhysMind improves accuracy by 38.23 points on CLEVRER and 8.08 points on Physion++. On counterfactual questions, it exceeds the strongest evaluated VLM baseline, GPT-5.5, by 19.25 points.

Abstract

Human Motion Understanding

Explore similar work

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning

ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs