Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution. A foundational data challenge is to measure and preserve this physical information as training supervision. In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. Rich physical experience in video does not automatically translate into sufficient language supervision. This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.
Figures & tables
\toprule Physical dimension
Evaluation focus
Annotation content
\midrule Event Coverage
Coverage of the main physical activities and events.
Event chain and linked timeline segments.
\addlinespace [2pt] Objects and Roles
Agreement on manipulated objects and specified actor or hand roles.
Entity identities and event participants.
\addlinespace [2pt] Actions and Motion
Agreement on actions, movement types, and directions.
Action labels, moving entities, motion types and directions.
\addlinespace [2pt] Contact and Interaction
Agreement on contact and interaction relations.
Contact participants, contact relations, and interaction events.
\addlinespace [2pt] States and Outcomes
Agreement on states, spatial relations, and resulting changes.
Initial and final states, state changes, and spatial relations.
\addlinespace [2pt] Physical Mechanisms and Causality
Agreement on explicitly stated mechanisms or causal explanations.
Mechanism statements, supporting events, and causal relations.
Table 1: Physical evaluation dimensions and annotation content. The mapping links dimensions to annotation content. Text-based scoring measures candidate agreement with and coverage of applicable reference content.
Figure 1: Dataset overview across three domains. (a) Category composition. (b) Representative video frames and synchronized handheld-gripper views. (c) Third-person scene types, egocentric action families, and handheld-gripper trajectory profiles, with sequence counts. Each domain contains 500 videos or demonstration episodes.
\toprule Data domain
Available observations
Annotation emphasis
\midrule Third-person physical scenes
Scene video showing participants and their evolving relations.
Scene entities, physical interactions, motion, and state evolution.
\addlinespace [2pt] Egocentric human manipulation
Egocentric video with hand-pose information available as annotation cues.
Hand identity and roles, hand–object interactions, and visible object changes.
Synchronized views and trajectory cues for locating frames to inspect.
Human-operated gripper identities and roles, and corresponding observations across views.
\bottomrule
Table 2: Three-domain data and annotation overview. Auxiliary observations support annotation; their availability to a captioning model is specified separately by the evaluation setting.
\toprule Domain
Data
Coverage and observations
\midrule Third-person
500 videos
Eight physical coverage categories and 16 scene categories; object motion, interactions, and state changes.
\addlinespace [2pt] Egocentric
500 videos
118 tasks; 15 scenario groups; RGB and hand tracking; approximately 2.21 hours.
\addlinespace [2pt] Handheld gripper
500 episodes
Nine task groups plus missing hints; three synchronized views and paired gripper trajectories.
\bottomrule
Table 3: Dataset coverage across the three domains. Multiple views belong to the same handheld-gripper episode.
Figure 2: Evidence-grounded physical annotation. (a) Domain-adapted observations guide frame selection and visual inspection across third-person, egocentric, and handheld-gripper data. (b) A vision–language annotator produces physical records linked to source frames, organizing entities, events, state changes, and relations. (c) Evidence checks, human cross-review, and structural validation refine the records and captions. (d) A bottle-opening example connects hand roles, event intervals, contact transitions, and object states to a physical caption.
Figure 3: Interaction-role cues for egocentric physical annotation. Source RGB frames (left) and HOI-DETR predictions (right). Red, yellow, and cyan boxes denote hands, directly manipulated objects, and objects affected through them; links show predicted interaction relations. Numbers beside boxes and links indicate detection and relation confidence, respectively. The role structure guides annotation of the acting hand, the manipulated object, and tool-mediated interactions, supporting the description of who acts on what.
Figure 4: Physical caption evaluation on the third-person dataset. Physis is highlighted in black. The seven axes correspond to physical evaluation dimensions, and each curve represents a captioning strategy. All strategies are evaluated against the same dimension-specific references. Scores range from 0 to 100, with higher scores farther from the center. The LingBot strategies are reconstructed from the paper description. Physis captioning is currently in internal testing.
\toprule Physical dimension
CogVideoX
Open-Sora v1.3
EgoVid Dense
EgoVid Action
Cosmos Ego
Physis
Meta caption
\midrule Event coverage
60.17
60.25
61.50
57.33
81.08
84.42
49.17
Objects and roles
61.17
62.92
63.08
57.25
78.50
84.83
46.67
Actions and motion
63.18
59.91
62.64
59.55
84.64
87.73
54.09
Contact and interaction
61.43
54.29
59.29
54.29
83.57
77.14
38.57
States and outcomes
71.88
68.25
70.88
60.62
80.12
92.75
55.00
Physical mechanisms and causality
25.83
31.42
30.00
35.83
46.50
49.58
7.08
Table 4: Physical caption evaluation on the egocentric dataset. All strategies are evaluated on the same egocentric clips using the same dimension-specific references. Scores range from 0 to 100; higher is better. Bold indicates the highest score in each row. The overall average is computed from per-video scores over applicable dimensions, as defined in Section 5.1 .
Figure 5: Physical captions improve video reconstruction across domains. LPIPS and PVCS are shown for third-person (top) and egocentric (bottom) videos using H3 and LingBot-World 2.0. Original captions are source metadata captions. Point labels report mean distances; green annotations show relative reductions under physical-caption conditioning. Lower is better.
Figure 6: Physical captions preserve interaction roles and state changes in reconstruction. Rows show source frames (top), H3 with the physical caption (middle; labeled Gold in the source experiment), and H3 with the metadata caption (bottom). Columns correspond to the same times. Physical conditioning better retains left-hand stabilization, right-hand shell removal, and the transition from a covered to an exposed interior. Metadata conditioning instead generates the assembly of a larger stool.
Figure 7: Physical captions preserve state transitions and interaction details in third-person reconstruction. Columns compare the source video, H3 conditioned on the physical caption (Gold), and H3 conditioned on the metadata caption (Meta). Each row uses the same timestamp. (a) After the board is lifted, Gold retains volumetric toy shapes, while Meta exposes flattened, sheet-like shapes; the 9.50 s row shows both generated conditions after unloading. (b) Gold better retains localized drilling debris and the final separation of the drill bit from the workpiece. (c) Gold more closely preserves the persistent white–turquoise plume and barrel orientation, while Meta emphasizes thin liquid sheets and spray.
\toprule Evaluation
Original
Enriched
Gain
Table 5: Physical-description quality and policy performance under instruction enrichment. (a) PhysCapBench scores of training descriptions. (b) Single-task success on 20 standard and 20 randomized shared scenes (40 overall), with demonstrations and training settings held fixed. (c,d) Official baseline and enriched-instruction success rates, averaged over six selected tasks in each benchmark. Gains are differences between displayed values, in score points (a) or percentage points (b–d). Table reports per-task results.