Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution. A foundational data challenge is to measure and preserve this physical information as training supervision. In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. Rich physical experience in video does not automatically translate into sufficient language supervision. This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.
Figures & tables
\toprule Physical dimension
Evaluation focus
Annotation content
\midrule Event Coverage
Coverage of the main physical activities and events.
Event chain and linked timeline segments.
\addlinespace [2pt] Objects and Roles
Agreement on manipulated objects and specified actor or hand roles.
Entity identities and event participants.
\addlinespace [2pt] Actions and Motion
Agreement on actions, movement types, and directions.
Action labels, moving entities, motion types and directions.
\addlinespace [2pt] Contact and Interaction
Agreement on contact and interaction relations.
Contact participants, contact relations, and interaction events.
\addlinespace [2pt] States and Outcomes
Agreement on states, spatial relations, and resulting changes.
Initial and final states, state changes, and spatial relations.
\addlinespace [2pt] Physical Mechanisms and Causality
Agreement on explicitly stated mechanisms or causal explanations.
Mechanism statements, supporting events, and causal relations.
Table 1: Physical evaluation dimensions and annotation content. The mapping links dimensions to annotation content. Text-based scoring measures candidate agreement with and coverage of applicable reference content.
Figure 1: Dataset overview across three domains. (a) Category composition. (b) Representative video frames and synchronized handheld-gripper views. (c) Third-person scene types, egocentric action families, and handheld-gripper trajectory profiles, with sequence counts. Each domain contains 500 videos or demonstration episodes.
\toprule Data domain
Available observations
Annotation emphasis
\midrule Third-person physical scenes
Scene video showing participants and their evolving relations.
Scene entities, physical interactions, motion, and state evolution.
\addlinespace [2pt] Egocentric human manipulation
Egocentric video with hand-pose information available as annotation cues.
Hand identity and roles, hand–object interactions, and visible object changes.
Synchronized views and trajectory cues for locating frames to inspect.
Human-operated gripper identities and roles, and corresponding observations across views.
\bottomrule
Table 2: Three-domain data and annotation overview. Auxiliary observations support annotation; their availability to a captioning model is specified separately by the evaluation setting.
\toprule Domain
Data
Coverage and observations
\midrule Third-person
500 videos
Eight physical coverage categories and 16 scene categories; object motion, interactions, and state changes.
\addlinespace [2pt] Egocentric
500 videos
118 tasks; 15 scenario groups; RGB and hand tracking; approximately 2.21 hours.
\addlinespace [2pt] Handheld gripper
500 episodes
Nine task groups plus missing hints; three synchronized views and paired gripper trajectories.
\bottomrule
Table 3: Dataset coverage across the three domains. Multiple views belong to the same handheld-gripper episode.
Figure 2: Evidence-grounded physical annotation. (a) Domain-adapted observations guide frame selection and visual inspection across third-person, egocentric, and handheld-gripper data. (b) A vision–language annotator produces physical records linked to source frames, organizing entities, events, state changes, and relations. (c) Evidence checks, human cross-review, and structural validation refine the records and captions. (d) A bottle-opening example connects hand roles, event intervals, contact transitions, and object states to a physical caption.
Figure 3: Interaction-role cues for egocentric physical annotation. Source RGB frames (left) and HOI-DETR predictions (right). Red, yellow, and cyan boxes denote hands, directly manipulated objects, and objects affected through them; links show predicted interaction relations. Numbers beside boxes and links indicate detection and relation confidence, respectively. The role structure guides annotation of the acting hand, the manipulated object, and tool-mediated interactions, supporting the description of who acts on what.
Figure 4: Physical caption evaluation on the third-person dataset. Physis is highlighted in black. The seven axes correspond to physical evaluation dimensions, and each curve represents a captioning strategy. All strategies are evaluated against the same dimension-specific references. Scores range from 0 to 100, with higher scores farther from the center. The LingBot strategies are reconstructed from the paper description. Physis captioning is currently in internal testing.
\toprule Physical dimension
CogVideoX
Open-Sora v1.3
EgoVid Dense
EgoVid Action
Cosmos Ego
Physis
Meta caption
\midrule Event coverage
60.17
60.25
61.50
57.33
81.08
84.42
49.17
Objects and roles
61.17
62.92
63.08
57.25
78.50
84.83
46.67
Actions and motion
63.18
59.91
62.64
59.55
84.64
87.73
54.09
Contact and interaction
61.43
54.29
59.29
54.29
83.57
77.14
38.57
States and outcomes
71.88
68.25
70.88
60.62
80.12
92.75
55.00
Physical mechanisms and causality
25.83
31.42
30.00
35.83
46.50
49.58
7.08
Table 4: Physical caption evaluation on the egocentric dataset. All strategies are evaluated on the same egocentric clips using the same dimension-specific references. Scores range from 0 to 100; higher is better. Bold indicates the highest score in each row. The overall average is computed from per-video scores over applicable dimensions, as defined in Section 5.1 .
Figure 5: Physical captions improve video reconstruction across domains. LPIPS and PVCS are shown for third-person (top) and egocentric (bottom) videos using H3 and LingBot-World 2.0. Original captions are source metadata captions. Point labels report mean distances; green annotations show relative reductions under physical-caption conditioning. Lower is better.
Figure 6: Physical captions preserve interaction roles and state changes in reconstruction. Rows show source frames (top), H3 with the physical caption (middle; labeled Gold in the source experiment), and H3 with the metadata caption (bottom). Columns correspond to the same times. Physical conditioning better retains left-hand stabilization, right-hand shell removal, and the transition from a covered to an exposed interior. Metadata conditioning instead generates the assembly of a larger stool.
Figure 7: Physical captions preserve state transitions and interaction details in third-person reconstruction. Columns compare the source video, H3 conditioned on the physical caption (Gold), and H3 conditioned on the metadata caption (Meta). Each row uses the same timestamp. (a) After the board is lifted, Gold retains volumetric toy shapes, while Meta exposes flattened, sheet-like shapes; the 9.50 s row shows both generated conditions after unloading. (b) Gold better retains localized drilling debris and the final separation of the drill bit from the workpiece. (c) Gold more closely preserves the persistent white–turquoise plume and barrel orientation, while Meta emphasizes thin liquid sheets and spray.
\toprule Evaluation
Original
Enriched
Gain
Table 5: Physical-description quality and policy performance under instruction enrichment. (a) PhysCapBench scores of training descriptions. (b) Single-task success on 20 standard and 20 randomized shared scenes (40 overall), with demonstrations and training settings held fixed. (c,d) Official baseline and enriched-instruction success rates, averaged over six selected tasks in each benchmark. Gains are differences between displayed values, in score points (a) or percentage points (b–d). Table reports per-task results.
Building omni-modal models with physical intelligence requires fine-grained supervision that captures physical evidence such as contact, support, deformation, and state transitions. However, existing omni-modal captioners primarily model general audiovisual semantics and often overlook transient or spatially localized physical evidence. We present a unified framework for physics-aware audiovisual captioning spanning data construction, training, and evaluation. Firstly, we build a data construction pipeline that identifies physics-rich clips and leverages OmniFysics-Agent to coordinate audio, visual, and physical-perception tools for collecting spatiotemporally aligned and traceable cross-modal evidence; within the Agent, a physical perception model (PPM) fine-tuned on approximately 2M image-level samples serves as a dedicated tool for extracting object-interaction and state-change cues. Secondly, we build the Daily-Physics 50K dataset and introduce the evidence-driven OmniPhysCap (OPC) benchmark to evaluate the recovery of physical and cross-modal evidence from generated captions. Finally, we train OmniFysics-Captioner from the resulting data. Our Captioner matches Gemini 3.1 Pro on audiovisual captioning, achieves state-of-the-art results on multiple video-captioning benchmarks, and substantially outperforms other open-source models. Ablations show that PPM evidence improves physical coverage and produces finer-grained, more reliable cross-modal descriptions.
Kaixiang Qiu, Minghao Han, Keliang Liu +7
Physical Superintelligence Lab, Fysics AI · College of Intelligent Robotics and Advanced Manufacturing, Fudan University
Video world models are expected to predict how the physical world evolves, yet they often produce visually plausible videos that violate basic physical principles. Existing approaches commonly assume that natural language is insufficient to represent the physical knowledge required for reliable generation, and therefore introduce additional visual, latent, numerical, or planning-based signals. We revisit this assumption and introduce Physis-Lang, a self-evolving framework that treats physical language as a shared and optimizable representation across data curation, model training, and video generation. Physis-Lang represents physical processes through language that describes their relevant entities, causes, interactions, governing principles, temporal evolution, and effects. To improve this representation, we construct PhysCapBench, which decomposes physical processes into atomic assertions and evaluates captions using recall and precision. An agentic loop iteratively analyzes assertion-level errors and refines the instruction used to produce physical captions. Physis-Lang further converts model deficiencies into textual descriptions and uses language-guided retrieval to identify visually diverse videos that cover missing physical processes. Experiments on four widely used physical video benchmarks with Wan and Cosmos backbones demonstrate consistent improvements in physical plausibility. Notably, starting from open-source Cosmos3-Nano backbones, our Physis-Lang-enhanced models surpass the leading proprietary Veo 3.1 model.
Modeling physical dynamics, including how objects move, interact, and change state, is central to video world models for embodied AI. We present StrucPhysVideo, a family of video world models that bridges physics-focused data curation with language- and action-conditioned prediction of scene evolution. Our data pipeline combines motion-aware video segmentation, quality and content filtering, and physical relevance verification with structured annotations of objects, materials, and temporally localized interactions. By disentangling camera motion from object behavior and explicitly describing contact, deformation, and state transitions, the pipeline provides supervision grounded in observable physical events. Building on these data, we introduce StrucPhysVideo-TI2V, a sparse Mixture-of-Experts (MoE) text-image-to-video model trained with a curriculum that progressively emphasizes physical dynamics while retaining general-domain video data. StrucPhysVideo-TI2V achieves state-of-the-art performance on Physics-IQ Verified, scoring 45.5% and outperforming Cosmos3-Super-Image2Video by 2.8 percentage points. Caption ablations across backbones further demonstrate the effectiveness of physics-focused supervision. We further extend StrucPhysVideo-TI2V to StrucPhysVideo-IA2V, an interactive image-action-to-video world model that predicts visual outcomes from robot end-effector commands. Action conditioning, causal autoregressive generation, and few-step distillation enable incremental robot rollouts with only four denoising steps. Together, StrucPhysVideo advances physical dynamics modeling from image- and language-conditioned video prediction toward action-driven interaction.