cs.CVSep 16, 2026
SaveStrucPhysVideo: Learning Physical Dynamics from Structured Captions and Robot Actions
Organizations: Awomo-WM Team
Abstract
Modeling physical dynamics, including how objects move, interact, and change state, is central to video world models for embodied AI. We present StrucPhysVideo, a family of video world models that bridges physics-focused data curation with language- and action-conditioned prediction of scene evolution. Our data pipeline combines motion-aware video segmentation, quality and content filtering, and physical relevance verification with structured annotations of objects, materials, and temporally localized interactions. By disentangling camera motion from object behavior and explicitly describing contact, deformation, and state transitions, the pipeline provides supervision grounded in observable physical events. Building on these data, we introduce StrucPhysVideo-TI2V, a sparse Mixture-of-Experts (MoE) text-image-to-video model trained with a curriculum that progressively emphasizes physical dynamics while retaining general-domain video data. StrucPhysVideo-TI2V achieves state-of-the-art performance on Physics-IQ Verified, scoring 45.5% and outperforming Cosmos3-Super-Image2Video by 2.8 percentage points. Caption ablations across backbones further demonstrate the effectiveness of physics-focused supervision. We further extend StrucPhysVideo-TI2V to StrucPhysVideo-IA2V, an interactive image-action-to-video world model that predicts visual outcomes from robot end-effector commands. Action conditioning, causal autoregressive generation, and few-step distillation enable incremental robot rollouts with only four denoising steps. Together, StrucPhysVideo advances physical dynamics modeling from image- and language-conditioned video prediction toward action-driven interaction.
Figures & tables
Figure 1: Physics-IQ Verified benchmark Results [ 1 ] . StrucPhysVideo achieves the highest Physics-IQ Verified score among all compared models in the text-image-to-video setting. * denotes our reproduction, and results for all other models are taken from the benchmark snapshot dated 16 September 2026 [ 2 ] .
Figure 2: Representative StrucPhysVideo generations under both the text-image-to-video (TI2V) and image-action-to-video (IA2V) settings, spanning general-domain, physical-world, and embodied scenarios.
Figure 3: StrucPhysVideo-TI2V overall architecture. The frozen Qwen3-VL-32B encoder jointly encodes the text prompt and reference image, while the frozen VAE encodes the target video. The first clean video latent is retained as a reference and the remaining latents are corrupted along the flow-matching path. Multimodal condition tokens and video tokens interact through joint self-attention in the StrucPhysVideo Transformer. Each block combines timestep-modulated attention with routed and shared experts, and the model predicts the latent velocity field used to supervise future-frame generation.
Figure 4: StrucPhysVideo-IA2V architecture. A causal action encoder maps raw action commands into latent action tokens and pooled contexts aligned with each video latent frame. In each DiT block, the pooled context modulates the video queries through AdaLN, while the corresponding action window provides the keys and values for cross-attention. A gated residual integrates action-conditioned features into the video stream.
Figure 5: Physical data pipeline. Source videos undergo Spatiotemporal Video Splitting, Technical Quality Filtering, and Content Purity Filtering. Physical Video Captioning first performs physical verification and generates a structured caption, then produces physical tags in a second model call. The resulting clips and annotations provide candidates for final dataset selection.
Figure 6: Examples of visual defects addressed by Technical Quality Filtering: solid-color frames, severe exposure defects, and persistent blur. Input-validity and temporal-quality checks are described in the text.
Figure 7: Face and Talking-Head Filtering rejects presenter-dominated clips while retaining clips with distant operators and human–object interactions.
Figure 8: VLM Text Review distinguishes text on physical objects from post-production overlays. Clips containing authentic scene text are retained for physical verification.
Figure 9: Physical verification in the first annotation call. Top row: rejected clips dominated by static product staging and decorative hand gestures. Bottom row: retained clips containing physical interactions. Retained clips receive complete structured captions and proceed to Closed-Set Tag Generation.
| Attribute | Vocabulary Size | Representative Values |
|---|---|---|
| Content category | 14 | Laboratory demonstration, educational explanation, daily life, industrial machinery, natural phenomena |
| Color | 14 | Warm, cool, saturated, neutral, high-contrast, monochrome |
| Shot scale | 7 | Extreme close-up, close-up, medium shot, full shot, long shot |
| Viewing angle | 6 | Eye-level, low-angle, high-angle, overhead, Dutch angle |
| Lens type | 5 | Ultra-wide, wide-angle, standard, telephoto, macro |
| Composition | 6 | Rule of thirds, centered, symmetrical, diagonal, leading lines |
Table 1: General video attributes, vocabulary sizes, and representative values.
Figure 10: Two-level taxonomy of physical phenomena. Five top-level categories contain 26 predefined leaf phenomena. An additional Other category covers physical interactions outside the vocabulary and has no predefined leaves.
Figure 11: Multi-label physical annotations for car washing, boiling water, and lathe cutting. Each example contains six frames and the associated phenomenon tags, grouped by top-level category. A filled star marks the primary phenomenon; outlined tags indicate additional phenomena.
Figure 12: Curation from source videos to annotated clips. Spatiotemporal Video Splitting produces candidate clips; Technical Quality Filtering and Content Purity Filtering select clips for Physical Video Captioning. Physical Verification selects physical clips, which receive structured captions and closed-set tags.
Figure 13: Action-conditioned data curation. Dataset-specific gates and adapters produce a unified EpisodeIR. Canonical versions define the training representations, with semantic checks during production and quality checks and independent readback before delivery of the corresponding training datasets.
Figure 14: Qualitative comparison among ground truth, Wan2.2-5B [ 4 ] , LingBot-Video-30B [ 6 ] , and StrucPhysVideo (Ours), with five frames shown from the start to the end of each sequence. In Case 1, the red boxes highlight substantial deformation of the track geometry, while the blue boxes indicate that the ball fails to roll along the shape of the track. In contrast, our model generates physically plausible ball-rolling motion that faithfully follows the track geometry. In Case 2, the blue boxes highlight the loss of the subject’s key facial features, whereas our model effectively preserves the subject’s identity and distinctive appearance.
Figure 15: Qualitative StrucPhysVideo-TI2V results for general-domain video generation. Each row shows five temporally ordered frames from one generated video, spanning people, vehicles, landscapes, and camera motion.
Figure 16: Qualitative StrucPhysVideo-TI2V results for physical-world video generation. Each row shows five temporally ordered frames from one generated video depicting a physical phenomenon or object interaction.
Figure 17: Qualitative StrucPhysVideo-TI2V results for embodied scenarios. Each row shows five temporally ordered frames from a generated robot manipulation sequence.
Figure 18: Caption ablation on Physics-IQ Verified. Wan2.2-5B [ 4 ] and LingBot-Video-30B [ 6 ] are evaluated under three settings: the reproduced baseline, incorporation of raw captions, and incorporation of physics-focused captions. Physics-focused captions yield the highest Verified Score for both backbones. Only Physics-IQ Verified scores are reported. ∗ denote scores reproduced by us.
| Method | PSNR | SSIM | LPIPS | Motion MAE | Traj. Acc. | Depth Acc. | Interaction |
|---|---|---|---|---|---|---|---|
| DreamDojo-AgiBot | 20.86 | 0.8277 | 0.1825 | 0.03509 | 0.7905 | 0.9736 | 3.366 |
| StrucPhysVideo-IA2V | 22.12 | 0.8365 | 0.1602 | 0.03368 | 0.8330 | 0.9854 | 3.470 |
Table 2: Quantitative performance on 5K video clips. Metrics exclude the reference frame. PSNR is in dB. Traj. Acc. and Depth Acc. are normalized to , with higher values indicating better agreement.
| Method | PSNR | SSIM | LPIPS | Motion MAE | Traj. Acc. | Depth Acc. | Interaction |
|---|---|---|---|---|---|---|---|
| A2World | 17.71 | 0.7110 | 0.2105 | 0.04316 | - | - | - |
| DreamDojo-AgiBot | 22.14 | 0.8415 | 0.1687 | 0.03338 | 0.7853 | 0.9685 | 3.365 |
| StrucPhysVideo-IA2V | 22.89 | 0.8571 | 0.1550 | 0.03172 | 0.8287 | 0.9827 | 3.470 |
Table 3: Quantitative comparison on head-view grid and frames 1–20 after the reference frame. A2World additionally receives initial wrist views and observed-pose-derived controls. Qualitative comparison is provided in Figure 19 .
Figure 19: AgiBot head-view rollouts at 0, 1, 2, 3, and 4 seconds. Each group shows GT, A2World, DreamDojo-AgiBot, and ours. All start from the same head image. All three models produce arm motion, but the generated trajectories and local robot geometry differ from ground truth. A2World also exhibits visible appearance distortion in these examples.
| Action dropout | Text dropout | Teacher CFG setting | Motion MAE | Traj. Acc. |
|---|---|---|---|---|
| 0% | 0% | w/o CFG | 0.03537 | 0.7913 |
| 15% | 0% | w/o CFG | 0.03455 | 0.7896 |
| 15% | 0% | Action CFG | 0.03449 | 0.8270 |
| 15% | 0% | Action CFG | 0.03517 | 0.8284 |
| 15% | 0% | Action CFG | 0.03583 | 0.8195 |
| 15% | 5% | Action CFG | 0.03432 | 0.8272 |
Table 4: Effects of teacher dropout and guidance on final distilled students. Dropout rates refer to bidirectional action-injection training. The reported action CFG scale is used both by the AR teacher to generate causal ODE distillation trajectories and by the bidirectional teacher to compute the real score during DMD. Bold and underlined values indicate the best and second-best values in each metric column, respectively.
More in this topic
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 257 papers