From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models
Authors: Guoqiang Liang, Yiqi Lin, Yanzhe Chen, Kevin Yuchen Ma, Mike Zheng Shou
Organizations: National University of Singapore · National University of Singapore Guangzhou Research Translation and Innovation Institute, Guangzhou, Guangdong 510700, China
Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM
Figures & tables
Figure 1: MetaWAM packs robot signals into one distribution-aware meta-frame. (a) Disjoint DCT bands route signals as inputs or denoising targets for policy, world-model, and value prediction. (b) Frozen-denoiser relative reconstruction error at σ=1.0 ( ↓ ): Spatial Duplication ( e.g. , Cosmos Policy [ 5 ] ) and VAE Encoding ( e.g. , AgiBot-World [ 7 ] ) require larger corrections, while our DCT Encoding is corrected least.
Figure 2: Robot-signal injection schemes used in the diagnostic study. (a–c) Single-signal encodings compare Spatial Duplication, VAE Encoding, and DCT Encoding. (d–g) Multi-signal packings compare Separate Spatial Duplication, Meta-Quadrants, Meta-Compact, and our Unified Meta-Frame, which assigns signals to disjoint channel bands and lifts them to one spatial latent by orthonormal DCT/iDCT.
Table 1: VDM-prior compatibility, measured as the relative reconstruction error of a frozen Cosmos-Predict2 [ 22 ] at noise level σ ( ↓ lower is better). Bold marks the best method per σ . † Separate Spatial Duplication is averaged over 3 frames.
Figure 3: Overview of MetaWAM. Normalized signals are packed into fixed coefficient slots and lifted by 2D inverse DCT. Slot-aware masks select clean inputs and prediction targets for policy, world-model, and value learning.
Method
Demos / Task
Avg. SR (%)
UVA [ 10 ]
50
50.0
DP-VLA [ 27 ]
3000
57.3
UWM [ 9 ]
1000
60.8
π0 [ 28 ]
300
62.5
GR00T-N1.5 [ 29 ]
300
64.1
Video Policy [ 30 ]
300
66.0
Table 2: RoboCasa benchmark results.
Model
Original
Camera
Robot
Lang.
Light
BG
Noise
Layout
Total
VLAs
π0 [ 28 ]
94.2
13.8
6.0
58.8
85.0
81.4
79.0
68.9
53.6
π0 -FAST [ 18 ]
85.5
65.1
21.6
61.0
73.2
73.2
74.4
68.8
61.6
OpenVLA-OFT [ 33 ]
97.6
55.6
21.7
81.0
92.7
91.0
78.6
68.7
67.9
UniVLA [ 34 ]
95.2
1.8
46.2
69.6
69.0
81.0
21.2
31.9
42.9
RIPT-VLA [ 35 ]
97.5
55.2
31.2
77.6
88.4
91.6
73.5
74.2
68.4
Table 3: LIBERO-Plus success rates (%). Baselines are reported in Table 4 of [ 32 ] ; Original denotes standard LIBERO. Lang.: language; BG: background. Bold marks the best result per column.
Figure 4: Real-world rollouts. For each of the three real-world tasks, we show a task reference, a representative generated trajectory, and a representative real rollout.
Method
In-Distribution (ID)
Out-of-Distribution (OOD)
Sweep
Place Cup
Build Block
SR%
Sweep
Place Cup
Build Block
SR%
π0 [ 28 ]
45.0
50.0
55.0
50.0
35.0
35.0
40.0
36.7
π0.5 [ 41 ]
70.0
70.0
75.0
71.7
45.0
45.0
50.0
46.7
Cosmos Policy [ 5 ]
75.0
80.0
80.0
78.3
60.0
60.0
70.0
63.3
MetaWAM (Ours)
80.0
85.0
90.0
85.0
65.0
65.0
75.0
68.3
Table 4: Real-world success rates (%) over 20 trials per task under ID and OOD settings.
SR
Latency
Signal recovery
World/value prediction
Encoding
Fr.
(%) ↑
(s) ↓
Gain ↑
Rel. L2↓
PSNR ↑
Prop. NEE ↓
Value MAE ↓
Separate Spatial Dup.
11
67.1
0.479
0.6361
0.3657
19.27
0.0250
0.072
Meta-Quadrants
8
65.9
0.291
0.6032
0.3988
18.86
0.0499
0.081
Meta-Compact
8
68.1
0.291
0.6563
0.3345
19.41
0.0193
0.065
Unified Meta-Frame
8
68.9
0.291
0.7146
0.2928
19.95
0.0143
0.046
Table 5: Ablation on robot-signal encoding. Fr.: latent frames per sequence. Signal recovery is measured with the frozen Cosmos-Predict2 denoiser of Tab. 1 ; world/value prediction is evaluated with the fine-tuned models. Prop. NEE: normalized error of the predicted future proprioception.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Table 6: Wan2.1-I2V-14B VDM-prior compatibility, measured by the relative reconstruction error of the frozen denoiser at noise level σ ( ↓ lower is better). Bold indicates the best method at each σ among the listed methods. † Sep. Spatial Dup. averages the reconstruction error over separate action, proprioception, and value latent frames.
Table 7: Wan2.2-TI2V-5B VDM-prior compatibility, measured by the relative reconstruction error of the frozen denoiser at noise level σ ( ↓ lower is better). Bold indicates the best method at each σ among the listed methods. † Sep. Spatial Dup. averages the reconstruction error over separate action, proprioception, and value latent frames.
Method
Spatial
Object
Goal
Long
Average
Diffusion Policy [ 42 ]
78.3
92.5
68.3
50.5
72.4
Dita [ 43 ]
97.4
94.8
93.2
83.6
92.3
π0 [ 28 ]
96.8
98.8
95.8
85.2
94.2
UniVLA [ 34 ]
96.5
96.8
95.6
92.0
95.2
π0.5 [ 41 ]
98.8
98.2
98.0
92.4
96.9
OpenVLA-OFT [ 33 ]
97.6
98.4
97.9
94.5
97.1
Appendix
Table 8: LIBERO simulation benchmark results.
Figure 5: Real-robot hardware setup. The platform consists of a 7-DoF Franka Emika Panda arm, a wrist-mounted Intel RealSense D405 camera, and two external Intel RealSense D435 cameras placed at the front and back of the workspace. The wrist camera provides observations near the end-effector, whereas the two external cameras provide third-person views.
Figure 6: Multi-view observations in the real-robot setup. We use three RGB-D camera views for real-world policy execution: a back-view camera, a front-view camera, and a wrist-mounted camera.
Figure 7: Visualization results in real-world OOD scenarios. We show representative key frames from real-robot rollouts under distribution shifts, including changed object locations, additional distractor objects, and background or layout variations. These examples complement the quantitative OOD evaluation and illustrate the robustness of MetaWAM under scene-level changes.
Figure 8: Visualization results on RoboCasa. We compare generated trajectories from Cosmos Policy and MetaWAM at a resolution of 224×224 . The examples show that MetaWAM preserves coherent future-observation generation while using a compact unified meta-frame to encode robot-related signals.
Figure 9: Frozen-VDM attention to encoded robot signals. The input carrier (left) and block-14 attention map (right) show an attention lift of 1.40× over the encoded region relative to the full-frame mean. White regions in the input are padding.
Figure 10: Action recovery with analytical and learned interfaces. Training action-chunk MAE for MetaWAM (left) and the MLP encoder/decoder control (right). Action L1 is a monitoring metric, not an additional loss. The two panels use different vertical scales.
World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.
Chenhao Qiu, Ruixiang Wang, Runyi Zhao +7
Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute
World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
Cascade world-action models (WAMs) use predicted visual futures to condition robot actions, but inherit costly iterative inference from diffusion-pretrained experts. We ask what a single-step predicted future must preserve to support accurate single-step action prediction. In the evaluated WAM, resampling video noise has little effect on downstream actions, yet naively truncating both streams to one step substantially reduces task success. Video-space supervision only partially closes this gap, and a local sensitivity analysis shows that vision prediction errors have highly unequal consequences for action accuracy. These findings motivate DirectWAM, which converts pretrained video and action experts into single-step predictors without stochastic sampling through action-sufficient distillation, while retaining the explicit future-to-action cascade. After learning a one-step action model, we train the future predictor by decoding actions from its own predicted futures and backpropagating demonstration action error through the expert, with an optional future-latent reconstruction loss. The resulting objective evaluates the student-generated future through its downstream control consequences rather than relying on representation matching alone. DirectWAM achieves 97.1% average success across four LIBERO suites and 85.4% on RoboTwin 2.0, outperforming Flash-WAM at the same one-evaluation-per-stream budget. This advantage has also extended to three real-world tasks that we designed. Compared with the full iterative teacher, it reduces inference latency by 28.0×.