From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models
Authors: Guoqiang Liang, Yiqi Lin, Yanzhe Chen, Kevin Yuchen Ma, Mike Zheng Shou
Organizations: National University of Singapore · National University of Singapore Guangzhou Research Translation and Innovation Institute, Guangzhou, Guangdong 510700, China
Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM
Figures & tables
Figure 1: MetaWAM packs robot signals into one distribution-aware meta-frame. (a) Disjoint DCT bands route signals as inputs or denoising targets for policy, world-model, and value prediction. (b) Frozen-denoiser relative reconstruction error at σ=1.0 ( ↓ ): Spatial Duplication ( e.g. , Cosmos Policy [ 5 ] ) and VAE Encoding ( e.g. , AgiBot-World [ 7 ] ) require larger corrections, while our DCT Encoding is corrected least.
Figure 2: Robot-signal injection schemes used in the diagnostic study. (a–c) Single-signal encodings compare Spatial Duplication, VAE Encoding, and DCT Encoding. (d–g) Multi-signal packings compare Separate Spatial Duplication, Meta-Quadrants, Meta-Compact, and our Unified Meta-Frame, which assigns signals to disjoint channel bands and lifts them to one spatial latent by orthonormal DCT/iDCT.
Table 1: VDM-prior compatibility, measured as the relative reconstruction error of a frozen Cosmos-Predict2 [ 22 ] at noise level σ ( ↓ lower is better). Bold marks the best method per σ . † Separate Spatial Duplication is averaged over 3 frames.
Figure 3: Overview of MetaWAM. Normalized signals are packed into fixed coefficient slots and lifted by 2D inverse DCT. Slot-aware masks select clean inputs and prediction targets for policy, world-model, and value learning.
Method
Demos / Task
Avg. SR (%)
UVA [ 10 ]
50
50.0
DP-VLA [ 27 ]
3000
57.3
UWM [ 9 ]
1000
60.8
π0 [ 28 ]
300
62.5
GR00T-N1.5 [ 29 ]
300
64.1
Video Policy [ 30 ]
300
66.0
Table 2: RoboCasa benchmark results.
Model
Original
Camera
Robot
Lang.
Light
BG
Noise
Layout
Total
VLAs
π0 [ 28 ]
94.2
13.8
6.0
58.8
85.0
81.4
79.0
68.9
53.6
π0 -FAST [ 18 ]
85.5
65.1
21.6
61.0
73.2
73.2
74.4
68.8
61.6
OpenVLA-OFT [ 33 ]
97.6
55.6
21.7
81.0
92.7
91.0
78.6
68.7
67.9
UniVLA [ 34 ]
95.2
1.8
46.2
69.6
69.0
81.0
21.2
31.9
42.9
RIPT-VLA [ 35 ]
97.5
55.2
31.2
77.6
88.4
91.6
73.5
74.2
68.4
Table 3: LIBERO-Plus success rates (%). Baselines are reported in Table 4 of [ 32 ] ; Original denotes standard LIBERO. Lang.: language; BG: background. Bold marks the best result per column.
Figure 4: Real-world rollouts. For each of the three real-world tasks, we show a task reference, a representative generated trajectory, and a representative real rollout.
Method
In-Distribution (ID)
Out-of-Distribution (OOD)
Sweep
Place Cup
Build Block
SR%
Sweep
Place Cup
Build Block
SR%
π0 [ 28 ]
45.0
50.0
55.0
50.0
35.0
35.0
40.0
36.7
π0.5 [ 41 ]
70.0
70.0
75.0
71.7
45.0
45.0
50.0
46.7
Cosmos Policy [ 5 ]
75.0
80.0
80.0
78.3
60.0
60.0
70.0
63.3
MetaWAM (Ours)
80.0
85.0
90.0
85.0
65.0
65.0
75.0
68.3
Table 4: Real-world success rates (%) over 20 trials per task under ID and OOD settings.
SR
Latency
Signal recovery
World/value prediction
Encoding
Fr.
(%) ↑
(s) ↓
Gain ↑
Rel. L2↓
PSNR ↑
Prop. NEE ↓
Value MAE ↓
Separate Spatial Dup.
11
67.1
0.479
0.6361
0.3657
19.27
0.0250
0.072
Meta-Quadrants
8
65.9
0.291
0.6032
0.3988
18.86
0.0499
0.081
Meta-Compact
8
68.1
0.291
0.6563
0.3345
19.41
0.0193
0.065
Unified Meta-Frame
8
68.9
0.291
0.7146
0.2928
19.95
0.0143
0.046
Table 5: Ablation on robot-signal encoding. Fr.: latent frames per sequence. Signal recovery is measured with the frozen Cosmos-Predict2 denoiser of Tab. 1 ; world/value prediction is evaluated with the fine-tuned models. Prop. NEE: normalized error of the predicted future proprioception.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Table 6: Wan2.1-I2V-14B VDM-prior compatibility, measured by the relative reconstruction error of the frozen denoiser at noise level σ ( ↓ lower is better). Bold indicates the best method at each σ among the listed methods. † Sep. Spatial Dup. averages the reconstruction error over separate action, proprioception, and value latent frames.
Table 7: Wan2.2-TI2V-5B VDM-prior compatibility, measured by the relative reconstruction error of the frozen denoiser at noise level σ ( ↓ lower is better). Bold indicates the best method at each σ among the listed methods. † Sep. Spatial Dup. averages the reconstruction error over separate action, proprioception, and value latent frames.
Method
Spatial
Object
Goal
Long
Average
Diffusion Policy [ 42 ]
78.3
92.5
68.3
50.5
72.4
Dita [ 43 ]
97.4
94.8
93.2
83.6
92.3
π0 [ 28 ]
96.8
98.8
95.8
85.2
94.2
UniVLA [ 34 ]
96.5
96.8
95.6
92.0
95.2
π0.5 [ 41 ]
98.8
98.2
98.0
92.4
96.9
OpenVLA-OFT [ 33 ]
97.6
98.4
97.9
94.5
97.1
Appendix
Table 8: LIBERO simulation benchmark results.
Figure 5: Real-robot hardware setup. The platform consists of a 7-DoF Franka Emika Panda arm, a wrist-mounted Intel RealSense D405 camera, and two external Intel RealSense D435 cameras placed at the front and back of the workspace. The wrist camera provides observations near the end-effector, whereas the two external cameras provide third-person views.
Figure 6: Multi-view observations in the real-robot setup. We use three RGB-D camera views for real-world policy execution: a back-view camera, a front-view camera, and a wrist-mounted camera.
Figure 7: Visualization results in real-world OOD scenarios. We show representative key frames from real-robot rollouts under distribution shifts, including changed object locations, additional distractor objects, and background or layout variations. These examples complement the quantitative OOD evaluation and illustrate the robustness of MetaWAM under scene-level changes.
Figure 8: Visualization results on RoboCasa. We compare generated trajectories from Cosmos Policy and MetaWAM at a resolution of 224×224 . The examples show that MetaWAM preserves coherent future-observation generation while using a compact unified meta-frame to encode robot-related signals.
Figure 9: Frozen-VDM attention to encoded robot signals. The input carrier (left) and block-14 attention map (right) show an attention lift of 1.40× over the encoded region relative to the full-frame mean. White regions in the input are padding.
Figure 10: Action recovery with analytical and learned interfaces. Training action-chunk MAE for MetaWAM (left) and the MLP encoder/decoder control (right). Action L1 is a monitoring metric, not an additional loss. The two panels use different vertical scales.