What Visual Representations Do World Action Models Need?
Authors: Kai Liu, Jiangran Lyu, Tingyu Deng, Yixin Zheng, Jingtian Wan, Wenbo Cui, Mi Yan, Yizhou Zhou, Haoran Li, Dongbin Zhao, Zhizheng Zhang, He Wang
Organizations: Institute of Automation, Chinese Academy of Sciences · Galbot · Peking University · University of Science and Technology of China · Individual Researcher
World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R2-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.
Figures & tables
Figure 1: Overview of our systematic study of visual representations in WAMs. Under a shared training protocol, we compare visual feature sources (RQ1), five integration mechanisms (RQ2), and feature compositions (RQ3).
Table 1: RQ1: single-source success rates (%) across all seven LIBERO-Plus ( Fei et al., 2026 ) perturbations under channel-wise concatenation. Sources are grouped as general-purpose or specialized, with subgroups indicating the dominant pretraining emphasis (Section 3.2 ). Results are reported as mean ± standard deviation across three evaluation seeds.
Figure 3: Feature-integration performance and mechanism analysis. (a) Average LP success rates across the thirteen representations in Table 1 . (b) DINOv3 ( Siméoni et al., 2026 ) prediction ablations within channel-wise concat. (c) Shared-PCA features overlaid on future RGB observations. (d) Local sensitivity of final normalized actions to current DINOv3 features, inspired by gradient-based sensitivity and attribution ( Simonyan et al., 2014 ; Hoffman et al., 2019 ; Yu et al., 2023 ) . We evaluate action sensitivity at three selected time steps in each trajectory, corresponding to approach, grasp, and placement. Each point reports the sensitivity of one action group in the predicted action chunk at a selected time step. Metric and computation details are in Appendix D.3 .
Figure 4: RQ3: progressive feature composition on LIBERO-Plus (mean ± SD over three evaluation seeds). Each bar in the two-, three-, and four-source groups adds its labeled source to the indicated base.
Figure 5: Real-robot evaluation on four physical manipulation tasks.
Formulation
LIBERO
LP
Direct action prediction
96.8
70.8
+ recipe
98.4
79.4
Video-then-action (IDM)
98.6
73.6
+ recipe
99.1
87.7
Table 2: Validation Results across WAM formulations (a) and real-robot tasks (b). Real-robot evaluation uses 20 trials per task; metric definitions are in Appendix C.3 .
Method
Success (%)
GR00T-N1.6 ( NVIDIA GEAR Team, 2025 )
47.6
DiT4DiT ( Ma et al., 2026b )
50.8
FLARE ( Zheng et al., 2025 )
55.0
LDA-1B ( Lyu et al., 2026 )
55.4
DIAL ( Chen et al., 2026c )
70.2
WALA ( Liu et al., 2026 )
75.2
Table 3: Comparison with selected reported methods on RoboCasa GR1 Tabletop ( Nasiriany et al., 2024 ) and RoboTwin 2.0 ( Chen et al., 2026b ) .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Attention masks for training and inference in our three WAM formulations. Rows are queries and columns are keys/values; filled cells permit attention and empty cells block it. v0 , v1:T , and a1:H denote current visual, future visual, and action tokens, respectively. Representative temporal positions are shown, with each visual entry representing a block of spatial tokens. Direct prediction omits future visual tokens at inference. Joint denoising and IDM share the displayed connectivity but differ in future-token conditioning: jointly denoised tokens versus clean ground-truth tokens during IDM training and previously generated, fixed tokens during IDM action inference.
Mechanism
Representation interface
Future-feature prediction
Transformer parameters
Auxiliary objective
Feature pathway at inference
REPA
hidden-state alignment
No
base WAM only
cosine alignment
removed
Gated cross-attention
current feature memory
No
added cross-attention
none
retained
Channel-wise concat.
feature channels
Yes
shared video expert
feature flow matching
retained during denoising
Token-wise concat.
feature tokens
Yes
shared video Transformer
feature flow matching
retained during denoising
Feature expert
feature tokens
Yes
independent expert
feature flow matching
retained during denoising
Appendix
Table 4: Implementation summary of the five feature-injection mechanisms.
Encoder
d=8
d=15
d=22
d=8
λ=0.1
λ=0.1
λ=0.1
λ=0.5
CLIP
77.51
76.56
75.98
77.47
DINOv3
78.05
77.36
77.86
78.02
SigLIP 2
76.28
78.57
77.25
76.10
VGGT- Ω
76.43
76.00
76.85
78.23
V-JEPA 2.1
75.36
75.50
75.89
73.73
Appendix
Table 5: REPA sensitivity to alignment depth and loss weight. LIBERO-Plus success rates (%) across five visual encoders.
Parameter
Value
Optimizer
AdamW
Initial learning rate
1×10−4
Minimum learning rate
1×10−6
Learning-rate schedule
Cosine decay
Warmup
Linear, first 5% of training
Weight decay
0.01
Appendix
Table 6: Common training settings for the simulation benchmarks.
Parameter
LIBERO
RoboCasa
RoboTwin 2.0
Training data
LIBERO (four suites)
GR1 Tabletop
Clean + randomized demonstrations
Number of tasks
40
24
50
Dataset size (demos)
1,712
24,000
27,500
RGB views
2
1 (ego)
3
Image resolution
224×224
224×224
240×320
Action dimension
7
47
14
Appendix
Table 7: Benchmark-specific training configurations. Image sizes are per view.
LIBERO
LIBERO-Plus
Source
Avg.
Camera
Robot
Lang.
Light
Bkg.
Noise
Layout
Total
VAE-only reference
97.9
51.2
77.8
91.1
94.3
62.0
66.6
82.4
74.7
REPA
DINOv3
98.4
59.0
75.5
92.2
96.2
67.8
75.1
83.0
78.1
V-JEPA 2.1
97.9
52.7
76.3
91.0
92.6
64.9
71.6
80.9
75.4
VC-1
97.5
53.7
75.5
90.6
95.7
60.5
69.6
81.1
74.9
Appendix
Table 8: Detailed LIBERO and LIBERO-Plus results for REPA, gated cross-attention, channel-wise concatenation, token-wise concatenation, and the feature expert.
Configuration
Current-feature input
Future-feature prediction
LP success (%)
Current only
Yes
No
76.5
Future only
No
Yes
81.9
Current + future
Yes
Yes
84.0
Appendix
Table 9: DINOv3 prediction ablations under channel-wise concatenation.
Source combination
Seed 41
Seed 42
Seed 43
Mean ± SD
D+V
85.13
84.62
85.03
84.93±0.28
D+L
86.01
85.50
85.97
85.83±0.28
D+S
84.93
83.80
84.75
84.49±0.61
D+G
84.38
84.29
84.55
84.40±0.13
D+H
85.31
83.91
84.23
84.48±0.74
D+U
86.05
85.85
86.47
86.12±0.32
Appendix
Table 10: RQ3 composition success rates on LIBERO-Plus across three evaluation seeds. All values are percentages; the standard deviation reflects evaluation-seed variability for a fixed trained checkpoint.
Task
Success (%)
Task
Success (%)
Cup to drawer (close)
54.0
Placemat to bowl
86.0
Potato to microwave (close)
86.0
Placemat to plate
88.0
Milk to microwave (close)
78.0
Placemat to tiered shelf
68.0
Bottle to cabinet (close)
88.0
Plate to bowl
90.0
Wine to cabinet (close)
76.0
Plate to cardboard box
76.0
Can to drawer (close)
96.0
Plate to pan
88.0
Appendix
Table 11: Task-level success rates (%) of R 2 -WAM on RoboCasa GR1 Tabletop. Each task is evaluated with 50 trials.
Task
Clean
Rand.
Task
Clean
Rand.
Adjust bottle
99.0
99.0
Place container plate
98.0
98.0
Beat block hammer
97.0
94.0
Place dual shoes
95.0
96.0
Blocks ranking RGB
100.0
97.0
Place empty cup
99.0
99.0
Blocks ranking size
66.0
87.0
Place fan
91.0
97.0
Click alarmclock
98.0
97.0
Place burger fries
100.0
99.0
Click bell
100.0
100.0
Place mouse pad
93.0
94.0
Appendix
Table 12: Task-level success rates (%) of R 2 -WAM on RoboTwin 2.0. Each of the 50 tasks is evaluated with 100 trials in each of the clean and randomized (Rand.) settings.
World Action Models (WAMs) offer a promising route for robot manipulation by using video generation models to model future scene evolution before producing control actions. However, our empirical observations reveal a phenomenon: generating plausible visual futures does not always guarantee the extraction of accurate actions. To diagnose this failure, we conduct action-head attention analysis and causal interventions. We find that the action decoder fails to focus on task-relevant interaction regions and remains sensitive to perturbations in task-irrelevant areas. This reveals a representation mismatch: hidden states optimized for visual reconstruction are not inherently organized in a form useful for low-level action control. In this paper, we propose AGRA, an Action-Grounded Representation Alignment objective that regularizes the world-action interface by aligning intermediate video diffusion features with spatially coherent semantic representations from a foundation visual encoder. We evaluate AGRA on real-world manipulation tasks. Experiments show that AGRA makes world model representations more action-grounded: by focusing the action decoder on the correct interaction regions, it improves object localization accuracy and affordance understanding, and makes the policy more robust to perturbations in task-irrelevant regions. As a result, AGRA consistently improves both in-distribution performance and out-of-distribution generalization over the baseline world action model.
This work presents RepWAM, a representation-centric world action model (WAM) built on representation visual-action tokenizers. Existing WAMs typically inherit reconstruction-oriented video tokenizers from pretrained video generation models. Although these tokenizers preserve visual fidelity, pixel reconstruction alone provides limited guidance for learning instruction-following dynamics that connect future prediction with robot control. To address this, we explore a semantic visual-action latent space for representation-centric world action modeling. Specifically, we train a representation visual-action tokenizer that maps visual inputs into aligned visual and latent action tokens. We then pretrain our WAM to jointly model future visual states and the latent actions that connect them under language instructions, followed by adaptation to real robot trajectories for closed-loop manipulation. Experiments on real-world manipulation tasks and simulation benchmarks show that RepWAM delivers strong performance across diverse manipulation settings, while ablations highlight the value of semantic visual-action tokenization over reconstruction-oriented alternatives. These results establish representation visual-action tokenization as a promising foundation for world action models and a step toward generalist robot policies. Code and weights will be available at https://github.com/wdrink/RepWAM.
Junke Wang, Qihang Zhang, Shuai Yang +5
Institute of Trustworthy Embodied AI, Fudan University · 2Robbyant, Ant Group · 3Hongkong University of Science and Technology
World Action Models (WAMs) commonly rely on video generation to bridge visual world modeling and robot control. However, video-based WAMs face three coupled limitations: dense multi-frame future tokens make inference costly, full video prediction spends capacity on action-irrelevant temporal and appearance details, and long-horizon future imagination may introduce errors that mislead action prediction. These issues raise a simple question: Does world action model really need video generation? We propose ImageWAM, a simple WAM framework that repurposes pretrained image editing models for robot action prediction. In contrast to video generation, image editing provides a better-matched prior: it only needs to model a target-frame transformation, focuses on action-relevant current-to-target visual differences, and grounds task instructions to localized visual changes through edit pretraining. In practice, ImageWAM does not decode the target frame at inference time; instead, it conditions a flow-matching action expert on the KV caches produced by image-editing denoising, using them as a compact world-action context. ImageWAM outperforms standard VLA baselines and matching competitive WAMs without additional policy pretraining across different simulator and real-world experiments. It also reduces FLOPs to 1/6 and latency to 1/4 of video-based WAMs. Attention analysis further shows that editing caches focus on task-relevant change regions, supporting image editing as an effective alternative to video-based world-action modeling.
Yuyang Zhang, Wenyao Zhang, Zekun Qi +7
Shanghai Jiao Tong University · Tsinghua University · Tencent Robotics X +1