What Visual Representations Do World Action Models Need?
Authors: Kai Liu, Jiangran Lyu, Tingyu Deng, Yixin Zheng, Jingtian Wan, Wenbo Cui, Mi Yan, Yizhou Zhou, Haoran Li, Dongbin Zhao, Zhizheng Zhang, He Wang
Organizations: Institute of Automation, Chinese Academy of Sciences · Galbot · Peking University · University of Science and Technology of China · Individual Researcher
World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R2-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.
Figures & tables
Figure 1: Overview of our systematic study of visual representations in WAMs. Under a shared training protocol, we compare visual feature sources (RQ1), five integration mechanisms (RQ2), and feature compositions (RQ3).
Table 1: RQ1: single-source success rates (%) across all seven LIBERO-Plus ( Fei et al., 2026 ) perturbations under channel-wise concatenation. Sources are grouped as general-purpose or specialized, with subgroups indicating the dominant pretraining emphasis (Section 3.2 ). Results are reported as mean ± standard deviation across three evaluation seeds.
Figure 3: Feature-integration performance and mechanism analysis. (a) Average LP success rates across the thirteen representations in Table 1 . (b) DINOv3 ( Siméoni et al., 2026 ) prediction ablations within channel-wise concat. (c) Shared-PCA features overlaid on future RGB observations. (d) Local sensitivity of final normalized actions to current DINOv3 features, inspired by gradient-based sensitivity and attribution ( Simonyan et al., 2014 ; Hoffman et al., 2019 ; Yu et al., 2023 ) . We evaluate action sensitivity at three selected time steps in each trajectory, corresponding to approach, grasp, and placement. Each point reports the sensitivity of one action group in the predicted action chunk at a selected time step. Metric and computation details are in Appendix D.3 .
Figure 4: RQ3: progressive feature composition on LIBERO-Plus (mean ± SD over three evaluation seeds). Each bar in the two-, three-, and four-source groups adds its labeled source to the indicated base.
Figure 5: Real-robot evaluation on four physical manipulation tasks.
Formulation
LIBERO
LP
Direct action prediction
96.8
70.8
+ recipe
98.4
79.4
Video-then-action (IDM)
98.6
73.6
+ recipe
99.1
87.7
Table 2: Validation Results across WAM formulations (a) and real-robot tasks (b). Real-robot evaluation uses 20 trials per task; metric definitions are in Appendix C.3 .
Method
Success (%)
GR00T-N1.6 ( NVIDIA GEAR Team, 2025 )
47.6
DiT4DiT ( Ma et al., 2026b )
50.8
FLARE ( Zheng et al., 2025 )
55.0
LDA-1B ( Lyu et al., 2026 )
55.4
DIAL ( Chen et al., 2026c )
70.2
WALA ( Liu et al., 2026 )
75.2
Table 3: Comparison with selected reported methods on RoboCasa GR1 Tabletop ( Nasiriany et al., 2024 ) and RoboTwin 2.0 ( Chen et al., 2026b ) .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Attention masks for training and inference in our three WAM formulations. Rows are queries and columns are keys/values; filled cells permit attention and empty cells block it. v0 , v1:T , and a1:H denote current visual, future visual, and action tokens, respectively. Representative temporal positions are shown, with each visual entry representing a block of spatial tokens. Direct prediction omits future visual tokens at inference. Joint denoising and IDM share the displayed connectivity but differ in future-token conditioning: jointly denoised tokens versus clean ground-truth tokens during IDM training and previously generated, fixed tokens during IDM action inference.
Mechanism
Representation interface
Future-feature prediction
Transformer parameters
Auxiliary objective
Feature pathway at inference
REPA
hidden-state alignment
No
base WAM only
cosine alignment
removed
Gated cross-attention
current feature memory
No
added cross-attention
none
retained
Channel-wise concat.
feature channels
Yes
shared video expert
feature flow matching
retained during denoising
Token-wise concat.
feature tokens
Yes
shared video Transformer
feature flow matching
retained during denoising
Feature expert
feature tokens
Yes
independent expert
feature flow matching
retained during denoising
Appendix
Table 4: Implementation summary of the five feature-injection mechanisms.
Encoder
d=8
d=15
d=22
d=8
λ=0.1
λ=0.1
λ=0.1
λ=0.5
CLIP
77.51
76.56
75.98
77.47
DINOv3
78.05
77.36
77.86
78.02
SigLIP 2
76.28
78.57
77.25
76.10
VGGT- Ω
76.43
76.00
76.85
78.23
V-JEPA 2.1
75.36
75.50
75.89
73.73
Appendix
Table 5: REPA sensitivity to alignment depth and loss weight. LIBERO-Plus success rates (%) across five visual encoders.
Parameter
Value
Optimizer
AdamW
Initial learning rate
1×10−4
Minimum learning rate
1×10−6
Learning-rate schedule
Cosine decay
Warmup
Linear, first 5% of training
Weight decay
0.01
Appendix
Table 6: Common training settings for the simulation benchmarks.
Parameter
LIBERO
RoboCasa
RoboTwin 2.0
Training data
LIBERO (four suites)
GR1 Tabletop
Clean + randomized demonstrations
Number of tasks
40
24
50
Dataset size (demos)
1,712
24,000
27,500
RGB views
2
1 (ego)
3
Image resolution
224×224
224×224
240×320
Action dimension
7
47
14
Appendix
Table 7: Benchmark-specific training configurations. Image sizes are per view.
LIBERO
LIBERO-Plus
Source
Avg.
Camera
Robot
Lang.
Light
Bkg.
Noise
Layout
Total
VAE-only reference
97.9
51.2
77.8
91.1
94.3
62.0
66.6
82.4
74.7
REPA
DINOv3
98.4
59.0
75.5
92.2
96.2
67.8
75.1
83.0
78.1
V-JEPA 2.1
97.9
52.7
76.3
91.0
92.6
64.9
71.6
80.9
75.4
VC-1
97.5
53.7
75.5
90.6
95.7
60.5
69.6
81.1
74.9
Appendix
Table 8: Detailed LIBERO and LIBERO-Plus results for REPA, gated cross-attention, channel-wise concatenation, token-wise concatenation, and the feature expert.
Configuration
Current-feature input
Future-feature prediction
LP success (%)
Current only
Yes
No
76.5
Future only
No
Yes
81.9
Current + future
Yes
Yes
84.0
Appendix
Table 9: DINOv3 prediction ablations under channel-wise concatenation.
Source combination
Seed 41
Seed 42
Seed 43
Mean ± SD
D+V
85.13
84.62
85.03
84.93±0.28
D+L
86.01
85.50
85.97
85.83±0.28
D+S
84.93
83.80
84.75
84.49±0.61
D+G
84.38
84.29
84.55
84.40±0.13
D+H
85.31
83.91
84.23
84.48±0.74
D+U
86.05
85.85
86.47
86.12±0.32
Appendix
Table 10: RQ3 composition success rates on LIBERO-Plus across three evaluation seeds. All values are percentages; the standard deviation reflects evaluation-seed variability for a fixed trained checkpoint.
Task
Success (%)
Task
Success (%)
Cup to drawer (close)
54.0
Placemat to bowl
86.0
Potato to microwave (close)
86.0
Placemat to plate
88.0
Milk to microwave (close)
78.0
Placemat to tiered shelf
68.0
Bottle to cabinet (close)
88.0
Plate to bowl
90.0
Wine to cabinet (close)
76.0
Plate to cardboard box
76.0
Can to drawer (close)
96.0
Plate to pan
88.0
Appendix
Table 11: Task-level success rates (%) of R 2 -WAM on RoboCasa GR1 Tabletop. Each task is evaluated with 50 trials.
Task
Clean
Rand.
Task
Clean
Rand.
Adjust bottle
99.0
99.0
Place container plate
98.0
98.0
Beat block hammer
97.0
94.0
Place dual shoes
95.0
96.0
Blocks ranking RGB
100.0
97.0
Place empty cup
99.0
99.0
Blocks ranking size
66.0
87.0
Place fan
91.0
97.0
Click alarmclock
98.0
97.0
Place burger fries
100.0
99.0
Click bell
100.0
100.0
Place mouse pad
93.0
94.0
Appendix
Table 12: Task-level success rates (%) of R 2 -WAM on RoboTwin 2.0. Each of the 50 tasks is evaluated with 100 trials in each of the clean and randomized (Rand.) settings.