cs.ROSep 30, 2026

EWAM: Emergent Depth-Wise Specialization in a Unified Embodied Model -- From Semantic Understanding through Visual Foresight to Action

Authors: Hao Wang, Jiajun Wen, Jingzhi Liu, Shuoshuo Xue, Zhiliang Chen, Min Lin, Yicheng Chang, Xiaoyu Guo, +16 more

Organizations: Yinwang Intelligent Technology Co. Ltd.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language-action (VLA) policies emphasize semantic understanding, whereas world-action models (WAMs) learn predictive representations of environment dynamics. Systems that expose a policy to both sources often still concentrate action computation on a single expert. We present EWAM, an action-centric unified embodied model whose asymmetric joint attention lets action tokens read semantic, current-visual, predicted-future, and action information at every layer while the perceptual experts retain their distinct roles. Without layer-wise supervision, EWAM develops an emergent depth-wise specialization: action queries attend mainly to vision-language features in shallow layers, to predicted future frames in intermediate layers, and to action tokens themselves in deep layers. This handoff replicates across tasks and is stable across denoising steps. Checkpoint tracking and causal interventions show that it is learned and that action generation depends on it. EWAM is pretrained in two separate regimes, one on cross-embodiment robot trajectories and one on human egocentric video. In simulation and real-robot experiments, it surpasses existing VLA, WAM, and hybrid baselines. Human egocentric data improve both cross-embodiment transfer and real-robot robustness, and subtask-phase supervision improves long-horizon completion. Together, these results suggest that unified embodied learning can induce an ordered internal progression from semantic understanding, through visual foresight, to action formation.

Figures & tables

Figure 1: Architectures and action-attention routing across embodied-model paradigms. Each panel pairs a simplified architecture with action-query attention maps. Representative VLA, WAM, and multi-expert policies concentrate action attention on one primary source. In EWAM, the dominant source changes from vision-language features in shallow layers, through predicted-future features in intermediate layers, to action self-attention in deep layers. Colour encodes attention mass on an absolute zero-to-one scale, the horizontal axis is the policy inference step at denoising step 0, and the vertical axis is the layer index. Appendix A gives further details.
Figure 2: Overview of our unified embodied model, EWAM. Language and image observations, noisy action sequences, and noisy video latents are tokenized into layer-aligned VL, action, and video streams. Action queries attend to VL, video, and action keys, while VL and video queries remain within their own streams. The model predicts an action sequence and future video, and optionally predicts the active subtask and its corresponding phase progress.
Figure 3: Fine-grained visual handoff across depth. (a) Action attention to observed and predicted video features and to VL text and image features on the Adjust Bottle task. (b) Cross-task VL-image and future-frame attention, macro-averaged over 50 tasks and five denoising steps.
Figure 4: Emergence of the visual handoff. Raw attention-mass difference between future-frame and VL-image tokens (percentage points) across depth at four checkpoints of the training run of Table 13 . Positive values indicate that action queries place more attention mass on predicted future frames than on VL-image tokens. At 150 steps, raw future-frame mass first exceeds VL-image mass at layer 8; at 40,000 steps, it exceeds VL-image mass at every layer from 13 onward.
Training results (without pretraining)Inference results (robot-pretrained model)
ConfigurationScoreΔ\DeltaConfigurationScoreΔ\Delta
Full unified model83–Full unified model92.80–
Symmetric full attention81−2-2Mask all video, layers 0–991.98−0.82-0.82
No VL access, layers 0–978−5-5Mask VL, layers 0–976.53−16.27-16.27
No video access, layers 10–1978−5-5Mask condition, layers 10–1981.40−11.40-11.40
Action only, layers 24–298300Mask future, layers 10–193.55−89.25-89.25
Table 1: Layer-targeted training and fixed-checkpoint interventions on RoboTwin Randomized. The left panel reports independently trained models without pretraining. The right panel applies action-attention masks only at inference time to one fixed cross-embodiment robot-pretrained checkpoint, without retraining. Scores and deltas are therefore relative to the separate full-model baseline in each panel and should not be compared directly across panels. “Action only” removes both VL and video keys from action queries in the specified layers.
TaskNo injectionInjection settingObserved behavior or outcomeSuccess
Adjust Bottle5/5Left-facing (43, 46)Donor-directed first grasp misses; both recover after cutoff2/2
 Right-facing (44, 45, 47)Donor-consistent right arm instead of required left arm; no recovery0/3
Place Empty Cup5/5All targets (43–47)All injected rollouts fail0/5
Stack Blocks Two5/5All targets (43–47)All injected rollouts fail0/5
Click Bell5/5All targets (43–47)Only seed 46 remains successful1/5
Grab Roller5/5All targets (43–47)Only seed 45 remains successful1/5
Table 2: Counterfactual future-feature injection across five tasks.
Figure 5: Egocentric demonstration collection system. A Lumos Ego headset records fisheye RGB, time-of-flight depth, and camera motion, while two wrist-mounted Vive Tracker 3.0 devices measure metric six-degree-of-freedom wrist motion.
Figure 6: Egocentric capture and observation–action trajectory construction. (a) Capture setup. (b) Hand-motion reconstruction from RGB, SLAM, and Tracker observations. (c) Grasp-frame construction and bimanual trajectory optimization. (d) Mask-based grasp-state estimation.
SessionTCP P95 (mm)Checks
 LeftRight 
01331256.792.92Pass
01332677.0114.65Pass
01333542.531.66Pass
01335218.631.30Pass
2226504.0719.69Pass
Table 3: Kinematic checks and TCP position errors on five retargeted demonstrations. TCP P95 measures the distance between FK-derived reference positions and human-derived tool targets. “Pass” indicates satisfaction of the specified constraint and consistency checks.
Figure 7: Human demonstration and robot replay. Rows show the human egocentric recording, simulation replay, and physical Unitree G1-D replay. Columns correspond to seven visually matched action phases: grasp, raise, two pouring stages, return, placement, and release. Yellow and green curves show projected trajectories of the source and receiver cup bases in simulation, respectively. The simulation represents the transferred material using rigid particles.
ParadigmModelC2C (%)C2R (%)Average (%)
VLAstarVLA [ 46 ]46.523.1624.84
GalaxeaVLA [ 25 ]62.7012.7237.71
Xiaomi Robotics-0 [ 9 ]62.9018.2040.55
X-VLA [ 75 ]68.0020.9044.45
ABot-M0 [ 61 ]57.4030.3643.88
π0.5\pi_{0.5} [ 4 ]70.7046.0058.35
Table 4: RoboTwin 2.0 clean-to-random performance. Policies are adapted on clean demonstrations only. Average is the mean of C2C and C2R.
ParadigmModelClean (%)Randomized (%)Average (%)
VLAπ0.5\pi_{0.5} [ 4 ]82.776.879.8
ABot-M0 [ 61 ]86.185.185.6
LingBot-VLA [ 56 ]88.686.787.7
JoyAI-RA [ 71 ]90.589.389.9
HyVLA-0.5 [ 68 ]90.990.190.5
ACE-Ego-0 [ 28 ]91.190.690.9
Table 5: RoboTwin 2.0 in-domain results. All methods are adapted on both clean and randomized demonstrations. Average is the mean of Clean and Randomized.
ParadigmModelSpatialObjectGoalLongAverage
VLAπ0.5\pi_{0.5} [ 4 ]98.898.298.092.496.9
ABot-M0 [ 61 ]98.899.899.096.698.6
Qwen-VLA-Instruct [ 51 ]––––97.9
WAMFast-WAM [ 67 ]98.2100.097.095.297.6
LingBot-VA [ 29 ]98.599.697.298.598.5
LaWAM [ 11 ]99.499.698.497.098.6
Table 6: LIBERO simulation results. Success rates (%); baselines are from published evaluations, and dashes indicate unreported suite-level scores.
Table 14
ConditionTrialsSPFSuccess (%)
In-domain baseline1082080.0
Cup instance1073070.0
Placement1072170.0
Background and clutter1064060.0
All held-out conditions30209166.7
Table 9: Stage-resolved generalization on the G1-D pour-beans task. Ten trials per condition, one factor varied at a time against the in-domain baseline. S denotes a trial completed with no stage error, P a trial in which at least one stage fails but the episode still runs to completion, and F an outright failure. The success rate counts S only. The held-out aggregate excludes the in-domain baseline.
TaskSettingGraspPlaceStackFull
Instructed Rankingw/o subtask94.089.5–83.5
w/ subtask93.597.5–91.0
Instructed Stackingw/o subtask91.5–44.544.5
w/ subtask98.5–58.558.5
Instructed Ranking & Stackingw/o subtask60.037.024.51.5
w/ subtask63.051.539.516.0
Table 10: Effect of subtask-phase supervision on long-horizon block tasks. Grasp, Place, and Stack are stage-wise rates; Full is complete-episode success. Dashes mark inapplicable stages.
Figure 8: Cross-embodiment transfer to Aloha-Agilex-2. Success rate over the 50 RoboTwin 2.0 tasks at each evaluated robot-training checkpoint, with and without human-pretrained initialization. This protocol is not comparable to Tables 4 and 5 .
Post-training dataPretrainingTrain lossHeld-out L2L_{2}Success (%)
90 robot episodesRobot0.0550.009870
60 robot + 30 humanRobot0.0680.023140
60 robot + 30 humanHuman0.0340.016460
Table 11: Human pretraining and co-training on G1-D pouring. Losses are means over the final five evaluations; success is measured over 10 physical trials. Training loss is comparable only between the two co-trained rows.
Figure 9: Effects of human pretraining and human co-training during G1-D real-robot post-training. Both runs are co-trained on the same mixture of 60 robot and 30 retargeted human episodes and differ only in initialization. (a) Flow-matching loss on the shared post-training data. (b) Action L2L_{2} error on the 30 held-out G1-D episodes. Both panels use logarithmic vertical axes. Human pretraining reduces both errors relative to robot pretraining under the same co-training mixture.
Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

 VL-imageFuture frame
LayersMass (%)EnrichmentMass (%)Enrichment
0–936.75 [36.45, 37.05]1.74 [1.73, 1.76]9.58 [9.49, 9.66]0.23 [0.23, 0.23]
1027.99 [27.72, 28.26]1.33 [1.32, 1.34]7.42 [7.34, 7.51]0.18 [0.17, 0.18]
1122.42 [22.22, 22.60]1.06 [1.06, 1.07]17.79 [17.64, 17.94]0.42 [0.42, 0.43]
1221.48 [21.30, 21.66]1.02 [1.01, 1.03]21.15 [20.89, 21.42]0.50 [0.50, 0.51]
1313.37 [13.15, 13.61]0.63 [0.62, 0.65]24.85 [24.65, 25.05]0.59 [0.58, 0.59]
Appendix
Table 12: Cross-task visual handoff before and after token-count normalization. Each entry is the macro mean over 50 RoboTwin 2.0 tasks with its 95% task-bootstrap confidence interval in brackets. We report individual layers around the transition and averages outside that interval. Enrichment is measured relative to a task-specific uniform-token reference.
 VL-image EEFuture frame EEShallow mass (%) 
Step0–910–1920–290–910–1920–29TextImageCrossover
502.071.742.330.400.560.282.944.2—
1001.110.950.750.560.860.7714.323.523 (71)
1501.100.610.610.530.890.8122.623.311 (84)
2001.020.270.250.270.550.4637.121.911 (95)
2501.040.210.310.150.460.4545.822.012 (89)
Appendix
Table 13: Checkpoint evolution of the visual handoff under the asymmetric mask. Token-normalized enrichment EE of action-query attention to VL-image and future-frame tokens by stage, with the shallow-stage raw mass of text and image attention. The crossover column gives the first layer at which token-normalized future enrichment exceeds image enrichment for three consecutive layers, followed in parentheses by the percentage of subsequent layers that maintain the ordering; “—” marks checkpoints at which no such layer exists; this token-normalized criterion can lag the raw-mass crossover of Figure 4 . Measured on one representative sequence averaged over five denoising steps, using the token counts recorded for each checkpoint.
Figure 11: Checkpoint evolution of depth-wise attention routing under the asymmetric mask. Stage-level quantities across the trajectory of Table 13 , laid out to match Figure 15 so that the two attention designs can be compared panel by panel. (a) Shallow-stage action attention over the five key sources. (b) Token-normalized VL-image enrichment at the two ends of the network, with the dotted line marking the uniform-token reference E=1E=1 . (c) Fraction of layers at or beyond the stable crossover for which token-normalized future-frame enrichment exceeds VL-image enrichment, with the curve broken at checkpoints that have no stable crossover. All circles belong to the same thirty-two-GPU training trajectory.
Figure 12: Complete raw cross-task action-attention allocation. Attention mass to all five sources is macro-averaged over 50 RoboTwin 2.0 tasks after averaging five denoising steps within each task; shaded regions show 95% task-bootstrap confidence intervals. Dashed vertical lines mark the shared shallow (0–9), intermediate (10–19), and deep (20–29) stages.
Stage-level action-attention mass (%)
StageVLVideoAction
Shallow (0–9)75.38 [75.24, 75.51]14.05 [13.93, 14.16]10.58 [10.54, 10.61]
Middle (10–19)36.55 [36.28, 36.79]24.31 [23.90, 24.76]39.14 [38.86, 39.41]
Deep (20–29)25.22 [25.02, 25.40]15.32 [14.83, 15.84]59.46 [59.00, 59.90]
Mechanism prevalence and transition location
CriterionCoverageEstimateUncertainty
Appendix
Table 14: Cross-task replication of depth-wise specialization. Stage allocations are macro means with 95% task-bootstrap confidence intervals. Mechanism prevalence is measured across 50 tasks; its uncertainty uses 95% Wilson intervals. Each task contributes one representative sequence.
Figure 13: Attention allocation under a fully bidirectional mask at 40,000 steps. Rows are the thirty transformer layers. All panels use an absolute colour scale from zero to one, so intensities are comparable across panels within this figure. (a) Action-query allocation over the three key sources, the bidirectional counterpart of Figure 1 (d). (b) Vision-language-query allocation, which has no counterpart in the main text because the asymmetric mask fixes this stream to self-attention at every depth. (c) Action-query allocation resolved into the two video sub-blocks and the two vision-language sub-blocks, the bidirectional counterpart of Figure 3 (a). The vision-language column of (a) is dark at every depth and the image column of (c) is uniformly pale, in contrast to the shallow-to-deep gradient that the asymmetric mask produces.
MaskShallow (0–9)Middle (10–19)Deep (20–29)Shallow/Deep
Asymmetric (ours)1.750.560.0443.8
Fully bidirectional0.810.700.671.2
Appendix
Table 15: Depth-differentiated use of VL-image tokens under the two masks. Token-normalized enrichment of action-query attention to VL-image tokens, by stage. Values above one indicate that a source receives more attention than a uniform allocation over all keys would give. Both rows are converged 40,000-step runs without robot pretraining; asymmetric values are the 40,000-step row of Table 13 .
Figure 14: Vision-language-query attention allocation across depth under a fully bidirectional mask. The vision-language stream retains self-attention only in shallow layers; from roughly layer 11 onward video tokens become its dominant key source, reaching 82.0% in the deep stage. The dashed line marks the asymmetric mask of Eq. 3 , under which this curve is fixed at 100% self-attention by construction. Curves are averaged over ten denoising steps at 40,000 steps.
Figure 15: Attention routing under a fully bidirectional mask across training. (a) Vision-language-query allocation, split into self-attention and attention to video, for the deep stage (layers 20–29, solid) and the shallow stage (layers 0–9, dashed). (b) Shallow-stage action-query allocation. (c) Fraction of layers at or beyond the stable crossover for which token-normalized future-frame enrichment exceeds VL-image enrichment. Circles joined by lines are the four-GPU checkpoint trajectory; stars to the right of the dashed rule are the independently trained thirty-two-GPU run at 40,000 steps, which uses a different global batch size and is therefore shown as a separate measurement rather than as a continuation of the trajectory.
ProtocolTasksEvaluation/taskTest domain
Standard Clean50100Clean
Standard Randomized50100Randomized
Clean-to-Random50100Randomized Hard
Appendix
Table 16: RoboTwin evaluation protocols used in this work.
 ACE-Ego-0 [ 28 ]Fast-WAM [ 67 ]WLA-0 [ 62 ]EWAM
TaskCleanRand.CleanRand.CleanRand.CleanRand.
Adjust Bottle100100100100100100100100
Beat Block Hammer9892999795879495
Blocks Ranking RGB989710010098989997
Blocks Ranking Size8991949893857685
Click Alarmclock523810010099100100100
Appendix
Table 17: RoboTwin 2.0 per-task success rates (%). Each baseline is the strongest Randomized-setting representative of its paradigm. EWAM is evaluated on 100 episodes per task and setting.
PlatformEmbodimentTasks
FrankaSingle armStack bowls; object into box
DobotDual armPour water; tidy desk; towel manipulation
Unitree G1-DHumanoid dual armKettle pouring; pour beans
Appendix
Table 18: Overview of the seven real-world evaluation tasks. The evaluation spans single-arm, dual-arm, and humanoid dual-arm embodiments.
Figure 16: Seven real-world manipulation tasks evaluated on Franka, Dobot, and Unitree G1-D platforms.
Figure 17: Dual-Basket Sorting on Unitree G1-D. Top: a successful rollout with subtask-phase supervision that clears the table (initial state, grasp marker, grasp doll, grasp bottle, grasp tissue, and clear table). Bottom: two failure modes that occur frequently without this supervision. The first three frames show grasp height errors; the last three show grasp pose-and-position errors. These errors lead to repeated attempts on the same object and stalled task progress.
Configuration10 steps (ms)5 steps (ms)
Baseline inference1668–
Without future-frame VAE decoding1128578
+ graph-compatible RoPE and full-graph compilation461254
+ chunk-overlap correction464258
Appendix
Table 19: Inference latency per action chunk on a single RTX 5090. Each chunk contains 16 actions; step counts refer to denoising. The baseline includes future-frame VAE decoding and does not use full-graph compilation. Subsequent rows apply the listed changes cumulatively.
Model5 steps (ms)10 steps (ms)Per step (ms)
VLA
π0.5\pi_{0.5} [ 4 ]50601.9
Xiaomi Robotics-1 [ 58 ]18632127.1
LingBot-VLA-4B [ 56 ]29350442.3
WAM
Fast-WAM [ 67 ]21640137.1
Appendix
Table 20: Action-output inference latency under a common benchmark configuration. All models use a single RTX 5090, batch size one, bfloat16, and chunks of 16 actions with 16 dimensions each. Baseline weights are randomly initialized. The final column reports the marginal cost of one denoising step.
CorpusSource materialHoursAnchor camera pose from
VITRA‑1M [ 30 ]Ego4D, EPIC‑Kitchens, SSv2283Inverted extrinsics
EgoDex [ 23 ]Apple Vision Pro829Camera‑to‑world transform
Xperience [ 44 ]Head‑mounted stereo rig972SLAM pose and calibration
Total 2,084 
Appendix
Table 21: Human egocentric corpora used for human pretraining. The last column names the quantity from which each source’s anchor camera pose is obtained; all three then enter the identical computation of Eq. 14 . VITRA‑1M is used through four of its five subsets; its EgoExo4D subset is excluded.
Figure 18: Source composition and retained duration distribution of the high‑quality robotic manipulation data used for EWAM pretraining.
Figure 19: Robot platforms, manipulation tasks, and interacted objects covered by the EWAM pretraining data.
ModeRobot PlatformJoint Dim.
Single ArmFranka8
UR57
ARX-57
Dual ArmFranka16
UR514
Agilex14
Appendix
Table 22: Joint-state dimensions by robot platform.

Explore similar work

CardsList

  1. World Action Models: The Next Frontier in Embodied AI

    May 12, 2026Siyin Wang, Junhao Shi, Zhaoyang Fu +11Recent World-Action ModelsWorld Models

    1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore

  2. JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

    Aug 10, 2026Yihan Lin, Jiawei He, Shifeng Bao +6Efficient World-Action ModelRobot Policies

    School of Information, Renmin University of China, Beijing, China · XYZ Embodied AI, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China +3

  3. UniWAM: Unified World-Action Model

    Oct 1, 2026Jiayi Chen, Wenxuan Song, Jingbo Wang +13Action PredictionWorld Models