EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming
Authors: Yaru Niu, Binghong Chen, Changwei Yao, Shangtao Li, Zhenlong Fang, Revanth Krishna Senthilkumaran, Shuai Zhou, Yuemin Mao, Kalie Ching, Po-Ju Chao, Yudong Luo, Peiqi Yu, Heng Zhang, Hao Zhang, Bingqing Chen, Chen Qiu, H. Eric Tseng, Ye Yuan, Umar Iqbal, Changliu Liu, Jonathan Francis, Ding Zhao
Organizations: Carnegie Mellon University · University of California, Berkeley · Bosch Center for Artificial Intelligence · The University of Texas at Arlington · NVIDIA
Generalist manipulation policies can benefit from the complementary experience of humans, real robots, and simulated agents. However, differences in morphology, sensing, and control interfaces make it difficult to learn effectively from their combined demonstrations. We introduce EgoX, an egocentric cross-embodiment learning framework that organizes heterogeneous demonstrations through semantically corresponding observation and action modalities. EgoX pretrains a Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) on this mixture and fine-tunes it on target-embodiment demonstrations. Within each embodiment, MXT+ assigns a dedicated tokenizer to each observation modality and a dedicated expert to each action modality, with a Transformer trunk shared across embodiments. A training-only embodiment-dreaming objective predicts latent representations of future visual and proprioceptive states, encouraging the shared trunk to model each embodiment's dynamics. Across 13 task-embodiment pairs spanning five tasks and six robot embodiments, including five physical robots and one simulated humanoid, cross-embodiment pretraining raises MXT+'s average normalized task progress from 0.63 to 0.80 over training from scratch. The gains are particularly pronounced under out-of-distribution conditions, where average task progress increases from 0.53 to 0.77. Ablations demonstrate the benefits of broader pretraining mixtures, modular interfaces, and embodiment dreaming, with dreaming producing more similar latent geometry across embodiments. Project page: egox-dream.github.io.
Figures & tables
Figure 1: Overview of EgoX. EgoX learns from highly diverse embodiments, including humans, a humanoid, a wheeled bimanual robot, a quadruped, tabletop arms, and a simulated humanoid, with dexterous hands, grippers, and tools as end-effectors. It first pretrains on egocentric cross-embodiment data from humans and robots (top left), then fine-tunes on a new embodiment–task domain (bottom left). Its model, MXT+, routes each embodiment’s observations through modality tokenizers into a shared Transformer trunk and decodes actions with embodiment-specific action experts, while a training-only dreaming expert predicts future embodiment latents (center). The resulting policies perform manipulation across embodiments and long-horizon loco-manipulation on a humanoid (right, top and middle). Embodiment dreaming and cross-embodiment pretraining both improve the average task progress score and success rate (right, bottom).
Figure 2: Shared task structure across heterogeneous embodiments. Rows show a Human-Dex5 demonstration and rollouts of robot policies trained with EgoX for G1-Dex5 , G1-Sim-Dex3 , and R1Lite-Gripper on Kitchen-Bread . Columns show corresponding task stages. Soft spotlights highlight the active interaction regions.
Figure 3: MXT+ architecture and EgoX embodiments. Top: embodiment-specific modality tokenizers map each embodiment’s observations ( egocentric images , embodiment-specific proprioception , and left / right wrist, left / right end-effector-link, and torso poses) into tokens that a shared Transformer encoder-decoder trunk fuses across embodiments. Embodiment action experts decode the trunk output into embodiment-specific action chunks, while embodiment-dreaming experts predict future embodiment latents supervised by EMA copies of the corresponding modality tokenizers. Bottom: the embodiments used in EgoX, from left to right: Human-Dex5 , G1-Dex5 , xArm7-Dex4 , G1-Sim-Dex3 , LocoMan-Dex2 , R1Lite-Gripper , and Z1-Tool . Each panel marks the egocentric frame Fego (on the body carrying the main camera, or on the arm base for the fixed-base xArm7-Dex4 and Z1-Tool ), in which left / right wrist poses and targets are expressed, and the wrist frames, in which left / right end-effector link poses are expressed. Left and right end-effectors are shown in olive and orange , matching the modality colors above, alongside a close-up of each end-effector and its end-effector-link keypoints.
Figure 4: Unified egocentric frames and shared token spaces (Fig. 3 colors). Solid: now (frames, camera view); translucent: one action chunk later (static bodies shifted sideways). MXT+’s modality tokenizers (solid arrows) map observations into one inner token space: right / left wrists at Fego , right / left end-effector links at their Fwrist , torso at Froot ; pale purple parts (embodiment-specific proprioception, if any) and camera views into frame-free proprioception and image tokens (bottom) and embodiment tokens (top). MXT+’s shared trunk (ring) fuses all into outer action tokens; MXT+’s action (dashed) and dreaming (purple) experts decode the chunk (steel-blue tokens at Froot , arrowed discs: velocity commands , if any) and each body’s training-only future latents .
Figure 5: Manipulation performance across tasks and embodiments. Normalized task progress of ACT, π0.5 , MXT+ without pretraining, and MXT+ with cross-embodiment pretraining across 13 task–embodiment pairs spanning five tasks and six embodiments (five physical robots and the simulated G1-Sim-Dex3 ). Each panel reports ID and OOD performance.
Figure 6
Figure 8: Coarse ablation. Arrows go from w/o to w/ pretraining; violins show each mean’s uncertainty (from its SEM).
Table 1: Training hyperparameters. Shared settings apply to all experiments; the remaining blocks list the task-specific settings. Pretraining batch sizes count the samples drawn from each embodiment per optimizer step. Values in parentheses are for R1Lite-Gripper . Parameter counts exclude the EMA teachers and cover the trunk together with the modules of every embodiment in pretraining and of the target embodiment otherwise. The largest per-embodiment counts are for Kitchen-Bread and Kitchen-Cleanup , whose target embodiments add two wrist cameras.
Table 2: Baseline hyperparameters. For parameters and batch size, the first value is for Sort , Insert , and Scoop , and the second is for Kitchen-Bread and Kitchen-Cleanup , where ACT adds a ResNet-18 for each of the two wrist cameras.
Figure 10: Physical robot embodiments used in EgoX.
Embodiment
Side
Robot Joint Dim.
EEF Joint Dim.
Torso Action
Wrist Pose Action
Fingertip Targets
G1-Dex5
Bimanual
29
6+6
Yes
L/R
5 / hand
R1Lite-Gripper
Bimanual
15
1+1
No
L/R
2 / gripper
xArm7-Dex4
Right
7
16
No
Right
4
LocoMan-Dex2
Right
18
1
Yes
Right
2
Z1-Tool
Right
6
–
No
Right
–
Appendix
Table 3: Embodiment-specific interfaces used in EgoX. EgoX represents manipulation actions using semantically corresponding geometric modalities while allowing each embodiment to expose only the modalities applicable to its morphology. EEF denotes end effector.
Embodiment
Root frame Froot
Egocentric frame Fego
Torso pose
Wrist axes
Human-Dex5
Ground point below the head at episode start
Tracked head, carrying the stereo camera
x , y , z , rpy
Tracked wrist
G1-Dex5
Stance frame at the feet, from leg kinematics
Torso link, origin shifted by (0.154,0,0.406) m
z , rpy
L: (x,−z,y) R: (x,z,−y)
G1-Sim-Dex3
Same as G1-Dex5
R1Lite-Gripper
Wheeled chassis
Floating base (top torso link)
z , pitch
L: (x,−z,y) R: (x,z,−y)
LocoMan-Dex2
Ground projection of the trunk at mode entry
Trunk
z , rpy
(−z,x,−y)
xArm7-Dex4
Arm base
Arm base (static camera)
–
(x,−y,−z)
Appendix
Table 4: Root and egocentric frames of each embodiment. All frames use x forward, y left, and z up. “Torso pose” lists the torso-pose components marked valid in the recorded data (rpy: roll, pitch, and yaw). “Wrist axes” lists the native wrist axes that become the EgoX wrist (x,y,z) axes; for the human, the tracked wrist frame defines the convention.
Figure 11: Shared representation and embodiment-specific execution. Human-Dex5 motion is recorded using an Apple Vision Pro with a mounted stereo camera. Human recordings and robot trajectories share the pose/action representation. Robot teleoperation and policy rollout use embodiment-specific controllers. Colors follow the action experts in Fig. 3 ( olive : left, orange : right). Action modalities vary by embodiment; frequencies below each robot indicate its low-level control rate.
Embodiment
Arm
Hand / gripper
Body / base
Rate
G1-Dex5
IK, 14 arm joints
DexPilot retargeting
Lower-body policy
50 Hz
R1Lite-Gripper
RelaxedIK, 2×6 joints
Fingertip distance
Chassis velocity
200 Hz
xArm7-Dex4
IK, 7 joints
DexPilot retargeting
–
400 Hz
LocoMan-Dex2
Whole-body controller
Fingertip distance
Whole-body controller
400 Hz
Z1-Tool
IK, 6 joints
–
–
400 Hz
Appendix
Table 5: Low-level control used for teleoperation and deployment. Every embodiment receives targets at 30 Hz; the listed rate is the frequency of the embodiment controller.
Embodiment
Domain
Tasks
Train
Val.
Total
Hours
Human-Dex5
Human
Insert, Kitchen-Bread, Scoop, Sort
2,198
198
2,396
9.26
G1-Dex5
Real
Insert, Kitchen-Bread, Scoop, Sort
622
88
710
6.70
xArm7-Dex4
Real
Insert, Scoop, Sort
398
39
437
3.64
G1-Sim-Dex3
Sim
Kitchen-Bread, Kitchen-Cleanup
691
58
749
7.51
LocoMan-Dex2
Real
Scoop
34
3
37
0.35
R1Lite-Gripper
Real
Insert, Kitchen-Bread, Sort
398
40
438
6.36
Appendix
Table 6: Dataset statistics. Number of training and validation trajectories and total interaction duration for each embodiment used in EgoX.
Figure 12: Composition of the EgoX dataset across embodiments and tasks. Number of demonstrations (top) and hours (bottom) for each task–embodiment pair, sorted from high to low in each panel.
Figure 13: Scoop across four embodiments. Each row progresses from left to right through grasping or positioning the scoop, collecting waste, emptying it into the disposal bin, and returning the scoop. Z1-Tool uses a fixed scoop and does not perform a grasp. The LocoMan-Dex2 example shows an OOD trial.
Figure 14: Sort across three embodiments. Each row shows the robot grasping blocks and placing them into the color-matched bins, with time progressing from left to right.
Figure 15: Insert across two embodiments. Each row shows the robot approaching, grasping, aligning, and inserting the T-shaped block, with time progressing from left to right.
Figure 16: Kitchen-Bread across three embodiments. The upper three rows show container pickup and transport; the lower three rows continue the same rollouts through container placement, tool pickup, dipping, spreading, and tool return. Time progresses from left to right within each row. R1Lite-Gripper retrieves the container from a counter rather than a refrigerator.
Figure 17: Kitchen-Cleanup on G1-Sim-Dex3 . From left to right, the robot approaches the spreader, grasps and lifts it, and places it into the container.
Task
Embodiment
Stages and points
Smax
Scoop
G1-Dex5 , xArm7-Dex4 , LocoMan-Dex2
Grasp the scoop: 1; move the scoop to the target object: 1; scoop and lift the object: 2; drop off the object into the bin: 1.
5 + 1
Z1-Tool
The same stages without the grasp, since the scoop is fixed to the arm: approach: 1; scoop and lift: 2; deposit: 1.
4 + 1
Sort
G1-Dex5 , xArm7-Dex4 , R1Lite-Gripper
Each of the n blocks is scored independently: pick up the block: 1; approach a bin: 3 for the bin of the block’s color and 1 for the other bin; place the block into a bin: 3 for the bin of the block’s color and 1 for the other bin. All trials use n=3 blocks, except R1Lite-Gripper OOD trials with more blocks, e.g., n=4 ; each trial is normalized by its own Smax .
7n + 1
Insert
G1-Dex5 , xArm7-Dex4
Approach the T-shaped block: 1; grasp the block: 2; insert the block into the slot: 2 for a full insertion, 1 for a near-successful insertion, and 0 otherwise.
5 + 1
Kitchen-Bread
R1Lite-Gripper
Pick up the sauce container: 1; drive to the table: 1; place the container: 1; pick up the spreading tool: 1; dip the tool into the sauce: 1; spread the sauce onto the bread: 1; put the tool back: 1.
7 + 1
G1-Dex5 , G1-Sim-Dex3
Open the refrigerator: 1; grasp the sauce container: 1; close the refrigerator: 1; walk to the counter: 1; place the container: 1; pick up the spreading tool: 1; dip the tool into the sauce: 1; spread the sauce onto the bread: 1; put the tool back: 1.
9 + 1
Appendix
Table 7: Task progress scoring used for evaluation. A trial earns the points of every stage it completes and a one-point success bonus if it succeeds. Smax is the full stage score plus the bonus, and the normalized task progress score is P=(s+b)/Smax (Appendix A.9 ).
Figure 18: Real objects of the tasks. Each panel shows the objects of one task. The lower Insert photo shows the blocks from the side to compare their sizes. Which objects appear in ID and which in OOD trials differs between embodiments and is given in the description of each task.
Figure 20: Success rates. Success rate of ACT, π0.5 , MXT+ without pretraining, and MXT+ with cross-embodiment pretraining on each task–embodiment pair under ID and OOD conditions, evaluated on the same trials as Fig. 5 .
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · University of Washington