Ego-CEVR: Learning Cross-Embodiment Visual Representations from Human Egocentric Videos
Authors: Keru Zhou, Zhirui Fang, Xinrun Xu, Yaopeng Lou, Zeyuan He, Zihao Zhu, Xinze Niu, Chen Wang, Tianfu Li, Samith Va, Bin Qian, Junwei Li, Wenjia Ding, Yicheng Xu, Chang Zhou, Ziwei Wang, Li Liu, Xiu Li, Maoqing Yao
Organizations: Maniformer · Tsinghua University · Nanyang Technological University · Huazhong University of Science and Technology · The Chinese University of Hong Kong, Shenzhen · Harbin Institute of Technology
As egocentric video becomes increasingly available for embodied learning, an important research question is how to turn this low-cost, scalable source of data into robot manipulation capabilities. Egocentric videos naturally provide rich context for high-level task understanding, while real-robot demonstrations with wrist observations and action annotations provide supervision for fine-grained action alignment. We present Ego-CEVR, a data-native framework for hierarchical visual representation learning that assigns distinct learning objectives to global and interaction representations. The global module learns object relations, task state, and temporal progress through future prediction on egocentric video. The Interaction module interprets wrist observations in global context and learns local alignment and contact cues through robot action supervision. Separate residual paths expose both representations to the action policy, preserving direct access to task context alongside fine-grained execution information. Ego-CEVR supports independent pretraining of its global video module on large-scale human egocentric videos and videos from diverse robot embodiments, without requiring robot action labels or paired wrist observations.
Figures & tables
Method
Spatial
Object
Goal
Long
Avg.
π0 [ 3 ]
95.8
96.2
98.8
85.2
94.2
π0.5 [ 4 ]
98.8
98.2
98.0
92.4
96.9
OpenVLA-OFT [ 49 ]
97.9
97.6
98.4
94.5
97.1
VLA-JEPA [ 40 ]
96.2
99.6
97.2
95.8
97.2
Fast-WAM [ 35 ]
98.2
100.0
97.0
95.2
97.6
Ego-CEVR (ours)
99.2
99.8
98.2
93.6
97.7
Table 1: LIBERO success rate (%). Avg. retains the aggregate reported with each result. Bold indicates the best listed score in each column.
Method
Camera
Robot
Language
Light
Background
Noise
Layout
Average
π0.5 [ 4 ]
75.8
79.4
83.3
95.5
95.0
89.6
87.0
85.7
ACoT-VLA [ 50 ]
72.6
82.6
87.5
97.7
96.5
87.8
86.6
86.6
VLA-JEPA [ 40 ]
63.3
67.1
85.4
95.6
93.6
66.3
85.1
79.5
JEPA-WAM [ 51 ]
79.2
59.2
68.2
93.3
94.6
83.6
76.1
79.2
Abot-M0.5 [ 52 ]
70.5
87.4
88.6
94.0
89.7
75.5
85.2
83.4
Being-H0.7 [ 53 ]
82.0
59.0
82.8
97.8
90.0
93.5
88.5
84.8
Table 2: LIBERO-Plus success rate (%) by perturbation type. Average retains each method’s reported aggregation and is not recomputed as an unweighted mean of the displayed categories. Bold indicates the best listed score in each column.
Method
Easy
Hard
Average
π0.5 [ 4 ]
70.7
46.0
58.4
Fast-WAM [ 35 ]
77.8
1.9
39.9
UniWAM [ 19 ]
75.2
67.6
71.4
WorldScape Policy 2.0 [ 55 ]
84.9
38.5
61.7
4D-WAM [ 56 ]
81.5
41.8
61.6
JEPA-WAM [ 51 ]
79.9
36.9
58.4
Table 3: RoboTwin success rate (%) under Easy and Hard conditions. Average retains each method’s reported aggregation.
Method
LIBERO
LIBERO-Plus
Spatial
Object
Goal
Long
Avg.
Camera
Robot
Language
Light
Background
Noise
Layout
Average
π0.5 [ 4 ]
98.8
98.2
98.0
92.4
96.9
75.8
79.4
83.3
95.5
95.0
89.6
87.0
85.7
Fast-WAM [ 35 ]
98.2
100.0
97.0
95.2
97.6
17.7
45.3
60.1
52.2
22.8
28.5
61.2
41.3
Flat fusion + Head prediction
98.4
98.2
97.0
94.0
96.9
66.7
63.2
77.9
81.1
80.2
71.4
80.6
73.8
Hierarchy + Head/Wrist prediction
99.4
99.6
98.0
94.4
97.9
88.8
70.8
84.9
96.6
97.4
92.2
86.1
87.4
Hierarchy + Head prediction (Ego-CEVR)
99.2
99.8
98.2
93.6
97.7
91.1
78.3
86.6
97.7
98.8
95.4
85.7
90.2
Table 4: Ablations of representation structure and predictive supervision with matched video sources and robot demonstrations. Results are success rates (%).
Video source
Plug Insertion
Water Pouring
Package Sorting
Robot Head only
20
60
60
Robot Head + 10 h Robot Head
35
95
85
Robot Head + 10 h Human Ego
30
75
80
Table 5: Video-source ablations on three real-robot manipulation tasks. All settings use the same robot demonstrations for policy training. Results are success rates (%).
Evaluation task
G2 Task 1 Full
None
Ego Task 2
G2 Task 2 Head
G2 Task 2 Full
G2 Task 1 SR
60
65
65
75
G2 Task 2 SR
0
10
20
55
Table 6: Task and robot transfer results. Task 1 is Water Pouring , and Task 2 is Block Stacking . SR denotes success rate (%).
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.