Scaling robot manipulation models requires large amounts of interaction data, yet collecting diverse real-robot experience remains costly. Human manipulation videos, in contrast, provide abundant and diverse task experience that could potentially benefit robot learning. However, differences in morphology, action spaces, and control mechanisms make human motions unsuitable as direct supervision for robot actions, raising a scientific question: what knowledge in human videos should be transferred across embodiments to robot manipulation? To address this question, we analyze subtask transfer through the lens of causal beams, focusing on which contextual conditions and interaction dependencies should be retained to support local task effects. Based on this perspective, we propose Causal-Beam-Guided Subtask Transfer (CBST). CBST learns manipulation-related representations from human videos and grounds them in task-level semantic variables using robot demonstrations. Within a predefined manipulation model, causal-beam constraints guide contextual information selection by requiring retained assignments to sustain local mechanism outputs while preserving nontrivial responsiveness to retained inputs. The selected semantic information conditions a world-action model, allowing the robot to generate actions from its own observations without requiring human-robot motion correspondence. Experiments on robot manipulation benchmarks show improvements in task success over the corresponding world-action model baselines.
Figures & tables
Figure 1: The cross-embodiment transfer problem. Human and robot demonstrations share task objectives but differ in execution. Examples of placing a bowl, opening a lid, and lifting a book motivate the question: what task information should transfer beyond embodiment-specific motion and goal endpoints?
Figure 2: (a) Candidate local manipulation SCM for a fixed task goal, showing the endogenous parent structure. X and I are boundary inputs, and the mechanisms for D and Y may differ across embodiments. Variable groups are instantiated into task-specific variables before beam selection. (b) Illustrative human and robot motion realizations of the same bowl-placement task.
Figure 3: Overview of Causal-Beam-Guided Subtask Transfer (CBST). Human-video pretraining and robot adaptation support state prediction and contextual beam selection. The resulting semantic conditions guide a frozen world–action model to predict actions and future states. Solid and dashed arrows denote inference and training/supervision, respectively.
Table 4Table 5
Figure 4: A LingBot-VA + CBST execution: operating the switch, turning on the stove, and placing the moka pot on it. Frames follow temporal order; timestamps indicate video playback time.
Figure 5: CBST execution of cup placement on the AgileX platform. One arm lifts the cup and transfers it to the receiving gripper, which transports it to the target location and releases it on the tabletop. Handover enables cross-workspace transport but is not the task goal. Labels describe observed events, and timestamps indicate video playback time.
Figure 6: Temporal-order comparison for ResNet18–GRU pretraining on SSv2. (a) Training classification loss: bold lines show sample-weighted means over non-overlapping blocks of 120 updates; faint traces show the original values. (b) Final top-1 accuracy on 24,777 validation clips after three epochs, with one run per condition.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Role in CBST
Scope / WAM
SSv2
Human-video representation pretraining
174 action categories; planner initialization
RoboTwin 2.0
Robot adaptation and evaluation
50 tasks; LingBot-VA
LIBERO-Long
Robot adaptation and evaluation
10 tasks; LingBot-VA and Cosmos Policy (Predict2.5)
RoboCasa
Robot adaptation and evaluation
24 tasks; Cosmos Policy (Predict2.5)
Appendix
Table 5: Datasets and evaluation scope. The RoboCasa entry refers to the 24-task protocol used by Cosmos Policy, rather than the complete RoboCasa task collection.
Split
Videos
Annotation availability
Training
168,913
Action labels
Validation
24,777
Action labels
Test
27,157
Labels withheld
Total
220,847
174 action categories
Appendix
Table 6: Official SSv2 split sizes. These are release-level counts; the number of clips retained after preprocessing is reported separately in Table 14 .
Interaction type
Representative distinctions
Object displacement
Directional movement; motion toward or away from the camera.
Placement and removal
Moving objects onto surfaces, into receptacles, or away from their previous locations.
Lifting and release
Partial versus complete lifting; holding versus releasing.
Contact and motion
Contact that moves an object versus contact with little observable displacement.
Support and stability
Tilting a support with or without causing the supported object to fall.
Attachment
Successful attachment versus an unsuccessful attempt.
Appendix
Table 7: Representative action distinctions in SSv2. Descriptions are abbreviated semantic summaries rather than verbatim labels or an exhaustive class list.
Horizon
Tasks
Task names
1
30
Adjust Bottle; Beat Block Hammer; Click Alarmclock; Click Bell; Dump Bin Bigbin; Grab Roller; Lift Pot; Move Can Pot; Move Pillbottle Pad; Move Playingcard Away; Move Stapler Pad; Open Laptop; Open Microwave; Place A2B Left; Place A2B Right; Place Bread Basket; Place Container Plate; Place Empty Cup; Place Fan; Place Mouse Pad; Place Object Scale; Place Object Stand; Place Phone Stand; Place Shoe; Press Stapler; Rotate QRcode; Shake Bottle Horizontally; Shake Bottle; Stamp Seal; Turn Switch.
2
15
Handover Block; Handover Mic; Hanging Mug; Pick Diverse Bottles; Pick Dual Bottles; Place Bread Skillet; Place Burger Fries; Place Can Basket; Place Cans Plasticbox; Place Dual Shoes; Place Object Basket; Put Object Cabinet; Scan Object; Stack Blocks Two; Stack Bowls Two.
Table 8: The 50 RoboTwin tasks grouped by the horizon labels used in the LingBot-VA report. Names are abbreviated from the official task identifiers.
Suite
Tasks
Main variation
Spatial
10
Spatial relations identifying a target object.
Object
10
Object identity under a shared placement goal.
Goal
10
Desired outcomes within related environments.
LIBERO-90
90
Diverse manipulation tasks.
Long / LIBERO-10
10
Tasks requiring extended execution.
Appendix
Table 9: LIBERO suite composition. LIBERO-90 and LIBERO-Long are the two partitions of LIBERO-100, not additional copies of that collection.
Suite
Task variants
Spatial
Move the black bowl to the plate. The bowl starts: between plate and ramekin; beside ramekin; at table center; on cookie box; inside top drawer; on ramekin; beside cookie box; on stove; beside plate; on cabinet.
Object
Place each target in the basket: alphabet soup; cream cheese; salad dressing; barbecue sauce; ketchup; tomato sauce; butter; milk; chocolate pudding; orange juice.
Goal
Open middle drawer; bowl to stove; wine bottle to cabinet top; bowl inside top drawer; bowl to cabinet top; plate toward stove front; cream cheese into bowl; activate stove; bowl to plate; wine bottle to rack.
Appendix
Table 10: Task composition of the Spatial, Object, and Goal suites. Shared instructions are factored out for compactness.
ID
Scene
Goal summary
L1
LR2
Basket: alphabet soup and tomato sauce.
L2
LR2
Basket: cream cheese and butter.
L3
K3
Activate stove; place moka pot on it.
L4
K4
Black bowl inside bottom drawer; drawer closed.
L5
LR5
White mug on left plate; yellow-and-white mug on right plate.
L6
S1
Book in caddy’s rear compartment.
Appendix
Table 11: The ten LIBERO-Long tasks. LR, K, and S denote living-room, kitchen, and study scenes, respectively.
Table 12: The 24 RoboCasa tasks used in the Cosmos Policy evaluation protocol. Task identifiers follow its environment release. PnP denotes pick-and-place.
Method
Model / interface
Role
Benchmark
LingBot-VA
Video–action generation
Backbone
R, L
Cosmos Policy–Predict2.5
Joint action and future prediction
Backbone
L, C
Cosmos Policy–Predict2
Earlier video-model configuration
Reference
L, C
X-VLA
Soft-prompted VLA
Reference
R, L
π0
Flow-based VLA
Reference
R, L, C
π0.5
Generalist VLA
Reference
R, L
Appendix
Table 13: Roles of the principal comparison methods. R, L, and C denote RoboTwin, LIBERO, and RoboCasa. Benchmark entries indicate the comparisons included in this paper, not the complete capabilities of each method.
Item
Configuration
Visual encoder
ResNet18
Temporal encoder
GRU
Camera-feature fusion
Mean pooling over available camera views
Human pretraining
174-class action classification on SSv2 using cross-entropy loss
Transferred components
Visual encoder, temporal module, and representation projection
Robot prediction heads
Semantic states, interval variables, and candidate parent subsets
Appendix
Table 14: Planner architecture and training configuration.
RoboTwin 2.0
LIBERO-Long
RoboCasa
Item
LingBot-VA
LingBot-VA
Predict2.5
Predict2.5
Tasks
50
10
10
24
Episodes per task
100
150
150
150
Total episodes
5,000
1,500
1,500
3,600
Appendix
Table 15: Evaluation scope and episode budgets. RoboTwin counts apply separately to Easy and Hard. LIBERO-Long and RoboCasa counts aggregate evaluation seeds. Each WAM is evaluated independently.
Method
Horizon 1
Horizon 2
Horizon 3
All 50 tasks
X-VLA ∗
81.6 / 82.5
59.3 / 55.9
61.2 / 66.0
72.9 / 72.8
π0
66.5 / 61.6
66.1 / 54.7
61.6 / 50.2
65.9 / 58.4
π0.5
85.1 / 80.2
79.3 / 73.0
78.6 / 67.4
82.7 / 76.8
Motus
91.0 / 90.6
85.2 / 80.9
85.0 / 84.2
88.7 / 87.0
LingBot-VA
94.18 / 93.56
90.35 / 86.95
93.22 / 93.28
92.93 / 91.55
LingBot-VA + CBST
94.73 / 93.97
91.93 / 88.80
95.80 / 95.20
94.00 / 92.54
Appendix
Table 16: RoboTwin success rates (%) grouped by task horizon. Each cell reports Easy / Hard. Values follow Table 1 of [ Li et al., 2026 ] ; X-VLA ∗ is adopted there from Motus.
Method
Source
Spatial
Object
Goal
Long
Average
Octo
L
78.9
85.7
84.6
51.1
75.1
Seer
L
–
–
–
87.7
–
MoDE
L
–
–
–
94.0
–
SuSIE
L
–
–
–
76.3
–
SpatialVLA
L
88.2
89.9
78.6
55.5
78.1
TraceVLA
L
84.6
85.2
75.1
54.1
74.8
Appendix
Table 17: Complete published LIBERO comparison. All values are success rates (%). L: Table 2 of [ Li et al., 2026 ] ; C: Table 1 of [ Kim et al., 2026 ] ; K: official Cosmos Cookbook [ Kim and Gu, 2026 ] . Source-reported averages and uncertainty values are preserved.
Method
Demos per task
Average SR (%)
GR00T-N1
300
49.6
UVA
50
50.0
DP-VLA
3000
57.3
GR00T-N1 + DreamGen
300 (+10000 synthetic)
57.6
GR00T-N1 + DUST
300
58.5
UWM
1000
60.8
Appendix
Table 18: Complete RoboCasa comparison over 24 tasks. Baseline scores and demonstration counts follow [ Kim et al., 2026 ] . Predict2.5 results follow the official Cosmos Cookbook [ Kim and Gu, 2026 ] . Synthetic data are identified separately.
Figure 7: Additional recorded LingBot-VA executions:(a) stacking three bowls in RoboTwin, (b) handing over a block in RoboTwin, and (c) placing a black bowl in a drawer and closing it in LIBERO. Each row shows six temporally ordered frames;timestamps indicate video playback time.
Variant
Retrain
Change relative to CBST
Without human pretraining
Yes
Omit SSv2 pretraining while retaining the common visual initialization and robot adaptation procedure.
Shuffled human pretraining
Yes
Shuffle sampled frames within each human clip before pretraining; retain labels, frame content, and pretraining budget.
State-only
No
Render admissible goals and predicted semantic states without the selected interaction-dependency fields.
Full structure
No
Render all candidate parent fields instead of the contextually selected subset.
Set λm=0 while preserving the remaining supervision and reference mechanisms.
Appendix
Table 19: Definitions of the principal ablation variants. “Retrain” refers to robot-side planner adaptation; the WAM remains frozen in every variant.
LingBot-VA
Variant
RoboTwin Easy
RoboTwin Hard
LIBERO-Long
Backbone
92.93
91.55
98.5±0.5
CBST
94.00
92.54
98.60
Without human pretraining
92.80
91.38
98.00
Shuffled human pretraining
92.94
91.54
98.47
State-only
92.82
91.36
98.13
Appendix
Table 20: Principal ablation results across benchmarks. All values are success rates (%). RoboTwin and LingBot-VA reference values follow [ Li et al., 2026 ] ; Predict2.5 reference values follow [ Kim and Gu, 2026 ] .
Temporal module
Parameters
LingBot-VA SR
Predict2.5 SR
Planner latency
MLP
12.0
98.53
97.13
9
TCN
12.3
98.53
97.20
11
GRU
12.5
98.60
97.33
12
Transformer
13.2
98.66
97.53
13
Appendix
Table 21: Temporal architecture comparisons on LIBERO-Long. Parameter counts refer to the planner; latency refers to a planner update with the same input specification and accelerator.
Factor
values
Reference
Fixed quantities
Sampled frame count
{4,8,16,32}
16
Lookback duration and temporal sampling strategy.
Lookback duration (s)
{1,2,4,8}
4
Frame count and recent/history allocation ratio.
λsize
{0,10−3,10−2,10−1}
10−2
Other loss weights, reference mechanisms, and training budget.
Appendix
Table 22: Sensitivity-analysis configurations. Default values correspond to the full CBST configuration.
ID
Task
Placement objective and success condition
T1
Remote-control placement
Move the remote from the tabletop into the designated open container; it must remain fully inside after gripper release.
T2
Cup placement
Move the cup to the designated tabletop region and leave it stable after release; handover may be used to connect the source and target workspaces.
T3
Plate racking
Place the plate into the designated rack slot, with the rack supporting the plate after both grippers disengage.
T4
Apple placement
Place an apple inside the designated fruit basket and release it.
T5
Banana placement
Place a banana within the designated tray and release it.
T6
Orange placement
Place an orange inside the designated bowl and release it.
Appendix
Table 23: Task inventory.
Cosmos Policy (Predict2.5)
+ CBST
ID
Task
Successes
SR (%)
Successes
SR (%)
Gain
T1
Remote
(17/20)
85
(18/20)
90
5
T2
Cup
(14/20)
70
(17/20)
85
15
T3
Plate
(13/20)
65
(15/20)
75
10
T4
Apple
(10/20)
50
(12/20)
60
10
T5
Banana
(15/20)
75
(16/20)
80
5
Appendix
Table 24: Detailed real-world results. Each method has 20 trials per task. Gains are measured in percentage points.
Figure 8: Additional real-world executions on the AgileX platform: (b) plate racking and (c) remote-control placement. Frames are shown in temporal order, with timestamps indicating video playback time. Annotations describe observed manipulation events.
Large-scale manipulation data is essential for robot learning, yet collecting robot demonstrations remains expensive and difficult to scale. Meanwhile, abundant egocentric human manipulation videos provide rich behavioral experiences, but transferring them across embodiments remains challenging due to differences between human hands and robotic end-effectors. Recent advances in video world models offer a promising pathway to synthesize robot-centric manipulation videos from human observations, while their cross-embodiment transfer capability remains largely unexplored. Therefore, we introduce H2R-Bench, a benchmark for evaluating cross-embodiment human-to-robot manipulation video generation, where models transform egocentric human demonstrations into robot manipulation videos under specified embodiments. Each benchmark instance contains a human demonstration video, target embodiment constraints, and source-grounded annotations covering task goals, action events, functional contacts, and object responses. H2R-Bench evaluates generated videos through five dimensions, including goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general video quality. We benchmark eleven state-of-the-art video generation models across six manipulation families and two robot embodiments. Our evaluation reveals that current video world models remain limited in human-to-robot manipulation transfer: even leading models often fail in embodiment consistency, functional interaction, and task execution. H2R-Bench provides a systematic diagnostic framework for evaluating whether video world models can bridge the human-to-robot embodiment gap and convert human manipulation observations into robot-centric training resources.
Dingyi Rong, Yue Shi, Chaofan Ma +6
1Shanghai Jiao Tong University · 2Shanghai Artificial Intelligence Laboratory
Learning robotic manipulation from human videos is a promising solution to the data bottleneck in robotics, but the distribution shift between humans and robots remains a critical challenge. Existing approaches often produce entangled representations, where task-relevant information is coupled with human-specific kinematics, limiting their adaptability. We propose a generative framework for cross-embodiment video editing that directly addresses this by learning explicitly disentangled task and embodiment representations. Our method factorizes a demonstration video into two orthogonal latent spaces by enforcing a dual contrastive objective: it minimizes mutual information between the spaces to ensure independence while maximizing intra-space consistency to create stable representations. A parameter-efficient adapter injects these latent codes into a frozen video diffusion model, enabling the synthesis of a coherent robot execution video from a single human demonstration, without requiring paired cross-embodiment data. Experiments show our approach generates temporally consistent and morphologically accurate robot demonstrations, offering a scalable solution to leverage internet-scale human video for robot learning.
Zhiyuan Li, Wenyan Yang, Wenshuai Zhao +4
Department of Electrical Engineering and Automation, Aalto University, Finland · Department of Computer Science, Aalto University, Finland · Hong Kong University of Science and Technology, China +1
Embodied foundation models have achieved significant breakthroughs in robotic manipulation, yet they still depend heavily on large-scale robot demonstrations. Although recent works have explored leveraging human data to alleviate this dependency, effectively extracting transferable knowledge remains a significant challenge due to the inherent embodiment gap between human and robot. We argue that the intention underlying human actions can serve as a powerful intermediate representation for bridging this gap. In this paper, we introduce a novel framework that explicitly learns and transfers human intention to facilitate robotic manipulation. Specifically, we model intention through gaze, as it naturally precedes physical actions and serves as an observable proxy for human intent. Our model is first pretrained on a large-scale egocentric human dataset to capture human intention and its synergy with action, followed by finetuning on a small set of robot and human data. During inference, the model adopts a Chain-of-Thought reasoning paradigm, sequentially predicting intention before executing the action. Extensive evaluations in simulation and real-world settings, across long-horizon and fine-grained tasks, and under few-shot and robustness benchmarks, show that our method consistently outperforms strong baselines, generalizes better, and achieves state-of-the-art performance. Project page: https://gazevla.github.io .
Chengyang Li, Kaiyi Xiong, Yuan Xu +3
Shanghai Jiao Tong University · Peking University · ShanghaiTech University +1