Cross-Embodiment Causal Transfer from Human Videos to Robot Manipulation

Xingyu Zhang; Yingchao Li; Peizheng Guo; Jie Hu; Changwen Zheng; Wenwen Qiang

cs.ROOct 15, 2026

Cross-Embodiment Causal Transfer from Human Videos to Robot Manipulation

Authors: Xingyu Zhang, Yingchao Li, Peizheng Guo, Jie Hu, Changwen Zheng, Wenwen Qiang

Organizations: University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · Meituan

Abstract

Scaling robot manipulation models requires large amounts of interaction data, yet collecting diverse real-robot experience remains costly. Human manipulation videos, in contrast, provide abundant and diverse task experience that could potentially benefit robot learning. However, differences in morphology, action spaces, and control mechanisms make human motions unsuitable as direct supervision for robot actions, raising a scientific question: what knowledge in human videos should be transferred across embodiments to robot manipulation? To address this question, we analyze subtask transfer through the lens of causal beams, focusing on which contextual conditions and interaction dependencies should be retained to support local task effects. Based on this perspective, we propose Causal-Beam-Guided Subtask Transfer (CBST). CBST learns manipulation-related representations from human videos and grounds them in task-level semantic variables using robot demonstrations. Within a predefined manipulation model, causal-beam constraints guide contextual information selection by requiring retained assignments to sustain local mechanism outputs while preserving nontrivial responsiveness to retained inputs. The selected semantic information conditions a world-action model, allowing the robot to generate actions from its own observations without requiring human-robot motion correspondence. Experiments on robot manipulation benchmarks show improvements in task success over the corresponding world-action model baselines.

Figures & tables

Figure 1: The cross-embodiment transfer problem. Human and robot demonstrations share task objectives but differ in execution. Examples of placing a bowl, opening a lid, and lifting a book motivate the question: what task information should transfer beyond embodiment-specific motion and goal endpoints?

Figure 2: (a) Candidate local manipulation SCM for a fixed task goal, showing the endogenous parent structure. $X$ and $I$ are boundary inputs, and the mechanisms for $D$ and $Y$ may differ across embodiments. Variable groups are instantiated into task-specific variables before beam selection. (b) Illustrative human and robot motion realizations of the same bowl-placement task.

Figure 3: Overview of Causal-Beam-Guided Subtask Transfer (CBST). Human-video pretraining and robot adaptation support state prediction and contextual beam selection. The resulting semantic conditions guide a frozen world–action model to predict actions and future states. Solid and dashed arrows denote inference and training/supervision, respectively.

Table 4

Table 5

Figure 4: A LingBot-VA + CBST execution: operating the switch, turning on the stove, and placing the moka pot on it. Frames follow temporal order; timestamps indicate video playback time.

Figure 5: CBST execution of cup placement on the AgileX platform. One arm lifts the cup and transfers it to the receiving gripper, which transports it to the target location and releases it on the tabletop. Handover enables cross-workspace transport but is not the task goal. Labels describe observed events, and timestamps indicate video playback time.

Figure 6: Temporal-order comparison for ResNet18–GRU pretraining on SSv2. (a) Training classification loss: bold lines show sample-weighted means over non-overlapping blocks of 120 updates; faint traces show the original values. (b) Final top-1 accuracy on 24,777 validation clips after three epochs, with one run per condition.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Role in CBST	Scope / WAM
SSv2	Human-video representation pretraining	174 action categories; planner initialization
RoboTwin 2.0	Robot adaptation and evaluation	50 tasks; LingBot-VA
LIBERO-Long	Robot adaptation and evaluation	10 tasks; LingBot-VA and Cosmos Policy (Predict2.5)
RoboCasa	Robot adaptation and evaluation	24 tasks; Cosmos Policy (Predict2.5)

Appendix

Table 5: Datasets and evaluation scope. The RoboCasa entry refers to the 24-task protocol used by Cosmos Policy, rather than the complete RoboCasa task collection.

Split	Videos	Annotation availability
Training	168,913	Action labels
Validation	24,777	Action labels
Test	27,157	Labels withheld
Total	220,847	174 action categories

Appendix

Table 6: Official SSv2 split sizes. These are release-level counts; the number of clips retained after preprocessing is reported separately in Table 14 .

Interaction type	Representative distinctions
Object displacement	Directional movement; motion toward or away from the camera.
Placement and removal	Moving objects onto surfaces, into receptacles, or away from their previous locations.
Lifting and release	Partial versus complete lifting; holding versus releasing.
Contact and motion	Contact that moves an object versus contact with little observable displacement.
Support and stability	Tilting a support with or without causing the supported object to fall.
Attachment	Successful attachment versus an unsuccessful attempt.

Appendix

Table 7: Representative action distinctions in SSv2. Descriptions are abbreviated semantic summaries rather than verbatim labels or an exhaustive class list.

Horizon	Tasks	Task names
1	30	Adjust Bottle; Beat Block Hammer; Click Alarmclock; Click Bell; Dump Bin Bigbin; Grab Roller; Lift Pot; Move Can Pot; Move Pillbottle Pad; Move Playingcard Away; Move Stapler Pad; Open Laptop; Open Microwave; Place A2B Left; Place A2B Right; Place Bread Basket; Place Container Plate; Place Empty Cup; Place Fan; Place Mouse Pad; Place Object Scale; Place Object Stand; Place Phone Stand; Place Shoe; Press Stapler; Rotate QRcode; Shake Bottle Horizontally; Shake Bottle; Stamp Seal; Turn Switch.
2	15	Handover Block; Handover Mic; Hanging Mug; Pick Diverse Bottles; Pick Dual Bottles; Place Bread Skillet; Place Burger Fries; Place Can Basket; Place Cans Plasticbox; Place Dual Shoes; Place Object Basket; Put Object Cabinet; Scan Object; Stack Blocks Two; Stack Bowls Two.
3	5	Blocks Ranking RGB; Blocks Ranking Size; Put Bottles Dustbin; Stack Blocks Three; Stack Bowls Three.

Appendix

Table 8: The 50 RoboTwin tasks grouped by the horizon labels used in the LingBot-VA report. Names are abbreviated from the official task identifiers.

Suite	Tasks	Main variation
Spatial	10	Spatial relations identifying a target object.
Object	10	Object identity under a shared placement goal.
Goal	10	Desired outcomes within related environments.
LIBERO-90	90	Diverse manipulation tasks.
Long / LIBERO-10	10	Tasks requiring extended execution.

Appendix

Table 9: LIBERO suite composition. LIBERO-90 and LIBERO-Long are the two partitions of LIBERO-100, not additional copies of that collection.

Suite	Task variants
Spatial	Move the black bowl to the plate. The bowl starts: between plate and ramekin; beside ramekin; at table center; on cookie box; inside top drawer; on ramekin; beside cookie box; on stove; beside plate; on cabinet.
Object	Place each target in the basket: alphabet soup; cream cheese; salad dressing; barbecue sauce; ketchup; tomato sauce; butter; milk; chocolate pudding; orange juice.
Goal	Open middle drawer; bowl to stove; wine bottle to cabinet top; bowl inside top drawer; bowl to cabinet top; plate toward stove front; cream cheese into bowl; activate stove; bowl to plate; wine bottle to rack.

Appendix

Table 10: Task composition of the Spatial, Object, and Goal suites. Shared instructions are factored out for compactness.

ID	Scene	Goal summary
L1	LR2	Basket: alphabet soup and tomato sauce.
L2	LR2	Basket: cream cheese and butter.
L3	K3	Activate stove; place moka pot on it.
L4	K4	Black bowl inside bottom drawer; drawer closed.
L5	LR5	White mug on left plate; yellow-and-white mug on right plate.
L6	S1	Book in caddy’s rear compartment.

Appendix

Table 11: The ten LIBERO-Long tasks. LR, K, and S denote living-room, kitchen, and study scenes, respectively.

Family	Count	Tasks
Pick-and-place	8	PnPCounterToCab; PnPCabToCounter; PnPCounterToSink; PnPSinkToCounter; PnPCounterToMicrowave; PnPMicrowaveToCounter; PnPCounterToStove; PnPStoveToCounter.
Cabinet doors	4	OpenSingleDoor; CloseSingleDoor; OpenDoubleDoor; CloseDoubleDoor.
Drawers	2	OpenDrawer; CloseDrawer.
Sink	3	TurnOnSinkFaucet; TurnOffSinkFaucet; TurnSinkSpout.
Stove	2	TurnOnStove; TurnOffStove.
Coffee	3	CoffeeSetupMug; CoffeeServeMug; CoffeePressButton.

Appendix

Table 12: The 24 RoboCasa tasks used in the Cosmos Policy evaluation protocol. Task identifiers follow its environment release. PnP denotes pick-and-place.

Method	Model / interface	Role	Benchmark
LingBot-VA	Video–action generation	Backbone	R, L
Cosmos Policy–Predict2.5	Joint action and future prediction	Backbone	L, C
Cosmos Policy–Predict2	Earlier video-model configuration	Reference	L, C
X-VLA	Soft-prompted VLA	Reference	R, L
$\pi_{0}$	Flow-based VLA	Reference	R, L, C
$\pi_{0.5}$	Generalist VLA	Reference	R, L

Appendix

Table 13: Roles of the principal comparison methods. R, L, and C denote RoboTwin, LIBERO, and RoboCasa. Benchmark entries indicate the comparisons included in this paper, not the complete capabilities of each method.

Item	Configuration
Visual encoder	ResNet18
Temporal encoder	GRU
Camera-feature fusion	Mean pooling over available camera views
Human pretraining	174-class action classification on SSv2 using cross-entropy loss
Transferred components	Visual encoder, temporal module, and representation projection
Robot prediction heads	Semantic states, interval variables, and candidate parent subsets

Appendix

Table 14: Planner architecture and training configuration.

	RoboTwin 2.0	LIBERO-Long		RoboCasa
Item	LingBot-VA	LingBot-VA	Predict2.5	Predict2.5
Tasks	50	10	10	24
Episodes per task	100	150	150	150
Total episodes	5,000	1,500	1,500	3,600

Appendix

Table 15: Evaluation scope and episode budgets. RoboTwin counts apply separately to Easy and Hard. LIBERO-Long and RoboCasa counts aggregate evaluation seeds. Each WAM is evaluated independently.

Method	Horizon 1	Horizon 2	Horizon 3	All 50 tasks
X-VLA ∗	81.6 / 82.5	59.3 / 55.9	61.2 / 66.0	72.9 / 72.8
$\pi_{0}$	66.5 / 61.6	66.1 / 54.7	61.6 / 50.2	65.9 / 58.4
$\pi_{0.5}$	85.1 / 80.2	79.3 / 73.0	78.6 / 67.4	82.7 / 76.8
Motus	91.0 / 90.6	85.2 / 80.9	85.0 / 84.2	88.7 / 87.0
LingBot-VA	94.18 / 93.56	90.35 / 86.95	93.22 / 93.28	92.93 / 91.55
LingBot-VA + CBST	94.73 / 93.97	91.93 / 88.80	95.80 / 95.20	94.00 / 92.54

Appendix

Table 16: RoboTwin success rates (%) grouped by task horizon. Each cell reports Easy / Hard. Values follow Table 1 of [ Li et al., 2026 ] ; X-VLA ∗ is adopted there from Motus.

Method	Source	Spatial	Object	Goal	Long	Average
Octo	L	78.9	85.7	84.6	51.1	75.1
Seer	L	–	–	–	87.7	–
MoDE	L	–	–	–	94.0	–
SuSIE	L	–	–	–	76.3	–
SpatialVLA	L	88.2	89.9	78.6	55.5	78.1
TraceVLA	L	84.6	85.2	75.1	54.1	74.8

Appendix

Table 17: Complete published LIBERO comparison. All values are success rates (%). L: Table 2 of [ Li et al., 2026 ] ; C: Table 1 of [ Kim et al., 2026 ] ; K: official Cosmos Cookbook [ Kim and Gu, 2026 ] . Source-reported averages and uncertainty values are preserved.

Method	Demos per task	Average SR (%)
GR00T-N1	300	49.6
UVA	50	50.0
DP-VLA	3000	57.3
GR00T-N1 + DreamGen	300 (+10000 synthetic)	57.6
GR00T-N1 + DUST	300	58.5
UWM	1000	60.8

Appendix

Table 18: Complete RoboCasa comparison over 24 tasks. Baseline scores and demonstration counts follow [ Kim et al., 2026 ] . Predict2.5 results follow the official Cosmos Cookbook [ Kim and Gu, 2026 ] . Synthetic data are identified separately.

Figure 7: Additional recorded LingBot-VA executions:(a) stacking three bowls in RoboTwin, (b) handing over a block in RoboTwin, and (c) placing a black bowl in a drawer and closing it in LIBERO. Each row shows six temporally ordered frames;timestamps indicate video playback time.

Variant	Retrain	Change relative to CBST
Without human pretraining	Yes	Omit SSv2 pretraining while retaining the common visual initialization and robot adaptation procedure.
Shuffled human pretraining	Yes	Shuffle sampled frames within each human clip before pretraining; retain labels, frame content, and pretraining budget.
State-only	No	Render admissible goals and predicted semantic states without the selected interaction-dependency fields.
Full structure	No	Render all candidate parent fields instead of the contextually selected subset.
Random selection	No	Render randomly selected candidate parent fields, matching the learned subset cardinality.
Without mechanism consistency	Yes	Set $\lambda_{m}=0$ while preserving the remaining supervision and reference mechanisms.

Appendix

Table 19: Definitions of the principal ablation variants. “Retrain” refers to robot-side planner adaptation; the WAM remains frozen in every variant.

	LingBot-VA
Variant	RoboTwin Easy	RoboTwin Hard	LIBERO-Long
Backbone	92.93	91.55	$98.5\pm 0.5$
CBST	94.00	92.54	98.60
Without human pretraining	92.80	91.38	98.00
Shuffled human pretraining	92.94	91.54	98.47
State-only	92.82	91.36	98.13

Appendix

Table 20: Principal ablation results across benchmarks. All values are success rates (%). RoboTwin and LingBot-VA reference values follow [ Li et al., 2026 ] ; Predict2.5 reference values follow [ Kim and Gu, 2026 ] .

Temporal module	Parameters	LingBot-VA SR	Predict2.5 SR	Planner latency
MLP	12.0	98.53	97.13	9
TCN	12.3	98.53	97.20	11
GRU	12.5	98.60	97.33	12
Transformer	13.2	98.66	97.53	13

Appendix

Table 21: Temporal architecture comparisons on LIBERO-Long. Parameter counts refer to the planner; latency refers to a planner update with the same input specification and accelerator.

Factor	values	Reference	Fixed quantities
Sampled frame count	$\{4,8,16,32\}$	$16$	Lookback duration and temporal sampling strategy.
Lookback duration (s)	$\{1,2,4,8\}$	$4$	Frame count and recent/history allocation ratio.
$\lambda_{\mathrm{size}}$	$\{0,10^{-3},10^{-2},10^{-1}\}$	$10^{-2}$	Other loss weights, reference mechanisms, and training budget.

Appendix

Table 22: Sensitivity-analysis configurations. Default values correspond to the full CBST configuration.

ID	Task	Placement objective and success condition
T1	Remote-control placement	Move the remote from the tabletop into the designated open container; it must remain fully inside after gripper release.
T2	Cup placement	Move the cup to the designated tabletop region and leave it stable after release; handover may be used to connect the source and target workspaces.
T3	Plate racking	Place the plate into the designated rack slot, with the rack supporting the plate after both grippers disengage.
T4	Apple placement	Place an apple inside the designated fruit basket and release it.
T5	Banana placement	Place a banana within the designated tray and release it.
T6	Orange placement	Place an orange inside the designated bowl and release it.

Appendix

Table 23: Task inventory.

ID	Task	Successes	SR (%)	Successes	SR (%)	Gain
		Cosmos Policy (Predict2.5)		+ CBST
T1	Remote	$(17/20)$	$85$	$(18/20)$	$90$	$5$
T2	Cup	$(14/20)$	$70$	$(17/20)$	$85$	$15$
T3	Plate	$(13/20)$	$65$	$(15/20)$	$75$	$10$
T4	Apple	$(10/20)$	$50$	$(12/20)$	$60$	$10$
T5	Banana	$(15/20)$	$75$	$(16/20)$	$80$	$5$

Appendix

Table 24: Detailed real-world results. Each method has 20 trials per task. Gains are measured in percentage points.

Figure 8: Additional real-world executions on the AgileX platform: (b) plate racking and (c) remote-control placement. Frames are shown in temporal order, with timestamps indicating video playback time. Annotations describe observed manipulation events.

Cross-Embodiment Robot Learning

Cross-embodiment robot learning transfers skills and policies across robots with different bodies, sensors, dynamics, and action spaces.

See all 347 papers

H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models
Aug 13, 2026Dingyi Rong, Yue Shi, Chaofan Ma +6World Models for Robotics Cross-Embodiment Robot Learning
1Shanghai Jiao Tong University · 2Shanghai Artificial Intelligence Laboratory
Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing
May 5, 2026Zhiyuan Li, Wenyan Yang, Wenshuai Zhao +4Disentangled Representation Learning Cross-Embodiment Robot Learning
Department of Electrical Engineering and Automation, Aalto University, Finland · Department of Computer Science, Aalto University, Finland · Hong Kong University of Science and Technology, China +1
GazeVLA: Learning Human Intention for Robotic Manipulation
Apr 24, 2026Chengyang Li, Kaiyi Xiong, Yuan Xu +3Long-Horizon Robotic Manipulation Robot Skill Learning
Shanghai Jiao Tong University · Peking University · ShanghaiTech University +1

Aug 13, 2026J/K move · Enter open · S save

H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models

Dingyi Rong, Yue Shi, Chaofan Ma, Jiezhang Cao+5

1Shanghai Jiao Tong University · 2Shanghai Artificial Intelligence Laboratory

World Models for Robotics Cross-Embodiment Robot Learning Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Large-scale manipulation data is essential for robot learning, yet collecting robot demonstrations remains expensive and difficult to scale. Meanwhile, abundant egocentric human manipulation videos provide rich behavioral experiences, but transferring them across embodiments remains challenging due to differences between human hands and robotic end-effectors. Recent advances in video world models offer a promising pathway to synthesize robot-centric manipulation videos from human observations, while their cross-embodiment transfer capability remains largely unexplored. Therefore, we introduce H2R-Bench, a benchmark for evaluating cross-embodiment human-to-robot manipulation video generation, where models transform egocentric human demonstrations into robot manipulation videos under specified embodiments. Each benchmark instance contains a human demonstration video, target embodiment constraints, and source-grounded annotations covering task goals, action events, functional contacts, and object responses. H2R-Bench evaluates generated videos through five dimensions, including goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general video quality. We benchmark eleven state-of-the-art video generation models across six manipulation families and two robot embodiments. Our evaluation reveals that current video world models remain limited in human-to-robot manipulation transfer: even leading models often fail in embodiment consistency, functional interaction, and task execution. H2R-Bench provides a systematic diagnostic framework for evaluating whether video world models can bridge the human-to-robot embodiment gap and convert human manipulation observations into robot-centric training resources.

Abstract

Cross-Embodiment Robot Learning

Explore similar work

H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models

Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing

GazeVLA: Learning Human Intention for Robotic Manipulation