Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs

Junhyeong Park; Wook Jung; Taeyoung Kim; Huiwon Jang; Jinwoo Shin

cs.AINov 3, 2026

Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs

Authors: Junhyeong Park, Wook Jung, Taeyoung Kim, Huiwon Jang, Jinwoo Shin

Organizations: KAIST · RLWRLD

Abstract

Pretrained Vision-Language-Action (VLA) policies provide a strong starting point for deployment, but adapting them to a target robot and environment still requires costly embodiment-specific demonstrations. Human demonstrations are far easier to collect, yet directly leveraging them is challenging due to the substantial human-robot embodiment gap, particularly when the underlying VLA has been pretrained primarily on robot data. We introduce H2RB (Human-To-Robot Bridge), a human-robot co-training framework for adapting pretrained VLA policies from unpaired human and robot demonstrations. H2RB bridges the embodiment gap at both interfaces: geometry-anchored alignment grounds human and robot observations in a compatible representation, while a decoupled-decoder tokenizer shares action latents without forcing both domains through a single action mapping. This design allows human demonstrations to provide useful behavioral supervision while retaining the rich visual, semantic, and control priors of the pretrained VLA. Across real-world tasks, H2RB substantially improves object transfer performance from 8.0 to 51.8 and enables skill transfer to score 62.7, in settings where co-train baselines totally fail to transfer human-demonstrated skills. The benefit also persists under object-placement shifts, improving out-of-distribution transfer from 6.0 to 45.3. H2RB suggests a way for human demonstrations to serve as a scalable supervision source for adapting pretrained robotic policies, reducing the dependence on costly target-robot data collection.

Figures & tables

Figure 1: Human-to-robot task transfer with H2RB. H2RB transfers task experience from human videos to robots using pretrained VLAs by bridging the cross-embodiment perception and action gaps, without target-robot demonstrations or paired demonstrations.

Panel of Figure 2 — Figure 2: Overview of H2RB. (a) A shared temporal encoder maps human and robot action chunks into a common latent action space, while embodiment-specific decoders reconstruct actions for each domain. (b) Hand geometry-anchored alignment grounds VLA conditioning across embodiments, while the frozen encoder provides latent action targets.

Figure 3: Qualitative human-to-robot transfer rollouts. (a) H2RB is trained jointly on human and robot demonstrations. (b) The trained policy executes target tasks introduced through human demonstrations, including object transfer on box, followed by skill transfer on opening.

	Robot in-domain	Human transfer
Method	Box PnP	Open	Close	Avg.
Base policy human-only	0.0	36.0	22.0	29.0
Base policy in-domain-only	80.5	0.0	0.0	0.0
Base policy co-train	16.5	0.0	0.0	0.0
+ HaWoR-shared action ( Zhang et al., 2025a )	19.5	0.0	0.0	0.0
+ Ego-Pi-shared action ( Kim et al., 2026b )	70.5	0.0	0.0	0.0

Table 2: Human-to-robot skill transfer. Robot in-domain reports performance on a robot-data-trained task, while Human transfer evaluates target tasks introduced through human demonstrations. Each entry is the weighted task score (0-100).

Table 3: Analysis of the decoupled-decoder latent tokenizer. (a) Effect of the action tokenizer and cross-domain latent regression on object transfer, skill transfer, and normalized human-robot latent gap. (b) Motion speed and average success of embodiment-specific reconstruction decoders.

Panel of Figure 8 — Figure 8: Failure case analysis of human-to-robot transfer. (a) Fails to complete the grasp. (b) Unrelated to the target object location. (c) Fails to produce a hand pose suitable for grasping. (d) Executes a robot-observed pick-and-place motion in human-observed task environment.

Panel of Figure 9 — Figure 9: PCA visualization of cross-embodiment representation adaptation. (a) Vision-language conditioning features with hand geometry-anchored alignment alone (left) and H2RB (right). Arrows show adapter-induced displacements for the same observation. (b) Action-tokenizer latents without (left) and with (right) cross-domain latent regression, using per-tokenizer PCA. Human I and II denote human tasks trained in within-skill object transfer and skill transfer, respectively.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Configuration
Backbone	RLDX-1-IMG ( Kim et al., 2026a )
Policy inputs	One current egocentric RGB frame, proprioceptive state, and instruction
Conditioning representation	64 cognition tokens
Physical action / current geometry	15 / 24 coordinates per hand
Data rate / action chunk	20 Hz / $K=16$ timesteps
Latent action target	$K\times D_{z}$ posterior-mean coordinates ( $K=16$ , $D_{z}=64$ )

Appendix

Table 4: Architecture and representations.

Setting	Tokenizer	OpenArm FT	HT
Updates	100k	30k	200k
Global batch size	512	64	128
Optimizer	AdamW	AdamW	AdamW
Learning rate	$5\times 10^{-5}$	$10^{-4}$	$10^{-4}$
Geometry-loss weight	Not applicable	0.2	0.2

Appendix

Table 5: Optimization settings for tokenizer training, task-specific OpenArm fine-tuning (FT), and heterogeneous training (HT).

Module	OpenArm FT	HT
Vision encoder	Frozen	Frozen
Language model	Frozen	Top four layers trainable
Cognition-token parameters	Frozen	Trainable
Geometry adapter / readout	Trainable / Trainable	Trainable / Trainable
Tokenizer encoder / decoders	Frozen	Frozen where used

Appendix

Table 6: Parameter updates during policy learning. Tokenizer training is a separate stage; its encoder and decoders are frozen during OpenArm FT.

Item	Object transfer	Skill transfer
Human target tasks	Box / Ball / Towel PnP	Air-fryer Open / Close
Robot in-domain evaluation	Doll PnP	Box PnP
Exact robot training task list	Bottle / Cup / Doll / Snack PnP	Box / Ball / Towel PnP
Retained robot episodes	208 (64 / 48 / 32 / 64)	300 (100 / 100 / 100)
Retained human episodes per task	Box: 100; Ball: 100; Towel: 100	Open: 50; Close: 50

Appendix

Table 7: Task and training-data conditions. PnP denotes pick-and-place. Policy FT uses all retained demonstrations without a held-out validation split; evaluation uses robot rollouts.

Condition	Adaptation sources	Action / conditioning interface
Base policy human-only	Human targets	Human demonstrations only, using HaWoR-extracted fingertip and wrist motions as the action representation.
Base policy in-domain-only	Source robot	Robot demonstrations only, using the native LeRobot robot action representation.
Base policy co-train	Source robot + human targets	Human and robot data retain their respective action interfaces and use separate embodiment IDs; the target robot ID is used at inference.
HaWoR-shared action ( Zhang et al., 2025a )	Source robot + human targets	Shared fingertip and wrist actions. Human actions come from HaWoR, while robot actions are mapped through forward kinematics; both use the same action decoder.
Ego-Pi-shared action ( Kim et al., 2026b )	Source robot + human targets	Shared physical wrist–hand actions without geometry grounding or a latent tokenizer.
Ego-Pi-shared action ( Kim et al., 2026b ) + hand geometry-anchored alignment	Source robot + human targets	Shared wrist–hand actions with our hand geometry-anchored alignment.

Appendix

Table 8: Baseline and variant definitions for the main comparison.

$w_{i}$	Pick-and-place	Open	Close
0.0	Failure	Failure	Failure
0.3	Reach to object	Reach to handle	Reach to handle
0.6	Valid grasping of object	Total grasp of handle	Total grasp of handle
0.8	The object lifted up	Opened halfway	Closed halfway
1.0	Complete task	Complete task	Complete task

Appendix

Table 9: Task-progress weights and milestones.

Evaluation group	ID	OOD	Total
I: Object transfer
Object target: Box PnP	20	10	30
Object target: Ball PnP	20	10	30
Object target: Towel PnP	20	10	30
Object target subtotal	60	30	90
Robot in-domain: Doll PnP	Separate source-task evaluation		20

Appendix

Table 10: Evaluation counts per policy in the main transfer comparison. The robot in-domain trials are separate from the target-task averages.

Panel of Figure 10 — Figure 10: OpenArm robot setup and evaluation distributions. (a) OpenArm robot platform used for real-world evaluation. (b) For object transfer, in-distribution (ID) rollouts are sampled from four workspace quadrants (right-upper, right-lower, left-upper, and left-lower), with five rollouts per quadrant and randomized object orientation. Out-of-distribution (OOD) rollouts are sampled outside the ID placement region (red area), again with randomized orientation. For skill transfer, the initial openness of the air fryer is randomized for all 30 trials.

Policy	$n$	$R_{\geq 0.3}$	$R_{\geq 0.6}$	$R_{\geq 0.8}$	$R_{=1.0}$	$S_{\mathrm{w}}$
I: Object transfer
Ego-Pi-shared action	90	68.9	12.2	12.2	2.2	27.2
H2RB	90	95.6	46.7	33.3	12.2	51.8
H2RB + HT	90	96.7	60.0	30.0	21.1	57.2
II: Skill transfer
Ego-Pi-shared action	30	0.0	0.0	0.0	0.0	0.0

Appendix

Table 11: Cumulative target-task progress (%) and weighted task scores (0-100).

R_{\geq 0.3}

R_{\geq 0.6}

R_{\geq 0.8}

, and

R_{=1.0}

denote increasing completion thresholds; the last is full-task success.

	Robot in-domain	Human transfer
Method	Doll PnP	ID	OOD	Avg.
Base policy co-train + Ego-Pi-shared action ( Kim et al., 2026b )	30.0	30.2	21.3	27.2
+ Ego-Pi-shared action ( Kim et al., 2026b ) + Hand geometry-anchored alignment	47.5	36.0	32.0	34.7
+ H2RB (Ours)	49.0	55.0	45.3	51.8

Appendix

Table 12: Effect of geometry grounding on object transfer. Robot in-domain evaluates the familiar Doll PnP task, while human transfer reports weighted task scores under in-distribution (ID) and out-of-distribution (OOD) object placements. All values are weighted task scores (0-100).

Reconstruction decoder	Box	Ball	Towel	Avg.
Robot decoder	37.0	16.0	45.0	32.7
Human decoder	53.0	49.0	53.3	51.8

Appendix

Table 13: Inference-decoder selection for the within-skill object transfer latent policy. Values are weighted task scores (0-100).

Figure 11: Qualitative transferred-task rollouts. Representative successful H2RB executions on box, ball, towel, open, and close, spanning both object-level and skill-level transfer.

Data source	Episodes	Frames
EgoDex ( Hoque et al., 2026 )	10,000	1,714,342
In-house human data	2,410	1,754,308
Single-hand OpenArm	1,035	236,751
Bimanual OpenArm	623	249,488
In-house ALLEX robot data	480	1,059,641
Total	14,548	5,014,530

Appendix

Table 14: Heterogeneous training data by collection source.

	Robot in-domain	Human transfer
Method	Doll	Box	Ball	Towel	Avg.
H2RB	49.0	53.0	49.0	53.3	51.8
HT + H2RB	69.5	56.7	53.3	61.7	57.2

Appendix

Table 15: Effect of heterogeneous training on object transfer. Robot in-domain reports performance on a robot-data-trained task, while Human transfer evaluates target tasks introduced through human demonstrations. Each entry is the weighted task score (0-100).

	Robot in-domain	Human transfer
Method	Box PnP	Open	Close	Avg.
H2RB	81.0	50.7	74.7	62.7
HT + H2RB	80.5	64.7	98.7	81.7

Appendix

Table 16: Effect of heterogeneous training on skill transfer. Robot in-domain reports performance on a robot-data-trained task, while Human transfer evaluates target tasks introduced through human demonstrations. Each entry is the weighted task score (0-100).

Panel of Figure 12 — Figure 12: RB-Y1 experimental setup. (a) Bimanual robot platform. (b) Boxes and polybags used for parcel handling.

VLA Model Adaptation

See all 48 papers

ReWeight: Leveraging Human Data for VLA Post-Training via Demonstration Retrieval and Sample Weighting
Sep 12, 2026Chenwei Wang, Dianye Huang, Match W. L. Ko +2Robot Learning Vision-Language-Action Models
Department of mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI), China Telecom, Shanghai 200232, China · Shenzhen Research Institute of Northwestern Polytechnical University, Xi’an 710072, China
HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model
May 29, 2026Xiang Zhu, Puzhen Yuan, Yichen Liu +1Cross-Embodiment Robot Learning Robotic Manipulation
Institute for Interdisciplinary Information Sciences, Tsinghua University, China · Shanghai Qi Zhi Institute, China
Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
Jun 7, 2026Linqi Yin, Shiduo Zhang, Shenling Qiu +11VLM Adaptation Vision-Language-Action Models
1Fudan University · 2Shanghai Innovation Institute

Sep 12, 2026J/K move · Enter open · S save

ReWeight: Leveraging Human Data for VLA Post-Training via Demonstration Retrieval and Sample Weighting

Chenwei Wang, Dianye Huang, Match W. L. Ko, Chenjia Bai+1

Department of mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI), China Telecom, Shanghai 200232, China · Shenzhen Research Institute of Northwestern Polytechnical University, Xi’an 710072, China

Robot Learning Vision-Language-Action Models Cross-Embodiment Robot Learning Data-Efficient Robot Learning Learning from Demonstration

Open paper AskPDF arXiv

Post-training vision-language-action (VLA) models for specific robots and tasks requires in-domain demonstrations, yet collecting diverse robot data is costly. Egocentric human demonstrations provide a scalable alternative, but directly mixing human and robot data can introduce cross-embodiment discrepancies and degrade policy performance. To address this challenge, we introduce ReWeight, a framework that incorporates human data into VLA post-training through demonstration-level retrieval and sample-level weighting. ReWeight learns a cross-embodiment visuomotor representation that combines visual observations with future actions to measure behavioral similarity between human and robot demonstrations. Based on optimal transport, it retrieves human demonstrations relevant to the target robot data and assigns larger weights to samples with smaller cross-embodiment discrepancies. We evaluate ReWeight using

π_{0.5}

across eight simulation tasks and four real-world tasks under both clean and randomized settings. In simulation, ReWeight improves the average success rate of post-trained

π_{0.5}

from 39% with only robot data and 44% with randomly mixed human-robot data to 57%. In the physical experimental setting, it achieves an average success rate of 68.8%, outperforming the baselines by 28.8% and 13.8%, respectively. Overall, ReWeight provides an effective paradigm for transforming abundant egocentric human experience into transferable supervision for robot learning. (Project webpage: https://reweight-vla.github.io/)

Abstract

VLA Model Adaptation

Explore similar work

ReWeight: Leveraging Human Data for VLA Post-Training via Demonstration Retrieval and Sample Weighting

HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model

Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data