Nexus-VLA: Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space

Biaoliang Guan; Tingyu Yuan; Wen Ye; Ziming Ji; Xu Yang; Yan Huang; Jihua Zhu; Chaoyang Zhao

cs.ROOct 30, 2026

Nexus-VLA: Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space

Authors: Biaoliang Guan, Tingyu Yuan, Wen Ye, Ziming Ji, Xu Yang, Yan Huang, Jihua Zhu, Chaoyang Zhao

Organizations: XJTU · CASIA · UCAS · BUPT

Abstract

Large-scale egocentric human videos provide rich and scalable manipulation experience for robot policy learning, yet substantial embodiment gaps in visual appearance, kinematic structure, and action spaces hinder effective knowledge transfer. Existing methods often decouple cross-embodiment knowledge alignment from robot policy learning, making it difficult for human manipulation knowledge to directly and fully contribute to robot action learning. To address this issue, we propose Nexus-VLA, a hybrid policy learning framework that enables cross-embodiment knowledge transfer through end-to-end human-robot joint training, allowing human manipulation experience to directly contribute to robot policy learning. We first introduce a hierarchical cross-embodiment semantic alignment paradigm that progressively establishes shared manipulation semantics from task intent to fine-grained interactions, enabling more complete use of the manipulation knowledge contained in human videos. Building on this hierarchy, we design a shared manipulation latent representation that jointly encodes multi-level shared knowledge into an internal representation directly used for continuous action generation, improving policy learning efficiency while effectively leveraging human knowledge. Across RoboCasa-GR1, OOD settings, and real-world robot experiments, Nexus-VLA achieves an 80.5% task success rate on RoboCasa-GR1 and demonstrates strong advantages in human-data utilization, learning efficiency, and cross-scenario generalization.

Figures & tables

Figure 1: Nexus-VLA bridges human and robot experiences through a Hierarchical Semantic Latent, enabling cross-embodiment knowledge transfer and real-world generalization.

Figure 2: Overview of the Nexus-VLA framework. Hierarchical semantic latents enable joint human–robot learning and action generation.

Type	Method	Close	Cuttingboard	Placemat	Tray	Plate	Avg.
Foundation VLA	GR00T-N1.6	24.2	56.9	51.9	55.1	57.6	47.6
	ABot-M0	61.7	61.6	51.0	56.8	58.0	58.3
Semantic Enhanced	VP-VLA	54.3	60.8	54.5	46.0	53.5	53.8
	IntentVLA	59.7	54.8	55.5	49.6	66.5	57.0
	ZR-0	41.7	83.6	74.8	72.0	84.2	69.3
Human Experience	DyPES-VLA	55.3	63.2	51.5	62.4	64.0	59.3

Table 1: Comparison on the RoboCasa-GR1 benchmark. We report task success rates (%), averaged within each of the five task groups. Avg. denotes the unweighted mean across all 24 tasks. Nexus-VLA is evaluated using the checkpoint at 150K training steps.

Method	Illumination	Visual Noise	Cross-Config.	Obj.-Receptacle	Ego-Object	Fully Unseen	Avg.
Baseline	62.0	58.4	51.2	45.0	40.8	31.6	48.2
Nexus-VLA	76.8	73.6	71.0	66.8	69.4	55.6	68.9

Table 2: OOD generalization on RoboCasa-GR1 under illumination changes, visual noise, novel spatial configurations and object-receptacle compositions, objects seen only in EgoDex, and fully unseen objects, with 500 rollouts per setting.

Method	Human-Robot	Hierarchical	Semantic	Success Rate (%) $\uparrow$
	Co-training	Alignment	Representation
Baseline	–	–	–	72.42%
+ Hierarchical Alignment	–	✓	–	75.67%
+ Human–Robot Co-training	✓	–	–	71.75%
+ Independent Latents	–	✓	Independent	76.92%
+ Unified Latent	–	✓	Unified	78.25%

Table 3: Robot-only variants are evaluated at 120K training steps, whereas variants using human demonstrations are evaluated at 180K steps with a human-to-robot sampling ratio of 1:2, matching the expected robot-data exposure across variants.

Figure 3: Training efficiency of different semantic modeling strategies.

Figure 4: Bidirectional cross-embodiment manipulation-stage retrieval using the Hierarchical Semantic Latent. Rows represent query stages and columns represent top-1 retrieved stages, with diagonal entries indicating correct matches.

Figure 5: Human data scaling on RoboCasa-GR1 using nested subsets of filtered EgoDex demonstrations. The zero-data point corresponds to the robot-only reference; co-training uses a fixed human-to-robot sampling ratio of 1:2.

Figure 6: Cross-embodiment transfer from human experience to robot execution.

Method	Standard	Object Distractor	Spatial Layout	Motion Constraint	Unseen Object	Avg.
$\pi_{0.5}$	64.0	48.0	50.0	44.0	40.0	49.2
GR00T-N1.6	58.0	42.0	44.0	38.0	34.0	43.2
Baseline	70.0	50.0	52.0	46.0	42.0	52.0
+ Human-Robot Co-training	74.0	56.0	58.0	52.0	48.0	57.6
+ Hierarchical Alignment + Unified Latent	78.0	68.0	66.0	62.0	58.0	66.4
Nexus-VLA	82.0	76.0	74.0	70.0	68.0	74.0

Table 4: Real-world manipulation and OOD generalization. We evaluate six real-world manipulation tasks under the standard setting and four controlled distribution shifts. Each setting is evaluated over 50 robot rollouts. All numbers are success rates (%).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Overview of the four Structured Manipulation Experts for constructing Intent, Spatial, Motion, and Interaction supervision. All SMEs are frozen and used only for offline supervision generation.

Projector	Architecture	Semantic Target
Subtask Projector	Linear ( $2048{\rightarrow}768$ ) + 1-layer causal Transformer	Local subtask
Grounding Projector	MLP ( $2048{\rightarrow}512{\rightarrow}8$ ) + visibility head	Manipuland-target grounding
Motion Projector	MLP ( $2048{\rightarrow}1024{\rightarrow}64$ ) + visibility head	Future bimanual wrist trajectory
Contact Projector	MLP ( $2048{\rightarrow}512{\rightarrow}8$ ) + existence head	Hand-object contact geometry

Appendix

Table 5: Architecture of the hierarchical semantic projectors.

Configuration	Value	Configuration	Value
Action horizon	16	Training steps	200K
Per-GPU batch size	32	Gradient accumulation	1
Global batch size	512	Learning rate	$3\times 10^{-5}$
Optimizer	AdamW	LR schedule	Cosine decay
Warmup ratio	0.05	Weight decay	$1\times 10^{-5}$
Human : Robot sampling	$1:2$	Semantic branch weights $\lambda_{k}$	$1.0$

Appendix

Table 6: Training and optimization configuration of Nexus-VLA.

Configuration	VLM Backbone	Action Head
Backbone	Qwen2.5-VL-3B-Instruct	Flow-Matching DiT
Transformer layers	36 (LLM) / 32 (ViT)	32
Residual width	2048 / 1280	1536
Attention heads $\times$ dim.	$16\times 128$ / $16\times 80$	$32\times 48$
Positional encoding	M-RoPE	Learned 1D + sinusoidal time
FFN intermediate width	11008 / 3420	6144

Appendix

Table 7: Architecture configuration of Nexus-VLA.

Task	ZR-0	JoyAI-RA 0.1	ACE-Ego-0	Nexus-VLA
CupToDrawerClose	20.0	48.0	36.0	48.0
PotatoToMicrowaveClose	59.0	70.0	58.0	80.0
MilkToMicrowaveClose	45.0	84.0	56.0	74.0
BottleToCabinetClose	39.0	84.0	78.0	88.0
WineToCabinetClose	40.0	54.0	56.0	56.0
CanToDrawerClose	47.0	90.0	70.0	76.0

Appendix

Table 8: Per-task success rates (%) on all 24 RoboCasa-GR1 tasks. Tasks follow the Close, Cuttingboard, Placemat, Tray, and Plate group ordering in Table 1 . Best and second-best values are boldfaced and underlined, respectively; ties receive the same formatting.

Figure 8: Qualitative visualization of eight RoboCasa-GR1 manipulation tasks. Each row shows six time-ordered frames, with colored overlays highlighting task-relevant objects, targets, and motion or interaction cues.

Figure 9: Real-robot executions under the standard setting. Rows T1–T6 correspond to the six task instructions printed below the image sequences.

Retrieval	Queries	Candidates	R@1	P@5	mAP
Human $\rightarrow$ Human	400	380	88.25	83.30	51.92
Robot $\rightarrow$ Robot	160	140	90.00	84.75	79.06
Human $\rightarrow$ Robot	400	160	47.00	45.15	49.07
Robot $\rightarrow$ Human	160	400	75.63	70.50	49.19

Appendix

Table 9: Intra- and cross-embodiment manipulation-stage retrieval results. All metrics are reported in percentages (%). Intra-embodiment retrieval excludes samples from the query demonstration; cross-embodiment retrieval uses domain-wise mean-centered representations.

Figure 10: Representative real-robot scenes under four OOD conditions. Rows indicate the distribution shift, and columns T1–T6 correspond to the standard tasks in Figure 9 . One frame is shown for each condition-task combination.

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 391 papers

EXE: Egocentric Cross-Embodiment Pretraining via Factorized Latent Actions for VLA Models
Oct 24, 2026Qingyuan Cai, Bo Zhang, Yongzhen Huang +1Cross-Embodiment Robot Learning Robot Learning from Video
ROBOTERA. · Beijing Normal University. · Tsinghua University.
Cross-Embodiment Transfer via Behavior-Aligned Representations
Jul 30, 2026Ajay Sridhar, Jensen Gao, Jonathan Yang +3Representation Learning Vision-Language-Action Models
1Stanford University · 2Toyota Research Institute
HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model
May 29, 2026Xiang Zhu, Puzhen Yuan, Yichen Liu +1Cross-Embodiment Robot Learning Robotic Manipulation
Institute for Interdisciplinary Information Sciences, Tsinghua University, China · Shanghai Qi Zhi Institute, China

Oct 24, 2026J/K move · Enter open · S save

EXE: Egocentric Cross-Embodiment Pretraining via Factorized Latent Actions for VLA Models

Qingyuan Cai, Bo Zhang, Yongzhen Huang, Jianyu Chen

ROBOTERA. · Beijing Normal University. · Tsinghua University.

Cross-Embodiment Robot Learning Robot Learning from Video VLMs for Robotics Latent Action Learning Motion Representation Learning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models are bottlenecked by the scarcity of robot data. Egocentric human manipulation videos offer an abundant, diverse, and semantically-rich alternative that inherently captures manipulation intent. However, the embodiment gap makes it difficult to extract action-relevant representations from these videos. From the perspective of action execution, human hands and robots share analogous manipulation primitives---reaching, grasping, placing---yet exhibit distinct kinematics tied to their embodiments. This might motivate our key approach: egocentric video pretraining can yield shared action representations capturing transferable manipulation skills, while human-specific motion patterns should be disentangled to avoid interfering with robot policy learning. To this end, we propose EXE, an Egocentric Cross(X)-Embodiment pretraining framework that learns a factorized latent action representation from egocentric human videos, separating shared manipulation priors from embodiment-specific signals. By leveraging the shared action space, the robot policy acquires generalizable manipulation semantics from diverse human demonstrations without being biased by human-specific kinematics. This shared knowledge transfers across embodiments with different kinematic structures, enabling the same pretraining to benefit both parallel grippers and dexterous hands. EXE yields consistent gains over baseline without pretraining on RoboTwin 2.0, RoboCasa GR1, and real-world manipulation tasks.

Abstract

Robot Policy Learning

Explore similar work

EXE: Egocentric Cross-Embodiment Pretraining via Factorized Latent Actions for VLA Models

Cross-Embodiment Transfer via Behavior-Aligned Representations

HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model