Cross-Hand Latent Representation for Vision-Language-Action Models

Guangqi Jiang; Yutong Liang; Jianglong Ye; Jia-Yang Huang; Changwei Jing; Rocky Duan; Pieter Abbeel; Xiaolong Wang

cs.ROMar 10, 2026

Cross-Hand Latent Representation for Vision-Language-Action Models

Authors: Guangqi Jiang, Yutong Liang, Jianglong Ye, Jia-Yang Huang, Changwei Jing, Rocky Duan, Pieter Abbeel, Xiaolong Wang, +1 more

Organizations: UC San Diego · Amazon FAR · UC Berkeley

Abstract

Dexterous manipulation is essential for real-world robot autonomy, mirroring the central role of human hand coordination in daily activity. Humans rely on rich multimodal perception--vision, sound, and language-guided intent--to perform dexterous actions, motivating vision-based, language-conditioned manipulation systems for robots. However, training reliable vision-language-action (VLA) models for dexterous manipulation requires large-scale demonstrations across many robotic hands. In addition, as new dexterous embodiments appear rapidly, collecting data for each becomes costly and impractical, creating a need for scalable cross-embodiment learning. We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands. This embodiment-invariant latent space is directly pluggable into standard VLA architectures, enabling seamless cross-embodiment training and efficient reuse of both existing and newly collected data. Experimental results demonstrate that XL-VLA consistently outperforms baseline VLA models operating in raw joint spaces, establishing it as an effective solution for scalable cross-embodiment dexterous manipulation.

Figures & tables

Paper	Data	Deployment		Input			Output
	Type(s)	Embodiment	EEF $\leftrightarrow$ EEF	Vision	Lang	Prop	Decoder	ZS
UniVLA [ 8 ]	human video; teleoperation	1 arm+1 gripper	gripper $\leftrightarrow$ gripper	✓	✓	✗	✓	✗
ATE [ 71 ]	teleoperation; simulation	2 arms+2 grippers	—	✓	✓	✗	✗	✗
LAD [ 3 ]	teleoperation	1 arm+1 hand/gripper	hand $\leftrightarrow$ hand/gripper	✓	✗	✗	✓	✗
EgoBridge [ 45 ]	human video; teleoperation	2 arms+2 pushers	—	✓	✗	✓	✗	✗
CoMo [ 65 ]	internet video; teleoperation	1 arm+1 gripper	—	✓	✗	✗	✗	✗

Table 1: Related Work Summary. Summary of related work comparing data sources, deployment settings, and input/output capabilities for latent-based cross-embodiment methods. Data indicates the training modalities used in each work. Deployment specifies the robot embodiments evaluated and whether cross–end-effector transfer is supported. Input denotes which modalities (vision, language, proprioception) are used for training. Output reports whether a method includes a cross-embodiment decoder and whether it enables zero-shot transfer to unseen embodiments.

Figure 2: Model Pipeline. XL-VLA builds on $\pi_{0}$ [ 6 ] with vision and language encoders paired with an action expert that operates in a shared latent action space for cross-embodiment control. During VLA training, the action expert is finetuned while the pretrained latent encoders and decoders remain frozen.

Method	Hand	PF	SC	SoC	HB	RL	PS	RB	PuS	PoS	PC	Mean
$\pi_{0}$ [ 6 ]	Ability	0.10	0.10	0.00	0.70	0.20	0.80	0.60	0.30	0.30	0.60	0.37
	Inspire	0.10	0.20	0.00	0.30	0.10	0.50	0.30	0.20	0.20	0.80	0.27
	Paxini	0.40	0.40	0.30	0.20	0.00	0.80	0.60	0.30	0.10	0.40	0.35
	XHand	0.20	0.40	0.00	0.40	0.10	0.60	0.30	0.20	0.30	0.40	0.29
	Mean	0.20	0.28	0.08	0.40	0.10	0.68	0.45	0.25	0.23	0.55	0.32
XL-VLA	Ability	0.80	0.80	0.40	1.00	0.70	1.00	0.70	0.30	0.90	0.70	0.73

Table 2: Vision-Language-Action Modeling. We compare XL-VLA with

\pi_{0}

under cross-embodiment training. Although

\pi_{0}

can handle different embodiments by adjusting sequence length, our method achieves consistently higher success rates across all hands and tasks. The first row PF, SC, SoC, etc. denote each task introduced in Task & Dataset, and each task is executed for 10 times, and we compute the success rate for each task cross hand.

Figure 3: Latent space pretraining pipeline. For each hand type, joint positions $\mathbf{q}_{h}$ are mapped through an encoder MLP into a shared latent space and reconstructed by a decoder MLP. The diagram also indicates the placement of the reconstruction loss $L_{1}$ , retargeting loss $L_{2}$ via differentiable forward kinematics, and latent regularization loss $L_{3}$ .

Figure 4: Zero-shot Unseen Tasks Generalization . For each hand, we randomly select some tasks as unseen tasks, whose data are held out from the training dataset. Then we test the unseen tasks with model trained on other data. Results show that by training with an aligned latent action space, XL-VLA gets the ability to generalize to novel hand-task combination in a zero-shot manner. PSR stands for “Partial Success Rate”, where policy is rewarded with half success if only one arm finishes its task.

	Ability	Inspire	X-Hand1	Paxini DexH13
#Fingers	5	5	5	4
#DoF(mimic)	12(6)	12(6)	12	16(3)

Table 3: Dexterous Hand Comparison.

Figure 5: G1 Cross-Robot Performance. Co-training with latent xArm and humanoid data outperforms using raw actions.

Model	Combination	PF	SC	SoC	HB	RL	PS	RB	PuS	PoS	PC	Mean
LAD [ 3 ]	Ability+Inspire	0.8	0.4	0.5	0.4	0.6	0.6	0.6	0.5	0.7	0.9	0.60
	Paxini+XHand	0.7	0.5	0.6	0.7	0.5	0.8	0.7	0.4	0.6	0.6	0.61
XL-VLA	Ability+Inspire	0.9	0.7	0.8	0.7	0.8	0.7	0.8	0.8	1.0	1.0	0.82
	Paxini+XHand	0.8	0.7	0.9	0.8	0.6	0.9	0.9	0.6	1.0	0.9	0.81

Table 4: Latent replay comparison . We compare our latent space with Latent Action Diffusion (LAD) [ 3 ] . For each hand combination, teleoperated trajectories collected on one source hand are encoded into the latent space, decoded onto the target hand, and replayed on real hardware. A replay is counted as successful if the encoded–decoded sequence can be executed without breaking contact or causing self-collisions. Higher replay success indicates better cross-embodiment consistency of the latent representation.

	Reconstruction $\downarrow$		Cross Embodiment $\downarrow$				Latent Continuity $\downarrow$		Interpolation $\downarrow$
Exp	Joint	Tip	PT ${}^{\text{dir}}$	PT ${}^{\text{dist}}$	RT ${}^{\text{dir}}$	RT ${}^{\text{dist}}$	Joint	Tip	Accel.	Jerk
Ours	$5.476$	$3.703$	$11.857$	$1.872$	$10.492$	$6.295$	$4.492$	$8.534$	$8.683$	$9.659$
$-L_{1}$	61.672	$39.400$	$11.741$	$1.857$	$10.398$	$6.375$	$24.784$	$58.858$	$12.028$	$16.852$
$-L_{2}^{\text{dist}}$	$5.195$	$3.580$	$3.972$	$4.413$	$6.788$	24.488	$4.073$	$8.168$	$8.525$	$9.240$
$-L_{2}^{\text{dir}}$	$4.966$	$3.378$	$46.217$	$2.251$	53.546	$5.518$	$4.451$	$9.217$	$8.742$	$9.551$
$-L_{2}$ (both)	$3.781$	$2.602$	$62.733$	$8.080$	71.765	$62.809$	$2.823$	$6.757$	$8.602$	$9.426$

Table 5: Ablations. Ablation results comparing reconstruction accuracy, cross-embodiment retargeting, latent-space continuity, and interpolation smoothness. Exp denotes model variants: removing losses (

-L_{2}

-L_{2}^{dist}

-L_{2}^{dir}

, or both), changing hidden sizes (

H^{b}_{a}

), or changing latent dimension (

L_{d}

). Metrics include joint and tip RMSE for reconstruction; pinch- and random-motion direction/distance errors for retargeting; joint/tip latent continuity; and mean acceleration/jerk for interpolation.

Panel of Figure 6 — Figure 6: Latent Visualizations. Latent decoding results cross embodiment.

Panel of Figure 7 — Figure 7: More Latent Visualizations. Latent decoding results cross embodiment.

Figure 8: xArm Camera Setup. We use a single RealSense L515 camera with the front view. Note that the D435 camera here is not used for XL-VLA.

Figure 9: Dexterous Hands. We use 4 kinds of hands, with various shapes, scales, degrees of freedom, and actuated joints.

Figure 10: xArm Camera View. This is what our camera sees and also the input for XL-VLA and all the baseline methods.

Figure 11: G1 Scene. We mount an L515 camera near the neck of G1 to have an egocentric view.

Figure 13: Objects. We use various everyday objects from existing datasets. They vary in scale, shape, texture, weight, etc., thus requiring the manipulation policy to be robust.

Figure 14: Apple Vision Por for Data Collection. Apple Vision Pro is used to track the human teleoperator’s hands and wrists. Then the tracked data is processed with retargeting and Inverse Kinematics to control the real-world robots.

Figure 15: G1 Teleoperation System. We build the G1 upper-body teleoperation system from HOMIE [ 4 ] . We use a pair of MANUS Mocap glove to track the human hand pose.

Panel of Figure 16 — Figure 16: Task Visualizations. We design 10 various tasks to test all the models. The tasks require varied manipulation skills and have different difficulties.

Figure 17: Latent Visualization of a Grasping Trajectory. A trajectory is shown here with all the robot hands.

Method	PF	HB	PS	PoS	Mean
$\pi_{0}$ [ 6 ]	0.4	0.6	0.5	0.6	0.525
XL-VLA	0.7	0.9	0.9	0.8	0.825 +57%

Table 6: G1 Policy Performances.

Explore similar work

Dexora: Open-source VLA for High-DoF Bimanual Dexterity
May 18, 2026Zongzheng Zhang, Jingrui Pang, Zhuo Yang +22Dexterous Manipulation Bimanual Robotic Manipulation
Tsinghua University · Beijing Academy of Artificial Intelligence · The University of Hong Kong +3
DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation
Aug 6, 2026Junfeng Li, Junjie He, Zhide Zhong +12Cross-Embodiment Robot Learning Robotics
1The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · 2COCO Matrix, Shanghai, China
METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model
Nov 21, 2025Yankai Fu, Ning Chen, Junkai Zhao +5Robotic Manipulation VLMs for Robotics
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence

May 18, 2026J/K move · Enter open · S save

Dexora: Open-source VLA for High-DoF Bimanual Dexterity

Zongzheng Zhang, Jingrui Pang, Zhuo Yang, Kun Li+21

Tsinghua University · Beijing Academy of Artificial Intelligence · The University of Hong Kong+3

Dexterous Manipulation Bimanual Robotic Manipulation Vision-Language-Action Models Robot Imitation Learning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently become a central direction in embodied AI, but current systems are restricted to either dual-gripper control or single-arm dexterous hand manipulation. While low-dimensional gripper control can often be handled with simpler methods, high-dimensional dexterous hand control benefits greatly from full end-to-end VLA learning. In this work, we introduce Dexora, the first open-source VLA system that natively targets dual-arm, dual-hand high-DoF manipulation. We design a hybrid teleoperation pipeline that decouples gross arm kinematics (captured with a custom exoskeleton backpack) from fine finger motion (markerless hand tracking via Apple Vision Pro), and that drives both a physical dual-arm dual-hand platform and an identical MuJoCo digital twin. Using that interface, we assemble a large training corpus: an embodiment-matched synthetic corpus (100K simulated trajectories, 6.5M frames) and a real-world dataset of 10K teleoperated episodes (2.92M frames). To mitigate noisy teleoperation demonstrations, we propose a data-quality-aware training recipe: an offline discriminator provides clip-level weights for diffusion-transformer policy training, down-weighting low-quality demonstrations. Empirically, Dexora outperforms competitive VLA baselines on both basic and dexterous benchmarks (e.g., average dexterous success 66.7% vs. 51.7%), attains 90% success on basic tasks, and shows robust out-of-distribution and cross-embodiment generalization. Ablations confirm the importance of real data and the discriminator for dexterity.

Abstract

Explore similar work

Dexora: Open-source VLA for High-DoF Bimanual Dexterity

DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation

METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model