While Vision-Language-Action (VLA) models have demonstrated promising generalization capabilities in robotic manipulation, deploying them on specific and complex downstream tasks still demands effective post-training. In parallel, Human-in-the-Loop (HiL) learning has proven to be a powerful mechanism for refining robot policies. However, extending this paradigm to dexterous manipulation remains challenging: multi-finger control is high-dimensional, contact-intensive, and exhibits execution distributions that differ markedly from standard arm motions, leaving existing dexterous VLA systems limited in reliability and adaptability. We present DexHiL, the first integrated arm-hand human-in-the-loop framework for dexterous VLA models, enabling coordinated interventions over the arm and the dexterous hand within a single system. DexHiL introduces an intervention-aware data sampling strategy that prioritizes corrective segments for post-training, alongside a lightweight teleoperation interface that supports instantaneous human corrections during execution. Real-robot experiments demonstrate that DexHiL serves as an effective post-training framework, yielding a substantial performance leap, outperforming standard offline-only fine-tuning baselines by an average of 25% in success rates across distinct tasks. Project page: https://chenzhongxi-sjtu.github.io/dexhil/
Figures & tables
Fig. 1 : While scaling offline data for VLA models yields slow accuracy gains and performance plateaus, DexHiL integrates offline training with online Human-in-the-Loop interventions. By strategically reweighting offline and corrective online data, our approach achieves high data efficiency and rapid accuracy growth.
Fig. 2 : The DexHiL Framework. Below we will introduce our overall framework of DexHiL from data acquisition system, human-in-the-loop intervention paradigm, dexterous manipulation VLA model structure to offline-to-online training process. ( a ) We propose an arm-hand data collection system both supporting teleoperation offline data collection and online human-in-the-loop policy intervention data collection. We also propose a two-stage training method for precise hand joint retargeting. ( b ) We propose an asynchronous human-in-the-loop policy intervention mechanism for online data collection, here we examplify the “Plush Toy Grasping” case. ( c ) Our dexterous manipulation VLA policy follows Being-H0.5 [ 11 ] structure, which utilizing MoT (Mixture of Transformer) to relate understanding model with action expert for multi-modal reasoning and action generation and inherit the open-source pretrained weights of it. ( d ) We propose a two-phase training framework that ultilize both offline dataset and online dataset. In the first warm-up phase, we finetuned the pretrained weights into warm-up model. In the DAgger loop, we utilize the system above to acquire online dataset and use reweighting training to update the policy which will be applied in the next DAgger loop.
Fig. 3 : Real-world rollouts of dexterous manipulation tasks. (Up) Tissue Extraction: The system achieves precise fingertip alignment and vertical retraction to extract the tissue. (Down) Plush Toy Grasping: The controller executes a synchronized multi-joint flexion to securely envelop and lift the deformable object.
Fig. 4 : Visualization of retargeting results for four representative gestures. We show the input human hand poses and the corresponding configurations generated by Dex-retargeting [ 35 ] , GeoRT [ 24 ] , and our method .Compared to other methods, our retargeting algorithm generates more accurate, smooth, and coordinated hand poses.
Fig. 5 : Quantitative performance and training loss analysis. (Left) Success rates across three consecutive training rounds for both Tissue Extraction and Plush Toy Grasping tasks. (Right) Initial training loss at step 10 for both DexHiL and DAgger*. While the loss previously plateaued between 0.002 and 0.008, it significantly increases after incorporating human corrective trajectories.
State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University · Shanghai Key Laboratory of Intelligent Robotics, Meta Robotics Institute, Shanghai Jiao Tong University, Shanghai 200240, China · ByteDance Seed +1