DEFT-VLA: Stage-Localized Force Specialization for Contact-Rich Robotic Manipulation
Authors: Zan Huang, Yibo Qiu, Kan Ding, Xuanping Zhang, Ao Tian, Ronald X Xu, Mingzhai Sun
Organizations: the Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China, and also with the School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, China. · the School of Robotics, Xi’an Jiaotong-Liverpool University, Suzhou, Jiangsu, China.
Contact-rich manipulation poses a stage-dependent sensory conflict: vision and proprioception are essential for localization and approach, when force is largely uninformative and potentially distracting, whereas force becomes critical after contact, when irrelevant visual details and correlated robot states may become shortcuts. We propose DEFT-VLA (Dual-Expert Force-specialized Training for Vision-Language-Action Manipulation), which retains the task-adapted base expert for vision-dominated non-contact behavior and introduces a force-aware contact expert for critical contact stages, localizing force-conditioned adaptation to where it is needed. We further introduce a contact-localized force-attending curriculum that applies force-attending post-training only to the contact expert, progressively restoring vision and masking state tokens to suppress shortcuts. Real-robot experiments on pipette tip attachment, bimanual pipette aspiration, and whiteboard wiping show that DEFT-VLA effectively exploits force feedback to improve contact-stage performance while maintaining strong non-contact behavior and robustness to held-out grasp heights. It improves overall task success by 20-40 percentage points over the strongest baseline on each task.
Figures & tables
Fig. 2: Overview of DEFT-VLA. A shared VLM conditions a task-adapted base expert for non-contact behavior and a force-specialized contact expert for critical contact stages. The contact expert is initialized from the base expert and receives an additional force token. During contact-expert post-training, the contact-localized curriculum gradually restores image clarity and the original state tokens to suppress visual and proprioceptive shortcuts. At inference, a lightweight stage estimator fuses task-conditioned visual features, robot state, and causal wrench history, while a contact-stage router selects the active expert. Snowflake and flame symbols denote frozen and trainable modules, respectively.
Fig. 3: Robot platform and sensor configuration. The Mercury B2 dual-arm robot is equipped with two arm-mounted cameras, one third-person camera, and two wrist-mounted six-axis force/torque sensors.
Fig. 4: Three contact-rich manipulation tasks and their stage decomposition. In each row, the first three frames show the vision-dominated preparatory stage handled by the task-adapted base expert, while the last three show the critical contact stage handled by the force-specialized contact expert. The central boundary denotes the transition from the base expert to the contact expert.
Fig. 5: Representative non-contact-stage outcomes. The top and bottom rows show failed and successful visual alignment, respectively. (a) Alignment between the left end-effector and the pipette plunger button in bimanual aspiration. (b) Alignment between the pipette tip cone and the target tip opening in tip attachment.
Method
Pipette tip attachment
Bimanual pipette aspiration
Whiteboard wiping
Approach & Align
Press & Retract
Overall
Position & Align
Press & Release
Overall
Volume MAE ( μL ) ↓
Approach & Grasp
Contact & Wipe
Overall
π0.5
90%
33.3%
30%
95%
5.3%
5%
120.0
100%
70%
70%
π0.5 + Force Token
40%
25%
10%
100%
20%
20%
132.5
100%
45%
45%
π0.5 + Force Token + Global FAC
85%
29.4%
25%
35%
14.3%
5%
420.0
50%
60%
30%
DEFT-VLA w/o Contact-Localized FAC
80%
25%
20%
100%
30%
30%
130.0
100%
55%
55%
DEFT-VLA (Ours)
75%
93.3%
70%
100%
50%
50%
104.0
100%
90%
90%
TABLE I: Stage-wise and overall success rates across three contact-rich manipulation tasks.
Fig. 6: Height generalization protocol. Height labels refer to the gripper’s grasp position along the pipette. (a) Tip attachment at the Normal and held-out Low positions. (b) Bimanual aspiration at the Normal and held-out Low and High positions. Red dashed lines indicate the Normal reference in the magnified views.
Method
Pipette tip attachment
Bimanual pipette aspiration
π0.5
0%
0%
π0.5 + Force Token
10%
10%
π0.5 + Force Token + Global FAC
0%
0%
DEFT-VLA w/o Contact-Localized FAC
0%
40%
DEFT-VLA (Ours)
50%
50%
TABLE II: Success rates under height generalization.
Method
Visual Curriculum
State Masking
Success Rate
DEFT-VLA w/o Contact-Localized FAC
–
–
20%
DEFT-VLA w/o State Masking
✓
–
15%
DEFT-VLA (Ours)
✓
✓
70%
TABLE III: Ablation of state-token masking on pipette tip attachment.
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
Embodied intelligence for contact-rich manipulation has predominantly relied on position control, while explicit awareness and regulation of interaction forces remain under-explored, limiting stability, precision, and robustness in real-world tasks. We propose ForceVLA2, an end-to-end vision-language-action framework that equips robots with hybrid force-position control and explicit force awareness. ForceVLA2 introduces force-based prompts into the VLM expert to construct force-aware task concepts across stages, and employs a Cross-Scale Mixture-of-Experts (MoE) in the action expert to adaptively fuse these concepts with real-time interaction forces for closed-loop hybrid force-position regulation. To support learning and evaluation, we construct ForceVLA2-Dataset, containing 1,000 trajectories over 5 contact-rich tasks, including wiping, pressing, and assembling, with multi-view images, task prompts, proprioceptive state, and force signals. Extensive experiments show that ForceVLA2 substantially improves success rates and reliability in contact-rich manipulation, outperforming pi0 and pi0.5 by 48.0% and 35.0%, respectively, across the 5 tasks, and mitigating common failure modes such as arm overload and unstable contact, thereby actively advancing force-aware interactive physical intelligence in VLAs. The project page is available at https://sites.google.com/view/force-vla2/home.
Yang Li, Zhaxizhuoma, Hongru Jiang +11
Tongji University · Shanghai AI Laboratory · Lumos Robotics +6
Vision-Language-Action (VLA) models have emerged as general-purpose robotic manipulation policies. However, in dexterous hand manipulation, contact forces are typically provided as separate states or force-specific representations, making it difficult to explicitly represent the spatial correspondence between force and their corresponding visual locations. In this work, we propose VisForce, which visually grounds the current and desired forces at their corresponding fingertip locations. VisForce renders current and desired visual force cues on the current wrist image and a task-specific goal image, and combines the two representations through goal-conditioned cross-attention to generate force-aware actions. We evaluate VisForce using a real UR10 robot equipped with an RH56F1 dexterous hand through force-conditioned grasping and three multi-stage manipulation tasks. In force-conditioned grasping experiments, VisForce exhibited a consistent grip-force response as the desired force increased, and achieved grasp-and-lift success rates of 70% and 80% for an egg and a toothpaste tube, respectively. It further achieved final success rates of 70%, 55%, and 40% on cup insertion/bottle pouring, tong-assisted bread transfer, and slip-modulated peg-in-hole, respectively. These results show that fingertip-aligned visual force representations can be effectively used for force-aware conditioning in VLA-based dexterous hand manipulation.
Force-aware Vision-Language-Action (VLA) policies improve contact-rich manipulation, but typically combine task-level motion and contact-dependent adjustment in a single action prediction. Demonstrations provide no explicit labels for decomposing that prediction into a reusable reference action and a correction. We present ForceDelta-VLA, a correction-distillation framework that constructs an explicit force-correction target using paired predictions from a frozen teacher's force-conditioned and learned force-agnostic modes. A separate delay-correction target accounts for reference-action mismatch and the change in reference state. Training uses asynchronous schedule replay with the cached task context available during execution. The resulting lightweight policy adjusts the reference actions using recent force history and robot state, responding to contact changes between reference-action updates without regenerating complete action chunks. Across nine single-arm and bimanual contact-rich tasks, ForceDelta-VLA achieves an 82.2% mean success rate, compared with 54.4% for the original ForceVLA baseline. Direct execution of our Stage-1 Temporal Teacher achieves 70.6%. Relative to ForceVLA, the complete system reduces mean peak contact force over successful trials by approximately 26% on both platforms.
Ju Dong, Yu Fu, Jian Chen +9
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg, Hamburg, Germany. · University of Science and Technology of China, Hefei, China. · Technical University of Munich, Germany.