ForceRFT: Refining VLA Actions through Force-Guided Residual Reinforcement Learning
Authors: Yichen Wang, Chaoyang Zhang, Xuqi Su, Jun Ma, Haiyue Zhu, Xiaocong Li
Organizations: College of Information Science and Technology, Eastern Institute of Technology, Ningbo, Ningbo 315200, China. · Jiangsu Key Laboratory of Advanced Food Manufacturing Equipment and Technology, Jiangnan University, Wuxi, Jiangsu, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China. · Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China. · Department of Electrical and Computer Engineering, National University of Singapore, Singapore 117583. · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, Eastern Institute of Technology, Ningbo, Ningbo 315200, China.
Force-conditioned vision-language-action (VLA) policies can respond to contact, but when trained solely on demonstrations, their recovery behavior may be limited by demonstration coverage, and they do not learn from deployment outcomes. Human corrective imitation provides additional recovery examples, but its objective matches local action targets without explicitly optimizing task return. We present ForceRFT, a force-guided residual reinforcement learning framework that learns contact-dependent corrections from human supervision and autonomous task outcomes. A frozen, demonstration-trained SmolVLA-based prior generates force-conditioned action chunks, while a lightweight residual actor refines individual end-effector pose commands using wrist feedback acquired during chunk execution. The decision-time wrist wrench, its temporal change, and the selected base motion condition both residual correction and value estimation. Human corrections supervise the residual actor, while verified autonomous transitions train the twin critics and support value-guided updates to the same actor. Bootstrapping is restricted to autonomous segments, preventing TD credit from crossing human-intervention boundaries. Real-robot experiments on plug insertion, ring-on-peg assembly, and whiteboard wiping show higher autonomous success rates than the evaluated demonstration-trained and residual-imitation baselines. Comparisons with residual imitation support value-guided residual optimization, while plug-insertion ablations indicate the benefit of direct execution-time wrist feedback.
Figures & tables
Fig. 1: Overview of ForceRFT. A demonstration-trained force-conditioned prior generates action chunks and remains frozen online. At each execution decision, the selected TCP target is re-anchored to the current robot state and refined by a bounded residual conditioned on the state, base motion, and wrist feedback. Only autonomous transitions train the twin critics and support value-guided actor updates; human corrections supervise the same actor through residual imitation. Takeover transfers control to the operator and blocks cross-takeover bootstrapping ( βn=0 ); release triggers fresh replanning. Solid arrows denote control/data flow; dashed arrows denote learning updates.
Fig. 2: Experimental platform. Real-robot setup with dual-view RGB cameras and wrist force/torque sensing.
Fig. 3: Task visualization. Representative stages of plug insertion, ring-on-peg assembly, and whiteboard wiping.
Method
Plug insertion ↑
Ring-on-peg assembly ↑
Whiteboard wiping ↑
SmolVLA-SFT
7/30
10/30
2/30
Force-conditioned prior
12/30
17/30
4/30
Residual imitation
19/30
22/30
8/30
ForceRFT
25/30
27/30
11/30
TABLE I: Real-robot performance. Autonomous successes out of 30 trials for each task.
Configuration
Plug insertion ↑
w/o wrench increment
23/30
w/o residual wrist feedback
19/30
Full (ForceRFT)
25/30
TABLE II: Wrist-feedback ablation on plug insertion. Autonomous successes out of 30 trials.
Fig. 4: Online adaptation across three tasks. Successes out of 30 autonomous trials per task for the frozen prior and ForceRFT checkpoints. Connecting lines indicate checkpoint order, not equal cycle intervals.
Force-aware Vision-Language-Action (VLA) policies improve contact-rich manipulation, but typically combine task-level motion and contact-dependent adjustment in a single action prediction. Demonstrations provide no explicit labels for decomposing that prediction into a reusable reference action and a correction. We present ForceDelta-VLA, a correction-distillation framework that constructs an explicit force-correction target using paired predictions from a frozen teacher's force-conditioned and learned force-agnostic modes. A separate delay-correction target accounts for reference-action mismatch and the change in reference state. Training uses asynchronous schedule replay with the cached task context available during execution. The resulting lightweight policy adjusts the reference actions using recent force history and robot state, responding to contact changes between reference-action updates without regenerating complete action chunks. Across nine single-arm and bimanual contact-rich tasks, ForceDelta-VLA achieves an 82.2% mean success rate, compared with 54.4% for the original ForceVLA baseline. Direct execution of our Stage-1 Temporal Teacher achieves 70.6%. Relative to ForceVLA, the complete system reduces mean peak contact force over successful trials by approximately 26% on both platforms.
Ju Dong, Yu Fu, Jian Chen +9
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg, Hamburg, Germany. · University of Science and Technology of China, Hefei, China. · Technical University of Munich, Germany.
Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.
Shuyi Zhang, Yunfan Lou, Hongyang Cheng +8
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China
Pretrained vision-language-action (VLA) policies provide strong language-conditioned manipulation knowledge, but they remain largely vision-driven and can struggle once manipulation enters contact states where the scene is occluded, depth is ambiguous, or small force errors push execution off the offline demonstration distribution. We present LIFT (Late Reactive Injection of Force for VLA Post-Training), a force-aware post-training framework that adds contact reactivity to a pretrained VLA policy while preserving its output at initialization. LIFT grafts a reactive action expert beside the original action expert, initializes it from pretrained action weights, and injects recent 6D end-effector force through causal force memory and zero-initialized cross attention, enabling actions to be refreshed during execution. To address the policy-dependent distribution shift of contact feedback, LIFT further couples reactive force injection with an online DAgger loop that trains on a mixture of offline task-alignment data and human-corrected online rollouts. Across towel folding, book insertion, and Hanoi ring placement, LIFT learns faster and reaches higher peak and final performance than vision-only DAgger, while ablations show that reactive force memory and online corrective data are both important for robust contact-rich manipulation. Our code is publicly available at https://github.com/y-wng/lift
Yi Wang, Wendi Chen, Zimo Wen +8
Shanghai Jiao Tong University · Shanghai Innovation Institute · Zhiyuan College, Shanghai Jiao Tong University +2