IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation

Zhian Su; Weijie Kong; Haonan Dong; Huixu Dong

cs.ROFeb 24, 2026

IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation

Authors: Zhian Su, Weijie Kong, Haonan Dong, Huixu Dong

Organizations: Grasp Laboratory, Mechanical Engineering Department, Zhejiang University, Hangzhou 310027, China. · Torch Kernel Co., Ltd., Hangzhou 310000, China.

Abstract

Vision-Language-Action (VLA) models have demonstrated significant potential for generalist robotic policies; however, they struggle to generalize to long-horizon complex tasks in novel real-world domains due to distribution shifts and the scarcity of high-quality demonstrations. Although reinforcement learning (RL) offers a promising avenue for policy improvement, applying it to real-world VLA fine-tuning faces challenges regarding exploration efficiency, training stability, and sample cost. To address these issues, we propose IG-RFT, a novel Interaction-Guided Reinforced Fine-Tuning system designed for flow-based VLA models. Firstly, to facilitate effective policy optimization, we introduce Interaction-Guided Advantage Weighted Regression (IG-AWR), an RL algorithm that dynamically modulates exploration intensity based on the robot's interaction status. Furthermore, to address the limitations of sparse or task-specific rewards, we design a novel hybrid dense reward function that integrates the trajectory-level reward and the subtask-level reward. Finally, we construct a three-stage RL system comprising SFT, Offline RL, and Human-in-the-Loop RL for fine-tuning VLA models. Extensive real-world experiments on four challenging long-horizon tasks demonstrate that IG-RFT achieves an average success rate of 85.0%, significantly outperforming SFT (18.8%) and standard Offline RL baselines (40.0%). Ablation studies confirm the critical contributions of IG-AWR and hybrid reward shaping. In summary, our work establishes and validates a novel reinforced fine-tuning system for VLA models in real-world robotic manipulation.

Figures & tables

Fig. 1: Overview of the Interaction-Guided Reinforced Fine-tuning System (IG-RFT). The system integrates a multi-head Q-Former-based critic model to provide dense value and interaction guidance. The supervision target of the dense value is our designed general hybrid value function. We use the interaction signal to modulate the flow matching initial noise to balance RL exploration and exploitation. The training process progresses through three stages: Supervised Fine-Tuning (SFT), data-efficient Offline RL using the proposed IG-AWR algorithm, and final Human-in-the-Loop refinement to master long-horizon real-world manipulation tasks.

Fig. 2: Illustration of Dynamic Uncertainty Modulation in IG-AWR. This mechanism modulates the initial sampling noise $\epsilon$ of the flow ODE based on the extracted interaction signal $I_{t}$ . During non-interaction phases (top), a higher variance $\sigma_{\text{high}}$ is applied to encourage diverse trajectory exploration. Conversely, during interaction phases (bottom), the variance is reduced to $\sigma_{\text{low}}$ to ensure precision and stability for contact-rich manipulation.

Fig. 3: Visualization of the Reward Signal and Value Function. We illustrate the proposed reward shaping mechanism on the ”Parcel Packing” task. (A) The reward signal $r_{t}$ combines a constant trajectory-level reward with weighted subtask-level rewards (peaks), providing dense feedback at key milestones. (B) The Value (Return-to-Go) function $V_{t}$ monotonically decreases from 1.0 to 0 as the agent progresses through subtasks, serving as a precise progress indicator for the Critic.

Fig. 4: Execution Sequences of Long-Horizon Tasks. From top to bottom, we visualize the successful execution flows for four challenging tasks: Parcel Packing, Fruit Bagging, Block Stacking, and Drink Shelving. Arrows indicate the progression of subgoals. Note: The text labels above each stage are abbreviated for visualization clarity and differ from the actual natural language instructions used by the model. For the detailed sequence of subtask instructions for each task, please refer to Appendix C .

Method	Parcel Packing	Fruit Bagging	Block Stacking	Drink Shelving	Avg.
SFT	15.0	30.0	25.0	5.0	18.8
IQL	40.0	50.0	45.0	20.0	38.8
AWR	35.0	55.0	45.0	25.0	40.0
Ours	85.0	95.0	90.0	70.0	85.0

TABLE I: Main Results on Real-world Long-horizon Tasks. We compare the success rate (%) of our system against baselines across four tasks. Results are averaged over 20 trials per task. Best results are highlighted in BOLD .

Configuration	Parcel Packing	Drink Shelving	Avg. Drop
Ours (Full System)	85.0	70.0	-
w/o Interaction Guidance	70.0	55.0	$\downarrow$ 15.0
w/o Trajectory-level Reward	75.0	60.0	$\downarrow$ 10.0
w/o Subtask-level Reward	55.0	45.0	$\downarrow$ 27.5

TABLE II: Ablation Study Results. We evaluate the contribution of Interaction Guidance (IG) and Reward Function on the two most challenging tasks.

Reward Configuration	Parcel Packing	Drink Shelving	Avg. Progress
Ours (Hybrid)	0.96	0.88	0.92
w/o Trajectory Reward	0.84	0.78	0.81
w/o Subtask Reward	0.68	0.54	0.61

TABLE III: Subtask Completion Progress. Instead of binary success rates, we report the normalized progress score (0.0-1.0), representing the average proportion of subtasks completed per episode. This metric reveals when the policy fails.

Fig. 5: Data Efficiency Analysis in Human-in-the-Loop Fine-tuning. We report the mean success rates averaged over the Parcel Packing and Drink Shelving tasks. The curves show that IG-RFT (Ours) achieves superior sample efficiency, converging to 77.5% success with only 40 interventions, significantly outperforming baselines. Shaded regions indicate the min-max range across tasks.

Training Stage / Configuration	Parcel Packing	Fruit Bagging	Block Stacking	Drink Shelving	Avg.	Avg. Drop
Stage I: SFT	15.0	30.0	25.0	5.0	18.8	$\downarrow$ 66.2
Stage II: Offline RL	40.0	55.0	45.0	20.0	40.0	$\downarrow$ 45.0
SFT $\to$ HIL (w/o Stage II)	65.0	75.0	70.0	45.0	63.8	$\downarrow$ 21.2
Stage III: Full System	85.0	95.0	90.0	70.0	85.0	-

TABLE IV: Policy Evolution and Stage Analysis. We compare the performance at different training stages. Results are averaged over 20 trials per task. HIL denotes Human-in-the-Loop. Best results are highlighted in BOLD .

Fig. 6: Visualization of Interaction Signal Extraction. We illustrate the process of quantifying interaction intensity. For each sample frame, the left panel shows the raw RGB observation from global camera view, while the right panel displays the dense optical flow with the robot body masked out (black region). The “Moving Pix” count represents the magnitude of environmental motion $\sum(1-M_{t})\cdot\mathbb{I}(||F_{t}||^{2}>\delta_{\text{flow}})$ , which serves as the threshold metric for determining the binary interaction signal $I_{t}$ .

Fig. 7: Visualization of Value during one rollout in Long-Horizon Tasks. We plot the critic-model-predicted state value (Return-to-Go) trajectories for (A) Parcel Packing and (B) Drink Shelving tasks. The dashed grey line represents the reference value function designed in Section IV-A, while the solid red line shows the predicted value during real-world execution. The images from left to right sequentially represent a successful task; the blue-highlighted images correspond to value peaks, indicating the policy took suboptimal actions. For each suboptimal case, we describe the potential reasons with text above the images. The corresponding subtask stages are annotated in grey italic text along the timeline.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

General Optimization
Parameter	Value
Optimizer	AdamW
LR Schedule	CosineDecay
Learning Rate (Actor)	$5\times 10^{-5}$
Learning Rate (Critic)	$5\times 10^{-6}$
Batch Size	64

Appendix

TABLE V: Hyperparameters for IG-RFT Training

RL Fine-Tuning

Reinforcement learning fine-tuning adapts pretrained models and policies through task rewards, preference feedback, or environment interaction.

See all 231 papers

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation
Jun 24, 2026Shuyi Zhang, Yunfan Lou, Hongyang Cheng +8RL for Robotics Vision-Language-Action Models
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China
EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models
May 25, 2026Perry Dong, Kuo-Han Hung, Tian Gao +2Robot Skill Learning Robotic RL
Stanford University
Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning
Sep 29, 2026Ruixiao Xu, Wong Lik Hang Kenny, Zhiqian Liu +12Vision-Language-Action Models VLMs for Robotics
Beihang University · The Chinese University of Hong Kong · PKU-Psibot Lab +4

Jun 24, 2026J/K move · Enter open · S save

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo+7

Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China

RL for Robotics Vision-Language-Action Models RL Fine-Tuning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.

Abstract

RL Fine-Tuning

Explore similar work

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning