IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation

Zhian Su; Weijie Kong; Haonan Dong; Huixu Dong

cs.ROFeb 24, 2026

IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation

Authors: Zhian Su, Weijie Kong, Haonan Dong, Huixu Dong

Organizations: Grasp Laboratory, Mechanical Engineering Department, Zhejiang University, Hangzhou 310027, China. · Torch Kernel Co., Ltd., Hangzhou 310000, China.

Abstract

Vision-Language-Action (VLA) models have demonstrated significant potential for generalist robotic policies; however, they struggle to generalize to long-horizon complex tasks in novel real-world domains due to distribution shifts and the scarcity of high-quality demonstrations. Although reinforcement learning (RL) offers a promising avenue for policy improvement, applying it to real-world VLA fine-tuning faces challenges regarding exploration efficiency, training stability, and sample cost. To address these issues, we propose IG-RFT, a novel Interaction-Guided Reinforced Fine-Tuning system designed for flow-based VLA models. Firstly, to facilitate effective policy optimization, we introduce Interaction-Guided Advantage Weighted Regression (IG-AWR), an RL algorithm that dynamically modulates exploration intensity based on the robot's interaction status. Furthermore, to address the limitations of sparse or task-specific rewards, we design a novel hybrid dense reward function that integrates the trajectory-level reward and the subtask-level reward. Finally, we construct a three-stage RL system comprising SFT, Offline RL, and Human-in-the-Loop RL for fine-tuning VLA models. Extensive real-world experiments on four challenging long-horizon tasks demonstrate that IG-RFT achieves an average success rate of 85.0%, significantly outperforming SFT (18.8%) and standard Offline RL baselines (40.0%). Ablation studies confirm the critical contributions of IG-AWR and hybrid reward shaping. In summary, our work establishes and validates a novel reinforced fine-tuning system for VLA models in real-world robotic manipulation.

Figures & tables

Fig. 1: Overview of the Interaction-Guided Reinforced Fine-tuning System (IG-RFT). The system integrates a multi-head Q-Former-based critic model to provide dense value and interaction guidance. The supervision target of the dense value is our designed general hybrid value function. We use the interaction signal to modulate the flow matching initial noise to balance RL exploration and exploitation. The training process progresses through three stages: Supervised Fine-Tuning (SFT), data-efficient Offline RL using the proposed IG-AWR algorithm, and final Human-in-the-Loop refinement to master long-horizon real-world manipulation tasks.

Fig. 2: Illustration of Dynamic Uncertainty Modulation in IG-AWR. This mechanism modulates the initial sampling noise $\epsilon$ of the flow ODE based on the extracted interaction signal $I_{t}$ . During non-interaction phases (top), a higher variance $\sigma_{\text{high}}$ is applied to encourage diverse trajectory exploration. Conversely, during interaction phases (bottom), the variance is reduced to $\sigma_{\text{low}}$ to ensure precision and stability for contact-rich manipulation.

Fig. 3: Visualization of the Reward Signal and Value Function. We illustrate the proposed reward shaping mechanism on the ”Parcel Packing” task. (A) The reward signal $r_{t}$ combines a constant trajectory-level reward with weighted subtask-level rewards (peaks), providing dense feedback at key milestones. (B) The Value (Return-to-Go) function $V_{t}$ monotonically decreases from 1.0 to 0 as the agent progresses through subtasks, serving as a precise progress indicator for the Critic.

Fig. 4: Execution Sequences of Long-Horizon Tasks. From top to bottom, we visualize the successful execution flows for four challenging tasks: Parcel Packing, Fruit Bagging, Block Stacking, and Drink Shelving. Arrows indicate the progression of subgoals. Note: The text labels above each stage are abbreviated for visualization clarity and differ from the actual natural language instructions used by the model. For the detailed sequence of subtask instructions for each task, please refer to Appendix C .

Method	Parcel Packing	Fruit Bagging	Block Stacking	Drink Shelving	Avg.
SFT	15.0	30.0	25.0	5.0	18.8
IQL	40.0	50.0	45.0	20.0	38.8
AWR	35.0	55.0	45.0	25.0	40.0
Ours	85.0	95.0	90.0	70.0	85.0

TABLE I: Main Results on Real-world Long-horizon Tasks. We compare the success rate (%) of our system against baselines across four tasks. Results are averaged over 20 trials per task. Best results are highlighted in BOLD .

Configuration	Parcel Packing	Drink Shelving	Avg. Drop
Ours (Full System)	85.0	70.0	-
w/o Interaction Guidance	70.0	55.0	$\downarrow$ 15.0
w/o Trajectory-level Reward	75.0	60.0	$\downarrow$ 10.0
w/o Subtask-level Reward	55.0	45.0	$\downarrow$ 27.5

TABLE II: Ablation Study Results. We evaluate the contribution of Interaction Guidance (IG) and Reward Function on the two most challenging tasks.

Reward Configuration	Parcel Packing	Drink Shelving	Avg. Progress
Ours (Hybrid)	0.96	0.88	0.92
w/o Trajectory Reward	0.84	0.78	0.81
w/o Subtask Reward	0.68	0.54	0.61

TABLE III: Subtask Completion Progress. Instead of binary success rates, we report the normalized progress score (0.0-1.0), representing the average proportion of subtasks completed per episode. This metric reveals when the policy fails.

Fig. 5: Data Efficiency Analysis in Human-in-the-Loop Fine-tuning. We report the mean success rates averaged over the Parcel Packing and Drink Shelving tasks. The curves show that IG-RFT (Ours) achieves superior sample efficiency, converging to 77.5% success with only 40 interventions, significantly outperforming baselines. Shaded regions indicate the min-max range across tasks.

Training Stage / Configuration	Parcel Packing	Fruit Bagging	Block Stacking	Drink Shelving	Avg.	Avg. Drop
Stage I: SFT	15.0	30.0	25.0	5.0	18.8	$\downarrow$ 66.2
Stage II: Offline RL	40.0	55.0	45.0	20.0	40.0	$\downarrow$ 45.0
SFT $\to$ HIL (w/o Stage II)	65.0	75.0	70.0	45.0	63.8	$\downarrow$ 21.2
Stage III: Full System	85.0	95.0	90.0	70.0	85.0	-

TABLE IV: Policy Evolution and Stage Analysis. We compare the performance at different training stages. Results are averaged over 20 trials per task. HIL denotes Human-in-the-Loop. Best results are highlighted in BOLD .

Fig. 6: Visualization of Interaction Signal Extraction. We illustrate the process of quantifying interaction intensity. For each sample frame, the left panel shows the raw RGB observation from global camera view, while the right panel displays the dense optical flow with the robot body masked out (black region). The “Moving Pix” count represents the magnitude of environmental motion $\sum(1-M_{t})\cdot\mathbb{I}(||F_{t}||^{2}>\delta_{\text{flow}})$ , which serves as the threshold metric for determining the binary interaction signal $I_{t}$ .

Fig. 7: Visualization of Value during one rollout in Long-Horizon Tasks. We plot the critic-model-predicted state value (Return-to-Go) trajectories for (A) Parcel Packing and (B) Drink Shelving tasks. The dashed grey line represents the reference value function designed in Section IV-A, while the solid red line shows the predicted value during real-world execution. The images from left to right sequentially represent a successful task; the blue-highlighted images correspond to value peaks, indicating the policy took suboptimal actions. For each suboptimal case, we describe the potential reasons with text above the images. The corresponding subtask stages are annotated in grey italic text along the timeline.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

General Optimization
Parameter	Value
Optimizer	AdamW
LR Schedule	CosineDecay
Learning Rate (Actor)	$5\times 10^{-5}$
Learning Rate (Critic)	$5\times 10^{-6}$
Batch Size	64

Appendix

TABLE V: Hyperparameters for IG-RFT Training

RL Fine-Tuning

Reinforcement learning fine-tuning adapts pretrained models and policies through task rewards, preference feedback, or environment interaction.

See all 231 papers

Jun 24, 2026

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.

Shuyi Zhang, Yunfan Lou, Hongyang Cheng +8

Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China

RL for Robotics Vision-Language-Action Models RL Fine-Tuning

Ask PDF

May 25, 2026

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

The ability to efficiently and reliably learn new tasks has been a foundational challenge in robotics. Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse manipulation tasks, yet pretrained policies consistently fall short of the reliability required for real-world deployment. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches either train from scratch without fully leveraging pretrained priors, or fine-tune VLAs without achieving the sample efficiency and success rates that practical deployment demands. We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies that closes this gap. Our system solves a suite of challenging manipulation tasks, including routing string lights and inserting the plug to light it up, striking a pool ball into a pocket, and inserting a flower into a wine bottle, each requiring combinations of high precision, dynamic actions, and robustness to varied initial states. Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data, outperforming both prior RL-from-scratch and VLA finetuning approaches. We release an open-source codebase with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.

Perry Dong, Kuo-Han Hung, Tian Gao +2

Stanford University

Robot Skill Learning Robotic RL Efficient VLA Models Robotic Manipulation Data-Efficient Robot Learning Vision-Language-Action Models+2

Ask PDF

Sep 29, 2026

Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning

We study reinforcement learning (RL) methods for cooperative multi-agent Vision-Language-Action (VLA) models. This problem is challenging because VLAs are pretrained on large-scale single-agent data and therefore lack the fine-grained coordination skills required for inter-robot collaboration. Supervised fine-tuning (SFT) on multi-robot demonstrations partially bridges this gap, but its performance is bounded by the demonstration data and cannot improve from its own experience. We present a three-stage reinforced fine-tuning (RFT) pipeline for multi-agent VLAs. First, initialization-aware data collection sweeps over initial configurations and invokes human demonstrations only when the pretrained VLA repeatedly fails, yielding robustness to initialization shift with reduced human cost. Second, offline credit-filtered tuning assigns credit to individual agents and fine-tunes on per-agent trajectories with positive advantage rather than on entire joint rollouts. Third, we find existing online RL for VLAs are less effective for hard multi-agent tasks, which we attribute to noisy co-exploration and unstable updates. We instead use online latent-space fine tuning, which freeze the VLA and perform RL in its latent noise space. We evaluate our multi-agent VLA with both

π_0

and

π_{0.5}

backbones across 11 tasks in RoboTwin, RoboFactory and real-world manipulation with two Franka robots. Our multi-agent VLA improves the average success rate by

+23.1\%

+16.4\%

, and

+44\%

on RoboTwin, RoboFactory, and real-world tasks, respectively. Code available at https://anonymous.4open.science/r/mavla_rft-2BC0/.

Ruixiao Xu, Wong Lik Hang Kenny, Zhiqian Liu +12

Beihang University · The Chinese University of Hong Kong · PKU-Psibot Lab +4

Vision-Language-Action Models VLMs for Robotics RL Fine-Tuning Cooperative MARL

Ask PDF

Abstract

RL Fine-Tuning

Explore similar work

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning