Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning

Ruixiao Xu; Wong Lik Hang Kenny; Zhiqian Liu; Jianing Guo; Hanxiao Li; Kejian Shi; Shuning Zhang; Pu Feng

cs.ROSep 29, 2026

Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning

Authors: Ruixiao Xu, Wong Lik Hang Kenny, Zhiqian Liu, Jianing Guo, Hanxiao Li, Kejian Shi, Shuning Zhang, Pu Feng, +7 more

Organizations: Beihang University · The Chinese University of Hong Kong · PKU-Psibot Lab · Tsinghua University · Zhongguancun Laboratory · Li Auto Inc. · Peking University

Abstract

We study reinforcement learning (RL) methods for cooperative multi-agent Vision-Language-Action (VLA) models. This problem is challenging because VLAs are pretrained on large-scale single-agent data and therefore lack the fine-grained coordination skills required for inter-robot collaboration. Supervised fine-tuning (SFT) on multi-robot demonstrations partially bridges this gap, but its performance is bounded by the demonstration data and cannot improve from its own experience. We present a three-stage reinforced fine-tuning (RFT) pipeline for multi-agent VLAs. First, initialization-aware data collection sweeps over initial configurations and invokes human demonstrations only when the pretrained VLA repeatedly fails, yielding robustness to initialization shift with reduced human cost. Second, offline credit-filtered tuning assigns credit to individual agents and fine-tunes on per-agent trajectories with positive advantage rather than on entire joint rollouts. Third, we find existing online RL for VLAs are less effective for hard multi-agent tasks, which we attribute to noisy co-exploration and unstable updates. We instead use online latent-space fine tuning, which freeze the VLA and perform RL in its latent noise space. We evaluate our multi-agent VLA with both $π_0$ and $π_{0.5}$ backbones across 11 tasks in RoboTwin, RoboFactory and real-world manipulation with two Franka robots. Our multi-agent VLA improves the average success rate by $+23.1\%$ , $+16.4\%$ , and $+44\%$ on RoboTwin, RoboFactory, and real-world tasks, respectively. Code available at https://anonymous.4open.science/r/mavla_rft-2BC0/.

Figures & tables

Figure 1: The three-stage recipe for reinforcement fine-tuning multi-agent VLAs. We first perform Initialization-Aware Data Collection to broaden coverage of environment conditions, followed by Offline Credit-Filtered Tuning to decompose agent-wise advantages and reinforce high-contribution behaviors. Finally, we extend DSRL to the multi-agent setting and perform Online Latent-Space Reinforcement Learning to enable further improvement through trial and error.

Figure 2: Success rate (%, $\uparrow$ ) under 8 different simulation tasks from RoboTwin and RoboFactory with backbone models $\pi_{0}$ and $\pi_{0.5}$ .

Task	CHORUS	+ Offline RL w/o IA	+Offline RL +Online RL w/o IA	+Offline RL	+Offline RL +Online RL
Lift Pot	24	30	46	42	50
Handover Mic	43	46	63	52	73
Three Robots Stack Cubes	49	57	58	61	65
Take Photo	55	68	79	69	88
Average	42.8	50.3	61.5	56.0	69.0

Table 1: Ablation study on the initialization-aware data collection. We report success rate (%,

\uparrow

Figure 3: Visualization of success rates under different environment initializations. Initialization-Aware Data Collection identifies and compensates for challenging configurations in which agents struggle to acquire successful behaviors through autonomous exploration.

Figure 4: Visualization of agent-wise credit assignments and the behaviors of them.

Task	Offline RL	+Flow-SDE	+Flow-Noise	+DSRL
Lift Pot	42	43	43	50
Handover Mic	52	56	62	73
Three Robots Stack Cubes	61	62	63	65
Take Photo	69	82	78	88
Average	56.0	60.1	61.5	69.0

Table 2: Comparison of different online RL methods. We report success rate (%,

\uparrow

Figure 7: The success rate (%, $\uparrow$ ) and the overview under 3 different collaborative tasks on 2 Franka robots. Our training recipe effectively enhances the collaborative performance of multi-agent VLAs.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Batch size	32
Training steps	30,000
VLM	Full-parameter
Action expert	Full-parameter
Chunk horizon	50 for simulation and 20 for real-world

Appendix

Table 3: Training settings of the VLA policies and value functions.

Hyperparameter	Value	Hyperparameter	Value	Hyperparameter	Value
Batch size	16	Actor lr	$1\times 10^{-4}$	Critic lr	$3\times 10^{-4}$
Training episodes	200	Temperature lr	$3\times 10^{-4}$	UTD ratio	1
Hidden dimensions	$(128,128,128)$	Latent dimension	50	Discount factor	0.999
Target update rate	0.005	Number of Q-functions	10	Critic reduction	Mean
CNN features	$(32,32,32,32)$	CNN strides	$(2,1,1,1)$	CNN padding	VALID
Encoder type	Small	Encoder normalization	Group	Bottleneck	True

Appendix

Table 4: Training hyperparameters of DSRL.

Hyperparameter	Value	Hyperparameter	Value	Hyperparameter	Value
Training episodes	200	Advantage type	embodied_grpo	Train expert only	True
Top-k	50	Loss type	embodied_grpo	Reward type	chunk_level
Parallel Num	8	PPO clip high	0.2	Separate models	True
Discount factor	0.99	PPO clip low	0.2	Action chunks	50
Chunk steps	16	PPO clip c	3.0	Flow steps	10
Eval chunk steps	16	Actor lr	$5\times 10^{-6}$	Noise level	0.1 (Flow-SDE) 0.01 (Flow-Noise)

Appendix

Table 5: Training hyperparameters of Flow-SDE and Flow-Noise.

Explore similar work

ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning
Apr 20, 2026Tuan Van Vo, Tan Q. Nguyen, Khang Nguyen +5Diffusion-Based Vision-Language-Actions Multimodal Reasoning
VinRobotics, Hanoi. Vietnam · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE. · University of Texas at Arlington, Texas, USA +4
Co-VLA: Consensus-based Federated Training for Vision-Language-Action Models
Sep 17, 2026Haolong Li, Guner Dilsad Er, Michael Muehlebach +1Diffusion-Based Vision-Language-Actions Scalable Robot Learning
Intelligent Perception in Technical Systems, University of Augsburg, Germany · Max Planck Institute for Intelligent Systems, Germany
RL Token: Bootstrapping Online RL with Vision-Language-Action Models
Apr 24, 2026Charles Xu, Jost Tobias Springenberg, Michael Equi +4Diffusion-Based Vision-Language-Actions Reinforcement Fine-Tuning
Physical Intelligence

Apr 20, 2026, cs.ROJ/K move · Enter open · S save

ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning

Tuan Van Vo, Tan Q. Nguyen, Khang Nguyen, Nhat Xuan Tran+4

VinRobotics, Hanoi. Vietnam · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE. · University of Texas at Arlington, Texas, USA+4

Diffusion-Based Vision-Language-Actions Multimodal Reasoning Robot Policies Refinevla

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have gained much attention from the research community thanks to their strength in translating multimodal observations with linguistic instructions into desired robotic actions. Despite their advancements, VLAs often overlook explicit reasoning and learn the functional input-action mappings, omitting crucial logical steps, which are especially pronounced in interpretability and generalization for complex, long-horizon manipulation tasks. In this work, we propose ReFineVLA, a multimodal reasoning-aware framework that fine-tunes VLAs with teacher-guided reasons. We first augment robotic datasets with reasoning rationales generated by an expert teacher model, guiding VLA models to learn to reason about their actions. Then, we fine-tune pre-trained VLAs with the reasoning-enriched datasets with ReFineVLA, while maintaining the underlying generalization abilities and boosting reasoning capabilities. We also conduct attention map visualization to analyze the alignment among visual observation, linguistic prompts, and to-be-executed actions of ReFineVLA, reflecting the model is ability to focus on relevant tasks and actions. Through this additional step, we explore that ReFineVLA-trained models exhibit a meaningful agreement between vision-language and action domains, highlighting the enhanced multimodal understanding and generalization. Evaluated across a suite of simulated manipulation benchmarks on SimplerEnv with both WidowX and Google Robot tasks, ReFineVLA achieves state-of-the-art performance, in success rate over the second-best method on the both the WidowX benchmark and Google Robot Tasks.

Abstract

Explore similar work

ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning

Co-VLA: Consensus-based Federated Training for Vision-Language-Action Models

RL Token: Bootstrapping Online RL with Vision-Language-Action Models