Multi-agent reinforcement learning is a key method for training multi-robot systems. Through rewarding or punishing robots over a series of episodes according to their performance, they can be trained and then deployed in the real world. However, poorly trained policies can lead to unsafe behaviour during early training stages. We introduce Multi-Agent Reinforcement Learning guided by language-based Inter-robot Negotiation (MARLIN), a hybrid framework in which large language models provide high-level planning before the reinforcement learning policy has learned effective behaviours. Robots use language models to negotiate actions and generate plans that guide policy learning. The system dynamically switches between reinforcement learning and language-model-based negotiation during training, enabling more effective exploration. MARLIN is evaluated using both simulated and physical robots with local and remote language models. Results show that, compared to standard multi-agent reinforcement learning, the hybrid approach achieves higher performance in early training with limited impact on the final performance. The code is available at https://github.com/SooratiLab/MARLIN.
Figures & tables
Fig. 1: Diagrams of the scenarios used for evaluation; Asymmetrical Two Slot Corridor (a), Maze Like Corridor (b), Single Slot Corridor (c), Symmetrical Two Slot Corridor (d), Two Path Corridor (e).
Fig. 2: Distribution of end locations throughout training (LLM results are static).
Fig. 3: Median performance of the MARLIN and MARL for different scenarios in simulation during the beginning and towards the end of training. The boxplots show the distributions of performance scores for trials using the LLM-based negotiation.
Fig. 4: Distribution of performance scores of the MARLIN and standard MARL at different times during training.
Fig. 5: Environment setup (a) and TurtleBot3 robot platform (b) used for the physical experiments in the Maze Like Corridor.
Scenario
Ep. 100
Ep. 850
Ep. 1600
MARL
MARLIN
MARL
MARLIN
MARL
MARLIN
(a)
0.8543
1.0000
0.9814
0.9957
0.9843
0.9929
(b)
0.4300
0.4271
0.8529
0.9643
0.8214
1.0000
(c)
0.7157
0.9071
0.9329
0.9943
0.9700
1.0000
(d)
0.8671
0.9714
0.9929
1.0000
0.9857
0.9952
(e)
0.7771
0.9271
0.9929
1.0000
1.0000
0.9667
TABLE I: Average performance throughout training. Bold indicates significance ( p<0.001 over a 50 run paired t-test);
Fig. 6: Median performance of the system for the Maze Like Corridor when executed on physical hardware. The box plot shows the performance distribution when the LLM-only system was evaluated on physical hardware.
Fig. 7: A scenario with a larger number of robots is used to demonstrate our method at scale. Agents negotiate in pairs to pass corridors with no alternative route (e.g. blue areas highlighting our examined corridors). The green cell is the exit, and the grey cells indicate starting positions.
Large language model (LLM) agents struggle to autonomously evolve coordination strategies in dynamic environments, largely because coarse global outcomes obscure the causal signals needed for local policy refinement. We identify this bottleneck as a multi-agent credit assignment problem, which has long been studied in classical multi-agent reinforcement learning (MARL) but remains underaddressed in LLM-based systems. Building on this observation, we propose LangMARL, a framework that brings credit assignment and policy gradient evolution from cooperative MARL into the language space. LangMARL introduces agent-level language credit assignment, pioneers gradient evolution in language space for policy improvement, and summarizes task-relevant causal relations from replayed trajectories to provide dense feedback and improve convergence under sparse rewards. Extensive experiments across diverse cooperative multi-agent tasks demonstrate improved sample efficiency, interpretability, and strong generalization.
Huaiyuan Yao, Longchao Da, Xiaoou Liu +3
Arizona State University · Cisco Research · University of North Carolina at Chapel Hill
Multi-agent reinforcement learning (MARL) commonly relies on a centralized critic to estimate the value function. However, learning such a critic from scratch is highly sample-inefficient and often lacks generalization across environments. At the same time, large vision-language-action models (VLAs) trained on internet-scale data exhibit strong multimodal reasoning and zero-shot generalization capabilities, yet directly deploying them for robotic execution remains computationally prohibitive, particularly in heterogeneous multi-robot systems with diverse embodiments and resource constraints. To address these challenges, we propose Multi-Agent Vision-Language-Critic Models (MA-VLCM), a framework that replaces the learned centralized critic in MARL with a pretrained vision-language model fine-tuned to evaluate multi-agent behavior. MA-VLCM acts as a centralized critic conditioned on natural language task descriptions, visual trajectory observations, and structured multi-agent state information. By eliminating critic learning during policy optimization, our approach significantly improves sample efficiency while producing compact execution policies suitable for deployment on resource-constrained robots. Results show good zero-shot return estimation on models with differing VLM backbones on in-distribution and out-of-distribution scenarios in multi-agent team settings
Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) offer a promising approach to collaborative tasks, but applying reinforcement learning to these systems remains challenging because classical multi-agent reinforcement learning assumptions do not fully capture their dependency structures, heterogeneous roles, and autoregressive outputs. This article provides a tutorial-style overview of the transition from reinforcement learning to reinforcement fine-tuning in single-agent and multi-agent settings, highlighting the differences between classical MARL and LLM-based multi-agent optimization. We introduce Multi-Agent Reinforcement Fine-Tuning (MARFT) and formalize LaMAS optimization as a Flexible Markov Game (Flex-MG). The MARFT framework combines predecessor-conditioned policy optimization, a centralized critic, parameter-efficient adapters, and token-level credit assignment. Our token-level analysis clarifies how within-action discounting affects returns and establishes consistency with action-level returns when the token discount is one. Experiments on fixed two- to four-agent mathematical and coding workflows compare MARFT with and without policy parameter sharing against IPPO. Both variants report higher mean evaluation scores in all 24 benchmark-team-size settings, while the variant without parameter sharing shows the clearest advantage in final training reward. Role-specification ablations examine how prescribed roles affect training. We conclude with application perspectives and open challenges, including dynamic workflows, sample efficiency, and system-level reliability. Code: https://github.com/jwliao-ai/MARFT.
Junwei Liao, Muning Wen, Jun Wang +1
Shanghai Jiao Tong University · Shanghai Innovation Institute · OPPO Research Institute