cs.CLApr 1, 2026
SaveLangMARL: Natural Language Multi-Agent Reinforcement Learning
Organizations: Arizona State University · Cisco Research · University of North Carolina at Chapel Hill
Abstract
Large language model (LLM) agents struggle to autonomously evolve coordination strategies in dynamic environments, largely because coarse global outcomes obscure the causal signals needed for local policy refinement. We identify this bottleneck as a multi-agent credit assignment problem, which has long been studied in classical multi-agent reinforcement learning (MARL) but remains underaddressed in LLM-based systems. Building on this observation, we propose LangMARL, a framework that brings credit assignment and policy gradient evolution from cooperative MARL into the language space. LangMARL introduces agent-level language credit assignment, pioneers gradient evolution in language space for policy improvement, and summarizes task-relevant causal relations from replayed trajectories to provide dense feedback and improve convergence under sparse rewards. Extensive experiments across diverse cooperative multi-agent tasks demonstrate improved sample efficiency, interpretability, and strong generalization.
Figures & tables
Figure 1: Challenges in multi-agent credit assignment. Global evaluation fails to pinpoint individual contributions, leading to ambiguous reflections. LangMARL addresses this by decomposing team performance into agent-specific credits.
Figure 2: An Easy-to-Use Toolkit for LangMARL. LangMARL mirrors the syntax and abstractions of classical MARL libraries (e.g., TorchRL), redefining core components in natural language space, making LLM-based multi-agent optimization as straightforward to implement as standard deep RL pipelines.
| Paradigm | Method | Self-Evolving | Central Critic | Text Gradient | Credit Assignment |
| Static Prompting | CoT [ 12 ] | ✘ | ✘ | ✘ | ✘ |
| Agents [ 13 ] | ✘ | ✘ | ✘ | ✘ | |
| Single-Agent Self-Evolving | Auto Prompt Engineer (AutoPE) [ 14 ] | ✔ | ✔ | ✘ | ✘ |
| DSPy [ 15 ] | ✔ | ✘ | ✘ | ✘ | |
| Reflexion [ 16 ] | ✔ | ✔ | ✘ | ✘ | |
| TextGrad [ 17 ] | ✔ | ✔ | ✔ | ✘ |
Table 1: Comparison across prompting and self-evolving paradigms. LangMARL provides the most comprehensive framework by unifying self-evolution, centralized critique, and explicit credit assignment.
Figure 3: The LangMARL System Pipeline. The framework follows a CTDE paradigm: (i) Language Policy Actors execute decentralized actions, (ii) a Centralized Language Critic assigns trajectory-level causal credits, and (iii) the Language Policy Optimizer updates policies in natural language.
| Category | Static-Prompting | Self-Evolving | Ours | |||||
| Method | CoT | Agents | AutoPE | DSPy | Reflexion | TextGrad | Symbolic | LangMARL |
| Natural Language Benchmarks (Accuracy / Pass Rate) | ||||||||
| HotPotQA | 38.8 | 37.5 | 39.8 | 43.9 | 59.1 | 57.3 | 44.8 | 60.2 |
| MATH | 23.2 | 23.8 | 22.5 | 17.3 | 49.4 | 53.8 | 38.8 | 56.0 |
| HumanEval | 59.2 | 59.5 | 63.5 | 66.7 | 70.1 | 68.9 | 64.5 | 73.2 |
| Multi-Agent Game: Overcooked-AI (Mean Reward) | ||||||||
Table 2: Comprehensive Comparison across QA, Reasoning, Coding and Multi-Agent Environments. All reported values represent the mean performance after 5 training iterations using the same backbone LLMs.
Figure 4: Impact of credit assignment on the convergence quality of LangMARL. The learning curves across five benchmark tasks demonstrate that credit assignment is pivotal for efficient policy optimization and superior final performance. Without this mechanism (green lines), the models exhibit sub-optimal learning rates and significant instability, particularly in complex reasoning and multi-agent coordination scenarios.
Figure 5: Emergent role specialization in LangMARL. Top: In the initial symmetric setup, both agents operate as generic problem solvers without predefined role differentiation. Bottom: After iterative trajectory-level credit assignment and language-based policy optimization, agents self-organize into complementary roles, such as in the coding scenario, Agent 1 specializing in structured implementation and Agent 2 in critical evaluation and refinement. This division of labor is not explicitly specified in the prompt but emerges through centralized language credits.
Figure 6: Sensitivity analysis of LangMARL components. (a) Performance comparison of various LLM backbones against the best baseline Reflexion. (b-c) Impact of Monte Carlo rollout counts on HumanEval and Pistonball.
Figure 7: Ablation study on credit assignment.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Comparison of optimization paradigms in multi-agent systems. While existing methods rely on direct updates (1), shared memory (2), or collective global evaluations (3), LangMARL (4) introduces an explicit individual credit assignment mechanism. This allows the centralized critic to decompose global feedback into agent-specific guidance, facilitating more precise policy optimization in collaborative tasks.
Figure 9: Overcooked-AI Layouts. From left to right: Asymmetric Advantages tests the adoption of high-level strategies based on individual strengths; Cramped Room presents low-level coordination challenges within a confined, collision-prone space; Coordination Ring requires synchronized navigation between opposite corners; Forced Coordination removes collision friction to focus on joint strategy, as neither agent can serve dishes alone; and Counter Circuit necessitates a non-obvious strategy of passing ingredients over counters rather than carrying them.
Figure 10: Learning curves on Overcooked-AI layouts. Comparison of LangMARL with (orange) and without (blue) credit assignment. Results across five maps show that language-level credit assignment significantly improves mean reward and convergence stability, especially in layouts requiring high-degree coordination (e.g., Coord. Ring and Forced Coord.).
Figure 11: Pistonball Layouts. The upper panel illustrates a failure case where poor coordination among piston agents leads to an uneven surface, trapping the ball and resulting in a significantly low negative score due to the accumulated time penalty. In contrast, the lower panel demonstrates successful coordination, where agents synchronously adjust their vertical positions to smoothly propel the ball toward the target.
Figure 12: Examples of language-based credit assignment and policy optimization in LangMARL. (a) Overcooked: the centralized language critic analyzes a full trajectory and produces agent-specific credits, which are then used to revise the language policy of Player Green, resulting in improved coordination with Player Blue. (b) Pistonball: the language critic identifies a causally responsible piston from the trajectory and assigns targeted credit, leading to a refined local policy that corrects blocking behavior.
More in this topic
RL for LLM Agents
Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.
See all 603 papers