Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning

Dogan Urgun; Gokhan Gungor

cs.LGMar 25, 2026

Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning

Authors: Dogan Urgun, Gokhan Gungor

Organizations: Department of Electrical and Electronics Engineering, Karabuk University, 78050 Karabuk, Türkiye · Department of Mechatronics Engineering, Karabuk University, 78050 Karabuk, Türkiye

Abstract

Designing effective auxiliary rewards for cooperative multi-agent systems remains challenging, as misaligned incentives can induce suboptimal coordination, particularly when sparse task rewards provide insufficient grounding for coordinated behavior. This study introduces an autonomous reward design framework that uses large language models (LLMs) to synthesize executable reward programs from environment instrumentation. The procedure constrains candidate programs within a formal validity envelope and trains policies from scratch using Multi-Agent Proximal Policy Optimization (MAPPO) under a fixed computational budget. The candidates are then evaluated on the basis of their performance, and selection across generations solely based on the sparse task returns. The framework is evaluated in four Overcooked-AI layouts characterized by varying levels of corridor congestion, handoff dependencies, and structural asymmetries. The proposed reward design approach consistently yields higher task returns and delivery counts, with the most pronounced gains observed in environments dominated by interaction bottlenecks. Diagnostic analysis of the synthesized shaping components reveals stronger interdependence in action selection and improved signal alignment in coordination-intensive tasks. These results demonstrate that the proposed LLM-guided reward search framework mitigates the need for manual engineering while producing shaping signals compatible with cooperative learning under finite budgets.

Figures & tables

Figure 1: Overview of the proposed LLM-guided reward design framework. The framework consists of an LLM-based reward generator that synthesizes candidate reward programs.

Panel of Figure 2 — Figure 2: Illustration of the CTDE paradigm for MARL. (a) Centralized training: a shared critic ( V ) utilizes global state information and joint observations to guide policy updates; (b) Decentralized execution: individual actors ( $\pi_{i}$ ) rely only on local observations for action selection, ensuring scalability in partially observable environments.

Panel of Figure 3 — Figure 3: The Overcooked-AI layouts and coordination challenges. Each environment isolates a specific facet of multi-agent cooperation: (a) Cramped Room evaluates spatial efficiency and collision avoidance in shared workspaces; (b) Forced Coordination requires strict functional specialization and inter-agent handoffs; (c) Coordination Ring tests movement synchronization to prevent bottlenecks in circular corridors; (d) Asymmetric Advantages introduces resource-specific disparities, requiring strategic role allocation based on proximity to dispensers.

Hyperparameter	Value
Actor Learning Rate	$5\times 10^{-4}$
Critic Learning Rate	$1\times 10^{-3}$
Discount Factor ( $\gamma$ )	0.99
GAE Parameter ( $\alpha$ )	0.95
PPO Clip Ratio	0.2
Entropy Coefficient	0.01

Table 1: The MAPPO hyperparameters used for training all candidates.

Layout	$J$			Deliveries			Invalid Deliveries
Layout	Baseline	Gen 1	Gen 2	Baseline	Gen 1	Gen 2	Baseline	Gen 1	Gen 2
Cramped Room	$148\pm 39$	$180\pm 25$	$188\pm 10$	$7.40$	$9.00$	$9.40$	$1.15$	$0.25$	$0.75$
Forced Coordination	$32\pm 14$	$55\pm 26$	$103\pm 29$	$1.60$	$2.75$	$5.15$	$0.80$	$1.25$	$0.30$
Coordination Ring	$13\pm 5$	$102\pm 28$	$153\pm 24$	$0.15$	$5.10$	$7.65$	$11.30$	$3.50$	$0.85$
Asymmetric Advantages	$231\pm 38$	$322\pm 36$	$381\pm 25$	$11.55$	$16.10$	$19.05$	$6.15$	$3.45$	$1.55$

Table 2: Evaluations on four Overcooked layouts.

J

denotes sparse return (mean

\pm

std across evaluation episodes). Deliveries and invalid deliveries are episode means.

Panel of Figure 4 — Figure 4: Learning curves of the sparse return $J$ during evaluation. Performance comparison between the baseline and selected candidates from the first and second generations across four layouts: (a) Cramped Room, (b) Forced Coordination, (c) Coordination Ring, and (d) Asymmetric Advantages. Shaded regions indicate variability across evaluation episodes.

Figure 5: Candidate promotion diagram. Nodes summarize evaluated candidates and objective scores, and edges indicate the promotion path used to condition subsequent generations.

Panel of Figure 6 — Figure 6: Empirical coordination diagnostics across Overcooked-AI layouts: (a) Cramped Room: action coupling (NMI) and incentive alignment ( $\rho$ ) trends indicate improved coordination; (b) Forced Coordination: a decline in payoff imbalance ( $\Delta$ ) suggests a more balanced distribution of shaping signals; (c) Coordination Ring: increases in $\rho$ and NMI reflect more synchronized behavior; and (d) Asymmetric Advantages: the NMI trajectory indicates the emergence of structured coordination patterns. The plots illustrate the progression from the baseline to Generation 2. Blue solid lines represent NMI, green dashed lines show $\rho$ , and orange dotted lines indicate $\Delta$ .

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 259 papers

LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning
May 28, 2026Xiaoguang Wu, Zhi Zheng, Hui XiongReward Shaping LLM-Guided RL
University of Science and Technology of China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · 2Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)
Training Small LLMs as Spatial Multi-Agent Policies
Aug 2, 2026Yi Mao, Andrew PerraultHierarchical RL Multi-Agent Collaboration
Department of Computer Science and Engineering, The Ohio State University
Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents
Nov 11, 2025Chih-Hsuan, Yang, Tanwi Mallick +5Multi-Agent LLM Systems Cooperative Game Theory
Bella · Argonne National Laboratory · & University of Chicago

May 28, 2026J/K move · Enter open · S save

LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning

Xiaoguang Wu, Zhi Zheng, Hui Xiong

University of Science and Technology of China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · 2Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)

Reward Shaping LLM-Guided RL Sparse-Reward RL Cooperative MARL Reward Design for RL

Open paper AskPDF arXiv

Effective training-time guidance is central to multi-agent reinforcement learning (MARL), yet remains difficult in sparse-reward settings where weak supervision limits coordination and policy improvement, and existing methods often require substantial domain expertise or manual design effort. Large language models (LLMs) provide a promising alternative for flexible learning-signal design, yet existing LLM-based methods remain largely single-agent-oriented, one-shot, or weakly validated for the evolving training dynamics of cooperative MARL. To address these limitations, we propose LLM-ALSO, an iterative LLM-driven adaptive learning-signal optimization framework for MARL. Rather than directly deploying LLM-generated rewards, LLM-ALSO decomposes adaptation into iterative diagnosis, proposal, and validation: a Critic LLM diagnoses stage-specific learning and coordination failures from sparse-return metrics and compact behavior evidence, a Generator LLM proposes candidate reward-shaping configurations conditioned on the diagnosis, and branch-validation feedback refines candidates before they affect the main training trajectory. Through short-horizon validation and stage-aware adaptation, LLM-ALSO promotes only validated updates into training, reducing the risk of unreliable LLM-generated modifications. Experiments on sparse-reward cooperative MARL tasks show that LLM-ALSO improves sparse-evaluation performance and learning efficiency.

Abstract

Reward Design for RL

Explore similar work

LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning

Training Small LLMs as Spatial Multi-Agent Policies

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents