GARDEN: Generalist Autonomous Reward Design Through Evidence Navigation for Cooperative Multi-Agent Reinforcement Learning

Dogan Urgun

cs.AIOct 27, 2026

GARDEN: Generalist Autonomous Reward Design Through Evidence Navigation for Cooperative Multi-Agent Reinforcement Learning

Authors: Dogan Urgun

Organizations: Department of Electrical and Electronics Engineering, Karabuk University, 78050 Karabuk, Türkiye

Abstract

Cooperative multi-agent reinforcement learning (MARL) enables teams of agents, such as robots and autonomous vehicles, to learn coordinated behavior, but what such teams learn is largely determined by their reward function. Because task success alone provides too little guidance for learning, intermediate rewards are usually designed by hand, and this effort must often be repeated whenever the environment or the team changes. Large language models (LLMs) can automate the effort by writing reward functions as code, yet the resulting rewards are typically designed and tested in a single setting. To enable their reuse across settings, GARDEN (Generalist Autonomous Reward Design through Evidence Navigation) is proposed, in which one reward program is developed for a family of related scenarios and transferred, unchanged, to scenarios that were not available during its design. During development, environment code, training records, and agent behavior are investigated by an LLM reward engineer, and draft rewards are tested in trial training runs, while the learning algorithm and the rule for selecting the final reward are held fixed. On unseen scenarios of the Overcooked and SMAX benchmarks, task returns were increased by 58.2% and 52.5% over the default rewards on two kitchen layouts, and the mean win rate in two combat scenarios was raised from 51.0% to 89.9%. Under matched candidate and probe budgets, this exceeded the 84.2% and 79.2% obtained with fixed diagnostic feedback and Eureka-inspired reward reflection. These results show that a single autonomously engineered reward can be reused effectively across related cooperative scenarios.

Figures & tables

Fig. 1: Overview of GARDEN for one task family. Candidates are developed in independent sessions from shared development evidence and private learning probes (1, 2), assessed with the fixed learner (3), and added to the evidence (4). After

G

generations, one program is selected by a fixed rule (5), frozen (6), and used to train new policies on the held-out scenarios, with native shaping as the reference (7). Green marks reward programs, and red marks information withheld from the reward engineer.

Approach	Agents	Main source of reward-design evidence
Text2Reward [ 7 ]	Single	Environment abstractions and human feedback
Eureka [ 8 ]	Single	Environment code and reward-component statistics from training
Self-alignment [ 10 ]	Single	Comparisons of behavior under candidate rewards
RF-Agent [ 11 ]	Single	Earlier designs and training feedback organized by tree search
Reward Design Agent [ 12 ]	Single	Trajectory assessment by a vision-language model
ARBO [ 9 ]	Single, multi	Source inspection and tool-based access to optimization history

TABLE I: Evidence Available to the LLM in Representative Reward-Design Approaches

Overcooked			SMAX
Scenario	Role	Agents	Scenario	Role	Allied vs. enemy units
Cramped Room	Development	2	3s5z vs 3s6z	Development	8 vs. 9
Bottleneck (adapted layout)	Development	2	SMACv2 5 units	Development	5 vs. 5
Coordination Ring	Development	2	5m vs 6m	Development	5 vs. 6
Forced Coordination	Held-out	2	10m vs 11m	Held-out	10 vs. 11
Counter Circuit	Held-out	2	SMACv2 10 units	Held-out	10 vs. 10

TABLE II: Development and Held-Out Scenarios

Panel of Figure 4 — Fig. 2: Held-out scenarios. (a), (b) Overcooked layouts, in which two cooks must coordinate across a dividing counter or around a central counter. (c), (d) SMAX battles with allied units in blue and enemy units in orange, against a larger marine team in (c) and with randomized unit types and positions in (d).

Fig. 3: MAPPO architectures. (a) Overcooked: feedforward convolutional networks on full-grid observations. (b) SMAX: recurrent networks with action masking. Actor parameters are shared across agents, and critics are used only during training.

Setting	Overcooked	SMAX	Setting	Overcooked	SMAX
Hidden width	64	128	PPO minibatches	16	4
Environment steps per training	4 997 120	4 997 120	Initial learning rate	$5\times 10^{-4}$	$2\times 10^{-3}$
Parallel environments	64	128	Entropy coefficient	0.01	0
Rollout length	256	128	Gradient norm bound	0.50	0.25
Episode limit	400	100	Evaluation episodes	64	256

TABLE III: MAPPO Training Settings

	Overcooked, task return (% of native)		SMAX, win rate (%)
Training reward	Forced Coordination	Counter Circuit	10m vs 11m	SMACv2 10 units
Native shaping	$100.0\pm 3.4$	$100.0\pm 4.3$	$27.3\pm 0.96$	$74.6\pm 1.80$
GARDEN	$158.2\pm 6.8$	$152.5\pm 15.8$	$96.1\pm 0.90$	$83.6\pm 0.86$
Gain over native	$+58.2$ %	$+52.5$ %	$+68.8$ pp	$+9.0$ pp

TABLE IV: Held-Out Performance of Native Shaping and Frozen GARDEN Rewards

Method	10m vs 11m	SMACv2 10 units	Mean	Model calls	Input tokens (M)
Native shaping	$27.3\pm 0.96$	$74.6\pm 1.80$	51.0	–	–
Fixed diagnostics	$91.2\pm 0.74$	$77.1\pm 0.88$	84.2	18	3.5
Eureka-inspired reflection	$87.6\pm 0.92$	$70.8\pm 0.94$	79.2	18	3.3
GARDEN	$96.1\pm 0.90$	$83.6\pm 0.86$	$89.9$	155	15.0

TABLE V: SMAX Held-Out Win Rate and LLM Usage of the Reward-Design Methods

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 265 papers

Mar 25, 2026

Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning

Designing effective auxiliary rewards for cooperative multi-agent systems remains challenging, as misaligned incentives can induce suboptimal coordination, particularly when sparse task rewards provide insufficient grounding for coordinated behavior. This study introduces an autonomous reward design framework that uses large language models (LLMs) to synthesize executable reward programs from environment instrumentation. The procedure constrains candidate programs within a formal validity envelope and trains policies from scratch using Multi-Agent Proximal Policy Optimization (MAPPO) under a fixed computational budget. The candidates are then evaluated on the basis of their performance, and selection across generations solely based on the sparse task returns. The framework is evaluated in four Overcooked-AI layouts characterized by varying levels of corridor congestion, handoff dependencies, and structural asymmetries. The proposed reward design approach consistently yields higher task returns and delivery counts, with the most pronounced gains observed in environments dominated by interaction bottlenecks. Diagnostic analysis of the synthesized shaping components reveals stronger interdependence in action selection and improved signal alignment in coordination-intensive tasks. These results demonstrate that the proposed LLM-guided reward search framework mitigates the need for manual engineering while producing shaping signals compatible with cooperative learning under finite budgets.

Dogan Urgun, Gokhan Gungor

Department of Electrical and Electronics Engineering, Karabuk University, 78050 Karabuk, Türkiye · Department of Mechatronics Engineering, Karabuk University, 78050 Karabuk, Türkiye

Reward Design for RL Cooperative MARL Multi-Agent Reinforcement Learning LLM-Guided RL

Ask PDF

May 28, 2026

LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning

Effective training-time guidance is central to multi-agent reinforcement learning (MARL), yet remains difficult in sparse-reward settings where weak supervision limits coordination and policy improvement, and existing methods often require substantial domain expertise or manual design effort. Large language models (LLMs) provide a promising alternative for flexible learning-signal design, yet existing LLM-based methods remain largely single-agent-oriented, one-shot, or weakly validated for the evolving training dynamics of cooperative MARL. To address these limitations, we propose LLM-ALSO, an iterative LLM-driven adaptive learning-signal optimization framework for MARL. Rather than directly deploying LLM-generated rewards, LLM-ALSO decomposes adaptation into iterative diagnosis, proposal, and validation: a Critic LLM diagnoses stage-specific learning and coordination failures from sparse-return metrics and compact behavior evidence, a Generator LLM proposes candidate reward-shaping configurations conditioned on the diagnosis, and branch-validation feedback refines candidates before they affect the main training trajectory. Through short-horizon validation and stage-aware adaptation, LLM-ALSO promotes only validated updates into training, reducing the risk of unreliable LLM-generated modifications. Experiments on sparse-reward cooperative MARL tasks show that LLM-ALSO improves sparse-evaluation performance and learning efficiency.

Xiaoguang Wu, Zhi Zheng, Hui Xiong

University of Science and Technology of China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · 2Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)

Reward Shaping LLM-Guided RL Sparse-Reward RL Cooperative MARL Reward Design for RL

Ask PDF

Apr 1, 2026

LangMARL: Natural Language Multi-Agent Reinforcement Learning

Large language model (LLM) agents struggle to autonomously evolve coordination strategies in dynamic environments, largely because coarse global outcomes obscure the causal signals needed for local policy refinement. We identify this bottleneck as a multi-agent credit assignment problem, which has long been studied in classical multi-agent reinforcement learning (MARL) but remains underaddressed in LLM-based systems. Building on this observation, we propose LangMARL, a framework that brings credit assignment and policy gradient evolution from cooperative MARL into the language space. LangMARL introduces agent-level language credit assignment, pioneers gradient evolution in language space for policy improvement, and summarizes task-relevant causal relations from replayed trajectories to provide dense feedback and improve convergence under sparse rewards. Extensive experiments across diverse cooperative multi-agent tasks demonstrate improved sample efficiency, interpretability, and strong generalization.

Huaiyuan Yao, Longchao Da, Xiaoou Liu +3

Arizona State University · Cisco Research · University of North Carolina at Chapel Hill

RL for LLM Agents Multi-Agent Reinforcement Learning Cooperative MARL

Ask PDF

Abstract

Reward Design for RL

Explore similar work

Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning

LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning

LangMARL: Natural Language Multi-Agent Reinforcement Learning