GARDEN: Generalist Autonomous Reward Design Through Evidence Navigation for Cooperative Multi-Agent Reinforcement Learning

Dogan Urgun

cs.AIOct 27, 2026

GARDEN: Generalist Autonomous Reward Design Through Evidence Navigation for Cooperative Multi-Agent Reinforcement Learning

Authors: Dogan Urgun

Organizations: Department of Electrical and Electronics Engineering, Karabuk University, 78050 Karabuk, Türkiye

Abstract

Cooperative multi-agent reinforcement learning (MARL) enables teams of agents, such as robots and autonomous vehicles, to learn coordinated behavior, but what such teams learn is largely determined by their reward function. Because task success alone provides too little guidance for learning, intermediate rewards are usually designed by hand, and this effort must often be repeated whenever the environment or the team changes. Large language models (LLMs) can automate the effort by writing reward functions as code, yet the resulting rewards are typically designed and tested in a single setting. To enable their reuse across settings, GARDEN (Generalist Autonomous Reward Design through Evidence Navigation) is proposed, in which one reward program is developed for a family of related scenarios and transferred, unchanged, to scenarios that were not available during its design. During development, environment code, training records, and agent behavior are investigated by an LLM reward engineer, and draft rewards are tested in trial training runs, while the learning algorithm and the rule for selecting the final reward are held fixed. On unseen scenarios of the Overcooked and SMAX benchmarks, task returns were increased by 58.2% and 52.5% over the default rewards on two kitchen layouts, and the mean win rate in two combat scenarios was raised from 51.0% to 89.9%. Under matched candidate and probe budgets, this exceeded the 84.2% and 79.2% obtained with fixed diagnostic feedback and Eureka-inspired reward reflection. These results show that a single autonomously engineered reward can be reused effectively across related cooperative scenarios.

Figures & tables

Fig. 1: Overview of GARDEN for one task family. Candidates are developed in independent sessions from shared development evidence and private learning probes (1, 2), assessed with the fixed learner (3), and added to the evidence (4). After

G

generations, one program is selected by a fixed rule (5), frozen (6), and used to train new policies on the held-out scenarios, with native shaping as the reference (7). Green marks reward programs, and red marks information withheld from the reward engineer.

Approach	Agents	Main source of reward-design evidence
Text2Reward [ 7 ]	Single	Environment abstractions and human feedback
Eureka [ 8 ]	Single	Environment code and reward-component statistics from training
Self-alignment [ 10 ]	Single	Comparisons of behavior under candidate rewards
RF-Agent [ 11 ]	Single	Earlier designs and training feedback organized by tree search
Reward Design Agent [ 12 ]	Single	Trajectory assessment by a vision-language model
ARBO [ 9 ]	Single, multi	Source inspection and tool-based access to optimization history

TABLE I: Evidence Available to the LLM in Representative Reward-Design Approaches

Overcooked			SMAX
Scenario	Role	Agents	Scenario	Role	Allied vs. enemy units
Cramped Room	Development	2	3s5z vs 3s6z	Development	8 vs. 9
Bottleneck (adapted layout)	Development	2	SMACv2 5 units	Development	5 vs. 5
Coordination Ring	Development	2	5m vs 6m	Development	5 vs. 6
Forced Coordination	Held-out	2	10m vs 11m	Held-out	10 vs. 11
Counter Circuit	Held-out	2	SMACv2 10 units	Held-out	10 vs. 10

TABLE II: Development and Held-Out Scenarios

Panel of Figure 4 — Fig. 2: Held-out scenarios. (a), (b) Overcooked layouts, in which two cooks must coordinate across a dividing counter or around a central counter. (c), (d) SMAX battles with allied units in blue and enemy units in orange, against a larger marine team in (c) and with randomized unit types and positions in (d).

Fig. 3: MAPPO architectures. (a) Overcooked: feedforward convolutional networks on full-grid observations. (b) SMAX: recurrent networks with action masking. Actor parameters are shared across agents, and critics are used only during training.

Setting	Overcooked	SMAX	Setting	Overcooked	SMAX
Hidden width	64	128	PPO minibatches	16	4
Environment steps per training	4 997 120	4 997 120	Initial learning rate	$5\times 10^{-4}$	$2\times 10^{-3}$
Parallel environments	64	128	Entropy coefficient	0.01	0
Rollout length	256	128	Gradient norm bound	0.50	0.25
Episode limit	400	100	Evaluation episodes	64	256

TABLE III: MAPPO Training Settings

	Overcooked, task return (% of native)		SMAX, win rate (%)
Training reward	Forced Coordination	Counter Circuit	10m vs 11m	SMACv2 10 units
Native shaping	$100.0\pm 3.4$	$100.0\pm 4.3$	$27.3\pm 0.96$	$74.6\pm 1.80$
GARDEN	$158.2\pm 6.8$	$152.5\pm 15.8$	$96.1\pm 0.90$	$83.6\pm 0.86$
Gain over native	$+58.2$ %	$+52.5$ %	$+68.8$ pp	$+9.0$ pp

TABLE IV: Held-Out Performance of Native Shaping and Frozen GARDEN Rewards

Method	10m vs 11m	SMACv2 10 units	Mean	Model calls	Input tokens (M)
Native shaping	$27.3\pm 0.96$	$74.6\pm 1.80$	51.0	–	–
Fixed diagnostics	$91.2\pm 0.74$	$77.1\pm 0.88$	84.2	18	3.5
Eureka-inspired reflection	$87.6\pm 0.92$	$70.8\pm 0.94$	79.2	18	3.3
GARDEN	$96.1\pm 0.90$	$83.6\pm 0.86$	$89.9$	155	15.0

TABLE V: SMAX Held-Out Win Rate and LLM Usage of the Reward-Design Methods

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 265 papers

Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning
Mar 25, 2026Dogan Urgun, Gokhan GungorReward Design for RL Cooperative MARL
Department of Electrical and Electronics Engineering, Karabuk University, 78050 Karabuk, Türkiye · Department of Mechatronics Engineering, Karabuk University, 78050 Karabuk, Türkiye
LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning
May 28, 2026Xiaoguang Wu, Zhi Zheng, Hui XiongReward Shaping LLM-Guided RL
University of Science and Technology of China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · 2Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)
LangMARL: Natural Language Multi-Agent Reinforcement Learning
Apr 1, 2026Huaiyuan Yao, Longchao Da, Xiaoou Liu +3RL for LLM Agents Multi-Agent Reinforcement Learning
Arizona State University · Cisco Research · University of North Carolina at Chapel Hill

Mar 25, 2026J/K move · Enter open · S save

Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning

Dogan Urgun, Gokhan Gungor

Department of Electrical and Electronics Engineering, Karabuk University, 78050 Karabuk, Türkiye · Department of Mechatronics Engineering, Karabuk University, 78050 Karabuk, Türkiye

Reward Design for RL Cooperative MARL Multi-Agent Reinforcement Learning LLM-Guided RL

Open paper AskPDF arXiv

Designing effective auxiliary rewards for cooperative multi-agent systems remains challenging, as misaligned incentives can induce suboptimal coordination, particularly when sparse task rewards provide insufficient grounding for coordinated behavior. This study introduces an autonomous reward design framework that uses large language models (LLMs) to synthesize executable reward programs from environment instrumentation. The procedure constrains candidate programs within a formal validity envelope and trains policies from scratch using Multi-Agent Proximal Policy Optimization (MAPPO) under a fixed computational budget. The candidates are then evaluated on the basis of their performance, and selection across generations solely based on the sparse task returns. The framework is evaluated in four Overcooked-AI layouts characterized by varying levels of corridor congestion, handoff dependencies, and structural asymmetries. The proposed reward design approach consistently yields higher task returns and delivery counts, with the most pronounced gains observed in environments dominated by interaction bottlenecks. Diagnostic analysis of the synthesized shaping components reveals stronger interdependence in action selection and improved signal alignment in coordination-intensive tasks. These results demonstrate that the proposed LLM-guided reward search framework mitigates the need for manual engineering while producing shaping signals compatible with cooperative learning under finite budgets.

Abstract

Reward Design for RL

Explore similar work

Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning

LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning

LangMARL: Natural Language Multi-Agent Reinforcement Learning