Policy Search in Code Space for Multi-Agent Pathfinding

Matthias Eder; Pantelis Emmanouil; Vivek Farias; Fabian Fischer; Adam Jozefiak; Tianyi Peng; Thomas Ulz; Raphael Watschinger

cs.ROOct 19, 2026

Policy Search in Code Space for Multi-Agent Pathfinding

Authors: Matthias Eder, Pantelis Emmanouil, Vivek Farias, Fabian Fischer, Adam Jozefiak, Tianyi Peng, Thomas Ulz, Raphael Watschinger,

Organizations: KNAPP · MIT · Columbia University · University of British Columbia

Abstract

Developing coordination policies for complex multi-agent systems relies on either labor-intensive algorithm design or opaque task-specific reinforcement learning. With recent advances in autonomous coding agents, we show that policy search directly in code space offers a compelling third path that combines automated optimization with explicit interpretability. We investigate a human-directed workflow in which coding agents iteratively propose, implement, evaluate, diagnose, and eventually refine algorithms guided by simulation feedback and explicit correctness verification. We conducted two case studies and found that this workflow robustly outperformed any manually engineered solutions. On a challenging map in the 2024 League of Robot Runners competition, a two-day search discovered a policy that exceeds the stronger locally reproduced competition entry by 35.0% on the densest instance and generalizes to unseen map instances. In simulated real-world warehouses with irregular lanes, continuous robot geometry, and asynchronous motion, a week-long search discovered an entirely new class of multi-agent pathfinding algorithm that outperforms a calibrated proprietary solution in throughput by 58.9%. Under stated assumptions, we further prove that the new algorithm is collision-free and completes, in finite time, every accepted goal that is not withdrawn or replaced. Together, the studies show that policy search in code space powered by state-of-the-art coding agents offers a highly competitive path to building real engineering solutions that are both effective and analytically tractable.

Figures & tables

Figure 1: Throughput of the discovered policies across milestone commits. Human-directed coding agents iteratively refine coordination algorithms, producing policies that outperform strong manually engineered baselines in competitive multi-agent pathfinding problems. Maps above are illustrative.

Panel of Figure 2 — Figure 2: Policy search in code space. The user provides constraints, performance objectives, and an initial codebase. The coding agent iteratively modifies source code, benchmarks candidates in simulation, and inspects traces to guide subsequent revisions.

	Top LoRR Entries				PLATM
	NMS		KK		Mean $\pm$	S.D.	Min	Max	Gain
RANDOM-01	644.8 $\pm$	8.1	687.1 $\pm$	6.8	736.8 $\pm$	10.4	724	754	$+7.22\%$
RANDOM-02	1,207.6 $\pm$	16.1	1,219.1 $\pm$	18.0	1,356.9 $\pm$	14.6	1,332	1,380	$+11.30\%$
RANDOM-03	2,384.1 $\pm$	42.3	1,956.6 $\pm$	20.1	2,647.5 $\pm$	31.4	2,606	2,703	$+11.05\%$
RANDOM-04	2,610.8 $\pm$	33.6	1,279.9 $\pm$	46.4	2,790.1 $\pm$	54.9	2,716	2,886	$+6.87\%$
RANDOM-05	3,150.5 $\pm$	69.7	2,065.3 $\pm$	25.5	4,253.3 $\pm$	59.8	4,170	4,331	$+35.00\%$

Table 1: Tasks completed by PLATM and the top LoRR entries, as mean

\pm

s.d. over eight paired trials.

Panel of Figure 4 — Figure 4: Policy search trajectory and computational cost of PLATM on RANDOM-05 (800 robots).

Figure 5: An illustrative warehouse layout. Black lanes connect waypoints, green markers denote storage nodes, and orange diamonds denote picking stations.

Figure 6: The six warehouse layouts. Maps are mirrored, and some rotated, for anonymity.

			KNAPP		CGAR
	$n$ 1	$d$ 1	Mean $\pm$	S.D.	Mean $\pm$	S.D.	Min	Max	$\mu/n$ 1	Gain
WAREHOUSE-01	65	51.8	487.9 $\pm$	51.9	775.3 $\pm$	25.4	733.0	808.5	11.93	$+58.9\%$
WAREHOUSE-02	17	43.5	(127.0 $\pm$	49.6) 2	278.6 $\pm$	2.7	275.0	282.0	16.39	( $+119.4\%$ ) 2
WAREHOUSE-03	8	21.4	(17.7 $\pm$	14.8) 2	135.1 $\pm$	5.8	128.5	144.0	16.89	( $+664.0\%$ ) 2
WAREHOUSE-04	19	49.3	(79.8 $\pm$	24.6) 2	269.4 $\pm$	8.7	254.5	285.0	14.18	( $+237.6\%$ ) 2
WAREHOUSE-05	13	38.3	(28.4 $\pm$	7.3) 2	191.4 $\pm$	7.5	177.5	201.5	14.73	( $+574.7\%$ ) 2

Table 2: Orders completed per hour by CGAR and KNAPP, as mean

\pm

s.d. over eight paired trials.

Panel of Figure 7 — Figure 7: Policy search trajectory and computational cost of CGAR on WAREHOUSE-01 (65 robots).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The core graph of WAREHOUSE-00, where

H=P

. Hollow purple circles mark its 17 core nodes; purple curves show core paths projected onto the ground network, omitting headings and rotations.

Panel of Figure 9 — Figure 9: CGAR motion selection. Committed motions must be collision-free and preserve a recovery plan. If a goal remains the oldest active goal for $D$ seconds, normal routing pauses for the fallback.

	$n$	Mean $\pm$	S.D.	Min	Max
RANDOM-01	100	736.8 $\pm$	10.4	724	754
RANDOM-02	200	1,356.9 $\pm$	14.6	1,332	1,380
RANDOM-03	400	2,647.5 $\pm$	31.4	2,606	2,703
RANDOM-04	700	2,790.1 $\pm$	54.9	2,716	2,886
RANDOM-05	800	4,253.3 $\pm$	59.8	4,170	4,331
RANDOM-06	100	746.4 $\pm$	10.9	724	761

Appendix

Table 3: Tasks completed by PLATM on the competition layout and, after map-specific calibration, on the additional layout, as mean

\pm

s.d. over eight paired trials. The RANDOM-09 and RANDOM-10 profiles received extended calibration.

Figure 11: WAREHOUSE-00, the synthetic layout evaluated here. Black lanes connect waypoints, green markers denote storage nodes, and orange diamonds denote picking stations. Blue vehicles illustrate poses and are not the evaluation’s initial state.

	$n$	$d$	Mean $\pm$	S.D.	Min	Max
WAREHOUSE-01	65	51.8	11.93 $\pm$	0.39	11.28	12.44
WAREHOUSE-02	17	43.5	16.39 $\pm$	0.16	16.18	16.59
WAREHOUSE-03	8	21.4	16.89 $\pm$	0.72	16.06	18.00
WAREHOUSE-04	19	49.3	14.18 $\pm$	0.46	13.39	15.00
WAREHOUSE-05	13	38.3	14.73 $\pm$	0.57	13.65	15.50
WAREHOUSE-06	32	74.2	9.52 $\pm$	0.39	9.06	10.02

Appendix

Table 4: Orders completed per hour per robot by CGAR, as mean

\pm

s.d. over eight trials. The symbols

n

and

d

are those of Table 2 .

Multi-Agent Path Finding

Multi-agent path finding (MAPF) plans collision-free routes for multiple agents sharing an environment.

See all 93 papers

MARS $^2$ : Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation
Apr 16, 2026Pengfei Li, Shijie Wang, Fangyuan Li +7RL for Code Generation Multi-Agent Collaboration
Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute · Harbin Institute of Technology +1
CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents
Mar 18, 2026Lintang Sutawika, Aditya Bharat Soni, Bharath Sriraam R R +8Code Localization Code Search
Carnegie Mellon University · Independent · OpenHands
Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models
Mar 10, 2026Daniel Hennes, Zun Li, John Schultz +1Multi-Agent Reinforcement Learning Interpretability in RL
Google DeepMind

Apr 16, 2026J/K move · Enter open · S save

MARS $^2$ : Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation

Pengfei Li, Shijie Wang, Fangyuan Li, Yikun Fu+6

Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute · Harbin Institute of Technology+1

RL for Code Generation Multi-Agent Collaboration Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Reinforcement learning (RL) paradigms have demonstrated strong performance on reasoning-intensive tasks such as code generation. However, limited trajectory diversity often leads to diminishing returns, which constrains the achievable performance ceiling. Search-enhanced RL alleviates this issue by introducing structured exploration, which remains constrained by the single-agent policy priors. Meanwhile, leveraging multiple interacting policies can acquire more diverse exploratory signals, but existing approaches are typically decoupled from structured search. We propose \textbf{MARS

^2

} (Multi-Agent Reinforced Tree-Search Scaling), a unified RL framework in which multiple independently-optimized agents collaborate within a shared tree-structured search environment. MARS

^2

models the search tree as a learnable multi-agent interaction environment, enabling heterogeneous agents to collaboratively generate and refine candidate solutions within a shared search topology. To support effective learning, we introduce a path-level group advantage formulation based on tree-consistent reward shaping, which facilitates effective credit assignment across complex search trajectories. Experiments on code generation benchmarks show that MARS

^2

consistently improves performance across diverse model combinations and training settings, demonstrating the effectiveness of coupling multi-agent collaboration with tree search for enhancing reinforcement learning. Our code is publicly available at https://github.com/TsinghuaC3I/MARTI.

Abstract

Multi-Agent Path Finding

Explore similar work

MARS2^22: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation

CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents

Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models

MARS $^2$ : Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation