Developing coordination policies for complex multi-agent systems relies on either labor-intensive algorithm design or opaque task-specific reinforcement learning. With recent advances in autonomous coding agents, we show that policy search directly in code space offers a compelling third path that combines automated optimization with explicit interpretability. We investigate a human-directed workflow in which coding agents iteratively propose, implement, evaluate, diagnose, and eventually refine algorithms guided by simulation feedback and explicit correctness verification. We conducted two case studies and found that this workflow robustly outperformed any manually engineered solutions. On a challenging map in the 2024 League of Robot Runners competition, a two-day search discovered a policy that exceeds the stronger locally reproduced competition entry by 35.0% on the densest instance and generalizes to unseen map instances. In simulated real-world warehouses with irregular lanes, continuous robot geometry, and asynchronous motion, a week-long search discovered an entirely new class of multi-agent pathfinding algorithm that outperforms a calibrated proprietary solution in throughput by 58.9%. Under stated assumptions, we further prove that the new algorithm is collision-free and completes, in finite time, every accepted goal that is not withdrawn or replaced. Together, the studies show that policy search in code space powered by state-of-the-art coding agents offers a highly competitive path to building real engineering solutions that are both effective and analytically tractable.
Figures & tables
Figure 1: Throughput of the discovered policies across milestone commits. Human-directed coding agents iteratively refine coordination algorithms, producing policies that outperform strong manually engineered baselines in competitive multi-agent pathfinding problems. Maps above are illustrative.
Figure 2: Policy search in code space. The user provides constraints, performance objectives, and an initial codebase. The coding agent iteratively modifies source code, benchmarks candidates in simulation, and inspects traces to guide subsequent revisions.
Top LoRR Entries
PLATM
NMS
KK
Mean ±
S.D.
Min
Max
Gain
RANDOM-01
644.8 ±
8.1
687.1 ±
6.8
736.8 ±
10.4
724
754
+7.22%
RANDOM-02
1,207.6 ±
16.1
1,219.1 ±
18.0
1,356.9 ±
14.6
1,332
1,380
+11.30%
RANDOM-03
2,384.1 ±
42.3
1,956.6 ±
20.1
2,647.5 ±
31.4
2,606
2,703
+11.05%
RANDOM-04
2,610.8 ±
33.6
1,279.9 ±
46.4
2,790.1 ±
54.9
2,716
2,886
+6.87%
RANDOM-05
3,150.5 ±
69.7
2,065.3 ±
25.5
4,253.3 ±
59.8
4,170
4,331
+35.00%
Table 1: Tasks completed by PLATM and the top LoRR entries, as mean ± s.d. over eight paired trials.
Figure 4: Policy search trajectory and computational cost of PLATM on RANDOM-05 (800 robots).
Figure 5: An illustrative warehouse layout. Black lanes connect waypoints, green markers denote storage nodes, and orange diamonds denote picking stations.
Figure 6: The six warehouse layouts. Maps are mirrored, and some rotated, for anonymity.
KNAPP
CGAR
n 1
d 1
Mean ±
S.D.
Mean ±
S.D.
Min
Max
μ/n 1
Gain
WAREHOUSE-01
65
51.8
487.9 ±
51.9
775.3 ±
25.4
733.0
808.5
11.93
+58.9%
WAREHOUSE-02
17
43.5
(127.0 ±
49.6) 2
278.6 ±
2.7
275.0
282.0
16.39
( +119.4% ) 2
WAREHOUSE-03
8
21.4
(17.7 ±
14.8) 2
135.1 ±
5.8
128.5
144.0
16.89
( +664.0% ) 2
WAREHOUSE-04
19
49.3
(79.8 ±
24.6) 2
269.4 ±
8.7
254.5
285.0
14.18
( +237.6% ) 2
WAREHOUSE-05
13
38.3
(28.4 ±
7.3) 2
191.4 ±
7.5
177.5
201.5
14.73
( +574.7% ) 2
Table 2: Orders completed per hour by CGAR and KNAPP, as mean ± s.d. over eight paired trials.
Figure 7: Policy search trajectory and computational cost of CGAR on WAREHOUSE-01 (65 robots).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The core graph of WAREHOUSE-00, where H=P . Hollow purple circles mark its 17 core nodes; purple curves show core paths projected onto the ground network, omitting headings and rotations.
Figure 9: CGAR motion selection. Committed motions must be collision-free and preserve a recovery plan. If a goal remains the oldest active goal for D seconds, normal routing pauses for the fallback.
n
Mean ±
S.D.
Min
Max
RANDOM-01
100
736.8 ±
10.4
724
754
RANDOM-02
200
1,356.9 ±
14.6
1,332
1,380
RANDOM-03
400
2,647.5 ±
31.4
2,606
2,703
RANDOM-04
700
2,790.1 ±
54.9
2,716
2,886
RANDOM-05
800
4,253.3 ±
59.8
4,170
4,331
RANDOM-06
100
746.4 ±
10.9
724
761
Appendix
Table 3: Tasks completed by PLATM on the competition layout and, after map-specific calibration, on the additional layout, as mean ± s.d. over eight paired trials. The RANDOM-09 and RANDOM-10 profiles received extended calibration.
Figure 11: WAREHOUSE-00, the synthetic layout evaluated here. Black lanes connect waypoints, green markers denote storage nodes, and orange diamonds denote picking stations. Blue vehicles illustrate poses and are not the evaluation’s initial state.
n
d
Mean ±
S.D.
Min
Max
WAREHOUSE-01
65
51.8
11.93 ±
0.39
11.28
12.44
WAREHOUSE-02
17
43.5
16.39 ±
0.16
16.18
16.59
WAREHOUSE-03
8
21.4
16.89 ±
0.72
16.06
18.00
WAREHOUSE-04
19
49.3
14.18 ±
0.46
13.39
15.00
WAREHOUSE-05
13
38.3
14.73 ±
0.57
13.65
15.50
WAREHOUSE-06
32
74.2
9.52 ±
0.39
9.06
10.02
Appendix
Table 4: Orders completed per hour per robot by CGAR, as mean ± s.d. over eight trials. The symbols n and d are those of Table 2 .
Reinforcement learning (RL) paradigms have demonstrated strong performance on reasoning-intensive tasks such as code generation. However, limited trajectory diversity often leads to diminishing returns, which constrains the achievable performance ceiling. Search-enhanced RL alleviates this issue by introducing structured exploration, which remains constrained by the single-agent policy priors. Meanwhile, leveraging multiple interacting policies can acquire more diverse exploratory signals, but existing approaches are typically decoupled from structured search. We propose \textbf{MARS2} (Multi-Agent Reinforced Tree-Search Scaling), a unified RL framework in which multiple independently-optimized agents collaborate within a shared tree-structured search environment. MARS2 models the search tree as a learnable multi-agent interaction environment, enabling heterogeneous agents to collaboratively generate and refine candidate solutions within a shared search topology. To support effective learning, we introduce a path-level group advantage formulation based on tree-consistent reward shaping, which facilitates effective credit assignment across complex search trajectories. Experiments on code generation benchmarks show that MARS2 consistently improves performance across diverse model combinations and training settings, demonstrating the effectiveness of coupling multi-agent collaboration with tree search for enhancing reinforcement learning. Our code is publicly available at https://github.com/TsinghuaC3I/MARTI.
Pengfei Li, Shijie Wang, Fangyuan Li +7
Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute · Harbin Institute of Technology +1
A prerequisite for coding agents to perform tasks on large repositories is code localization - the identification of relevant files, classes, and functions to work on. While repository-level code localization has been performed using embedding-based retrieval approaches such as vector search, recent work has focused on developing agents to localize relevant code either as a standalone precursor to or interleaved with performing actual work. Most prior methods on agentic code search equip the agent with complex, specialized tools, such as repository graphs derived from static analysis. In this paper, we demonstrate that, with an effective reinforcement learning recipe, a coding agent equipped with nothing more than a standard Unix terminal can be trained to achieve strong results. Our experiments on three benchmarks (SWE-Bench Verified, Pro, and Lite) reveal that our models consistently achieve superior or competitive performance over 2-18x larger base and post-trained LLMs and sometimes approach performance provided by closed models like Claude Sonnet, even when using specialized scaffolds. Our work particularly focuses on techniques for re-purposing existing coding agent environments for code search, reward design, and RL optimization. We release the resulting model family, CodeScout, along with all our code and data for the community to build upon.
Lintang Sutawika, Aditya Bharat Soni, Bharath Sriraam R R +8
Carnegie Mellon University · Independent · OpenHands
Recent advances in multi-agent reinforcement learning, particularly Policy-Space Response Oracles (PSRO), have enabled the computation of approximate game-theoretic equilibria in increasingly complex domains. However, these methods rely on deep reinforcement learning oracles that produce `black-box' neural network policies, making them difficult to interpret, trust or debug. We introduce Code-Space Response Oracles (CSRO), a novel framework that addresses this challenge by replacing RL oracles with Large Language Models (LLMs). CSRO reframes the best response computation as a code generation task, prompting an LLM to generate policies directly as human-readable code. This approach not only yields inherently interpretable policies but also leverages the LLM's pretrained knowledge to discover complex, human-like strategies. We explore multiple ways to construct and enhance an LLM-based oracle: zero-shot prompting, iterative refinement and AlphaEvolve, a distributed LLM-based evolutionary system. We demonstrate that CSRO achieves performance competitive with baselines while producing a diverse set of explainable policies. Our work presents a new perspective on multi-agent learning, shifting the focus from optimizing opaque policy parameters to synthesizing interpretable algorithmic behavior.