cs.MAOct 15, 2026

Learning Self-Orchestration in Decentralized Agent Swarms

Authors: Guowei Xu, Zhenting Qi, Amani Kiruga, Himabindu Lakkaraju, Yilun Du

Organizations: Harvard University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The capabilities of an agent swarm depend on both its individual members and how they coordinate. Yet existing multi-agent systems often train agents within fixed orchestration structures, separating the learning of individual capabilities from the organization of collective behavior. We introduce market-based post-training (MPT), a framework for jointly learning action policies and self-orchestration in decentralized agent swarms. Agents express their willingness to participate through bids, which determine who acts and how collective outcomes provide local training credit. Learning both action and bidding behavior allows individual capabilities and patterns of participation to adapt together without a central coordinating agent. We evaluate MPT on embodied crafting, robot navigation, and deep research. Results show gains over monolithic post-training and confirm that agent participation indeed evolves with learning. With the same post-trained action policies, learned self-orchestration outperforms alternative coordination strategies. Ablations show that both action and bidding benefit from learning, and that learned bids provide more effective action-training signals. Beyond the original training objectives, trained swarms adapt faster to related tasks and achieve better performance with larger populations at inference time. Larger populations also suffer smaller performance losses under agent dropout, demonstrating that learned self-orchestration supports reuse, scaling, and robustness in decentralized agent swarms. More details are available on our project page at https://guoweixu.com/mpt/.

Figures & tables

Figure 1: Learning self-orchestration in decentralized agent swarms. The three panels illustrate successive stages of training within the same agent population. Each shape represents an agent, darker colors indicate progressive parameter updates, and bidirectional arrows depict coordination patterns. Through experience, agents jointly learn how to act and when to participate through bidding. As individual capabilities improve, patterns of participation adapt in turn.
Figure 2: Self-orchestration through bidding in embodied crafting. (a) Selected turns from an Iron Pickaxe episode show the Logger, Miner, and Smith participating as resource gathering and crafting requirements change. (b) At each decision, agents submit bids, and the highest bidder acts. Here, the Smith wins with a bid of 0.8 and crafts a crafting table; the environment then returns a reward and the next state.
Figure 3: Learning to act through bidding-based credit assignment. Agents act sequentially during the forward rollout ( 1 ). The resulting reward is then propagated backward ( 2 ): at turn tt , the acting agent retains a fraction 1−bt1-b_{t} of the accumulated value as its credit and passes the remaining fraction btb_{t} to the preceding participant. Colored blocks associate credit with its recipient.
Figure 4: Learning to self-orchestrate through counterfactual comparisons. Starting from the same state, two alternative bids from the same agent induce different turn allocations. The alternative branch is continued from the restored state, and both branches are evaluated, producing returns that determine which bid is preferred. Preference learning then increases the relative likelihood of the bid associated with the better outcome, teaching the agent when to participate.
Figure 5: MPT improves performance at matched training compute. Training performance versus training FLOPs for monolithic RL (Single) and MPT (Society). The panels report mean episode reward for embodied crafting and robot navigation, and mean accuracy for deep research. Training FLOPs include all compute used for learning.
Figure 6: Self-orchestration evolves during training. Participation proportions of the Logger, Miner, and Smith on Iron Pickaxe. From left to right: after SFT and before RL, two intermediate RL checkpoints, and after RL. As training progresses, participation increasingly shifts toward the Miner and Smith at later decision steps.
StrategyMean Reward
Round-robin ( Wu et al., 2024 )23.8±4.223.8\pm 4.2
AgentNet ( Yang et al., 2025 )20.9±4.220.9\pm 4.2
GoA ( Yun et al., 2026 )25.4±4.625.4\pm 4.6
MPT41.5±5.9\mathbf{41.5\pm 5.9}
Table 2: Learned self-orchestration improves coordination. Episode reward (mean ±\pm SE) on Iron Pickaxe over 32 seeds. Left: comparison with alternative coordination strategies using the same policies. Right: contributions of learning bidding and action.
Figure 8: Trained swarms adapt faster to new objectives. Adaptation to (a) Torch and (b) Stone Bricks. Monolithic RL (Single) and MPT (Society) start from their respective Iron Pickaxe checkpoints and continue training separately. Curves show mean episode reward against training FLOPs.
Figure 9: Larger swarms improve inference-time performance. Pass@3 across five crafting milestones as the agent population increases from 3 to 12.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

CategoryHyperparameter == Value
SFT cold startepochs == 2; per_device_batch == 4; grad_accum == 4; learning_rate == 1 ⁣× ⁣10−51\!\times\!10^{-5} ; warmup_ratio == 0.03; weight_decay == 0; max_grad_norm == 1.0.
Post-traininglearning_rate == 1 ⁣× ⁣10−61\!\times\!10^{-6} ; train_batch_size == 32; grad_clip == 1.0; entropy_coeff == 0.
Generationtemperature == 1.0; top_p == 1.0;
 max_prompt_length == 3,072; max_response_length == 320 (action), 96 (bid).
Environmentdecisions_per_episode == 120; frames_per_instruction == 40.
Appendix
Table 3: Hyperparameters for the Embodied Crafting experiment.
CategoryHyperparameter == Value
SFT cold startepochs == 2; per_device_batch == 2; grad_accum == 8; learning_rate == 2 ⁣× ⁣10−52\!\times\!10^{-5} ; warmup_ratio == 0.03; weight_decay == 0; max_grad_norm == 1.0.
Post-traininglearning_rate == 1 ⁣× ⁣10−61\!\times\!10^{-6} ; train_batch_size == 16; grad_clip == 1.0; entropy_coeff == 0.
Generationtemperature == 1.0; top_p == 1.0; num_frames == 8;
 max_prompt_length == 4,096; max_response_length == 96.
Environmentdecisions_per_episode == 64; frame_size == 512×512512\times 512 .
Appendix
Table 4: Hyperparameters for the Robot Navigation experiment.
CategoryHyperparameter == Value
SFT cold startepochs == 2; per_device_batch == 1; grad_accum == 16; learning_rate == 1 ⁣× ⁣10−51\!\times\!10^{-5} ; warmup_ratio == 0.03; weight_decay == 0; max_grad_norm == 1.0.
Post-traininglearning_rate == 1 ⁣× ⁣10−61\!\times\!10^{-6} ; train_batch_size == 64; grad_clip == 1.0; entropy_coeff == 0.
Generationtemperature == 1.0; top_p == 1.0;
 max_prompt_length == 16,384; max_response_length == 4,096 (action), 128 (bid).
Environmentdecisions_per_episode == 64; search_results == 10; judge == qwen3.5-flash .
Appendix
Table 5: Hyperparameters for the Deep Research experiment.
Figure 11: Embodied Crafting. Cumulative reward over the 120 decisions of one evaluation episode of the trained population.
Figure 12: Decision 3 ( +1+1 , log). Bids: Logger 0.8, Miner 0, Smith 0; Logger wins and issues “chop down the nearest tree”.
Figure 13: Decision 8 ( +6+6 , planks). Bids: Logger 0.2, Miner 0, Smith 0.4; Smith wins and issues “craft planks”.
Figure 14: Decision 9 ( +6+6 , crafting table). Bids: Logger 0.2, Miner 0, Smith 0.8; Smith wins and issues “craft a crafting table”.
Figure 15: Decision 17 ( +8+8 , wooden pickaxe). Bids: Logger 0, Miner 0, Smith 0.8; Smith wins and issues “craft a wooden pickaxe”.
Figure 16: Decision 30 ( +12+12 , cobblestone). Bids: Logger 0, Miner 0.4, Smith 0; Miner wins and issues “dig down to find stone”.
Figure 17: Decision 52 ( +10+10 , iron ingot). Bids: Logger 0, Miner 0.2, Smith 0.8; Smith wins and issues “smelt iron”.
Figure 18: Decision 74 ( +30+30 , iron pickaxe). Bids: Logger 0, Miner 0.2, Smith 0.8; Smith wins and issues “craft iron pickaxe”.
Figure 19: Robot Navigation. (a) Cumulative reward and (b) geodesic distance to the goal over the 17 decisions of one episode of the trained population.
Figure 20: Decision 2 ( +0.75+0.75 , 6.9 m →\rightarrow 6.1 m). Bids: Turner 0.2, Mover 0.8, Stopper 0; Mover wins and issues “move forward 75 cm”.
Figure 21: Decision 6 ( +0.75+0.75 , 5.6 m →\rightarrow 4.9 m). Bids: Turner 0.2, Mover 0.4, Stopper 0; Mover wins and issues “move forward 75 cm”.
Figure 22: Decision 8 ( +0.74+0.74 , 4.1 m →\rightarrow 3.4 m). Bids: Turner 0.2, Mover 0.8, Stopper 0; Mover wins and issues “move forward 75 cm”.
Figure 23: Decision 13 ( +0.60+0.60 , 2.7 m →\rightarrow 2.1 m). Bids: Turner 0.2, Mover 0.8, Stopper 0; Mover wins and issues “move forward 75 cm”.
Figure 24: Decision 17 ( +2.50+2.50 , 2.0 m →\rightarrow 2.0 m, success). Bids: Turner 0.2, Mover 0.2, Stopper 0.8; Stopper wins and issues “stop”.
DecisionBids (S/O/F)WinnerAction
10.8 / 0 / 0.2Searchersearch("Legend of the Lost ... country")
20.8 / 0.8 / 0.2Openeropen(id=0) →\rightarrow Wikipedia: “…near Tripoli, Libya.”
30.2 / 0.8 / 0.8Finderfind("Libya")
40.2 / 0.4 / 0.8Finderfind("Ghadames")
50.2 / 0.4 / 0.8Finderfind("Libya")
60.2 / 0.2 / 0.8Finderfinal answer: Libya
Appendix
Table 6: Deep Research. All decisions of the episode. Bids are listed as Searcher / Opener / Finder .
More in this topic

Multi-Agent Orchestration

Multi-agent orchestration coordinates specialized AI agents, tools, and workflows to solve complex tasks efficiently and reliably.

See all 206 papers

Explore similar work

CardsList
Sep 28, 2026

Waggle: Learning One Anonymous Local Law for Self-Organizing LLM Swarms

As LLM agents increasingly collaborate on complex tasks, how to organize their interactions becomes a central design question. Existing multi-agent systems typically learn or adapt explicit roles, hierarchies, routing policies, or communication topologies. We shift the learning target to a reusable local law that can be shared across interchangeable agents and adapt coordination as populations or interaction conditions change, without redefining a global organization. We introduce Waggle, a shared anonymous policy over bounded local views that jointly selects task actions, semantic communication, and local commitment updates. Repeated execution of the same law allows coordination to form, persist, and reorganize online without explicit roles or global topology. To learn this law across interchangeable agents and evolving coordination, we develop Swarm-Consistent Distillation (SCD), combining anonymous-orbit consistency with rollout-grounded prediction of the next local coordination field, with no added inference-time components. Across diverse coordination settings, the same learned law remains effective as populations and interaction budgets change, retains over 96% of substrate-specific oracle quality, and transfers without retraining; SCD further improves reorganization after counterevidence. Together, these results show that LLM-agent organization can emerge and adapt through repeated execution of a learned local law.

Mingxi Zou, Wei Zhu, Zhuo Wang +4

Shanghai Academy of AI for Science (SAIS) · AI3 Institute, Fudan University · Yunnan University

Multi-Agent LLM SystemsMulti-Agent CoordinationMulti-Agent SystemsMulti-Agent Communication
AskPDF
Oct 6, 2026

Decoupled Multi-Agent Orchestration

Learned orchestration can automatically construct effective language-model multi-agent systems, but existing approaches couple planning to fixed worker pools and train decomposition and collaboration from the same terminal outcome, limiting transfer and obscuring credit assignment. We introduce DeOrch, which separates worker-agnostic planning from concrete worker selection. Its two-stage planner first decomposes the task without worker information, then chooses collaboration operations using compact, worker-identity-free matchability feedback from the pool, enabling conditional credit assignment to decomposition and collaboration decisions. A lightweight matcher estimates worker suitability from behavior on a fixed probe set and adapts online with a contextual bandit, allowing new workers to be incorporated without retraining the planner or matcher. Across diverse in- and out-of-distribution tasks, DeOrch outperforms prior automatic MAS orchestration methods with fewer worker calls than competing learned orchestrators, remains effective when transferred to an entirely unseen worker pool without retraining, and shows consistent gains from both components.

Xinle Wu, Yao Lu

National University of Singapore

Multi-Agent OrchestrationLLM Agent OrchestrationMulti-Agent PlanningTask Decomposition
AskPDF
Aug 31, 2026

SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?

Large language model-based multi-agent systems are evolving from fixed interaction topologies toward dynamically orchestrated Agent Swarms. However, existing benchmarks are still largely based on single-agent or general-purpose agent tasks, making it difficult to systematically evaluate key orchestration capabilities. We propose SwarmBench, a benchmark that evaluates model performance from multiple perspectives, including accuracy, efficiency, cost, and process quality. Experimental results show that current models exhibit substantial differences in orchestration capability. These differences are reflected not only in final accuracy, efficiency, and cost, but also in the overall quality of the orchestration process itself. Based on these findings, we further propose SwarmExp, a simple yet effective method based on experience extraction and experience replay, which consistently improves the orchestration performance of large language models.

Jinshan Gao, Zhuoran Jin, Tianyi Men +2

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

LLM Agent OrchestrationAI Agent BenchmarksMulti-Agent LLMs
AskPDF