cs.MAOct 15, 2026

Learning Self-Orchestration in Decentralized Agent Swarms

Authors: Guowei Xu, Zhenting Qi, Amani Kiruga, Himabindu Lakkaraju, Yilun Du

Organizations: Harvard University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The capabilities of an agent swarm depend on both its individual members and how they coordinate. Yet existing multi-agent systems often train agents within fixed orchestration structures, separating the learning of individual capabilities from the organization of collective behavior. We introduce market-based post-training (MPT), a framework for jointly learning action policies and self-orchestration in decentralized agent swarms. Agents express their willingness to participate through bids, which determine who acts and how collective outcomes provide local training credit. Learning both action and bidding behavior allows individual capabilities and patterns of participation to adapt together without a central coordinating agent. We evaluate MPT on embodied crafting, robot navigation, and deep research. Results show gains over monolithic post-training and confirm that agent participation indeed evolves with learning. With the same post-trained action policies, learned self-orchestration outperforms alternative coordination strategies. Ablations show that both action and bidding benefit from learning, and that learned bids provide more effective action-training signals. Beyond the original training objectives, trained swarms adapt faster to related tasks and achieve better performance with larger populations at inference time. Larger populations also suffer smaller performance losses under agent dropout, demonstrating that learned self-orchestration supports reuse, scaling, and robustness in decentralized agent swarms. More details are available on our project page at https://guoweixu.com/mpt/.

Figures & tables

Figure 1: Learning self-orchestration in decentralized agent swarms. The three panels illustrate successive stages of training within the same agent population. Each shape represents an agent, darker colors indicate progressive parameter updates, and bidirectional arrows depict coordination patterns. Through experience, agents jointly learn how to act and when to participate through bidding. As individual capabilities improve, patterns of participation adapt in turn.
Figure 2: Self-orchestration through bidding in embodied crafting. (a) Selected turns from an Iron Pickaxe episode show the Logger, Miner, and Smith participating as resource gathering and crafting requirements change. (b) At each decision, agents submit bids, and the highest bidder acts. Here, the Smith wins with a bid of 0.8 and crafts a crafting table; the environment then returns a reward and the next state.
Figure 3: Learning to act through bidding-based credit assignment. Agents act sequentially during the forward rollout ( 1 ). The resulting reward is then propagated backward ( 2 ): at turn tt , the acting agent retains a fraction 1−bt1-b_{t} of the accumulated value as its credit and passes the remaining fraction btb_{t} to the preceding participant. Colored blocks associate credit with its recipient.
Figure 4: Learning to self-orchestrate through counterfactual comparisons. Starting from the same state, two alternative bids from the same agent induce different turn allocations. The alternative branch is continued from the restored state, and both branches are evaluated, producing returns that determine which bid is preferred. Preference learning then increases the relative likelihood of the bid associated with the better outcome, teaching the agent when to participate.
Figure 5: MPT improves performance at matched training compute. Training performance versus training FLOPs for monolithic RL (Single) and MPT (Society). The panels report mean episode reward for embodied crafting and robot navigation, and mean accuracy for deep research. Training FLOPs include all compute used for learning.
Figure 6: Self-orchestration evolves during training. Participation proportions of the Logger, Miner, and Smith on Iron Pickaxe. From left to right: after SFT and before RL, two intermediate RL checkpoints, and after RL. As training progresses, participation increasingly shifts toward the Miner and Smith at later decision steps.
StrategyMean Reward
Round-robin ( Wu et al., 2024 )23.8±4.223.8\pm 4.2
AgentNet ( Yang et al., 2025 )20.9±4.220.9\pm 4.2
GoA ( Yun et al., 2026 )25.4±4.625.4\pm 4.6
MPT41.5±5.9\mathbf{41.5\pm 5.9}
Table 2: Learned self-orchestration improves coordination. Episode reward (mean ±\pm SE) on Iron Pickaxe over 32 seeds. Left: comparison with alternative coordination strategies using the same policies. Right: contributions of learning bidding and action.
Figure 8: Trained swarms adapt faster to new objectives. Adaptation to (a) Torch and (b) Stone Bricks. Monolithic RL (Single) and MPT (Society) start from their respective Iron Pickaxe checkpoints and continue training separately. Curves show mean episode reward against training FLOPs.
Figure 9: Larger swarms improve inference-time performance. Pass@3 across five crafting milestones as the agent population increases from 3 to 12.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

CategoryHyperparameter == Value
SFT cold startepochs == 2; per_device_batch == 4; grad_accum == 4; learning_rate == 1 ⁣× ⁣10−51\!\times\!10^{-5} ; warmup_ratio == 0.03; weight_decay == 0; max_grad_norm == 1.0.
Post-traininglearning_rate == 1 ⁣× ⁣10−61\!\times\!10^{-6} ; train_batch_size == 32; grad_clip == 1.0; entropy_coeff == 0.
Generationtemperature == 1.0; top_p == 1.0;
 max_prompt_length == 3,072; max_response_length == 320 (action), 96 (bid).
Environmentdecisions_per_episode == 120; frames_per_instruction == 40.
Appendix
Table 3: Hyperparameters for the Embodied Crafting experiment.
CategoryHyperparameter == Value
SFT cold startepochs == 2; per_device_batch == 2; grad_accum == 8; learning_rate == 2 ⁣× ⁣10−52\!\times\!10^{-5} ; warmup_ratio == 0.03; weight_decay == 0; max_grad_norm == 1.0.
Post-traininglearning_rate == 1 ⁣× ⁣10−61\!\times\!10^{-6} ; train_batch_size == 16; grad_clip == 1.0; entropy_coeff == 0.
Generationtemperature == 1.0; top_p == 1.0; num_frames == 8;
 max_prompt_length == 4,096; max_response_length == 96.
Environmentdecisions_per_episode == 64; frame_size == 512×512512\times 512 .
Appendix
Table 4: Hyperparameters for the Robot Navigation experiment.
CategoryHyperparameter == Value
SFT cold startepochs == 2; per_device_batch == 1; grad_accum == 16; learning_rate == 1 ⁣× ⁣10−51\!\times\!10^{-5} ; warmup_ratio == 0.03; weight_decay == 0; max_grad_norm == 1.0.
Post-traininglearning_rate == 1 ⁣× ⁣10−61\!\times\!10^{-6} ; train_batch_size == 64; grad_clip == 1.0; entropy_coeff == 0.
Generationtemperature == 1.0; top_p == 1.0;
 max_prompt_length == 16,384; max_response_length == 4,096 (action), 128 (bid).
Environmentdecisions_per_episode == 64; search_results == 10; judge == qwen3.5-flash .
Appendix
Table 5: Hyperparameters for the Deep Research experiment.
Figure 11: Embodied Crafting. Cumulative reward over the 120 decisions of one evaluation episode of the trained population.
Figure 12: Decision 3 ( +1+1 , log). Bids: Logger 0.8, Miner 0, Smith 0; Logger wins and issues “chop down the nearest tree”.
Figure 13: Decision 8 ( +6+6 , planks). Bids: Logger 0.2, Miner 0, Smith 0.4; Smith wins and issues “craft planks”.
Figure 14: Decision 9 ( +6+6 , crafting table). Bids: Logger 0.2, Miner 0, Smith 0.8; Smith wins and issues “craft a crafting table”.
Figure 15: Decision 17 ( +8+8 , wooden pickaxe). Bids: Logger 0, Miner 0, Smith 0.8; Smith wins and issues “craft a wooden pickaxe”.
Figure 16: Decision 30 ( +12+12 , cobblestone). Bids: Logger 0, Miner 0.4, Smith 0; Miner wins and issues “dig down to find stone”.
Figure 17: Decision 52 ( +10+10 , iron ingot). Bids: Logger 0, Miner 0.2, Smith 0.8; Smith wins and issues “smelt iron”.
Figure 18: Decision 74 ( +30+30 , iron pickaxe). Bids: Logger 0, Miner 0.2, Smith 0.8; Smith wins and issues “craft iron pickaxe”.
Figure 19: Robot Navigation. (a) Cumulative reward and (b) geodesic distance to the goal over the 17 decisions of one episode of the trained population.
Figure 20: Decision 2 ( +0.75+0.75 , 6.9 m →\rightarrow 6.1 m). Bids: Turner 0.2, Mover 0.8, Stopper 0; Mover wins and issues “move forward 75 cm”.
Figure 21: Decision 6 ( +0.75+0.75 , 5.6 m →\rightarrow 4.9 m). Bids: Turner 0.2, Mover 0.4, Stopper 0; Mover wins and issues “move forward 75 cm”.
Figure 22: Decision 8 ( +0.74+0.74 , 4.1 m →\rightarrow 3.4 m). Bids: Turner 0.2, Mover 0.8, Stopper 0; Mover wins and issues “move forward 75 cm”.
Figure 23: Decision 13 ( +0.60+0.60 , 2.7 m →\rightarrow 2.1 m). Bids: Turner 0.2, Mover 0.8, Stopper 0; Mover wins and issues “move forward 75 cm”.
Figure 24: Decision 17 ( +2.50+2.50 , 2.0 m →\rightarrow 2.0 m, success). Bids: Turner 0.2, Mover 0.2, Stopper 0.8; Stopper wins and issues “stop”.
DecisionBids (S/O/F)WinnerAction
10.8 / 0 / 0.2Searchersearch("Legend of the Lost ... country")
20.8 / 0.8 / 0.2Openeropen(id=0) →\rightarrow Wikipedia: “…near Tripoli, Libya.”
30.2 / 0.8 / 0.8Finderfind("Libya")
40.2 / 0.4 / 0.8Finderfind("Ghadames")
50.2 / 0.4 / 0.8Finderfind("Libya")
60.2 / 0.2 / 0.8Finderfinal answer: Libya
Appendix
Table 6: Deep Research. All decisions of the episode. Bids are listed as Searcher / Opener / Finder .
More in this topic

Multi-Agent Orchestration

Multi-agent orchestration coordinates specialized AI agents, tools, and workflows to solve complex tasks efficiently and reliably.

See all 206 papers

Explore similar work

CardsList

  1. Waggle: Learning One Anonymous Local Law for Self-Organizing LLM Swarms

    Sep 28, 2026Mingxi Zou, Wei Zhu, Zhuo Wang +4Multi-Agent LLM SystemsMulti-Agent Coordination

    Shanghai Academy of AI for Science (SAIS) · AI3 Institute, Fudan University · Yunnan University

  2. Decoupled Multi-Agent Orchestration

    Oct 6, 2026Xinle Wu, Yao LuMulti-Agent OrchestrationLLM Agent Orchestration

    National University of Singapore

  3. SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?

    Aug 31, 2026Jinshan Gao, Zhuoran Jin, Tianyi Men +2LLM Agent OrchestrationAI Agent Benchmarks

    The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China