Groups can achieve more than individuals in the same amount of time, but only if their members coordinate: they must decide who works on what and what to share. However, this organization in multi-agent systems today usually comes from arbitrary, static harnesses designed a priori without adapting to agent behaviors or task specifications. We ask whether agents can instead learn to organize themselves. We introduce s3, the first multi-agent reinforcement learning method in which agents decide for themselves how long to work alone and when to share findings or read teammates' messages, with no predefined organization or roles. Our s3 agents learn to take different approaches, read each other's work, exhibit emergent roles that complement each other in the swarm, and generalize to bigger teams. We study both coordination, where agents must combine private information, and parallel exploration, where sharing discoveries can make search more efficient. With matched training and inference budgets, s3, swarms evaluated with twice as many agents as in training solve over 25× more coordination problems than the strongest baseline, and in exploration, communicating agents reduce repeated work by nearly 50% and outperform best-of-N by up to 9.4 percentage points. Rather than being designed in advance, s3 enables organization to emerge from a shared goal. This opens the door to capable systems built from many models that learn to work together.
Figures & tables
Condition
Description
Solo (SFT+RL)
Train a single agent with individual reward; evaluate independent copies as best-of- N
s 3 − communication
Team reward with messaging tools disabled
s 3 : individual reward
Messaging enabled; each agent is rewarded for its own success
s 3 : online SFT
Train on successful communicating teams sampled from the current policy
SFT only / initial
Fine-tuned checkpoint or base model, without RL
s 3 − SFT
Team reward and messaging, trained from the base model
Table 1: Controls and baselines. Swarm controls are ablations of s 3 ; individual-reward swarms still communicate and learn together, whereas solo agents train independently. SFT denotes supervised fine-tuning. The SFT ablation applies to decentralized exploration; information aggregation starts from the base model. Team learners train with four agents and solo learners with one. Within each task, matched learners use the same problems, number of agent trajectories per update, per-agent token allowance, and optimizer. Rewards include the token cost in Equation 1 .
Figure 5: How coordination changes during training. (a) Information aggregation: swarms progress from incomplete information sharing, through repeated broadcasts, toward one broadcast per agent. Categories distinguish incomplete participation (at least one agent never sends) from swarms where everyone sends: with direct messages, with broadcasts only but some repeated, or with one broadcast per agent. (b) Agents that read incoming messages before sharing their own information. (c) Decentralized exploration: success according to which branch at the first junction contains the exit. Communication-disabled agents favor the first-listed branch; communicating swarms succeed more consistently across exit locations. Bands and bars show mean ± SE across groups of attempts on the same problem in ten-update bins (a, b) or across search trees (c).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Tool
Description given to the model
message(to, text)
“Send plans, questions, tentative ideas, dead ends, or checked results to one available teammate or all available teammates. Label uncertainty and verification status. Delivery is immediate. Sending does not read incoming messages or wait for replies; use read_messages when notified.” to is a teammate’s name or all .
read_messages()
“Read explicitly received unread peer messages.” Returns up to 16 unread messages with their senders.
check_notifications()
“Check unread message counts by sender without reading messages.” The harness also pushes this tool’s output on every arrival.
move(room) (decentralized exploration)
“Walk through a door of your current room into the named adjacent room. Returns what you see there. Your position is private: moving never moves a teammate.”
look() (decentralized exploration)
“Describe your current room again without moving.”
Appendix
Table 2: Tools available to each agent. Descriptions are shown to the model as tool schemas.
SFT on demonstrations
RL and evaluation
Depth d
2
3
Rooms
121
391
Dead ends (exit candidates)
9
27
Moves on the shortest path
20
30
Moves for a full depth-first search
240
780
Training / development / evaluation search trees
1,024 / 32 / 32
1,024 / 32 / 32
Appendix
Table 3: Exploration task settings. All search trees have branching b=3 , corridor length c=10 , and no loops.
Setting
Information aggregation
Decentralized exploration
Model
Qwen3-0.6B, thinking on
Qwen3-1.7B, thinking on, all reasoning retained
Initial policy
Base model
50 SFT updates on demonstrations (Appendix L )
RL updates
200
200
Teams per update
256 (32 problems × 8 attempts)
128 (16 problems × 8 attempts)
Team size in training
4
4
Per-agent allowance
8,192 work + 1,024 final
8,192 (+1 unused final token)
Appendix
Table 4: Training settings.
Learner
Problems
Episodes
Trajectories
Adv. group
Loss unit
s 3
16
128 teams of 4
512
8 teams
team
s 3 − communication
16
128 teams of 4
512
8 teams
team
s 3 : individual reward
16
128 teams of 4
512
32 agents
team
Solo learner
16
512 single agents
512
32 agents
agent
Appendix
Table 5: What each learner sees per update in decentralized exploration. Allowances are per agent.
Reward
Success (%)
Efficiency
Learner
Start
End
Start
End
Δ (pp)
Start
End
Sorting
s 3 (Ours)
0.027±0.005
0.926±0.002
3.0±0.5
93.3±0.3
+90.3
0.09±0.02
29.61±1.83
s 3 : individual reward
0.007±0.001
0.018±0.001
3.1±0.2
8.5±0.5
+5.4
0.09±0.01
0.11±0.01
s 3 : online SFT
0.025±0.003
0.262±0.012
2.8±0.3
28.7±1.3
+25.9
0.08±0.01
0.82±0.05
Mazes
Appendix
Table 6: Training metrics at the start and end of training for every learner. Start and end are means over epochs 1–20 and 181–200. Values are mean ± SE over four consecutive five-update block means within each window; these SEs describe variation within one training trajectory. Δ is the change in success rate in percentage points. Higher is better for all three metrics. Reward uses each condition’s reward signal; efficiency is solve rate divided by the mean fraction of the token allowance consumed.
Learner
Start success (%)
Final success (%)
Gain (pp)
Exploration: s 3 (Ours)
84.9±1.9
89.6±1.5
4.7±1.3
Exploration: Solo (SFT+RL)
81.8±1.9
80.2±2.1
−1.6±1.3
Exploration: s 3 − communication
69.8±2.6
70.3±2.4
0.5±2.0
Exploration: s 3 : individual reward
84.9±1.9
90.6±1.4
5.7±2.0
Exploration: s 3 − SFT
2.6±1.3
13.5±1.0
10.9±0.7
Exploration: Solo RL (no SFT)
12.0±1.0
15.1±1.3
3.1±1.1
Appendix
Table 7: Starting versus final policy at N=4 , evaluated on the same 32 held-out evaluation problems and six seeds. Means ± SE across seed means; gain uses paired seed differences. Starting policies are the corresponding SFT checkpoints or initial base models.
Team learner
N
Success gain (pp)
Efficiency gain
s 3 (Ours)
1
−8.3±1.9
−0.13±0.03
s 3 (Ours)
2
4.7±3.2
−0.00±0.08
s 3 (Ours)
4
9.4±3.1
0.09±0.17
s 3 (Ours)
6
7.3±1.7
0.00±0.22
s 3 (Ours)
8
0.5±1.9
−0.60±0.35
s 3 (Ours)
12
−1.6±0.7
−0.67±0.25
Appendix
Table 8: Paired decentralized exploration comparisons against solo-trained best-of- N : s 3 versus Solo (SFT+RL), and s 3 − SFT versus Solo RL (no SFT). Means ± SE of six within-seed success differences and efficiency differences. Positive differences favor the team.
Family / task
Contrast
N
Metric
Effect ± SE
dz
Before
After
Train / expl.
s 3 vs. Solo (SFT+RL)
4
Success
9.38±3.13
1.22
Yes
No (2)
Train / expl.
s 3 vs. Solo (SFT+RL)
4
Efficiency
0.09±0.17
0.23
No
No (2)
Scale / expl.
s 3 vs. Solo (SFT+RL)
6
Success
7.29±1.74
1.71
Yes
Yes (2)
Scale / expl.
s 3 vs. Solo (SFT+RL)
6
Efficiency
0.00±0.22
0.00
No
No (2)
Train / expl.
s 3 − SFT vs. Solo RL (no SFT)
4
Success
−1.56±1.34
-0.48
No
No (2)
Abl. / expl.
s 3 vs. s 3 : individual reward
4
Success
−1.04±1.55
-0.28
No
No (3)
Appendix
Table 9: Main comparisons (expl.: decentralized exploration; agg.: information aggregation; train: training-size comparison; scale: larger-population comparison; abl.: ablation; comm.: communication analysis) using two-sided paired t tests on six seed means. Before and after indicate significance at 0.05 before and after Bonferroni correction; parentheses give the family size. Families group contrasts within the same analysis family, task, metric, team size, and comparison type. Tests at different team sizes are pointwise. Effects are first minus second; success differences are percentage points. Cohen’s dz standardizes the paired difference.
Figure 13: Average information aggregation communication graph per learner (rows) and training phase (columns). Within each team, agents are ranked by the number of messages they send (#1 sends most). Arrow width is the mean number of direct messages per team from one rank to another; node area is the mean number of broadcasts sent, and node colour marks the learner. “Silent” is the share of teams in which that agent sends nothing.
Figure 14: Information aggregation communication over training. Top: share of teammate numbers that were both sent and read; share of messages that are broadcasts; share of direct-message pairs answered by a direct message in the other direction; inequality of messages sent across agents. Bottom: share of messages sent on the sender’s first turn; share of agents that read their inbox before their first send; median time from send to read; turns per agent. Mean ± SE across GRPO groups.
Figure 15: Share of information aggregation teams in each communication structure over training, for each learner. Mean ± SE across GRPO groups.
s 3
Structure
1–100
101–200
Share
An agent never messages
−12.9±0.5
−25.9±1.3
8%
Conversation (reference)
0
0
18%
Repeated broadcasts
+24.7±2.0
+4.4±0.7
26%
One broadcast each
+30.0±3.2
+6.9±0.7
48%
Appendix
Table 10: Information aggregation success by communication structure, relative to conversational teams in the same GRPO group (percentage points, SE clustered by group). The last column gives each structure’s share of s 3 teams in updates 101–200.
Figure 16: Information aggregation message language over training: characters per message; share of messages matching the five most common templates of that learner and 10-update bin (numbers and names replaced by placeholders); share of messages that are their sender’s first. Mean ± SE across GRPO groups.
Updates 1–20
s 3 , 101–200
s 3 : individual reward, 101–200
s 3 : online SFT, 101–200
Message content
Own number (“My private number is forty-two.”)
71%
90%
64%
74%
Request only (“What is your private number?”)
21%
3%
26%
16%
Own number stated as recipient’s
4%
1%
6%
3.5%
Misstated own number
1.6%
1.2%
1.7%
1.9%
Failure cause
Appendix
Table 11: Message content (share of messages) and failure causes (share of failed teams, first matching cause). “Own number stated as recipient’s” copies the agent’s own prompt line (“Your private number is X.”) into a message.
Figure 17: Information aggregation success by team feature, updates 101–200. The y-axis is a team’s success minus the mean success of its GRPO group (same problem and policy version), in percentage points. Teams are binned by each feature, with points at each bin’s median; bins with fewer than 100 teams are omitted. Mean ± SE across teams. Delivering and reading every teammate’s number goes with success under every learner; reading before sending, asking for numbers without giving one’s own, and wrong or garbled numbers go with failure. The text gives per-SD slopes for further features.
Figure 18: Information aggregation reasoning at update 195. Left: how each agent’s first turn ended, for agents that message and agents that never message (groups with at least 20 agents). Right: share of turns whose thought matches each theme under individual reward, for agents that message (turns before their first send) and agents that never message (all turns), mean ± SE across agents.
Figure 19: Decentralized exploration success by problem, updates 101–200. Left: by the listed position of the exit’s door at the first fork. Middle: by the number of moves a lone depth-first agent, taking doors in listed order, needs to reach the exit (quintiles). Right: gain from messages, the success of s 3 : individual reward and of s 3 minus that of s 3 − communication. Mean ± SE across search trees; every episode whose GRPO group maps to a search tree (78% of episodes).
Figure 20: Decentralized exploration communication over training. Top left: teams trained from the base model, by the number of agents that send. Other panels: messages per 100 moves; characters per message; share of messages matching the five most common templates; share of branch claims that duplicate a teammate’s claim; share of forward “exhausted” reports whose branch contains the exit. Mean ± SE across GRPO groups.
Figure 21: Decentralized exploration success by team feature, updates 101–200. Teams are split into fifths of each feature (points at each fifth’s median) or, for the two skewed features, into two groups. Success is compared within the same search tree and 10-update bin: each team’s success minus the mean of its search tree and bin, plus the overall mean. Panels run from the most positive to the most negative slope under s 3 . Mean ± SE across teams.
As LLM agents increasingly collaborate on complex tasks, how to organize their interactions becomes a central design question. Existing multi-agent systems typically learn or adapt explicit roles, hierarchies, routing policies, or communication topologies. We shift the learning target to a reusable local law that can be shared across interchangeable agents and adapt coordination as populations or interaction conditions change, without redefining a global organization. We introduce Waggle, a shared anonymous policy over bounded local views that jointly selects task actions, semantic communication, and local commitment updates. Repeated execution of the same law allows coordination to form, persist, and reorganize online without explicit roles or global topology. To learn this law across interchangeable agents and evolving coordination, we develop Swarm-Consistent Distillation (SCD), combining anonymous-orbit consistency with rollout-grounded prediction of the next local coordination field, with no added inference-time components. Across diverse coordination settings, the same learned law remains effective as populations and interaction budgets change, retains over 96% of substrate-specific oracle quality, and transfers without retraining; SCD further improves reorganization after counterevidence. Together, these results show that LLM-agent organization can emerge and adapt through repeated execution of a learned local law.
Mingxi Zou, Wei Zhu, Zhuo Wang +4
Shanghai Academy of AI for Science (SAIS) · AI3 Institute, Fudan University · Yunnan University
While large language model-based multi-agent systems have shown strong potential for complex reasoning, how to effectively organize multiple agents remains an open question. In this paper, we introduce OrgAgent, a company-style hierarchical multi-agent framework that separates collaboration into governance, execution, and compliance layers. OrgAgent decomposes multi-agent reasoning into three layers: a governance layer for planning and resource allocation, an execution layer for task solving and review, and a compliance layer for final answer control. By evaluating the framework across reasoning tasks, LLMs, execution modes, and execution policies, we find that multi-agent systems organized in a company-style hierarchy generally outperform other organizational structures. Besides, hierarchical coordination also reduces token consumption relative to flat collaboration in most settings. For example, for GPT-OSS-120B, the hierarchical setting improves performance over flat multi-agent system by 102.73% while reducing token usage by 74.52% on SQuAD 2.0. Further analysis shows that hierarchy helps most when tasks benefit from stable skill assignment, controlled information flow, and layered verification. Overall, our findings highlight organizational structure as an important factor in multi-agent reasoning, shaping not only effectiveness and cost, but also coordination behavior.
Yiru Wang, Xinyue Shen, Yaohui Han +3
The Chinese University of Hong Kong · CISPA Helmholtz Center for Information Security · IBM Research
Reinforcement learning (RL) solves the explore-exploit problem by training a learner on reward; its multi-agent, resource-sharing form has an equilibrium allocation as its optimum -- the ideal-free distribution (IFD), the Nash equilibrium of a congestion game. We show a minimal swarm approximates it with no learner at all. We present a decentralized, learner-free method for this problem: agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a degree-normalized threshold cascade, and disperse as a patch crowds. The method drives the collective to the shape of the equilibrium allocation (pooled spatial matching slope 1.14, R^2=0.61) with no training, no stored parameters, and no reward in its dynamics; it reaches this as a self-organized attractor and holds it, re-finding it when the resources move -- though it reaches the shape, not the exact equilibrium, and stays measurably exploitable (Nash-gap 1.46). Coupling recruitment to movement causes the allocation: removing it collapses the matching (slope 0.006). Against Proximal Policy Optimization (PPO) learners (feedforward, centralized-critic, graph message-passing, and recurrent) trained in the same sensing-limited environment, the method at zero training is the only arm left unbiased w.r.t. the IFD gradient (slope ~1, CI containing 1, vs learners 0.69-0.97, all under-concentrating) and ties the best learner's harvest where resource is clustered, while the learner collects more where resource is scarce (0.33 vs 0.22) and holds a tighter per-capita intake. Learning buys precision and transfer across reward structures, not a better allocation shape. A companion paper develops the mechanism.