Self-organizing reinforcement learning: a learner-free swarm approaches the explore-exploit optimum

Galen J. Wilkerson

cs.LGOct 18, 2026

Self-organizing reinforcement learning: a learner-free swarm approaches the explore-exploit optimum

Authors: Galen J. Wilkerson

Organizations: Komplexity AI LLC

Abstract

The explore/exploit trade-off is the core problem of reinforcement learning (RL). We show that a minimal foraging swarm approaches its optimum - the ideal-free / Nash allocation of a multi-agent resource-sharing game (an equilibrium, not a return maximum) - with no learner and no reward signal. Agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a threshold cascade on their proximity network, and disperse as a patch crowds; this cascade cycle balances exploration and exploitation and drives foragers toward the ideal free distribution. The swarm attains the shape of this optimum (committed foragers track richness with slope 0.94) but not its exact equilibrium - it stays measurably exploitable - reaching it dynamically (~150 steps) through the coupling of recruitment to movement: remove the coupling and matching collapses (0.94 -> 0.08). The RL sits at the level of the problem and the allocation reached, not the micro-rule: no step evaluates payoff (reward-free by construction, though the crowd signal is informationally tied to payoff), and the swarm reaches the allocation without learning it (quality plateaus and re-settles after resources move, no cumulative gain). Against a trained baseline the comparison is two-sided: in an easier, disembodied game a learner is more precise, while in the same embodied environment the reward-free swarm reaches ~65% of a validated oracle ceiling at zero training and leads until trained PPO and centralised-critic (CTDE) learners overtake it (~74-75%, 16 seeds, all above the swarm mean) after ~4,000 of 15,000 training steps - its advantage is in budget, not asymptote. In this one setting we read it not as beating RL but as a strong zero-shot, reward-free operating point: a demonstration that the self-organizing physics of a minimal swarm can carry a collective close to the optimum RL seeks, with none of RL's machinery.

Figures & tables

Figure 1: The physical model. Agents explore by Lévy flights (left). An agent that lands on a resource—a seed —triggers a recruitment cascade along the proximity network, drawing neighbours into local convergence on the find; as the neighbourhood crowds, degree-normalized activation

a/k

falls below threshold

\phi

and further recruitment is blocked (red). Schematic of the model described in § 3 ; the full model is developed in ( Wilkerson, 2026 ) .

Symbol	Meaning	Value
$N$	foragers	$200$
$L$	torus side length	$800$
$\rho_{r}=N(r/L)^{2}$	connectivity density	$2.1$ ( $r\approx 82$ )
$\phi$	activation threshold ( $a/k\geq\phi$ )	$0.18$ ( $2/\phi\approx 11$ )
$\alpha$	Lévy exponent	$1.0$
$C$	patches (resource clumps)	$10$

Table 1: Main model parameters (spatial foraging model, § 3 ). The abstract congestion game (§ 6 , App. C ) uses

K{=}10

patches and

N{=}200

agents with the same Pareto richness; the embodied comparison (App. D ) uses

N{=}200

over horizon

T{=}600

Figure 2: The self-organizing cycle balances exploration and exploitation. In the plane of network/spatial density (horizontal) and node activity (vertical), the swarm runs a drive–avalanche cycle: discovery seeds a recruitment cascade (explore

\to

recruit

\to

exploit); crowding blocks and disperses it (exploit

\to

released

\to

explore); aggregation ( drive ) versus dispersal ( dissipation ) hold the swarm in a poised window, with no parameter tuned. Schematic of the self-organizing cycle described in § 3 ; full treatment in ( Wilkerson, 2026 ) .

Figure 3: The swarm reaches the optimal allocation, and the coupling causes it. Committed foragers per patch vs. richness, pooled over

B{=}64

independent replica worlds (

n{=}591

patch-replica points). (a) Recruiting. Pooled slope

0.94

(

R^{2}{=}0.67

; per-replica mean slope

{\approx}1.0

); dashed is the ideal slope

1

, the small shortfall the “undermatching” real foragers show. (b) Coupling off. With recruitment removed, matching is destroyed (slope

0.08

): the optimal (ideal-free) allocation is caused by the recruitment–movement coupling.

Figure 4: The allocation is reached dynamically. Matching slope (regression of log committed foragers per patch on log richness, trailing window; replicas with bootstrap

95\%

CI) vs. time. With the recruitment coupling (green) the swarm climbs to the ideal-free plateau (slope

\to 1

, dashed) within

{\sim}150

steps and holds; without it (red) matching never develops.

Figure 5: The operating regime is self-organized, not tuned. (a) The single-seed recruitment cascade has an inverted-U window between

z_{c}^{\mathrm{RGG}}\approx 4.51

and the blocking bound

2/\phi

(reproduced here). (b) Swept over a

60\times

connectivity range, the mobile swarm holds a subcritical branching ratio

\sigma

nearly flat (

0.06

–

0.09

), with its broad minimum near the operating point used throughout (

\rho_{r}=2.1

); its bulk mean degree runs above the single-seed window (aggregation), so the criticality is local, not bulk. (c) Near-ideal IFD matching (slope

0.80

–

1.03

) holds across the whole range with no per-condition tuning.

Figure 6: Illustrative: recruitment tracks crowd, not payoff (reward-freeness itself is established by the rule’s construction, § 3 , not by this figure). Per-patch recruitment inflow

J^{+}_{i}

(new commits per step) vs. per-capita payoff

f_{i}

, richness

Q_{i}

, and occupancy

n_{i}

(quantile-binned mean, bootstrap

95\%

CI over

64

replicas). Inflow correlates positively with occupancy (

r=+0.11

)—the crowd signal the cascade rule uses—weakly negatively with payoff (

r=-0.11

), and shows no monotone trend with richness (non-monotone, linear

r\approx 0

). The correlations are small and, under equal sharing, non-identifying (App. B ); they illustrate rather than prove the reward-free construction.

Figure 7: The swarm reaches the optimum but does not learn it. Allocation informativeness (closeness

=1-

total-variation distance between committed occupancy and the resource distribution; replicas with bootstrap

95\%

CI) vs. time. It rises once and plateaus by

{\sim}120

steps; after the resources are relocated (

t{=}400

, dashed line) it dips and re-tracks to the same plateau, not a higher one—no cumulative improvement. This is a converged policy, not a learner.

Figure 8: Learner-free swarm vs. trained multi-agent RL, same optimum (abstract congestion game). (a) matching slope and (b) Nash-gap (exploitability) vs. learning steps for independent softmax policy-gradient and

Q

-learning (bootstrap

95\%

CI over

64

replicas), with the zero-training swarm and the analytic optimum as references: the swarm reaches the right allocation shape at zero training (a), but its Nash-gap (

0.58

) stays well above the trained learners’ (b)—it is not near-optimal on exploitability. The learners play an easier, disembodied problem (no spatial search); units differ (learner steps vs. swarm physical steps), as marked. (c) Closeness to the regime optimum in the congestion regime (optimum

=

ideal free distribution) and a non-congestion regime (fixed per-patch reward, optimum

=

richest patch). The swarm is near-optimal only in the congestion regime its mechanism matches; the learners adapt to both. Learning buys generality.

Figure 9: Embodied comparison to properly-trained learners: a budget separation, not an asymptotic one. Same spatial environment and matched sensing/movement/action set for all arms; fraction of a validated oracle ceiling;

16

seeds per learner. (a) learning curves (median and inter-quartile band) for independent PPO (IPPO) and a centralised-critic CTDE learner (MAPPO): both start below the reward-free swarm (dashed, zero training) and random (dotted), cross the swarm at

{\approx}4{,}000

environment steps (of

15{,}000

total), and plateau at

0.72

–

0.75

{\approx}10^{4}

steps (flat tail

\Rightarrow

convergence demonstrated, not assumed). (b) converged fractions: random

0.36

, swarm

0.65

(zero training, no reward), IPPO

0.75

, MAPPO

0.74

. (c) per-seed finals: all

32

learner seeds finish above the swarm mean; the swarm’s best seeds (

0.75

) reach into the learners’ lower tail. The swarm’s value is the zero-shot, reward-free operating point—it leads the low-budget regime and lands within

{\approx}0.10

of the converged learners—not asymptotic superiority.

To reach the optimum	Learner-free swarm (this work)	Trained multi-agent RL
Reward signal	none (reward-free, Fig. 6 )	per-agent scalar payoff, every step
Stored parameters	none	a policy (and critic) network
Training before use	none; acts at its operating point	${\approx}4{,}000$ env-steps to match, ${\approx}10^{4}$ to plateau (of $15{,}000$ )
Communication	local recruitment only (within range)	none, but needs a reward signal every step
Movement to coordinate	none beyond exploration (recruits reuse the exploring step)	learned from local observations
Precision, abstract game	right shape (slope $0.95$ ), but exploitable (Nash-gap $0.58$ )	near-exact (Nash-gap ${\approx}0$ )

Table 2: The two pay for different things. The learner-free swarm reaches the allocation shape with no reward signal, no parameters, and no training, and is competitive in the low-budget regime; the trained learner spends a reward signal, parameters, and training to buy precision (disembodied), a modestly higher converged allocation (embodied), and transfer. In the embodied environment the swarm leads at zero budget and the learner overtakes it only after

{\approx}4{,}000

steps (Fig. 9 ).

Multi-Agent Resource Allocation

See all 28 papers

Aug 28, 2026

Emergent aggregation from collective foraging

Collective behaviour in living systems is usually modelled as the outcome of a direct social drive: agents are rewarded, or hard-wired, to align with or approach their neighbours. Here we show that aggregation can instead emerge from an indirect objective. We let reinforcement learning foragers, initially performing a random walk, optimize their dynamics from a purely individual reward for finding replenishable targets, while perceiving only their conspecifics and never the targets themselves. As the visual range grows, the agents undergo a sharp crossover from an environment-tuned individual search to a scale-agnostic collective one, and this crossover coincides with the onset of spatial aggregation. Thus a collective phase arises as a by-product of optimal foraging, without any direct reward for grouping. A minimal analytical first-passage model reproduces the transition as a crossover between the two search strategies. Our results identify indirect, resource-driven reward as a generic route to emergent collective phenomena.

Gorka Muñoz-Gil, Andrea López-Incera, Vide Ramsten +3

University of Innsbruck, Department for Theoretical Physics, Technikerstr. 21a, A-6020 Innsbruck, Austria · Universitat Aut`onoma de Barcelona, Departamento de Did´actica de la Matem´atica y las Ciencias Experimentales, 08193 Bellaterra (Cerdanyola del Vall`es), Spain · Department of Physics, University of Gothenburg, Origovägen 6B, 41296, Göteborg, Sweden +1

Emergent Behavior in Multi-Agent Systems

Ask PDF

May 3, 2026

Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning

Cooperative multi-agent reinforcement learning (MARL) requires agents to discover joint strategies in a combinatorially large state-action space, yet effective coordination configurations are exceedingly rare. Intrinsic motivation, which augments task rewards with novelty bonuses, is a popular approach for driving exploration, but its effectiveness hinges on the exploration intensity

β

, where too large a value overwhelms the task signal and causes coordination collapse, while too small a value prevents discovery of rare strategies. We address two complementary challenges: adapting

β

globally over training, and allocating the exploration budget across agents whose intrinsic reward signals vary in reliability. Our framework combines a return-conditioned sigmoid schedule (RCB) for global intensity control with a per-agent Reward Signal Quality (RSQ) metric that concentrates the exploration budget on agents with reliable signals. The core insight is that agents receiving noisy intrinsic rewards should explore less aggressively, and this allocation can be determined automatically from signal-to-noise statistics. Successor Distance (SD), a quasimetric intrinsic reward, naturally produces distinguishable per-agent signal quality, completing the framework with convergence and ordering preservation guarantees. On seven cooperative benchmarks (MPE, SMAX, MABrax), our method achieves top-tier returns across all environments.

Dahyun Oh, Minhyuk Yoon, H. Jin Kim

Department of Aerospace Engineering, Seoul National University, Seoul, Republic of Korea

Multi-Agent Coordination Intrinsic Reward Methods Reinforcement Learning Intrinsic Motivation RL Exploration Cooperative MARL

Ask PDF

May 2, 2026

LLM-Foraging: Large Language Models for Decentralized Swarm Robot Foraging

Swarm foraging algorithms, such as the central-place foraging algorithm (CPFA), typically rely on offline parameter optimization using genetic algorithms (GA) or reinforcement learning, yielding policies tightly coupled to a specific combination of team size, arena size, and resource distribution. When deployment conditions change, performance degrades, and retraining is computationally expensive. We propose LLM-Foraging, a decentralized swarm controller that augments the CPFA state machine with a large language model (LLM) tactical decision-maker at three structured decision points, namely post-deposit, central-zone arrival, and search starvation. Each robot runs its own LLM client and queries it using only locally observable state, while the existing CPFA motion and sensing stack executes the selected action. Because the LLM serves as a general decision policy rather than parameters fitted to a single configuration, the controller is training-free at deployment and transfers across configurations without re-optimization. We evaluate LLM-Foraging in Gazebo with TurtleBot3 robots across 36 configurations spanning team sizes of 4 to 10 robots, arena sizes from 6x6 to 10x10 meters, and three resource distributions (clustered, powerlaw, random). LLM-Foraging collects more resources than the GA-tuned CPFA baseline across the evaluated configurations and is more consistent, a property that the GA's single-configuration tuning does not transfer.

Peihan Li, Joanna Gutierrez, Fabian Hernandez +2

Drexel University, Philadelphia, PA 19104, USA · The University of Texas Rio Grande Valley, Edinburg, Texas 78541, USA

Swarm Robotics Language Model-Based Control Robotics Multi-Robot Coordination Robot Policy Generalization

Ask PDF

Abstract

Multi-Agent Resource Allocation

Explore similar work

Emergent aggregation from collective foraging

Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning

LLM-Foraging: Large Language Models for Decentralized Swarm Robot Foraging