Emergent aggregation from collective foraging

Gorka Muñoz-Gil; Andrea López-Incera; Vide Ramsten; Giovanni Volpe; Thomas Müller; Hans J. Briegel

cond-mat.stat-mechAug 28, 2026

Emergent aggregation from collective foraging

Authors: Gorka Muñoz-Gil, Andrea López-Incera, Vide Ramsten, Giovanni Volpe, Thomas Müller, Hans J. Briegel

Organizations: University of Innsbruck, Department for Theoretical Physics, Technikerstr. 21a, A-6020 Innsbruck, Austria · Universitat Aut`onoma de Barcelona, Departamento de Did´actica de la Matem´atica y las Ciencias Experimentales, 08193 Bellaterra (Cerdanyola del Vall`es), Spain · Department of Physics, University of Gothenburg, Origovägen 6B, 41296, Göteborg, Sweden · Department of Philosophy, Box 17, University of Konstanz, 78457 Konstanz, Germany

Abstract

Collective behaviour in living systems is usually modelled as the outcome of a direct social drive: agents are rewarded, or hard-wired, to align with or approach their neighbours. Here we show that aggregation can instead emerge from an indirect objective. We let reinforcement learning foragers, initially performing a random walk, optimize their dynamics from a purely individual reward for finding replenishable targets, while perceiving only their conspecifics and never the targets themselves. As the visual range grows, the agents undergo a sharp crossover from an environment-tuned individual search to a scale-agnostic collective one, and this crossover coincides with the onset of spatial aggregation. Thus a collective phase arises as a by-product of optimal foraging, without any direct reward for grouping. A minimal analytical first-passage model reproduces the transition as a crossover between the two search strategies. Our results identify indirect, resource-driven reward as a generic route to emergent collective phenomena.

Figures & tables

Figure 1: Model. An agent (blue) moves by continue and turn actions and perceives conspecifics within a vision cone of half-width

\pi/8

and range

r_{v}

. Targets (green, radius

r

) are not visible to the agents. A tagged agent (red) has just collected a target and is visible as rewarded for a time

\tau_{R}

; grey agents are non-rewarded.

Figure 2: Foraging efficiency. Mean reward per step $\eta$ versus visual range $r_{v}$ in the cooperative regime. (a) Fixed tag time $\tau_{R}=9$ , several depletion times $\tau$ . (b) Fixed $\tau=2$ , several tag times $\tau_{R}$ . Below a $\tau$ - and $\tau_{R}$ -dependent visual range the agents recover the blind single-agent efficiency (dashed lines); above it, $\eta$ jumps and plateaus. Error bars: SEM over 4 runs of $N_{a}=50$ agents.

Figure 3: Learned policies. Continuing probability of the learned policies $\pi$ for the three visual states, in the competitive and cooperative scenarios, at $\tau=4$ , $\tau_{R}=9$ , $N_{a}=50$ . Dashed lines in no agent state: single-agent (blind) policy. Dotted lines: initial policy. Error bars: standard deviation over 4 runs and $N_{a}$ agents (collective) and over $12000$ agents (single).

Figure 4: Aggregation. Results for an environment with $N_{a}=25$ , $\tau_{R}=9$ and the cooperative scenario. a) Spatial distribution of agents over the final 5000 steps of a 15000 step run with $N_{t}=100$ with trained policies at parameter values marked by the corresponding symbols in c. Four exemplary agent trajectories are overlaid. b, c) Clark–Evans index $R$ for two target densities $N_{t}$ . Left panels: $R$ over the $(\tau,r_{v})$ plane; dashed lines show Eq. 1 with $\ell=5,8$ , respectively. Right panels: line cuts over previous plots; error bars: standard deviation over 4 runs and $N_{a}$ agents.

Figure 5: Role of the depletion rule. Clark–Evans index $R$ over the $(\tau_{R},r_{v})$ plane for the competitive and cooperative scenarios for $N_{a}=25$ , $\tau=2$ , and $N_{t}=100$ .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Single agent MFPT Scaling of the MFPT for the single agent as a function of the depletion time

\tau

, for

N_{t}=10

r=0.5

and different box side lengths

L

. Blue data points show averages over

5\cdot 10^{3}

simulations, with the errorbars showcasing the standard deviation of the mean. Solid lines are obtained from Eq. 5 with fitted parameters

A,B

shown in the plot. Dotted line shows the value of the mean path length Eq. 3 .

Figure 7: Single-Collective strategy crossover Blue line shows the crossover between the single and collective strategy when the former follows Eq. 10 . The red lines show Eq. 14 for different values of $\ell$ and $B=0.64$ .

Figure 8: Finite-size analysis of the aggregation onset. Dense-phase order parameter

m

measured over 8000 random initializations of policies for four independent trainings per

N_{a}

, with the system scaled at fixed agent and target density (

L\propto\sqrt{N_{a}}

). (a) Mean order parameter

\langle m\rangle

and (b) fraction of runs reaching the aggregated state (

m>0

) versus visual range

r_{v}

. (c)–(f) Distribution

P(m)

near

r_{v}^{*}

(

r_{v}=7

) for four independently trained policies per parameter set. (g) Width of the transition region (the

r_{v}

interval over which the occupied fraction rises from

0.1

0.9

) versus

1/L^{2}

; the fit extrapolates to a finite intercept

c\approx 0.09

. (h) Onset location

r_{v}^{*}(L)

versus

1/L^{2}

, converging to

r_{v}^{*}(L\to\infty)\approx 6.2

. Error bars: spread across the four policies.

Emergent Behavior in Multi-Agent Systems

Emergent behavior in multi-agent systems studies how local agent interactions produce collective patterns such as cooperation, norms, and instability.

See all 170 papers

Self-organizing reinforcement learning: a learner-free swarm approaches the explore-exploit optimum
Oct 18, 2026Galen J. WilkersonMulti-Agent Resource Allocation Exploration-Exploitation Tradeoff
Komplexity AI LLC
A Mechanistic Model for Collective Motion from Sensorimotor Regularities
May 15, 2026Vito Mengers, Bao Duc Cao, Oliver BrockAgent-Based Modeling Animal Behavior Analysis
Robotics and Biology Laboratory, Technische Universität Berlin, Germany · Science of Intelligence, Research Cluster of Excellence, Berlin, Germany · Robotics Institute Germany
Social-spatial dependencies for learning visual navigation
Jul 8, 2026Patrick Govoni, Pawel RomanczukVisual Navigation Multi-Agent Coordination
Institute for Theoretical Biology, Department of Biology, Humboldt Universität zu Berlin, Berlin, Germany · Science of Intelligence, Research Cluster of Excellence, Berlin, Germany · Bernstein Center for Computational Neuroscience, Berlin, Germany

Oct 18, 2026J/K move · Enter open · S save

Self-organizing reinforcement learning: a learner-free swarm approaches the explore-exploit optimum

Galen J. Wilkerson

Komplexity AI LLC

Multi-Agent Resource Allocation Exploration-Exploitation Tradeoff Multi-Agent Systems Reward-Free RL

Open paper AskPDF arXiv

The explore/exploit trade-off is the core problem of reinforcement learning (RL). We show that a minimal foraging swarm approaches its optimum - the ideal-free / Nash allocation of a multi-agent resource-sharing game (an equilibrium, not a return maximum) - with no learner and no reward signal. Agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a threshold cascade on their proximity network, and disperse as a patch crowds; this cascade cycle balances exploration and exploitation and drives foragers toward the ideal free distribution. The swarm attains the shape of this optimum (committed foragers track richness with slope 0.94) but not its exact equilibrium - it stays measurably exploitable - reaching it dynamically (~150 steps) through the coupling of recruitment to movement: remove the coupling and matching collapses (0.94 -> 0.08). The RL sits at the level of the problem and the allocation reached, not the micro-rule: no step evaluates payoff (reward-free by construction, though the crowd signal is informationally tied to payoff), and the swarm reaches the allocation without learning it (quality plateaus and re-settles after resources move, no cumulative gain). Against a trained baseline the comparison is two-sided: in an easier, disembodied game a learner is more precise, while in the same embodied environment the reward-free swarm reaches ~65% of a validated oracle ceiling at zero training and leads until trained PPO and centralised-critic (CTDE) learners overtake it (~74-75%, 16 seeds, all above the swarm mean) after ~4,000 of 15,000 training steps - its advantage is in budget, not asymptote. In this one setting we read it not as beating RL but as a strong zero-shot, reward-free operating point: a demonstration that the self-organizing physics of a minimal swarm can carry a collective close to the optimum RL seeks, with none of RL's machinery.

Abstract

Emergent Behavior in Multi-Agent Systems

Explore similar work

Self-organizing reinforcement learning: a learner-free swarm approaches the explore-exploit optimum

A Mechanistic Model for Collective Motion from Sensorimotor Regularities

Social-spatial dependencies for learning visual navigation