Authors: Gorka Muñoz-Gil, Andrea López-Incera, Vide Ramsten, Giovanni Volpe, Thomas Müller, Hans J. Briegel
Organizations: University of Innsbruck, Department for Theoretical Physics, Technikerstr. 21a, A-6020 Innsbruck, Austria · Universitat Aut`onoma de Barcelona, Departamento de Did´actica de la Matem´atica y las Ciencias Experimentales, 08193 Bellaterra (Cerdanyola del Vall`es), Spain · Department of Physics, University of Gothenburg, Origovägen 6B, 41296, Göteborg, Sweden · Department of Philosophy, Box 17, University of Konstanz, 78457 Konstanz, Germany
Collective behaviour in living systems is usually modelled as the outcome of a direct social drive: agents are rewarded, or hard-wired, to align with or approach their neighbours. Here we show that aggregation can instead emerge from an indirect objective. We let reinforcement learning foragers, initially performing a random walk, optimize their dynamics from a purely individual reward for finding replenishable targets, while perceiving only their conspecifics and never the targets themselves. As the visual range grows, the agents undergo a sharp crossover from an environment-tuned individual search to a scale-agnostic collective one, and this crossover coincides with the onset of spatial aggregation. Thus a collective phase arises as a by-product of optimal foraging, without any direct reward for grouping. A minimal analytical first-passage model reproduces the transition as a crossover between the two search strategies. Our results identify indirect, resource-driven reward as a generic route to emergent collective phenomena.
Figures & tables
Figure 1: Model. An agent (blue) moves by continue and turn actions and perceives conspecifics within a vision cone of half-width π/8 and range rv . Targets (green, radius r ) are not visible to the agents. A tagged agent (red) has just collected a target and is visible as rewarded for a time τR ; grey agents are non-rewarded.
Figure 2: Foraging efficiency. Mean reward per step η versus visual range rv in the cooperative regime. (a) Fixed tag time τR=9 , several depletion times τ . (b) Fixed τ=2 , several tag times τR . Below a τ - and τR -dependent visual range the agents recover the blind single-agent efficiency (dashed lines); above it, η jumps and plateaus. Error bars: SEM over 4 runs of Na=50 agents.
Figure 3: Learned policies. Continuing probability of the learned policies π for the three visual states, in the competitive and cooperative scenarios, at τ=4 , τR=9 , Na=50 . Dashed lines in no agent state: single-agent (blind) policy. Dotted lines: initial policy. Error bars: standard deviation over 4 runs and Na agents (collective) and over 12000 agents (single).
Figure 4: Aggregation. Results for an environment with Na=25 , τR=9 and the cooperative scenario. a) Spatial distribution of agents over the final 5000 steps of a 15000 step run with Nt=100 with trained policies at parameter values marked by the corresponding symbols in c. Four exemplary agent trajectories are overlaid. b, c) Clark–Evans index R for two target densities Nt . Left panels: R over the (τ,rv) plane; dashed lines show Eq. 1 with ℓ=5,8 , respectively. Right panels: line cuts over previous plots; error bars: standard deviation over 4 runs and Na agents.
Figure 5: Role of the depletion rule. Clark–Evans index R over the (τR,rv) plane for the competitive and cooperative scenarios for Na=25 , τ=2 , and Nt=100 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Single agent MFPT Scaling of the MFPT for the single agent as a function of the depletion time τ , for Nt=10 , r=0.5 and different box side lengths L . Blue data points show averages over 5⋅103 simulations, with the errorbars showcasing the standard deviation of the mean. Solid lines are obtained from Eq. 5 with fitted parameters A,B shown in the plot. Dotted line shows the value of the mean path length Eq. 3 .
Figure 7: Single-Collective strategy crossover Blue line shows the crossover between the single and collective strategy when the former follows Eq. 10 . The red lines show Eq. 14 for different values of ℓ and B=0.64 .
Figure 8: Finite-size analysis of the aggregation onset. Dense-phase order parameter m measured over 8000 random initializations of policies for four independent trainings per Na , with the system scaled at fixed agent and target density ( L∝Na ). (a) Mean order parameter ⟨m⟩ and (b) fraction of runs reaching the aggregated state ( m>0 ) versus visual range rv . (c)–(f) Distribution P(m) near rv∗ ( rv=7 ) for four independently trained policies per parameter set. (g) Width of the transition region (the rv interval over which the occupied fraction rises from 0.1 to 0.9 ) versus 1/L2 ; the fit extrapolates to a finite intercept c≈0.09 . (h) Onset location rv∗(L) versus 1/L2 , converging to rv∗(L→∞)≈6.2 . Error bars: spread across the four policies.
The explore/exploit trade-off is the core problem of reinforcement learning (RL). We show that a minimal foraging swarm approaches its optimum - the ideal-free / Nash allocation of a multi-agent resource-sharing game (an equilibrium, not a return maximum) - with no learner and no reward signal. Agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a threshold cascade on their proximity network, and disperse as a patch crowds; this cascade cycle balances exploration and exploitation and drives foragers toward the ideal free distribution. The swarm attains the shape of this optimum (committed foragers track richness with slope 0.94) but not its exact equilibrium - it stays measurably exploitable - reaching it dynamically (~150 steps) through the coupling of recruitment to movement: remove the coupling and matching collapses (0.94 -> 0.08). The RL sits at the level of the problem and the allocation reached, not the micro-rule: no step evaluates payoff (reward-free by construction, though the crowd signal is informationally tied to payoff), and the swarm reaches the allocation without learning it (quality plateaus and re-settles after resources move, no cumulative gain). Against a trained baseline the comparison is two-sided: in an easier, disembodied game a learner is more precise, while in the same embodied environment the reward-free swarm reaches ~65% of a validated oracle ceiling at zero training and leads until trained PPO and centralised-critic (CTDE) learners overtake it (~74-75%, 16 seeds, all above the swarm mean) after ~4,000 of 15,000 training steps - its advantage is in budget, not asymptote. In this one setting we read it not as beating RL but as a strong zero-shot, reward-free operating point: a demonstration that the self-organizing physics of a minimal swarm can carry a collective close to the optimum RL seeks, with none of RL's machinery.
Collective behavior in animals has long been modeled through self-propelled particle models, which reproduce striking group-level phenomena through abstract interaction forces. Yet these models are fundamentally descriptive: they leave open the question of how collective behavior is actually produced. Recent empirical work makes this gap concrete: locusts do not align with neighbors, sensory and cognitive mechanisms mediate interaction instead. A mechanistic model must therefore operate at the sensorimotor level, grounded in what individual organisms can actually perceive, estimate, and physically execute. We present such a model based on a modeling framework from robotics, extended here to collective motion. Each agent perceives neighbors through bearing and apparent-size cues within a limited field of view, maintains uncertain internal state estimates, and selects actions through gradient descent on a desired social distance -- without any prescribed interaction forces. This simple model produces diverse collective behaviors including polarized motion, milling, ring formations, and subgroup fragmentation. A global sensitivity analysis shows that behavioral transitions are governed by sensorimotor parameters corresponding to measurable biological quantities: field of view geometry, sensory noise, turning agility, and memory. Collective behavior can therefore be understood as the emergent outcome of interacting sensorimotor regularities, and differences across species as the emergent outcome of differences in embodiment and environment.
Vito Mengers, Bao Duc Cao, Oliver Brock
Robotics and Biology Laboratory, Technische Universität Berlin, Germany · Science of Intelligence, Research Cluster of Excellence, Berlin, Germany · Robotics Institute Germany
Navigation for social organisms rarely is a fully independent activity. Group structure and dynamics, as well as embodied interactions, critically influence useful behavior. Individual neural network controlled agents are trained to navigate in different social contexts, where social dependence and behavioral strategy learned is determined by relative task performance and spatial effect. Increasing high quality social information drives phase transitions from individual to following navigational strategy, and to collision avoidance in response to a crowded foraging patch. Predictable, nonstationary environmental dynamics drive behavioral hybridization between individual and social navigation, far and near the patch. Our findings challenge the approach of only inspecting individual behavior for social organisms and highlight the importance of taking a bottom-up approach in understanding how organisms behave.
Patrick Govoni, Pawel Romanczuk
Institute for Theoretical Biology, Department of Biology, Humboldt Universität zu Berlin, Berlin, Germany · Science of Intelligence, Research Cluster of Excellence, Berlin, Germany · Bernstein Center for Computational Neuroscience, Berlin, Germany