Robots that learn to evaluate models of collective behavior
Authors: Mathis Hocke, Andreas Gerken, David Bierbach, Jens Krause, Tim Landgraf
Organizations: Department of computer science, Freie Universität Berlin, 14195 Berlin, Germany. · SCIoI Excellence Cluster, Technische Universität Berlin, 10587, Berlin, Germany. · Faculty of Life Sciences, Humboldt-Universität zu Berlin, 10117, Berlin, Germany. · Department of Fish Biology, Fisheries, and Aquaculture, Leibniz Institute of Freshwater Ecology and Inland Fisheries, 12587 Berlin, Germany.
Understanding and modeling animal behavior is essential for studying collective motion, decision-making, and bio-inspired robotics. Yet, evaluating the accuracy of behavioral models still often relies on offline comparisons to static trajectory statistics. Here we introduce a reinforcement-learning-based framework that uses a biomimetic robotic fish (RoboFish) to evaluate computational models of live fish behavior through closed-loop interaction. We trained policies in simulation using four distinct fish models-a simple constant-follow baseline, two rule-based models, and a biologically grounded convolutional neural network model-and transferred these policies to the real RoboFish setup, where they interacted with live fish. Policies were trained to guide a simulated fish to goal locations, enabling us to quantify how the response of real fish differs from the simulated fish's response. We evaluate the fish models by quantifying the sim-to-real gaps, defined as the Wasserstein distance between simulated and real distributions of behavioral metrics such as goal-reaching performance, inter-individual distances, wall interactions, and alignment. The neural network-based fish model exhibited the smallest gap across goal-reaching performance and most other metrics, indicating higher behavioral fidelity than conventional rule-based models under this benchmark. More importantly, this separation shows that the proposed evaluation can quantitatively distinguish candidate models under matched closed-loop conditions. Our work demonstrates how learning-based robotic experiments can uncover deficiencies in behavioral models and provides a general framework for evaluating animal behavior models through embodied interaction.
Figures & tables
Figure 1: Workflow of our sim-to-real model evaluation framework. Live fish trajectories are recorded and used to train behavioral models. Each model is then used to train an RL control policy in simulation. The learned policy is transferred to real fish–robot interaction experiments, behavioral metrics are compared between simulation and reality, and the resulting sim-to-real discrepancy produces a model-realism estimate.
Figure 2: Experimental setup for guiding live guppies with a magnetically controlled replica. ( A ) Live guppies interacting with the magnetically controlled guppy replica in the tank. ( B ) Two-wheeled robot positioned below the tank that controls the replica’s movement. ( C ) 1 × 1 m experimental tank with the robot underneath.
Figure 3: Distributions of the number of goals reached. Goals reached in simulation (blue) and in the real environment with a live guppy (orange). Models MFollow , MZone and MForce allow the respective policies to learn strategies that perform much better in simulation than they do with real fish. The policy trained with MConvNet performs more similarly in simulation and reality. The distributions in the real setup are consistently wider than in simulation. Each distribution is based on n=18 trials per policy and environment. Total goals across all trials (simulation/real): πFollow 1087/177, πZone 733/104, πForce 917/378, πConvNet 212/232.
Figure 4: Sim-to-real gap across behavioral metrics. Annotated heatmap summarizing sim-to-real gaps for each policy (rows) and metric (columns). Each cell reports a point estimate of the sim-to-real gap based on trial-level Wasserstein distances, min–max normalized per metric, with the 95% bootstrap confidence interval shown below in brackets. The bold numbers on the left of each cell indicate the within-metric ranking of policies based on this point estimate (1 = smallest gap). Averaging ranks across all metrics except goals reached yields mean ranks of πFollow : 3.56, πZone : 3.11, πForce : 2.11, and πConvNet : 1.22, consistent with the ranking induced by the goals reached metric alone.
Figure 5: Sim-to-sim transfer between behavioral models. Wasserstein distances (median and 95% bootstrap CI) between the distributions of goals reached when a policy trained with a given behavioral model (“Training model”, columns) is rolled out using the same or a different model (“Rollout model”, rows). Each cell quantifies how well a policy trained on one model generalizes to another. Distances on the diagonal remain small but nonzero due to bootstrap variability, whereas off-diagonal distances are substantially larger, demonstrating that the four fish behavior models induce distinct interaction dynamics. Consequently, RL policies learn model-specific strategies that do not transfer across models, confirming that the models differ meaningfully in their behavioral assumptions.
Figure 6: Inter-individual distance (IID) in simulation (blue) and in the real environment with a live guppy (orange). Thin solid lines show per-trial kernel density estimates over per-time-step IID samples ( n=18 trials per policy and environment; 15 min at 25 Hz), highlighting inter-trial variability. Dotted lines indicate the pooled IID distributions across all trials.
Figure 7: Approach behavior toward a stationary fish model across policies. ( A ) Top-down view of the initial setup, with the robot facing the center of the tank, the goal positioned on its right, and a stationary fish model placed on its left. ( B ) Inter-individual distance (IID) over time. All policies except πFollow reduce their distance to the fish model to some extent. πForce repeatedly approaches very closely and spends more than half of the trial on the side of the tank nearest the fish model.
Figure 8: Mean goals reached per minute over time per policy in simulation and with live guppies. ( A ) Simulation results. ( B ) Results with live guppies. Lines show the mean across n=18 trials per policy; shaded areas show one standard deviation. Each trial starts with two minutes of milling, followed by 15 minutes of closed-loop control shown in the plot. Dashed lines indicate fitted negative-binomial regression trends for per-minute goal counts. See Appendix Table S3 for regression results.
Parameter
value
zone of repulsion
2 cm
zone of orientation
6 cm
zone of attraction
10 cm
speed
6.30 cm/s
field of view
2π
turn noise
0.2
Table S1 : Parameters for MZone .
Parameter
Value
Social interaction parameters
Interaction range (neighbor radius)
15 cm
Preferred inter-individual distance ( rd )
5.0 cm
Velocity alignment strength ( μalg )
0.1
Distance regulation strength ( μd )
0.3
Distance interaction slope ( md )
0.2
Table S2 : Parameters of MForce .
Env
Policy
RR/min (95% CI)
p
sim
total
1.006 [1.002, 1.010]
0.004
sim
πFollow
1.002 [1.000, 1.004]
0.011
sim
πZone
1.008 [0.998, 1.018]
0.107
sim
πForce
1.009 [1.000, 1.018]
0.056
sim
πConvNet
1.006 [0.984, 1.030]
0.588
real
total
0.974 [0.956, 0.992]
0.005
Table S3 : Temporal trend in goals reached per minute. We fit a negative-binomial regression (log link) of per-minute goal counts on time (minute index), separately for each policy and pooled across policies, using trial-clustered robust standard errors. The table reports the per-minute rate ratio (RR) with 95% CI and the p -value for the time coefficient.
This study investigates a method to guide and control fish schools using virtual fish trained with reinforcement learning. We utilize 2D virtual fish displayed on a screen to overcome technical challenges such as durability and movement constraints inherent in physical robotic agents. To address the lack of detailed behavioral models for real fish, we adopt a model-free reinforcement learning approach. First, simulation results show that reinforcement learning can acquire effective movement policies even when simulated real fish frequently ignore the virtual stimulus. Second, real-world experiments with live fish confirm that the learned policy successfully guides fish schools toward specified target directions. Statistical analysis reveals that the proposed method significantly outperforms baseline conditions, including the absence of stimulus and a heuristic "stay-at-edge" strategy. This study provides an early demonstration of how reinforcement learning can be used to influence collective animal behavior through artificial agents.
Yusuke Nishii, Hiroaki Kawashima
Faculty of Engineering, Kyoto University, Kyoto, Japan · Graduate School of Informatics, Kyoto University, Kyoto, Japan
Human behaviour models are increasingly used to provide simulated human interactions for training robots via reinforcement learning (RL). Existing approaches primarily evaluate these models according to how closely their generated behaviours resemble real human data, while their usefulness as training partners is less commonly assessed. This creates a potential objective mismatch: a model that performs well according to behavioural evaluation metrics may not necessarily provide useful training experiences for downstream robot policy learning. Across two human-robot interaction (HRI) domains, highway driving and object handover, we evaluate different types of human behaviour models using common generative-model evaluation metrics. We consider multiple human trajectory representations and compare the resulting evaluations with the downstream performance of RL policies trained using the corresponding models. We find that existing behavioural evaluation metrics do not consistently predict downstream policy performance across domains and trajectory representations, while their evaluations are sensitive to the choice of trajectory representation. We also show that even when the true testing behaviour distribution is known, training against a different distribution may yield better downstream performance than training directly against the true distribution.
Haoyang Jiang, Dana Kulic, Michael Burke +1
Department of Electrical and Computer Systems Engineering, Monash University, Clayton, Australia
Collective behavior of fish schools is usually modeled on the assumption that each individual follows specific rules of motion that depend on its position and velocity relative to its neighbors. Although these models reproduce many schooling patterns observed in nature, it remains unclear whether the assumed rules are realistic at the individual level. To address this question, we first analyzed a set of experiments in which a live fish interacted with four moving robotic fish in a tank, and measured the time it spent in each position relative to the robots. We then simulated this experiment, replacing the fish with an agent, and assessed the extent to which classical models agree with the experimental observations. An extensive exploration of the parameter space showed that these models closely matched the experimental observations, with a very specific choice of parameters. However, they were highly sensitive to the parameter values: a minor perturbation caused them to fail completely. To resolve this, we propose a new model that incorporates an additional exploratory term characterizing the agent's positional preference at every moment. This model proved robust to small errors in the parameters while successfully replicating the experiments. Furthermore, when generalised to multiple fish, the model reproduced schooling behaviour and common schooling patterns, while also exhibiting exploratory behaviour at both the individual and the school level. This shows that social cohesion coexists with individual exploratory behaviour, and that realistic models of collective behaviour should account for both social interactions and this exploratory component.
Sorbonne University Abu Dhabi, Abu Dhabi, UAE · BioRobotics Institute, Sant’Anna School of Advanced Studies, Pisa, Italy · Division of Computing and Mathematical Science, MBZUAI, Abu Dhabi, UAE