Robots that learn to evaluate models of collective behavior

Mathis Hocke; Andreas Gerken; David Bierbach; Jens Krause; Tim Landgraf

cs.ROApr 8, 2026

Robots that learn to evaluate models of collective behavior

Authors: Mathis Hocke, Andreas Gerken, David Bierbach, Jens Krause, Tim Landgraf

Organizations: Department of computer science, Freie Universität Berlin, 14195 Berlin, Germany. · SCIoI Excellence Cluster, Technische Universität Berlin, 10587, Berlin, Germany. · Faculty of Life Sciences, Humboldt-Universität zu Berlin, 10117, Berlin, Germany. · Department of Fish Biology, Fisheries, and Aquaculture, Leibniz Institute of Freshwater Ecology and Inland Fisheries, 12587 Berlin, Germany.

Abstract

Understanding and modeling animal behavior is essential for studying collective motion, decision-making, and bio-inspired robotics. Yet, evaluating the accuracy of behavioral models still often relies on offline comparisons to static trajectory statistics. Here we introduce a reinforcement-learning-based framework that uses a biomimetic robotic fish (RoboFish) to evaluate computational models of live fish behavior through closed-loop interaction. We trained policies in simulation using four distinct fish models-a simple constant-follow baseline, two rule-based models, and a biologically grounded convolutional neural network model-and transferred these policies to the real RoboFish setup, where they interacted with live fish. Policies were trained to guide a simulated fish to goal locations, enabling us to quantify how the response of real fish differs from the simulated fish's response. We evaluate the fish models by quantifying the sim-to-real gaps, defined as the Wasserstein distance between simulated and real distributions of behavioral metrics such as goal-reaching performance, inter-individual distances, wall interactions, and alignment. The neural network-based fish model exhibited the smallest gap across goal-reaching performance and most other metrics, indicating higher behavioral fidelity than conventional rule-based models under this benchmark. More importantly, this separation shows that the proposed evaluation can quantitatively distinguish candidate models under matched closed-loop conditions. Our work demonstrates how learning-based robotic experiments can uncover deficiencies in behavioral models and provides a general framework for evaluating animal behavior models through embodied interaction.

Figures & tables

Figure 1: Workflow of our sim-to-real model evaluation framework. Live fish trajectories are recorded and used to train behavioral models. Each model is then used to train an RL control policy in simulation. The learned policy is transferred to real fish–robot interaction experiments, behavioral metrics are compared between simulation and reality, and the resulting sim-to-real discrepancy produces a model-realism estimate.

Figure 2: Experimental setup for guiding live guppies with a magnetically controlled replica. ( A ) Live guppies interacting with the magnetically controlled guppy replica in the tank. ( B ) Two-wheeled robot positioned below the tank that controls the replica’s movement. ( C ) 1 $\times$ 1 m experimental tank with the robot underneath.

Figure 3: Distributions of the number of goals reached. Goals reached in simulation (blue) and in the real environment with a live guppy (orange). Models $M_{Follow}$ , $M_{Zone}$ and $M_{Force}$ allow the respective policies to learn strategies that perform much better in simulation than they do with real fish. The policy trained with $M_{ConvNet}$ performs more similarly in simulation and reality. The distributions in the real setup are consistently wider than in simulation. Each distribution is based on $n=18$ trials per policy and environment. Total goals across all trials (simulation/real): $\pi_{Follow}$ 1087/177, $\pi_{Zone}$ 733/104, $\pi_{Force}$ 917/378, $\pi_{ConvNet}$ 212/232.

Figure 4: Sim-to-real gap across behavioral metrics. Annotated heatmap summarizing sim-to-real gaps for each policy (rows) and metric (columns). Each cell reports a point estimate of the sim-to-real gap based on trial-level Wasserstein distances, min–max normalized per metric, with the 95% bootstrap confidence interval shown below in brackets. The bold numbers on the left of each cell indicate the within-metric ranking of policies based on this point estimate (1 = smallest gap). Averaging ranks across all metrics except goals reached yields mean ranks of $\pi_{Follow}$ : 3.56, $\pi_{Zone}$ : 3.11, $\pi_{Force}$ : 2.11, and $\pi_{ConvNet}$ : 1.22, consistent with the ranking induced by the goals reached metric alone.

Figure 5: Sim-to-sim transfer between behavioral models. Wasserstein distances (median and 95% bootstrap CI) between the distributions of goals reached when a policy trained with a given behavioral model (“Training model”, columns) is rolled out using the same or a different model (“Rollout model”, rows). Each cell quantifies how well a policy trained on one model generalizes to another. Distances on the diagonal remain small but nonzero due to bootstrap variability, whereas off-diagonal distances are substantially larger, demonstrating that the four fish behavior models induce distinct interaction dynamics. Consequently, RL policies learn model-specific strategies that do not transfer across models, confirming that the models differ meaningfully in their behavioral assumptions.

Figure 6: Inter-individual distance (IID) in simulation (blue) and in the real environment with a live guppy (orange). Thin solid lines show per-trial kernel density estimates over per-time-step IID samples ( $n=18$ trials per policy and environment; 15 min at 25 Hz), highlighting inter-trial variability. Dotted lines indicate the pooled IID distributions across all trials.

Figure 7: Approach behavior toward a stationary fish model across policies. ( A ) Top-down view of the initial setup, with the robot facing the center of the tank, the goal positioned on its right, and a stationary fish model placed on its left. ( B ) Inter-individual distance (IID) over time. All policies except $\pi_{Follow}$ reduce their distance to the fish model to some extent. $\pi_{Force}$ repeatedly approaches very closely and spends more than half of the trial on the side of the tank nearest the fish model.

Figure 8: Mean goals reached per minute over time per policy in simulation and with live guppies. ( A ) Simulation results. ( B ) Results with live guppies. Lines show the mean across $n=18$ trials per policy; shaded areas show one standard deviation. Each trial starts with two minutes of milling, followed by 15 minutes of closed-loop control shown in the plot. Dashed lines indicate fitted negative-binomial regression trends for per-minute goal counts. See Appendix Table S3 for regression results.

Parameter	value
zone of repulsion	2 cm
zone of orientation	6 cm
zone of attraction	10 cm
speed	6.30 cm/s
field of view	$2\pi$
turn noise	0.2

Table S1 : Parameters for

M_{Zone}

Parameter	Value
Social interaction parameters
Interaction range (neighbor radius)	15 cm
Preferred inter-individual distance ( $r_{d}$ )	5.0 cm
Velocity alignment strength ( $\mu_{\mathrm{alg}}$ )	0.1
Distance regulation strength ( $\mu_{d}$ )	0.3
Distance interaction slope ( $m_{d}$ )	0.2

Table S2 : Parameters of

M_{Force}

Env	Policy	RR/min (95% CI)	p
sim	total	1.006 [1.002, 1.010]	0.004
sim	$\pi_{Follow}$	1.002 [1.000, 1.004]	0.011
sim	$\pi_{Zone}$	1.008 [0.998, 1.018]	0.107
sim	$\pi_{Force}$	1.009 [1.000, 1.018]	0.056
sim	$\pi_{ConvNet}$	1.006 [0.984, 1.030]	0.588
real	total	0.974 [0.956, 0.992]	0.005

Table S3 : Temporal trend in goals reached per minute. We fit a negative-binomial regression (log link) of per-minute goal counts on time (minute index), separately for each policy and pooled across policies, using trial-clustered robust standard errors. The table reports the per-minute rate ratio (RR) with 95% CI and the

p

-value for the time coefficient.

Animal Behavior Analysis

Animal behavior analysis uses computational tools to measure how animals move, interact, communicate, and respond to their environments.

See all 37 papers

Controlling Fish Schools via Reinforcement Learning of Virtual Fish Movement
Mar 17, 2026Yusuke Nishii, Hiroaki Kawashima
Faculty of Engineering, Kyoto University, Kyoto, Japan · Graduate School of Informatics, Kyoto University, Kyoto, Japan
Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning
Oct 26, 2026Haoyang Jiang, Dana Kulic, Michael Burke +1Human Behavior Modeling Human Behavior Simulation
Department of Electrical and Computer Systems Engineering, Monash University, Clayton, Australia
Positional choice and robust collective behavior in fish schools: biohybrid experiments and modeling
Sep 28, 2026Vahagn Grigoryan, Donato Romano, Cesare Stefanini +1Agent-Based Modeling Animal Behavior Analysis
Sorbonne University Abu Dhabi, Abu Dhabi, UAE · BioRobotics Institute, Sant’Anna School of Advanced Studies, Pisa, Italy · Division of Computing and Mathematical Science, MBZUAI, Abu Dhabi, UAE

Mar 17, 2026J/K move · Enter open · S save

Controlling Fish Schools via Reinforcement Learning of Virtual Fish Movement

Yusuke Nishii, Hiroaki Kawashima

Faculty of Engineering, Kyoto University, Kyoto, Japan · Graduate School of Informatics, Kyoto University, Kyoto, Japan

Open paper AskPDF arXiv

This study investigates a method to guide and control fish schools using virtual fish trained with reinforcement learning. We utilize 2D virtual fish displayed on a screen to overcome technical challenges such as durability and movement constraints inherent in physical robotic agents. To address the lack of detailed behavioral models for real fish, we adopt a model-free reinforcement learning approach. First, simulation results show that reinforcement learning can acquire effective movement policies even when simulated real fish frequently ignore the virtual stimulus. Second, real-world experiments with live fish confirm that the learned policy successfully guides fish schools toward specified target directions. Statistical analysis reveals that the proposed method significantly outperforms baseline conditions, including the absence of stimulus and a heuristic "stay-at-edge" strategy. This study provides an early demonstration of how reinforcement learning can be used to influence collective animal behavior through artificial agents.

Abstract

Animal Behavior Analysis

Explore similar work

Controlling Fish Schools via Reinforcement Learning of Virtual Fish Movement

Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning

Positional choice and robust collective behavior in fish schools: biohybrid experiments and modeling