Video-based Locomotion Analysis for Fish Health Monitoring

Timon Palm; Clemens Seibold; Anna Hilsmann; Peter Eisert

cs.CVMar 5, 2026

Video-based Locomotion Analysis for Fish Health Monitoring

Authors: Timon Palm, Clemens Seibold, Anna Hilsmann, Peter Eisert

Organizations: Fraunhofer HHI, Berlin, Germany · Humboldt University of Berlin, Berlin, Germany

Abstract

Monitoring the health conditions of fish is essential, as it enables the early detection of disease, safeguards animal welfare, and contributes to sustainable aquaculture practices. Physiological and pathological conditions of cultivated fish can be inferred by analyzing locomotion activities. In this paper, we present a system that estimates the locomotion activities from videos using multi object tracking. The core of our approach is a YOLOv11 detector embedded in a tracking-by-detection framework. We investigate various configurations of the YOLOv11-architecture as well as extensions that incorporate multiple frames to improve detection accuracy. Our system is evaluated on a manually annotated dataset of Sulawesi ricefish recorded in a home-aquarium-like setup, demonstrating its ability to reliably measure swimming direction and speed for fish health monitoring. The dataset will be made publicly available upon publication.

Figures & tables

Figure 1: The Sulawesi ricefish : Its lucid appearance, small size and the large number of instances in a shoal makes accurate tracking difficult.

Figure 2: A schematic view of extracting quantitative directional information of fish observations from Tracking-By-Detection methods.

Panel of Figure 3 — Figure 3: Example frames with segmentation annotation from our Sulawesi ricefish dataset.

Video	Fish Count	#Tracks	Fish per Frame (avg)
A	9214	256	$\sim 91.2$
B	1969	45	$\sim 19.5$

Table 1: A summary of the Sulawesi ricefish dataset specification for video A and B

Panel of Table 2 — Table 2: Precision, Recall, mAP50 [ 6 ] and mAP50-95 [ 13 ] metrics for each context configuration in our experiments. Best values are shown in bold, second best are underlined.

Scheme	Size	Video A				Video B
Scheme	Size	mAP50-95 $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	HOTA $\uparrow$	mAP50-95 $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	HOTA $\uparrow$
GT Detections	-	1.000	0.756	0.991	0.803	1.000	0.815	0.981	0.856
Single Frame	nano	0.451	0.396	0.165	0.330	0.368	0.486	0.360	0.373
	medium	0.413	0.465	0.005	0.374	0.442	0.461	0.265	0.362
	large	0.416	0.439	0.012	0.361	0.370	0.427	0.193	0.339
xXx	nano	0.445	0.539	0.331	0.408	0.398	0.463	0.363	0.346

Table 3: Bytetrack: Tracking results for using different context windows for the YOLOv11 detection model. The best performance is highlighted in bold and the second best is underlined.

Scheme	Size	Video A				Video B
Scheme	Size	mAP50-95 $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	HOTA $\uparrow$	mAP50-95 $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	HOTA $\uparrow$
GT Detections	-	1.000	0.756	0.991	0.803	1.000	0.824	0.976	0.856
Single Frame	nano	0.451	0.407	0.071	0.361	0.368	0.532	0.455	0.423
	medium	0.413	0.507	0.130	0.428	0.442	0.509	0.383	0.421
	large	0.416	0.489	0.099	0.422	0.370	0.489	0.313	0.400
xXx	nano	0.445	0.604	0.363	0.481	0.398	0.493	0.427	0.386

Table 4: BoT-SORT: Tracking results for using different context windows for the YOLOv11 detection model. The best performance is highlighted in bold and the second best is underlined.

Figure 4: Histograms depicting the angular swimming directions extracted from tracking with model size nano (left), medium (middle) and large (right) for video A (top row) and video B (bottom row).

Figure 5: Histograms showing the aggregated magnitude of the swimming directions from tracking with model size nano (left), medium (middle) and large (right) for video A (top row) and video B (bottom row).

Aquaculture

See all 10 papers

Automated 3D Kinematic Monitoring for Circadian Activity and Anomaly Detection in Juvenile Fish
Jun 5, 2026Chih-Wei Huang, Chang-Wen Huang, Chung-Ping Chiang +1Aquaculture Anomaly Detection
AI Research Center, National Taiwan Ocean Univ., Keelung City 20224, Taiwan. · Dept. of Aquaculture, National Taiwan Ocean Univ., Keelung City 20224, Taiwan. · Center of Excellence for the Oceans, National Taiwan Ocean University, Keelung City 20224, Taiwan.
Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos
Sep 30, 2026Hangong Chen, Linfeng Cheng, Tahsin Zaman Jilan +4Underwater 3D Reconstruction Robot Locomotion
Department of Computer Science, North Carolina A&T State University · Department of Computer Science, University of Maryland, College Park
Non-invasive Growth Monitoring of Small Freshwater Fish in Home Aquariums via Stereo Vision
Mar 6, 2026Clemens Seibold, Anna Hilsmann, Peter EisertStereo Vision
Humboldt University of Berlin, Berlin, Germany · Fraunhofer HHI, Berlin, Germany

Jun 5, 2026J/K move · Enter open · S save

Automated 3D Kinematic Monitoring for Circadian Activity and Anomaly Detection in Juvenile Fish

Chih-Wei Huang, Chang-Wen Huang, Chung-Ping Chiang, Tsung-Wei Pan

AI Research Center, National Taiwan Ocean Univ., Keelung City 20224, Taiwan. · Dept. of Aquaculture, National Taiwan Ocean Univ., Keelung City 20224, Taiwan. · Center of Excellence for the Oceans, National Taiwan Ocean University, Keelung City 20224, Taiwan.

Aquaculture Anomaly Detection

Open paper AskPDF arXiv

Precision aquaculture faces a "phenotyping bottleneck" in tracking high-resolution behavioral traits, as conventional methods cannot quantify instantaneous three-dimensional (3D) physical exertion. To address this, we present a high-throughput 3D behavioral phenotyping framework integrating deep learning object detection with binocular stereo vision for real-time monitoring of juvenile tilapia in high-density environments. The system automates non-contact body length estimation and reconstructs 3D swimming trajectories from absolute spatial coordinates. By eliminating 2D perspective distortions, this approach precisely quantifies 3D velocity and acceleration, marking the first estimation of true physical swimming speeds in free-roaming juveniles. Results show the framework successfully establishes circadian locomotor baselines, serving as an early warning system for physiological stress and providing an objective metric for fish vitality.

Abstract

Aquaculture

Explore similar work

Automated 3D Kinematic Monitoring for Circadian Activity and Anomaly Detection in Juvenile Fish

Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos

Non-invasive Growth Monitoring of Small Freshwater Fish in Home Aquariums via Stereo Vision