CFC26: Building Evaluations for Deployment in Sonar-Based Fish Counting

Madison Van Horn; Suzanne Stathatos; Sevan Brodjian; Justin Kay; Kai Van Brunt; Erik Young; Sara Beery; Pietro Perona

cs.CVOct 15, 2026

CFC26: Building Evaluations for Deployment in Sonar-Based Fish Counting

Authors: Madison Van Horn, Suzanne Stathatos, Sevan Brodjian, Justin Kay, Kai Van Brunt, Erik Young, Sara Beery, Pietro Perona,

Organizations: California Institute of Technology · Massachusetts Institute of Technology · California Institute of Technology Volunteer

Abstract

Accurate counts of fish populations are essential for monitoring ecosystems and making decisions for conservation, harvest limits, and river management. In many field deployments, these counts are obtained from sonar videos that experts must inspect manually, making large-scale monitoring slow, expensive, and difficult. Automated counting systems have been proposed as a scalable alternative, but so far this promise has been difficult to realize in deployment. Developing evaluations that better reflect real-world deployment conditions will help close this gap and drive improvements in counting performance in practice. To achieve this, we propose Caltech Fish Counting 2026 (CFC26), a new dataset with improved evaluation protocols that simulate deployment to seen and novel rivers, testing in-distribution and out-of-distribution performance. We expand the CFC22 dataset to eleven deployment locations across nine Pacific salmon river systems from Alaska to Northern California, capturing a broader geographical and ecological range of data. We additionally introduce two new evaluation metrics for counting, nMNE and nMANE, that mirror the operational workflow that experts use and help surface errors that frame-level detection metrics do not capture. Using these metrics, we reveal previously-concealed directional bias in upstream vs downstream counts, which would have direct and significant ecological implications if undetected. The dataset and code are publicly available on CaltechDATA and GitHub.

Figures & tables

Figure 1: CFC26 data is diverse geographically and visually. Sonar sensors are deployed across nine distinct rivers spanning Alaska, British Columbia, Washington, and California, across eleven deployment locations. Blue markers indicate locations included in CFC22, where we have added additional data and annotations; orange markers indicate new CFC26 locations: (1) Chemainus, (2) Dungeness, (3) Eel, (4) Elwha, (5) Kenai-Channel, (6) Kenai-Leftbank, (7) Kenai-Rightbank, (8) Klamath, (9) Mad, (10) Nanaimo, (11) Nushagak.

Figure 2: CFC26 expands coverage with existing and newly added locations. The left chart shows the number of bounding boxes per location, highlighting each river’s contribution to the dataset. Elwha and Kenai-Channel include additional annotations not present in CFC22. The right plot shows how locations differ: CFC22 locations generally have higher densities and larger bounding boxes than newly added locations. Kenai locations are shown by stratum rather than aggregated, exposing variation across sonar range configurations within the same deployment site.

Figure 3: Example of temporal distribution of annotated clips for Kenai-Leftbank Stratum1. Data splits were constructed by identifying gaps in coverage to define temporally separated train, validation and test splits, reducing any potential data leakage. This was applied across all locations.

Location	# Clips	# Frames	# Tracks	Track Coverage (%)	# Boxes	Avg Track Length
Chemainus	22	16,287	449	41.82	8,488	22.63
Dungeness	6	3,118	8	95.48	3,273	493.00
Eel	27	2,900	104	48.31	3,136	35.09
Elwha	264	107,238	450	31.12	4,4703	92.38
Kenai-Channel	217	41,929	1,421	78.31	179,789	106.65
Kenai-Leftbank	546	192,652	1,970	41.31	150,599	101.94

Table 1: CFC26 scales and diversifies CFC22. Our dataset includes 2.5

\times

as many clips, 2

\times

as many frames, and 2

\times

as many bounding boxes than CFC22, and 6 additional locations that expand OOD evaluation to more realistically reflect deployment needs.

Figure 4: Counting errors per location across training conditions. Balanced Sampling (ID) achieves the lowest nMAE at most locations, while holding out the target river degrades performance substantially at harder sites such as Eel and Dungeness. Directional errors (nMANE) are negative across nearly all conditions, indicating consistent undercounting of the dominant movement direction; Chemainus and Eel show the largest directional bias. Note, the nMNE value for Chemainus under Balanced Sampling (OOD) is a large outlier (3.67) and is truncated from the figure for readability.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	YOLOv5		YOLOv26
Metric	Small	Medium	Small	Medium
Val [email protected]	0.6780	0.6926	0.6310	0.6610
Val nMAE	0.0729	0.0720	0.1857	0.1601
Val nMNE	-0.0325	-0.0355	-0.2441	-0.2003
Val nMANE	-0.0372	-0.0338	-0.1595	-0.1338
Test [email protected]	0.7346	0.7546	0.6555	0.6854

Appendix

Table 2: Model performance comparison

Architecture	Val mAP50	Val nMAE	Val nMNE	Val nMANE	Test mAP50	Test nMAE	Test nMNE	Test nMANE
YOLOv5m	0.6926	0.0720	-0.0355	-0.0338	0.7546	0.0802	-0.0397	-0.0506
RT-DETRv2-S	0.6892	0.0725	-0.0317	-0.0316	0.7480	0.0906	-0.0433	-0.0501

Appendix

Table 3: Detector architecture robustness. Detection and counting performance for YOLOv5m and RT-DETRv2-S on the validation and test sets.

Location	Epoch
Chemainus	173
Dungeness	516
Eel	616
Elwha	42
Kenai-Channel	43
Kenai-Left	20

Appendix

Table 4: Epoch Scaling for Target-Location Training

Configuration	Frames	Epochs	Upsample	mAP50	nMAE	nMNE	nMANE
10k, upsampling (selected)	10k	74	✓	0.74	0.07	-0.04	-0.05
10k, no upsampling	10k	74	$\times$	0.77	0.11	-0.05	-0.07
1k, no upsampling	1k	734	$\times$	0.69	0.09	0.08	-0.05

Appendix

Table 5: Sampling strategy comparison. 10k no-upsampling achieves higher mAP50 performance, but 10k-upsampling has lower counting errors.

Val Location	mAP50
Leftbank	0.80929
Rightbank	0.90179

Appendix

Table 6: Train on Kenai-Leftbank Model performance comparison

	nMAE			nMANE			nMNE
River	ID	OOD	Target	ID	OOD	Target	ID	OOD	Target
Nushagak	0.028	0.035	0.014	-0.028	0.028	-0.014	-0.028	0.028	-0.014
Kenai-Right	0.029	0.025	0.027	-0.021	-0.017	-0.015	-0.021	-0.017	-0.015
Kenai-Left	0.048	0.069	0.052	-0.033	-0.036	-0.048	-0.034	-0.041	-0.048
Kenai-Channel	0.143	0.286	0.143	0.030	0.000	0.030	0.032	-0.129	0.032
Elwha	0.147	0.265	0.206	-0.094	-0.219	-0.094	-0.115	-0.269	0.038

Appendix

Table 7: Per-river counting performance under balanced sampling for in-distribution (ID), balanced sampling out-of-distribution (OOD) evaluation, and on the target river only.

Figure 5: Counting error (nMAE) per location across training conditions. Balanced Sampling (ID), which includes target-location data, achieves the lowest error for most locations, while holding out the target river (OOD) degrades performance substantially at harder locations such as Eel and Dungeness. Training on CFC22 locations alone fails catastrophically on Eel with nMAE

\approx

1.0, demonstrating that in-distribution benchmark scores do not predict deployment performance.

Figure 6: Example echogram. A raw sonar clip (left) and its corresponding echogram (right). While the five fish are barely visible in the noisy, raw imagery, the echogram clearly reveals the tracks over time.

Object Counting

Object counting estimates the number and locations of target instances in images or video, from dense crowds to fine-grained categories.

See all 61 papers

A Paired RGB-Sonar Dataset for Fish Monitoring with Consumer Sonar
Oct 16, 2026Mourad Derbedrosian, Michael Hobley, Pietro PeronaMultimodal Dataset Underwater Imaging
California Institute of Technology · UC Santa Barbara
WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition
Aug 21, 2026Abigail G. Grassick, Jerome Tze-Hou Hsu, Ethan Lin +10Vision Foundation Models Video Action Recognition
Cornell University, 14850 Ithaca, NY · University of Colorado Boulder, 80309 Boulder, CO · HHMI Janelia Research Campus, 20147, Ashburn VA
Automated high-frequency quantification of fish communities and biomass using computer vision
May 11, 2026Kota Ishikawa, Takuma Masui, Keita Koeda +3Wildlife Monitoring
Graduate School of Life Sciences, Tohoku University, Sendai, Japan · Advanced Institute for Marine Ecosystem Change (WPI-AIMEC), Tohoku University, Sendai, Japan · Graduate School of Science and Engineering, University of the Ryukyus, Okinawa, Japan +1

Oct 16, 2026J/K move · Enter open · S save

A Paired RGB-Sonar Dataset for Fish Monitoring with Consumer Sonar

Mourad Derbedrosian, Michael Hobley, Pietro Perona

California Institute of Technology · UC Santa Barbara

Multimodal Dataset Underwater Imaging

Open paper AskPDF arXiv

Imaging sonar enables underwater observation in conditions where optical imaging is ineffective due to darkness and turbid water. However, imaging-sonar systems previously used for ecological monitoring have typically been expensive, limiting their accessibility. Consumer forward-looking sonar has the potential to make fish monitoring more accessible, but its noisy imagery makes annotation and the development of automated methods challenging. We introduce the Huntington and Caltech Koi RGB and Sonar dataset (HaCKRS), which pairs Garmin LiveScope imaging-sonar recordings with top-down RGB video of koi carp. The dataset contains more than 105,000 temporally matched frame pairs, spanning approximately 3.3 hours of recordings across four range and orientation configurations. We provide frame-wise metadata, sonar geometry and coordinate information, synchronization data for all four configurations, and planar registration for the three horizontal configurations. The release focuses on paired observations and calibration rather than task-specific fish annotations. We intend HaCKRS to support annotation and the development of fish detection, tracking, and counting methods, and investigations into the potential of consumer sonar for ecological monitoring.

Abstract

Object Counting

Explore similar work

A Paired RGB-Sonar Dataset for Fish Monitoring with Consumer Sonar

WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition

Automated high-frequency quantification of fish communities and biomass using computer vision