cs.CVOct 16, 2026
SaveA Paired RGB-Sonar Dataset for Fish Monitoring with Consumer Sonar
Organizations: California Institute of Technology · UC Santa Barbara
Abstract
Imaging sonar enables underwater observation in conditions where optical imaging is ineffective due to darkness and turbid water. However, imaging-sonar systems previously used for ecological monitoring have typically been expensive, limiting their accessibility. Consumer forward-looking sonar has the potential to make fish monitoring more accessible, but its noisy imagery makes annotation and the development of automated methods challenging. We introduce the Huntington and Caltech Koi RGB and Sonar dataset (HaCKRS), which pairs Garmin LiveScope imaging-sonar recordings with top-down RGB video of koi carp. The dataset contains more than 105,000 temporally matched frame pairs, spanning approximately 3.3 hours of recordings across four range and orientation configurations. We provide frame-wise metadata, sonar geometry and coordinate information, synchronization data for all four configurations, and planar registration for the three horizontal configurations. The release focuses on paired observations and calibration rather than task-specific fish annotations. We intend HaCKRS to support annotation and the development of fish detection, tracking, and counting methods, and investigations into the potential of consumer sonar for ecological monitoring.
Figures & tables
Figure 1: HaCKRS contains paired sonar and RGB frames of koi carp at the Huntington Gardens. Colored lines correspond to matched boundaries between modalities. The white circle denotes the sonar location. Crosshatching denotes areas that were not captured by the other modality.
| Run | Range | Orientation | Duration | Total Frames | Matched Pairs |
| 15H | 15ft / 4.57m | Horizontal | 1:03:08.018 | 36,028 | 35,984 |
| 40H | 40ft / 12.19m | Horizontal | 1:09:11.831 | 32,898 | 32,878 |
| 100H | 100ft / 30.48m | Horizontal | 0:47:25.255 | 20,641 | 20,635 |
| 20V | 20ft / 6.10m | Vertical | 0:18:18.507 | 15,869 | 15,865 |
Table 1: Recording durations and matched frame counts by run. Sonar frames were matched with the nearest RGB frame. Frames without an RGB observation within 100 ms were excluded from the paired release. Duration is in HH:mm:ss.fff.
| Run | RGB frame # | Sonar frame # | (ms) |
| 15H | 50,960 | 50,657 | |
| 15H | 50,960 | 50,658 | |
| 40H | 231,208 | 25,589 | |
| 40H | 231,208 | 25,590 | |
| 40H | 231,230 | 25,599 | |
| 40H | 231,230 | 25,600 |
Table 2: Cases where the same RGB frame was matched to two sonar frames. These occur around RGB capture boundaries where fewer RGB frames were captured due to file saving. Overall, HaCKRS contains 105,362 pair records for the 105,362 unique sonar frames linked to 105,358 unique RGB frames. A positive difference means the corrected RGB timestamp is later than the sonar timestamp.
| Run | Median | 95th% | Max | 33 ms |
| 15H | 8.30 ms | 15.86 ms | 82.63 ms | 99.989% |
| 40H | 8.67 ms | 15.86 ms | 91.91 ms | 99.982% |
| 100H | 8.20 ms | 15.88 ms | 63.12 ms | 99.990% |
| 20V | 8.05 ms | 15.85 ms | 76.86 ms | 99.987% |
Table 3: Temporal difference between the matched RGB and sonar frames for each run. Frames with 100 ms difference were not matched. 95th% refers to the 95th percentile.
| Frames per Second | ||||
| Run | 5th% | Mean | Median | 95th% |
| 15H | 6.2 | 9.50 | 8.6 | 15.2 |
| 40H | 5.0 | 7.92 | 6.4 | 13.4 |
| 100H | 5.4 | 7.25 | 7.6 | 8.6 |
| 20V | 14.0 | 14.44 | 14.4 | 15.0 |
Table 4: Frames per second of the sonar frames. 5th% and 95th% refer to the 5th and 95th percentiles. See fig. 8 for a graph of how the frame rate changes over the duration of each run, and see fig. 9 for a visualization of the frame rate distribution.
Figure 2: Horizontal and vertical viewpoints. The horizontal sonar approximates a top down view. The vertical run takes a thin profile slice through the water column. In the vertical setting, the sensor is near the pond floor. Significant reflections can be seen from both the water surface and the hard pond floor.
Figure 3: Example imaging-sonar frame before and after cartesian mapping. The same 3 ft (91 cm) range lines and 30 ∘ angular reference lines are shown in both.
Figure 4: After homographic warping, the two modalities are approximately aligned. The RGB frame is homographically warped (right) to align with the sonar frame (left). As discussed in section 4.5 , a planar mapping cannot fully capture depth-dependent displacement.
Figure 5: Example sonar-RGB correspondences used for homography estimation. From left to right: full cartesian sonar frame, cropped sonar region containing the calibration object with homography points, cropped RGB region containing the calibration object with homography points, and full RGB frame.
Figure 6: Spatial overlap between sonar and RGB imagery for the three horizontal configurations (15H, 40H, and 100H, left to right), estimated using the calibrated homographies. Top: RGB coverage in cartesian sonar coordinates, retaining the full recorded sonar footprint. Bottom: sonar coverage in RGB coordinates. Percentages indicate the fraction of each reference image covered by the other modality.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Resource | Fish-focused | Paired optics | Public data | Spatial calib. | Sonar class | Cross-modal ML intent |
| McCann 2018 [ 10 ] | ● | ● | ● | ◐ | Research | ◐ |
| Terayama 2019 [ 16 ] | ● | ● | ○/◐ | ◐ | Research | ● |
| RGBS50 [ 8 ] | ○/◐ | ● | ● | ◐ | Research | ● |
| DFKI UXO [ 3 ] | ○ | ● | ● | ● | Research | ● |
| SOLAQUA [ 12 ] | ●/◐ | ● | ● | ◐ | Research | ◐ |
| HODOR [ 19 , 6 ] | ● | ● | ● | ◐ | Research | ● |
Table 5: Comparison of relevant multimodal and imaging-sonar datasets and studies. ● indicates that the feature is a central, explicitly documented part of the resource; ◐ indicates that it is present but secondary, partial, or differently configured; and ○ indicates that it is absent or not central. “Paired optics” denotes synchronized optical imagery. “Cross-modal ML intent” indicates that the resource is explicitly framed for fusion, cross-modal learning, translation, or using one modality to inform another, rather than merely deploying two sensors simultaneously.
Table 6: Recording inventory for the paired sonar and RGB data. Each table lists the capture interval, duration, and number of extracted frames for each clip and recording configuration. For RGB, duration is calculated from the final precise timestamp rather than from the nominal frame count divided by 30.
Figure 7: Sonar polar range-line position measured from the bottom of the image to the center-column trough (illustrated right). Each test contains 1,000 randomly sampled frames (4,000 frames total). Std. Dev. (%) is relative to the median value.
Figure 8: Sonar frame rate over time for each recording configuration. Solid blue vertical lines mark time-synchronization events, when a pipe was rapidly inserted into the water to produce a recognizable event in both sonar and RGB video. Dashed pink lines mark frames containing the cross-shaped pipe calibration object used to identify correspondences for homography estimation. These annotations refer to physical objects and actions recorded in the videos, rather than electronic synchronization signals. Gray shading indicates intervals outside the selected usable footage; orange crosses mark sonar frames without an RGB match within 100ms.
Figure 9: Distributions of sonar frame rates estimated over rolling 5 s windows for the selected usable footage in each recording configuration. Violin width represents the density of frame-rate estimates; white bars mark the medians, and red diamonds mark the overall mean frame rates. The number of estimates is shown below each run label. The horizontal runs exhibit broader, multimodal distributions, whereas 20V has a narrower distribution.
| Run | Sonar frame | RGB frame | Point ID | Sonar residual (px) | RGB residual (px) |
| 15H | 38,797 | 32,107 | 3 | 5.17 | 11.36 |
| 15H | 39,206 | 32,757 | 4 | 6.49 | 16.62 |
| 40H | 14,609 | 206,883 | 2 | 2.60 | 14.24 |
| 40H | 76,014 | 342,979 | 3 | 2.34 | 15.13 |
| 40H | 76,014 | 342,979 | 4 | 2.36 | 15.28 |
| 40H | 77,530 | 346,337 | 2 | 3.85 | 13.34 |
Table 7: Points rejected by the stored RANSAC consensus. A rejected point may reflect a mis-click, parallax/depth, imperfect timing, or violation of the planar model. Consensus membership was preserved during coordinate migration; it is not equivalent to thresholding the final refined residual alone.
| RGB (pixels) | Sonar (cm) | ||||||||
| Run | # Frames | # Points | Outliers | Inlier rate | Median | RMSE | Median | RMSE | Inlier RMSE |
| 15H | 7 | 28 | 2 | 92.9% | 3.92 | 6.28 | 1.70 | 2.51 | 2.15 |
| 40H | 8 | 32 | 4 | 87.5% | 5.69 | 7.60 | 3.06 | 4.11 | 3.57 |
| 100H | 6 | 24 | 2 | 91.7% | 6.66 | 6.90 | 4.93 | 5.95 | 5.89 |
Table 8: Shared-homography agreement. # Frames and # Points refer to the number of homography frames and points used. RGB (pixels) and Sonar (cm) represent the error between the sonar and RGB points projected into the specified image space. Errors include all points, including outliers, in all columns other than Inlier RMSE. Outliers are detailed in table 7 . The corrected sonar images have a resolution of pixels for all three runs. The RGB calibration images for 15H and 100H have a resolution of pixels. For 40H, the RGB image is rotated , resulting in a calibration-image resolution of pixels.