Label uncertainty and annotator disagreement are common challenges in the field of computer vision, yet their study has largely been confined to the medical domain or to generic image-recognition datasets. Underwater datasets are particularly susceptible to these issues due to the need for domain expertise, degraded visibility conditions, and the inherent difficulty of establishing reliable ground truth in inaccessible environments. Despite these challenges, annotation uncertainty in underwater imagery remains largely unexplored. In this work, we present the first systematic multi-annotator study of segmentation in real underwater scenes, with over 100 participants, and across varying, controlled levels of turbidity. We show that underwater datasets face many of the same annotation challenges as other vision tasks, while turbidity introduces additional systematic errors. We further investigate the main factors driving label noise and explore ways to improve annotation quality in turbid underwater environments, including privileged information, individual effort and annotator ensembles. All (meta-) data collected in this study will be available on the project page: https://vap.aau.dk/tubcertainty
Figures & tables
Figure 1: We design a multi-annotator study and collect hundreds of confidence-weighted instance segmentation masks of objects in real underwater scenes across 3 increasing levels of turbidity. We analyze annotation error, variability, and uncertainty in relation to varying visibility.
Figure 2: Scenes are selected randomly for every user, while ensuring that Group 1 ( \Nearrow ), Group 2 ( \Searrow ), and Group 3 ( ⇒ ) see three different scenes. Group 4 ( ↗ ) see the same (random) scene but with increasing turbidity levels. We show example scenes here for illustration.
Figure 3: Screenshots of different parts of the annotation interface.
Figure 4: The matching protocols used when evaluating the collected annotations.
Figure 5: Randomly selected annotations for the same object in the same scene ( Exp30_Cam1 ), varying turbidity. Some users fail to annotate the object altogether, especially as turbidity increases.
Figure 6: Clear-image annotations (object contours and confidence), selected via visual inspection to showcase inter-annotator diversity. More examples can be found in Appendix 0.D.3 .
Figure 7: Example from a medium-turbidity scene ( Exp6_Cam1 ), where annotators separately contour different parts of the same rock. We do not observe this phenomenon in low-turbidity annotations. In high-turbidity images, annotators are more likely to roughly annotate the whole rock, miss it altogether, or only annotate the bright spots.
Figure 8
Figure 10: Instance segmentation IoU for different turbidity levels (one point per object), matching annotators → GT (penalizing FNs, but not FPs) or GT → annotators (penalizing FPs, but not FNs).
Figure 11: Average semantic segmentation performance across individual or ensemble masks, as a function of the confidence threshold. (B) only keeps the best annotator per image. With absolute thresholding, a pixel with confidence c is considered an object if c≥T . With relative thresholding, a pixel is considered an object if c≥cmax⋅T , where cmax is the maximum confidence in the image.
Figure 12: Instance segmentation IoU of individual annotated objects (FNs not penalized) and their corresponding confidence. See Appendix 0.D.2 for a binned version.
Figure 13: All five scenes with ground truth annotations and multi-annotator heatmaps (scale on the right). The low and high turbidity images are included for reference.
Figure 14: Annotation quality, inter-annotator agreement, and confidence of users with the same questionnaire answers. All metrics are computed at the image level.
Figure 15: Interplay between annotations, user group, image order and turbidity. For each user group, the turbidity levels are shown in the order that they were shown in the study.
Figure 16: Different facets of annotator behavior and effort under different turbidity levels.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 17: The 15 TUB [ 20 ] images used in this study (5 scenes at 3 turbidity levels).
Figure 18: Turbidity measurements for the 15 TUB [ 20 ] images used in this study (5 scenes at 3 turbidity levels). Measurements are based on TUB metadata, with avg ± std turbidity per image.
Figure 19: Screenshot of the starting screen
Figure 20: Screenshot of the post-annotation questionnaire
Figure 21: Ordered screenshots and explanations of the practice page flow.
Figure 22: How far did participants get? 190 participants started the study, while 104 fully completed it with non-empty annotations. Results and analysis are based on these 104 submissions.
Figure 23: Number of annotations obtained for each image. Note that each annotator is assigned to three different images, and that Group 3 ( ⇒ ), sees 3 medium turbidity images (vs. one image per turbidity level for the other groups) - hence the higher of number of annotations per medium turbidity image.
Figure 24: Participants’ browser timezone and screen size (104 participants).
Figure 26: Distribution of the 104 participants across the 4 groups
Figure 27: Illustration of how an annotator’s contours (left) are compared to GT contours (right) in terms of binary semantic segmentation, using an annotation of Exp18_Cam3_Stage06 as an example.
Figure 28: All annotations for the same object in the same scene ( Exp30_Cam1 ), varying turbidity. Some users fail to annotate the object altogether, especially as turbidity increases.
Figure 30: Distribution of object confidence for different turbidity levels.
Figure 32: Every point represents an annotated contour/polygon. The top plot is the same as Figure 12 (main text) but enlarged for convenience. The bottom plot is based on the same data points, but shows the average performance (with error bar) across 10 confidence bins.
Figure 33: Final change in the value of image adjustment sliders (final value − default value). 0 indicates no change. Each point represents an annotated image, and only images where the value was changed at least once during annotation are included in the plot. Note that the "contrast" sub-plot corresponds to Figure 16(b) in the main text.
Figure 34: Type and frequency of image adjustments, individually and in combination. : no sliders were adjusted. : only the zoom was adjusted. : only the brightness was adjusted. : only the contrast was adjusted. : both the contrast and brightness were adjusted. And so on. Note that this plot is the same as Figure 16(a) in the main text, just enlarged.
Figure 35: Distribution of annotation time per image ( left ) and distribution of annotation time per image normalized by the number of annotated objects ( right ). Outliers are not shown for readability, as they heavily distort the plot scale.
Figure 36: Does time per image help? Here we look at the time in minutes spent on every annotated image, for different turbidity levels. For each turbidity level, we create three (roughly) equally sized time groups ( fast, medium-speed, slow), split at the 33rd (100/3) and 66th (200/3) percentile. Note that this plot is the same as Figure 16(d) in the main text, just enlarged.
Figure 37: Does adjusting individual contours help? Here we look at the number of adjustments per annotated object for different turbidity levels. For each turbidity level, we create three (roughly) equally sized groups ( small, medium, large [number of adjustments]), split at the 33rd (100/3) and 66th (200/3) percentile.
Visibility in underwater environments degrades rapidly under turbid conditions, yet the effects on computer-vision models remain unclear. This issue is compounded by reliance on synthetic turbidity datasets, which may misrepresent real-world information loss. To address this gap, we introduce the Turbid Underwater Baseline (TUB) dataset, comprising 1,320 images captured under extreme turbidity and over 16,000 high-confidence ground-truth segmentation masks. We additionally propose PCD, a metric derived from phase congruency maps that is invariant to contrast and aims to capture the loss of structural information in real turbidity. We show that PCD correlates strongly with the performance of instance segmentation models on both real and synthetic turbid images, whereas common metrics in the field show weak to no correlation at all. The dataset and relevant code can be found on the project page: https://vap.aau.dk/pcd
Vasiliki Ismiroglou, Stefan H. Bengtson, Tasos Benos +2
Visual Analysis and Perception Laboratory, Aalborg University, Denmark · Pioneer Centre for Artificial Intelligence, Denmark
Underwater object detection is strongly affected by domain shift, where performance can vary significantly across different locations, habitats, and deployment conditions. However, detector performance is typically evaluated using aggregate metrics that hide failures in specific environments, while existing domain generalization benchmarks often rely on synthetic variations that do not reflect real-world conditions. We introduce a framework that characterizes underwater images by appearance, scene composition, and acquisition geometry to assign domain labels. Using this framework, we perform the first systematic study of how domain factors influence both human annotation quality in underwater object detection datasets and deep learning-based detector performance, revealing substantial domain-dependent discrepancies. By incorporating physically meaningful domain labels, domain shift becomes something we can characterize, measure, benchmark, and act on. We highlight how this can be used to guide data collection and annotation, design more informative benchmarks, and assess detector robustness across diverse underwater environments.
Melanie Wille, Dimity Miller, Tobias Fischer +1
QUT Centre for Robotics, Queensland University of Technology, Brisbane, Australia
Domain shift, where deviations between training and deployment data distributions degrade model performance, is a key challenge in underwater environments. Existing benchmarks testing performance for underwater domain shift simulate variability through synthetic style transfer. This fails to capture intrinsic scene factors such as visibility, illumination, scene composition, or acquisition factors, limiting analysis of real-world effects. We propose a labeling framework that defines underwater domains using measurable image, scene, and acquisition characteristics. Unlike prior benchmarks, it captures physically meaningful factors, enabling semantically consistent image grouping and supporting domain-specific evaluation of detection performance including failure analysis. We validate this on public datasets, showing systematic variations across domain factors and revealing hidden failure modes.
Melanie Wille, Dimity Miller, Tobias Fischer +1
QUT Centre for Robotics, Queensland University of Technology