Drowning in Noise: On Signal Saliency in Self-Supervised Learning for Ocean Acoustics
Organizations: Carnegie Mellon University, USA · Stanford, USA
Abstract
Self-supervised learning (SSL) for audio typically benefits from larger pre-training corpora, but this relationship assumes that training data carries meaningful learning signal. In ocean acoustics, this assumption is often violated: passive recordings are dominated by waves, wind, and silence, with biological vocalizations occupying a small fraction of the total duration. We show that this overrepresentation of noise causes SSL models to allocate representational capacity to acoustically trivial structure, degrading downstream performance. Using BEATs as a testbed, we demonstrate that pre-training on 30 hours of salient recordings (segments containing animal vocalizations) outperforms 1,240 hours of unfiltered data on a 31-class marine mammal classification benchmark. Under linear probing, the manually curated salient set achieves markedly better accuracy than the full corpus, despite using 41 times less data. Automatic saliency-based filtering improves over unfiltered pre-training, but still falls well short of the manually curated salient set. Our findings suggest that, in domains where informative content is temporally sparse, data curation matters more than data scale for SSL pre-training.
Figures & tables
| Setting | Pre-training Data | Volume (hr) | Tokenizer | Linear Probing | Full Fine-tuning |
|---|---|---|---|---|---|
| 0 | - | 0 | - | 33.6 | 77.9 |
| 1 | AudioSet | 5388 | Random Codebook | 65.5 | 93.2 |
| 2 | Watkins (Master Tapes) | 1240 | Random Codebook | 43.1 | 79.4 |
| 3 | Watkins (Saliency Filtered) | 613 | Random Codebook | 47.5 | 90.6 |
| 4 | Watkins (Cut Tapes) | 30 | Random Codebook | 72.0 | 91.2 |
| 5 | AudioSet | 5388 | BEATs Tokenizer | 65.8 | 92.3 |
| Watkins | BirdSet | |||||
|---|---|---|---|---|---|---|
| Method | Recall | Sel. | AUC | Recall | Sel. | AUC |
| MAAD-ROI [ 26 , 41 ] | 0.402 | 0.689 | 0.317 | 0.465 | 0.604 | 0.319 |
| PCEN-Peak [ 42 ] | 0.288 | 0.753 | 0.209 | 0.253 | 0.600 | 0.181 |
| Spectral Entropy [ 43 ] | 0.224 | 0.398 | 0.205 | 0.100 | 0.313 | 0.089 |
| Energy Template [ 44 ] | 0.259 | 0.618 | 0.152 | 0.161 | 0.588 | 0.098 |
Training Data Selection
Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.
See all 196 papers