Large-scale bioacoustic detection using semantic segmentation: a deep learning framework applied to fin whale calls in ocean-bottom seismometer recordings
Authors: Jocelyn Japnanto, Alex A. Saoulis, Miriam Romagosa, Rita Leitão, Gabrielle Arrieta, Mónica A. Silva, Matthew Graham, Ana M. G. Ferreira
Organizations: Department of Earth Sciences, University College London, London, United Kingdom · Department of Physics and Astronomy, University College London, London, United Kingdom · Instituto de Investigação em Ciências do Mar (OKEANOS), Universidade dos Açores, Horta, Portugal · Marine Geosciences Department, Institute of Marine Sciences, CSIC, 08003 Barcelona, Spain · Centre for Research into Ecological & Environmental Modelling, University of St Andrews, St Andrews, United Kingdom · Instituto do Mar (IMAR), Horta, Portugal · Advanced Research Computing Centre, University College London, London, United Kingdom
Ocean-bottom seismometers (OBS), originally deployed for geophysical research, continuously record low-frequency sound for months to years across broad areas of ocean, offering a largely untapped resource for passive acoustic monitoring (PAM) of baleen whales. Realising this potential requires automated detection methods that operate reliably across the varied conditions in large sensor networks. We present a deep learning semantic segmentation framework that detects the 20-Hz notes of fin whales (Balaenoptera physalus) in OBS spectrograms, assigning each pixel a probability of belonging to a call and converting the resulting probability maps into time-frequency bounding boxes describing individual detections. We trained the model on hydrophone data from one OBS deployment in the Azores-Madeira-Canaries region and applied it without retraining to vertical-component seismometer data from a second, geographically distinct deployment, showing that a single trained model generalises across sensor types and recording environments. Applied to 378,912 h of recordings from 46 OBS sites, the detector identified 6.3 million calls, forming the largest fin whale call catalogue assembled to date, with high precision (~97%) across both deployments. The resulting catalogue resolves call timing and spectral structure accurately enough to support ecological analyses, revealing coherent seasonal shifts in three persistent inter-note interval (INI) groups across the singing season and basin-scale patterns in calling activity. By transforming existing geophysical infrastructure into a scalable sensing network, our approach substantially expands the spatial and temporal reach of PAM without new hardware investment, offering a transferable framework for tracking other low-frequency vocalising species and informing conservation planning, marine spatial management, and abundance estimation across large scales.
Figures & tables
Figure 1: Map of the São Jorge (triangles) and UPFLOW (squares) OBS arrays located across the Azores-Madeira-Canaries region. Each symbol marks an individual OBS station, with those coloured red being used in this study. Hydrophone-component data from São Jorge (August 2022 to February 2023), and vertical-component data from UPFLOW (June 2021 and August 2022), were used to detect fin whale calls.
Figure 2: A 5-minute recording showing part of a fin whale song sequence at station SJO03, located west of the São Jorge Island. a) Waveform time series and b) spectrogram of the hydrophone component. Instrument response was removed and the trace was band-pass filtered between 12 to 35 Hz with a Butterworth filter. The spectrogram was calculated using a short-time Fourier transform with a window length of 2.0 s and 75% overlap. Manual annotations of 20-Hz notes are shown as white boxes.
Figure 3: Overview of the data processing and detection pipeline. a) Raw waveforms are segmented into five-minute windows. b) Greyscale spectrograms, computed via short-time Fourier transforms, serve as input to the model. c) The encoder-decoder machine learning (ML) model. d) Post-processing extracts individual calls from the model’s predictions. e) Resulting detections (white boxes) form the call catalogue.
Figure 4: Comparison of post-processing approaches to extract calls, applied to a 5-minute sample from the test dataset from station SJO03 (05-Dec-2022). a) Manual annotations of 20-Hz notes (pink). b) Corresponding 20-Hz note probability map, inferred by the detector. ci) A simple thresholding approach, where pixels exceeding a fixed probability threshold are grouped into connected components to form detections, cii) and the resulting calls after filtering, evaluated against annotations, showing true positives (TP; green), false positives (FP; red), and false negatives (FN; orange). di) Our method, where local maxima (“seeds”) in a smoothed response map initialise the formation of our candidate bounding boxes prior to non-maximum suppression (NMS), dii) and the resulting detections, processed and evaluated similarly. Around 03:02:30, simple thresholding fragments and discards noisy calls that our method recovers.
Network
Station
Total recording hours
Number of detections
São Jorge
SJO01
3,792
58,478
SJO02
3,792
248,155
SJO03
3,768
197,509
SJO04
3,768
217,934
SJO05
3,768
215,590
SJO06
3,768
216,157
Table 1: Number of 20-Hz notes detected at each station in the São Jorge and UPFLOW arrays.
Figure 5: Recall as a function of minimum SNR threshold applied to a) an aggregation of all UPFLOW annotations, and b) the held-out test set from station SJO03. The threshold sets the minimum SNR an annotated call must exceed to enter the evaluation set, and the grey shading shows the number of annotations retained at each threshold.
Network
Station
TP
FP
FN
Precision
Recall
F1 Score
mIoU
São Jorge
SJO03
4583
267
1137
0.94
0.80
0.87
0.61
UPFLOW
OBS17
114
2
196
0.98
0.37
0.53
0.13
UP06
93
2
247
0.98
0.27
0.43
0.13
UP09
187
1
250
0.99
0.43
0.60
0.12
UP11
46
1
447
0.98
0.09
0.17
0.13
UP16
199
2
275
0.99
0.42
0.59
0.13
Table 2: Detection performance on the held-out SJO03 dataset and independent validation dataset from 16 UPFLOW stations.
Figure 6: Weekly fin whale call counts across all 46 stations in this study. The heatmap shows the number of detected calls per station aggregated by week. Grey intervals correspond to periods where the instrument was not recording, while white intervals indicate weeks with no detections. Elevated call abundance is concentrated between October and February across most stations. North-western (located farther offshore in the central North Atlantic) and south-eastern -most stations (located closer to continental Portugal and Morocco) are highlighted in red and yellow, respectively.
Figure 7: INI distributions of 20-Hz notes at the a) São Jorge array (6 stations, August 2022 to February 2023) and the b) UPFLOW array (40 stations, June 2021 to August 2022). Three distinct INI groups are observed in both arrays.
Figure 8: INI group composition of fin whale calls on the UPFLOW array during peak singing season. Detections are clustered into one of three INI groups ( μ ). Stations with fewer than 1,000 INIs for a given month are excluded from the plot.
Figure 9: a) Number of calls detected and b) mean INI over time in the UPFLOW array, aggregated by week. Only peak singing months are included in the latter. North-western and south-eastern -most stations are highlighted in red and yellow, respectively. Grey dashed vertical lines indicate the two weeks that contrast the start and end of the singing season. These are the weeks used in c) , which shows a bubble plot of the number of calls detected at each station (indicated by the size of the bubble), and the mean INI (indicated by the bubble’s colour), in a single week beginning in ci) early December and cii) mid-February.
Figure 10: Heatmap of weekly mean INI values across all 46 stations in this study. Grey intervals correspond to periods where the instrument is not recording, while white intervals show insufficient INIs as only weeks with more than 100 valid INIs were considered. North-western and south-eastern -most stations are highlighted in red and yellow, respectively.
Network
Station
Deployment start
Recovery date
Sensor used
Sampling rate (Hz)
Station depth (km)
São Jorge
SJO01
26/08/2022
30/01/2023
HighTech HTI-96-Min hydrophone
250
1.24
SJO02
26/08/2022
30/01/2023
250
1.36
SJO03
26/08/2022
29/01/2023
250
1.24
SJO04
26/08/2022
29/01/2023
250
1.16
SJO05
26/08/2022
29/01/2023
250
1.16
SJO06
26/08/2022
29/01/2023
250
1.13
Table S1 : Recording characteristics of each station from the São Jorge and UPFLOW arrays used in this study.
Figure S1 : Samples from the training set. Top: Samples from station SJO03 with annotated calls (pink). Bottom: Background samples from multiple stations, showing various sources of noise.
Figure S2 : Example of spectrogram-based SNR estimation for a single detection from station UP09 (03-Feb-2022, 20:16:27). Left : Power spectrogram centred on the detection (yellow vertical line). SNR is computed within a predefined frequency band [fmin,fmax] (magenta lines). The signal window (green) is defined as a duration T=1.5s immediately following the detection onset, while the noise window (cyan) is the preceding interval of equal duration. A narrow frequency band (orange) is constructed symmetrically about the dominant frequency (orange dotted line) at the onset time. The signal level Ssig is taken as the median power within this narrow band over the signal window, whereas the local and global noise level Snoise,local and Snoise,global is the median power across the full frequency band within the noise window, and context window, respectively. Right : The power spectrum at the onset time and includes the peak frequency and the corresponding narrow-band selection used for SNR estimation.
Figure S3 : Recall as a function of minimum SNR threshold applied to ground-truth annotations, with results broken down at each annotated UPFLOW station.
Figure S4 : Comparisons of INIs derived from model detections Nd (green) and annotations Na (yellow). These songs show a a) singlet INI pattern centred at 16.5 s, b) doublet INI pattern alternating between 12.5 s and 16.5 s, and c) predominantly singlet INI pattern centred at 11.5 s.
Figure S5 : Example of anomalous noise signals in station UP28 masking fin whale vocalisations around ∼ 22 Hz. This noise appeared at multiple distinct frequency bands and was observed throughout the whole deployment.
Figure S6 : Broadband noise in station UP38 observed throughout the deployment, which may have affected the detectability of calls (white boxes)
Figure S7 : Broadband noise in station UP39, similar to those in station UP38.
Figure S8 : Detected 20-Hz notes (white boxes) overlaid on spectrograms of OBS recordings illustrating sources of missed detections. a) Calls at station SJO03 masked by a regional earthquake, visible as a broadband transient signal. b) A dense chorusing event at station UP25 in which overlapping calls from multiple individuals reduce call separability. c) Masking due to shipping noise at station UP41.
Submarine fiber-optic cables instrumented with distributed acoustic sensing (DAS) provide an effective approach for large-scale monitoring of fin whales. We present an end-to-end workflow for detecting, characterizing, and localizing fin whale notes, tested on two submarine telecom cables in the Strait of Gibraltar and western Alboran Sea. The workflow applies a kurtosis-value picker adapted to narrow-band fin whale notes. Channel-wise detections are grouped into individual notes using density-based spatio-temporal clustering, cluster agglomeration, and hyperbolic fitting to reject incoherent picks. The retained clusters are characterized through temporal, spectral, and energy-related descriptors that support note-type discrimination and estimation of inter-note intervals. Relative arrival times across DAS channels are then used in a grid-search procedure to estimate candidate source locations. Evaluation against manually annotated detections from six fin whale songs yielded median pick-level precision of 0.990 and recall of 0.744, and median cluster-level precision of 0.880 and recall of 0.806. Representative applications demonstrate separation of overlapping vocalizations, characterization of type-A and type-B notes, and the inference of apparent source movement. By transforming dense DAS recordings into compact note-level bioacoustic information, the workflow provides an integrated framework for fin whale monitoring and a basis for adaptation to other synchronized acoustic receiver arrays.
Passive acoustic monitoring of killer whales is particularly important for conservation of the endangered Southern Resident killer whale population, but requires accurate models that can operate in real time under severe class imbalance and deployment shift. We propose a lightweight ResNet-based two-stage cascade that first detects killer whale vocalizations and then classifies confident detections into five eastern North Pacific ecotypes, abstaining on ambiguous calls. We train and evaluate the pipeline on the DCLDE 2027 dataset, where the detector achieves 0.960 macro-F1 and the classifier 0.958, outperforming frozen Perch 2.0 embeddings on the five-ecotype benchmark. By separating detection from ecotype recognition, the end-to-end cascade improves seven-class macro-F1 from 0.919 for a single-stage model to 0.933, with the largest gain on the rare OKW ecotype. To assess transfer beyond the benchmark, we use active learning to adapt the Stage 1 to the acoustic environment of Puget Sound, WA, increasing killer whale detection F1 from 0.405 to 0.755 on manually verified detection windows. Finally, each stage processes a 3 s window in approximately 1.4 ms on an NVIDIA H100, enabling faster than real time inference. These results demonstrate that the proposed two-stage cascade pipeline enables reliable killer whale detection and classification, adaptation to new acoustic domains, and real-time monitoring for conservation applications.
Daniela Ruiz, Manuel Castellote, Zhongqi Miao +5
Microsoft AI for Good Research Lab, Redmond, Washington, USA · Universidad de los Andes, Bogota, Colombia
Passive hydroacoustic monitoring often generates large volumes of continuous recordings that are only partially exploited due to the cost of manual annotation. Supervised detection methods perform well but require large labeled datasets, seldom available for rare signals or understudied environments. This work proposes a self-supervised exploration pipeline to address this limitation in low-frequency settings. A Masked AutoEncoder (MAE) is pre-trained on a reconstruction pretext task, then used to extract patch-level representations from spectrograms. Within each spectrogram, adjacent informative patches are aggregated into event-level embeddings, enabling the disentanglement of overlapping events. These embeddings are then clustered at the dataset scale using the dimension reduction algorithm UMAP and the clustering algorithm HDBSCAN to identify hydroacoustic patterns. The pipeline was applied to a multi-year hydroacoustic dataset collected near Mayotte Island, Indian Ocean, containing marine mammal vocalizations, seismo-volcanic signals, and anthropogenic noise. The 317 clusters were manually mapped to 15 hydroacoustic classes or noise in less than one hour. The method was evaluated in two ways. Quantitatively, when used as a classifier, it achieved performance comparable to two existing detectors. Qualitatively, it recovered known seasonal patterns of marine mammal acoustic activity. It also identified patterns of previously unstudied signals, thereby demonstrating its practical value.
Pierre-Yves Raumer, Axel Marmoret, Dorian Cazau +6
Laboratoire de G´eologie, Ecole Normale Sup´erieure/CNRS UMR 8538, PSL Research University, Paris 75005, France · Universit´e de Brest, CNRS, Ifremer, UMR6538 Geo-Ocean, 29280 Plouzan´e, France · Lab-STICC – UMR 6285 CNRS, ENSTA IP Paris, Brest, France +5