cs.SDFeb 19, 2025
SaveLocalized time-frequency representation learning for bioacoustic classification in complex soundscapes
Organizations: ARL, Tropical Marine Science Institute, National University of Singapore · Department of Electrical & Computer Engineering, National University of Singapore · National Parks Board, Singapore
Abstract
Prevailing bioacoustic classifiers assign species labels to fixed time-frequency windows rather than to individual vocalizations. When multiple vocalizations occur within the same window, predictions cannot be unambiguously linked to specific calls, which limits analyses at the level of individual vocalizations. This work introduces a framework for time-frequency localized bird classification. A Local-Context Classifier (LCC) identifies species from localized time-frequency events (TFEs), while a Dual-Context Classifier (DCC) combines local and global acoustic context through a fine-tuned bioacoustic foundation model. On an in-distribution dataset from Singapore comprising 306 vocalization classes from 103 bird species, the LCC achieves an F1-micro score of 79.3%, while combining local and global context through the DCC yields the highest overall performance (94.6%). To reduce labeled data requirements, the LCC is pre-trained via self-supervised contrastive learning, achieving an 18.8% relative gain on an out-of-distribution dataset. A focused evaluation on continuous soundscape recordings further demonstrates the potential of the framework for long-term monitoring applications. By preserving the time-frequency localization of individual vocalizations, the proposed framework supports both ecological monitoring and vocalization-level studies of animal acoustic behavior.
Figures & tables
Figure 1: Left: first deployment at the SBG (site #1) from July 4, 2020 until September 20, 2020. The approximate locations of the recording units are marked as yellow circles, with the corresponding external microphones of each unit marked as orange circles. The six microphones surround a lake and cover an area of roughly 50 m 50 m. Right: a section of the elevated boardwalk used for the second deployment at the SBG (site #2) from September 20, 2020 until February 1, 2021. The same six microphones from site #1 were deployed along the circular boardwalk in a similar constellation, but without direct line of sight between units due to the dense vegetation. Photos from Google Maps.
Figure 2: An example illustrating extraction of TFEs from audio recordings. The left panel shows the spectrogram from a 5-second audio clip, with a few detected sounds enclosed by white rectangles. The right panels show the respective extracted TFEs. This example is non-exhaustive, i.e., not all detections in the audio clip are shown here.
Figure 3: Architecture of the self-supervised contrastive neural network. Features are extracted from the input TFE, compressed, and mapped to a 1024-dimensional embedding space. The embeddings are normalized to lie on the surface of a unit hypersphere, allowing similarity between embeddings to be measured by their dot product, or equivalently by the angle between them.
Figure 4: Examples of positive TFE pairs used to train the contrastive network. Pairs are obtained by associating the same sound event on two nearby microphones, allowing propagation effects to create natural “augmentations”.
Figure 5: Complete LCC architecture. TFEs extracted from raw recordings are mapped to embedding vectors using the pre-trained contrastive network, followed by a four-layer classification head that outputs class confidence scores. The embedding network and classifier are trained in alternating rounds to ensure that the embeddings retain their contrastive objective while adapting to the classification task: five epochs of supervised end-to-end training of the full model (i.e., the embedding network is not frozen) are followed by one epoch of self-supervised contrastive training of the embedding network.
| Input context | ||
| BirdNET parameter | Local | Global |
| Output dim | 306 | 103 |
| Hidden units | 256 | 512 |
| Dropout | 0.33 | 0.5 |
| Batch size | 64 | 64 |
| Learning rate | 0.001 | 0.001 |
Table 1: Non-default BirdNET hyperparameters used to fine-tune BirdNET on both local- and global-context inputs. With the exception of the minimum frequency and the minimum confidence score, which are kept fixed for both models, the parameters are obtained in a hyperparameter search over 20 trials and 5 runs per trial using BirdNET’s autotune option. The output dimension matches the number of target classes: 306 vocalization classes for local context and 103 species for global context.
| Dataset | Source | Rec. type | Size | Sample type | Location | #Classes | #Species |
| SSPT | Self-collected | Soundscape | 19,311 | TFE pairs | Singapore | — | — |
| SFT | Self-collected / XC | Mixed | 5,174 | TFEs | Singapore | 306 | 103 |
| OOD | XC | Focal | 11,092 | Recordings | Non-SG | 95 | 95 |
Table 2: Overview of three datasets used in this work: Self-supervised pre-training (SSPT), Supervised fine-tuning (SFT), and Out-of-distribution (OOD). TFE pairs for contrastive pre-training are derived from simultaneous recordings of the same vocalization on two microphones. The SFT set comprises a mixture of soundscape (36.5%) and focal (63.5%) recordings. Non-SG denotes recordings collected outside Singapore.
| Dataset | Aug. | Train | Val. | Test | Total |
| SSPT | No | 18,811 | 500 | — | 19,311 |
| SFT | No | 3,259 | 971 | 943 | 5,174 |
| SFT | Yes | 15,300 | 6,257 | — | 21,557 |
Table 3: Data splits for self-supervised pre-training (SSPT) and supervised fine-tuning (SFT) datasets, with and without augmentation. The SFT test set remains unaugmented and consists exclusively of natural vocalizations, ensuring evaluation on real-world data.
| Input context | Model | F1-micro | F1-macro |
| Local | LCC | 0.793 (0.767–0.819) | 0.685 (0.663–0.730) |
| BN-PT | 0.265 (0.238–0.294) | 0.296 (0.253–0.326) | |
| BN-FT | 0.780 (0.752–0.805) | 0.717 (0.699–0.766) | |
| LCC + BN-FT | 0.883 (0.861–0.902) | 0.820 (0.797–0.858) | |
| Global | BN-PT | 0.619 (0.587–0.650) | 0.622 (0.603–0.680) |
| BN-FT | 0.877 (0.852–0.894) | 0.921 (0.900–0.942) |
Table 4: Performance on the SFT test set. Values are reported as F1 scores with 95% bootstrap confidence intervals. LCC operates on local-context TFE inputs, whereas BirdNET is evaluated on both local- and global-context inputs. Models operating on local context are evaluated at the TFE level (except BN-PT), and all other models at the species level.
Figure 6: LCC performance by species on the OOD dataset showing hit rates (blue) and the number of non-augmented training samples for the corresponding species (orange).
Figure 7: Hourly detection rates for the 16 shortlisted species, averaged over a 10-day period at both SBG sites and grouped into diurnal and nocturnal species. The five species with the highest detection counts at each site are listed by name. Note that the y-axis ranges differ between the sites by an order of magnitude.
| Class | Training samples | Common name | Scientific name | Detections | TPs | Precision |
| 1 | 33 | Asian Glossy Starling | Aplonis panayensis | 50 | 44 | 0.88 |
| 2 | 6 | Black-naped Oriole | Oriolus chinensis | 14 | 10 | 0.71 |
| 3 | 4 | Blue-eared Kingfisher | Alcedo meninting | 50 | 30 | 0.60 |
| 4 | 34 | Brown-throated Sunbird | Anthreptes malacensis | 36 | 34 | 0.94 |
| 5 | 4 | Brown-throated Sunbird | Anthreptes malacensis | 50 | 41 | 0.82 |
| 6 | 3 | Changeable Hawk-eagle | Nisaetus cirrhatus | 2 | 2 | 1.00 |
Table 5: Performance of shortlisted classes from a full day of continuous soundscape recordings at site #1. Bird names follow the Checklist of Bird Species for Singapore [ 25 ] .
Figure 8: Example of extracted TFEs and human-labeled bird vocalizations (reproduced from [ 14 ] ). The TFE method detects both bird and non-bird acoustic events, including insect noise (top) and human speech (bottom).
| Dataset | Metric | Aggregation | LCC* | LCC | Rel. Gain |
| SFT | Accuracy | Micro | 82.4% | 81.0% | |
| Macro | 74.4% | 72.8% | |||
| OOD | Hit Rate | Micro | 23.4% | 27.8% | |
| Macro | 25.0% | 27.8% |
Table 6: Comparison of LCC and LCC without contrastive pre-training (LCC*) on the SFT and OOD test sets.
Figure 9: Examples from the SBG recordings illustrating the complementary strengths of local and global context. Left: A repeating call sequence from a Common Tailorbird ( Orthotomus sutorius ) together with a low-frequency call from an Asian Koel ( Eudynamys scolopaceus ) around 0.5–1 s. Global-context models such as BirdNET may detect both species but cannot associate predictions with individual vocalizations, whereas the LCC preserves time-frequency localization. Right: A dominant call from an Olive-winged Bulbul ( Pycnonotus plumosus ) together with several low-SNR background calls. Localized TFEs do not always capture complete vocalization sequences, illustrating how broader acoustic context can support classification.