cs.SDOct 30, 2026

Self-Supervised Object-Centric Representation for Multi-Task Audio Analysis

Authors: Parthasaarathy Sudarsanam, Tuomas Virtanen

Organizations: Audio Research Group, Tampere University, Finland

Abstract

This paper studies the capability of an object-centric audio representation to support multiple downstream tasks within a unified framework. The object-centric model is trained in a self-supervised setup using slot attention by reconstructing Audio Spectrogram Transformer (AST) patch embeddings from a set of latent slots. It is trained on synthetic audio mixtures, which provides controlled multi-source audio scenes. These slot representations are evaluated for the classification, detection, and separation of sound events by training task specific output layers. On the classification task, the model achieves an F1-score of 0.8, comparable to clip-level AST embeddings which achieve 0.7. For sound event detection, it reaches a macro F1-score of 0.7, outperforming clip-level AST embeddings at 0.4 and matching AST patch embeddings at 0.7, which encode finer time-frequency information. In source separation, it achieves an SI-SDR improvement (SI-SDRi) of 18.4 dB. Furthermore, in a zero-shot source separation setting, the model achieves an SI-SDRi of 14.5 dB. These results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.

Figures & tables

Explore similar work

CardsList
  1. AudioSphere: Towards Self-Supervised Spatial Audio Representation Models

    Nov 6, 2026Goksenin Yuksel, Marcel van Gerven, Kiki van der HeijdenAudio Representation LearningSpatial Audio

  2. AudioFM: Towards General Purpose Audio Representation Learning

    Oct 25, 2026Yamin Li, Ruidong Zhang, Anran Wang +19Audio Representation LearningTemporal Representation Learning

  3. MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning

    Oct 20, 2026Kohei Yamamoto, Kosuke OkusaAudio Representation LearningSelf-Supervised Pre-Training