Audio-Visual Representation Learning

Follow this topic Ask about this topic

Audio-visual representation learning studies how models can encode sound and video so that information in one modality helps interpret the other. Common approaches include contrastive alignment, masked or predictive self-supervision, cross-modal distillation, and attention-based fusion, with settings ranging from speech recognition and sound-event localization to video retrieval and spatial grounding. Evaluation spans benchmarks such as AudioSet, VGGSound, AVE, AVSBench, and LRS3, as well as tests with noisy, missing, mismatched, or unseen inputs. A central tension is how to learn broad shared semantics without losing the fine-grained timing and spatial detail needed for reliable predictions.

Latest papers

31 papers8 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization
Dec 2, 2024Hugo Malard, Michel Olvera, Stephane Lathuiliere +1Image Segmentation Audio-Visual Representation Learning
LTCI, Télécom Paris, Institut Polytechnique de Paris

Dec 2, 2024J/K move · Enter open · S save

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

Hugo Malard, Michel Olvera, Stephane Lathuiliere, Slim Essid

LTCI, Télécom Paris, Institut Polytechnique de Paris

Image Segmentation Audio-Visual Representation Learning

Open paper AskPDF arXiv

Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods.