Finetuning Strategies for Querying Sounds by Vocal Imitation

Aditya Bhattacharjee; Christos Plachouras; Sungkyun Chang; Emmanouil Benetos

cs.SDAug 19, 2026

Finetuning Strategies for Querying Sounds by Vocal Imitation

Authors: Aditya Bhattacharjee, Christos Plachouras, Sungkyun Chang, Emmanouil Benetos

Organizations: School of Electronic Engineering and Computer Science, Queen Mary University of London, UK

Abstract

This technical report describes our winning submission to the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation. We investigate two complementary fine-tuning strategies: contrastive learning with a frozen, pretrained CED encoder, and joint contrastive-triplet learning with semi-hard negatives using a MobileNetV3 encoder. This report has been updated for posterity to include details released after the challenge.

Figures & tables

Model Name	MRR ex	NDCG cat
random	0.0444	0.337
2DFT	0.1262	0.4793
Greif et al.	0.2726	0.6463
Submission #1	0.2876	0.6600
Submission #2	0.2932	0.6468

TABLE I: Performance Metrics

Supervised Fine-Tuning

Supervised fine-tuning adapts pretrained models by learning from labeled examples, instructions, demonstrations, and task-specific trajectories.

See all 223 papers

Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval
Apr 20, 2026HaeJun Yoo, Yongseop Shin, Insung Lee +2Audio-Language Model Evaluation Multimodal IR
Sogang University
Adaptive Perturbation Selection for Contrastive Audio Decoding
Jun 30, 2026Aaron Isidore Grace, Zhouyuan Huo, Weiran WangAudio-Language Model Evaluation Audio-Language Model Hallucination Detection
Department of Computer Science University of Iowa Iowa City, IA, USA · Google Mountain View, CA, USA
Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training
Apr 12, 2026Jielin Qiu, Ming Zhu, Wenting Zhao +11Audio Representation Learning Domain Adaptation
Salesforce AI Research

Apr 20, 2026J/K move · Enter open · S save

Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

HaeJun Yoo, Yongseop Shin, Insung Lee, Myoung-Wan Koo+1

Sogang University

Audio-Language Model Evaluation Multimodal IR Audio Understanding Audio-Language Models Retrieval Robustness Audio-Text Retrieval+1

Open paper AskPDF arXiv

Audio-text retrieval systems based on Contrastive Language-Audio Pretraining (CLAP) achieve strong performance on traditional benchmarks; however, these benchmarks rely on caption-style queries that differ substantially from real-world search behavior, limiting their assessment of practical retrieval robustness. We present Omni-Embed-Audio (OEA), a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding. To systematically evaluate robustness beyond caption-style queries, we introduce User-Intent Queries (UIQs) - five formulations reflecting natural search behaviors: questions, commands, keyword tags, paraphrases, and exclusion-based negative queries. For negative queries, we develop a hard negative mining pipeline and propose discrimination metrics (HNSR, TFR) assessing models' ability to suppress acoustically similar distractors. Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-audio retrieval performance to state-of-the-art M2D-CLAP, while demonstrating clear advantages in two critical areas: (1) dominant text-to-text retrieval (+22% relative improvement), and (2) substantially superior hard negative discrimination (+4.3%p HNSR@10, +34.7% relative TFR@10), revealing that LLM backbones provide superior semantic understanding of complex queries.

Abstract

Supervised Fine-Tuning

Explore similar work

Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

Adaptive Perturbation Selection for Contrastive Audio Decoding

Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training