cs.SDNov 6, 2026

Improving Large Audio-Language Model Embedding Baselines for Instruction-Aware Speech Retrieval

Authors: Chen-An Li, Cheng-Han Chiang, Hung-yi Lee

Organizations: National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)

Abstract

Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text instruction. Large audio-language models (LALMs) effectively process speech and text, and our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We introduce IRIS-Embed, which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. We construct about 900K training samples from real data and synthetic data, with instructions expressing requirements for one or more speech attributes. On INSPIRE, an instruction-aware speech retrieval benchmark, IRIS-Embed consistently outperforms the corresponding LALM baselines. The best model achieves 62.92% macro-average Recall@50, surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness and the benefits of audio layer weighting. On MAEB, a massive audio embedding benchmark, macro-average gains of 9.4-19.8 points over the corresponding baselines show generalization beyond speech retrieval.

Figures & tables

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval

    Aug 17, 2026Chen-An Li, Hung-yi LeeAudio-Text RetrievalAudio-Language Model Evaluation

  2. ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models

    Jun 27, 2026Fengjie Lu, Chenang Jiang, Jiarui Hai +2Audio Representation LearningAudio-Language Models

  3. SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data

    Oct 13, 2026Sangkwon Park, Dongwook Lee, Youngho Cho +3Audio-Text Retrieval