Improving Large Audio-Language Model Embedding Baselines for Instruction-Aware Speech Retrieval
Organizations: National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
Abstract
Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text instruction. Large audio-language models (LALMs) effectively process speech and text, and our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We introduce IRIS-Embed, which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. We construct about 900K training samples from real data and synthetic data, with instructions expressing requirements for one or more speech attributes. On INSPIRE, an instruction-aware speech retrieval benchmark, IRIS-Embed consistently outperforms the corresponding LALM baselines. The best model achieves 62.92% macro-average Recall@50, surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness and the benefits of audio layer weighting. On MAEB, a massive audio embedding benchmark, macro-average gains of 9.4-19.8 points over the corresponding baselines show generalization beyond speech retrieval.
Figures & tables
| Method | Model | DailyTalk | VCTK | Expresso | Synthetic | Avg. | ||||||||||
| R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | ||
| RAND | Random | 0.24 | 1.06 | 2.10 | 0.35 | 1.84 | 3.63 | 0.30 | 1.50 | 2.96 | 0.20 | 1.00 | 1.98 | 0.27 | 1.35 | 2.67 |
| ENC | WavLM-large | 16.50 | 24.50 | 35.50 | 17.50 | 35.00 | 43.75 | 9.50 | 17.69 | 24.00 | 1.70 | 4.48 | 7.22 | 11.30 | 20.42 | 27.62 |
| SENC | ECAPA-TDNN | 15.50 | 24.00 | 29.50 | 100.00 | 100.00 | 100.00 | 34.88 | 41.75 | 45.44 | 10.02 | 14.83 | 18.82 | 40.10 | 45.15 | 48.44 |
| emotion2vec-base | 11.50 | 23.00 | 30.50 | 3.13 | 6.25 | 13.75 | 0.88 | 4.38 | 8.00 | 1.61 | 4.53 | 6.82 | 4.28 | 9.54 | 14.77 | |
| CAS | Qwen3-Embedding-8B | 62.00 | 81.50 | 89.50 | 1.88 | 6.25 | 10.63 | 0.75 | 5.56 | 9.69 | 7.02 | 8.95 | 11.07 | 17.91 | 25.57 | 30.22 |
| Method | DailyTalk | VCTK | Expresso | Synthetic | Avg. |
|---|---|---|---|---|---|
| Layer 13 | 15.50 | 76.88 | 48.94 | 35.27 | 44.15 |
| Final Layer | 82.50 | 48.75 | 28.25 | 34.72 | 48.56 |
| Mean Layer | 70.00 | 71.25 | 43.13 | 44.90 | 57.32 |
| Finetuned | 32.50 | 91.25 | 47.38 | 43.35 | 53.62 |
| Static Weight | 66.00 | 78.75 | 43.19 | 45.28 | 58.31 |
| Inst. Weight (Ours) | 77.50 | 85.00 | 45.00 | 44.17 | 62.92 |
| Clf | Clust | PC | Rrnk | Rtrvl | |||||||||||||
| Method | Model | Beijing Opera | BirdCLEF | CommonLang Age | GTZAN Genre | IEMOCAP Gender | MInDS14 | Mridingham Tonic | SIBFLEURS | VoxPopuli LanguageID | Vehicle Sound | VoxPopuli Gender | NMSQA | VoxPopuli Accent | GTZAN | JamAlt Artist | Avg. |
| ENC | WavLM-large | 57.7 | 19.8 | 16.9 | 67.7 | 63.1 | 13.5 | 31.4 | 12.5 | 89.0 | 0.2 | 0.1 | 49.0 | 52.2 | 68.8 | 78.5 | 41.4 |
| CLAP | clap-htsat-fused | 92.8 | 18.0 | 15.3 | 67.9 | 93.6 | 9.2 | 47.0 | 9.1 | 32.0 | 4.7 | 52.7 | 47.9 | 53.2 | 61.3 | 87.7 | 46.2 |
| larger_clap_general | 93.6 | 17.0 | 20.5 | 84.5 | 89.3 | 8.0 | 49.4 | 9.4 | 84.6 | 3.4 | 3.3 | 47.4 | 53.5 | 66.8 | 95.6 | 48.4 | |
| LALM | Qwen2-Audio-7B | 97.4 | 37.1 | 17.6 | 93.1 | 93.0 | 24.4 | 61.2 | 15.4 | 99.0 | 5.5 | 0.1 | 48.9 | 52.9 | 80.8 | 37.6 | 50.9 |
| LCO-Embedding-Omni-3B | 93.2 | 31.6 | 16.9 | 81.0 | 59.4 | 88.9 | 39.5 | 45.1 | 63.4 | 2.2 | 1.0 | 96.9 | 50.7 | 75.4 | 67.1 | 54.2 | |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Attribute | |||||||||
| Source | Answer relevance | Speaking style | Gender | Speaker | Accent | Sentiment | Dialogue | Environmental sound | License |
| Real data | |||||||||
| VoxCeleb1 ( Nagrani et al., 2017 ) | CC BY-SA 4.0 | ||||||||
| ESD ( Zhou et al., 2021 ) | Custom | ||||||||
| MELD ( Poria et al., 2019 ) | GPL-3.0 | ||||||||
| CREMA-D ( Cao et al., 2014 ) | ODbL 1.0 | ||||||||
| TTS model | Voice names |
|---|---|
| Gemini | Achernar, Achird, Algenib, Algieba, Alnilam, Aoede, Autonoe, Callirrhoe, Charon, Despina, Enceladus, Erinome, Fenrir, Gacrux, Iapetus, Kore, Laomedeia, Leda, Orus, Puck, Pulcherrima, Rasalgethi, Sadachbia, Sadaltager, Schedar, Sulafat, Umbriel, Vindemiatrix, Zephyr, Zubenelgenubi |
| Qwen3 TTS | Ryan, Aiden |
| Style | Prompt |
|---|---|
| Happy | Voice: bright, cheerful, and upbeat, as if you are smiling while you speak. Tone: warm, joyful, and openly affectionate. Pacing: lively and energetic, with buoyant rhythm. Emotion: genuinely delighted and optimistic, emphasizing key words with an uplifted intonation. |
| Sad | Voice: soft, low, and melancholic, with a gentle tremor. Tone: heavy, sorrowful, and subdued, as if holding back tears. Pacing: slow and drawn out, with thoughtful pauses. Emotion: deep sadness and quiet grief, letting the weight of the words settle. |
| Angry | Voice: sharp, intense, and forceful, with hard consonants. Tone: confrontational, tense, and biting. Pacing: clipped and driving, with emphatic stress on charged words. Emotion: strong anger held just under control, barely restrained frustration. |
| Fearful | Voice: trembling, breathy, and slightly higher in pitch. Tone: anxious, uncertain, and vulnerable. Pacing: uneven, with nervous hesitations and quick shallow breaths. Emotion: palpable fear, as if something dangerous is nearby and you are trying not to be heard. |
| Surprised | Voice: wide-eyed and astonished, with sudden lifts in pitch. Tone: amazed and caught off guard. Pacing: quick bursts punctuated by short breathy intakes. Emotion: genuine shock and disbelief, emphasizing unexpected words with emphatic stress. |
| Disgusted | Voice: curled-lipped and repulsed, with nasal resonance. Tone: scornful, dismissive, and revolted. Pacing: slightly drawn-out on offensive words, with sharp cutoffs elsewhere. Emotion: strong aversion, as if the subject is physically distasteful. |
| Short instructions | Full-sentence instructions |
| You are generating {N} very short, informal retrieval cues for an audio retrieval system. Given a spoken query, the user wants to find another clip satisfying certain requirements. MATCHING constraints (same as spoken query): - the same {key} (e.g., {values}) as the given spoken query TARGET constraints (find a specific value): - a specific {key} (e.g., {values}) of "{value}" Each cue should be an informal, telegraphic phrase a user might type in a chat or voice interface: keyword-style fragments, not full sentences. Requirements: 1. Very short, typically 2 to 6 words 2. Lowercase is fine, no trailing punctuation 3. No full sentences; use fragments, comma-separated keywords, or elliptical phrases 4. For MATCH constraints use phrases such as "same emotion", "matching accent" 5. For TARGET constraints use phrases such as "female", "happy", "british accent" 6. A natural example for this pattern is "{example_cue}" 7. Vary wording, word order, and synonyms across cues; do not repeat the example 8. Do NOT mention technical terms such as "requirements", "attribute", or "metadata" Output ONLY the cues, one per line, numbered 1 to {N}. | You are generating diverse natural language retrieval instructions for an audio retrieval system. Given a spoken query, the user wants to find another clip satisfying certain requirements. MATCHING constraints (same as spoken query): - the same {key} (e.g., {values}) as the given spoken query TARGET constraints (find a specific value): - a specific {key} (e.g., {values}) of "{value}" Generate {N} diverse and natural retrieval instructions. Each instruction should: 1. Clearly indicate which attributes should MATCH the spoken query (e.g., "same emotion", "matching gender", "from the same speaker") 2. Clearly indicate which attributes should have a SPECIFIC TARGET value (e.g., "with a happy emotion", "in a British accent") 3. Use varied sentence structures, synonyms, and phrasings 4. Be natural and conversational 5. Not mention technical terms such as "requirements", "attribute", or "metadata" 6. Refer to "the spoken query" or "the given audio" when referring to the query Output ONLY the instructions, one per line, numbered 1 to {N}. |
| Requirements | Short instruction | Full-sentence instruction |
|---|---|---|
| Same speaker | identical voice | Fetch another clip of the speaker from this audio clip. |
| Same environmental sound | matching ambiance | Find audio that matches the background sound environment of this recording. |
| Same speaker, angry speaking style | anger, same voice | Search for an angry version of this person’s voice. |
| Source | Training | Validation |
| Real data | ||
| VoxCeleb1 ( Nagrani et al., 2017 ) | 294,303 | 2,974 |
| ESD ( Zhou et al., 2021 ) | 115,282 | 1,182 |
| MELD ( Poria et al., 2019 ) | 72,526 | 791 |
| CREMA-D ( Cao et al., 2014 ) | 56,754 | 584 |
| L2-ARCTIC ( Zhao et al., 2018 ) | 53,022 | 537 |
| Setting | Value |
|---|---|
| Learning rate | |
| Adam coefficients | , |
| Adam epsilon | |
| Weight decay | |
| Maximum gradient norm | |
| Schedule | Linear warmup, then cosine decay to zero |
| Qwen2.5-Omni-3B Query Prompt: | Qwen2.5-Omni-3B Document Prompt: |
|---|---|
| <|im_start|>user Instruction: [instruction] Query: <|audio_bos|> [audio_tokens] <|audio_eos|><|im_end|> <|im_start|>assistant | <|im_start|>user Document: <|audio_bos|> [audio_tokens] <|audio_eos|><|im_end|> <|im_start|>assistant |
| Gemma-4-E2B-it Query Prompt: | Gemma-4-E2B-it Document Prompt: |
| <bos><|turn>user Instruction: [instruction] Query: [audio_tokens] <turn|> <|turn>model | <bos><|turn>user Document: [audio_tokens] <turn|> <|turn>model |
| MiniLM Query Prompt: | MiniLM Document Prompt: |
| Instruction: [instruction] Query: | Document: |
| Task | Type | Instruction |
| BeijingOpera | Clf | Find recordings with the same percussion instrument as the query. |
| BirdCLEF | Clf | Find recordings with the same bird species as the query. |
| CommonLanguageAgeDetection | Clf | Find recordings with the same speaker age group as the query. |
| GTZANGenre | Clf | Find recordings with the same music genre as the query. |
| IEMOCAPGender | Clf | Find recordings with the same gender as the query. |
| MInDS14 | Clf | Find recordings with the same intent as the query. |
| Method | Model | DailyTalk | VCTK | Expresso | Synthetic | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Cont. | Spk. | Spk. | Sty. | Spk. Sty. | Spk. Sty. ∗ | Ans. | Spk. | Sty. | Env. | ||
| RAND | Random | 1.06 | 1.84 | 1.73 | 1.46 | 1.38 | 1.42 | 0.84 | 1.09 | 1.38 | 1.04 |
| ENC | WavLM-large | 24.50 | 35.00 | 11.50 | 30.75 | 27.50 | 1.00 | 0.75 | 0.75 | 8.75 | 7.50 |
| SENC | ECAPA-TDNN | 24.00 | 100.00 | 71.50 | 24.25 | 70.50 | 0.75 | 1.67 | 87.75 | 13.50 | 1.75 |
| emotion2vec-base | 23.00 | 6.25 | 3.25 | 6.25 | 7.25 | 0.75 | 16.75 | 2.25 | 18.75 | 2.25 | |
| CAS | Qwen3-Embedding-8B | 81.50 | 6.25 | 4.00 | 5.50 | 6.00 | 6.75 | 98.50 | 1.75 | 1.00 | 4.50 |
| Label | Short instruction | Full-sentence instruction |
|---|---|---|
| Cont. | Continuation. | Find speech that continues the conversation in the query. |
| Ans. | Find the answer. | Find speech that contains the answer to the question in the query. |
| Spk. | Same speaker. | Find speech from the same speaker as the query. |
| Sty. | Same speaking style. | Find speech with the same speaking style as the query. |
| Env. | Same environmental sound. | Find speech with the same environmental sound as the query. |
| Doc. | Document: | |
| Qwen2.5-Omni-3B | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Instruction | Form | DailyTalk | VCTK | Expresso | Synthetic | ||||
| Cont. | Spk. | Spk. | Sty. | Ans. | Spk. | Sty. | Env. | ||
| Continuation. | Short | 79.00 | 81.88 | 19.75 | 22.50 | 87.25 | 2.75 | 2.75 | 2.75 |
| Find speech that continues the conversation in the query. | Full | 77.50 | 83.75 | 31.00 | 15.75 | 63.00 | 18.75 | 2.50 | 2.00 |
| Find the answer. | Short | 82.50 | 73.75 | 8.00 | 4.25 | 90.25 | 2.00 | 1.75 | 1.25 |
| Find speech that contains the answer to the question in the query. | Full | 78.50 | 67.50 | 6.25 | 3.50 | 91.00 | 2.00 | 1.50 | 0.25 |
| Instruction | Qwen2.5-Omni-3B | Gemma-4-E2B-it | ||||||
| Spk. Sty. | Spk. Env. | Sty. Env. | Spk. Sty. Env. | Spk. Sty. | Spk. Env. | Sty. Env. | Spk. Sty. Env. | |
| Find the answer. | 0.75 | 1.75 | 3.25 | 2.00 | 1.00 | 1.25 | 1.75 | 1.50 |
| Continuation. | 1.50 | 4.50 | 5.25 | 6.50 | 2.75 | 3.75 | 3.25 | 4.50 |
| Same speaker. | 18.50 | 8.50 | 2.50 | 10.25 | 18.00 | 8.50 | 4.00 | 14.25 |
| Same speaking style. | 9.50 | 2.25 | 3.25 | 7.00 | 12.25 | 4.50 | 7.75 | 12.25 |
| Same environmental sound. | 1.50 | 10.75 | 15.00 | 11.75 | 2.50 | 6.75 | 9.25 | 10.50 |
| Layer Aggregation | DailyTalk | VCTK | Expresso | Synthetic | Avg. | ||||||||||
| R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | |
| Qwen2.5-Omni-3B | |||||||||||||||
| Layer 13 | 5.50 | 15.50 | 22.50 | 61.88 | 76.88 | 84.38 | 32.44 | 48.94 | 57.75 | 17.95 | 35.27 | 46.12 | 29.44 | 44.15 | 52.69 |
| Final Layer | 59.50 | 82.50 | 85.50 | 34.38 | 48.75 | 59.38 | 15.88 | 28.25 | 36.88 | 18.47 | 34.72 | 45.20 | 32.06 | 48.56 | 56.74 |
| Mean Layer | 47.50 | 70.00 | 78.50 | 57.50 | 71.25 | 81.25 | 25.75 | 43.13 | 52.63 | 26.03 | 44.90 | 55.00 | 39.20 | 57.32 | 66.85 |
| Finetuned | 14.50 | 32.50 | 42.00 | 76.25 | 91.25 | 95.00 | 31.19 | 47.38 | 55.50 | 24.90 | 43.35 | 54.49 | 36.71 | 53.62 | 61.75 |
| Layer Aggregation | DailyTalk | VCTK | Expresso | Synthetic | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Cont. | Spk. | Spk. | Sty. | Spk. Sty. | Spk. Sty. ∗ | Ans. | Spk. | Sty. | Env. | |
| Qwen2.5-Omni-3B | ||||||||||
| Layer 13 | 15.50 | 76.88 | 44.75 | 45.50 | 67.25 | 38.25 | 2.00 | 43.50 | 54.75 | 39.25 |
| Final Layer | 82.50 | 48.75 | 11.25 | 39.75 | 44.75 | 17.25 | 94.50 | 27.50 | 40.50 | 31.00 |
| Mean Layer | 70.00 | 71.25 | 37.75 | 42.00 | 57.75 | 35.00 | 87.75 | 50.75 | 45.50 | 40.50 |
| Finetuned | 32.50 | 91.25 | 51.50 | 42.25 | 65.00 | 30.75 | 68.25 | 52.50 | 54.50 | 43.25 |
| Audio Projection | DailyTalk | VCTK | Expresso | Synthetic | Avg. | ||||||||||
| R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | R@10 | R@50 | R@100 | |
| Qwen2.5-Omni-3B | |||||||||||||||
| Frozen | 69.00 | 83.00 | 87.50 | 24.38 | 41.88 | 56.88 | 14.13 | 26.50 | 36.50 | 16.83 | 30.80 | 42.00 | 31.09 | 45.55 | 55.72 |
| Trainable | 59.50 | 82.50 | 85.50 | 34.38 | 48.75 | 59.38 | 15.88 | 28.25 | 36.88 | 18.47 | 34.72 | 45.20 | 32.06 | 48.56 | 56.74 |
| Gemma-4-E2B-it | |||||||||||||||
| Frozen | 17.00 | 34.00 | 46.00 | 24.38 | 45.00 | 58.13 | 13.63 | 24.69 | 31.75 | 9.06 | 15.83 | 21.78 | 16.02 | 29.88 | 39.42 |
| Audio Projection | DailyTalk | VCTK | Expresso | Synthetic | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Cont. | Spk. | Spk. | Sty. | Spk. Sty. | Spk. Sty. ∗ | Ans. | Spk. | Sty. | Env. | |
| Qwen2.5-Omni-3B | ||||||||||
| Frozen | 83.00 | 41.88 | 14.50 | 34.25 | 39.25 | 18.00 | 92.25 | 24.00 | 39.75 | 29.25 |
| Trainable | 82.50 | 48.75 | 11.25 | 39.75 | 44.75 | 17.25 | 94.50 | 27.50 | 40.50 | 31.00 |
| Gemma-4-E2B-it | ||||||||||
| Frozen | 34.00 | 45.00 | 9.75 | 35.25 | 37.75 | 16.00 | 84.75 | 17.25 | 39.25 | 0.75 |
Audio-Text Retrieval
Audio-text retrieval finds audio clips or speech that match text queries, and text that corresponds to audio, using shared representations.
See all 28 papers