INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval

Chen-An Li; Hung-yi Lee

cs.SDAug 17, 2026

INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval

Authors: Chen-An Li, Hung-yi Lee

Organizations: National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)

Abstract

Existing speech retrieval systems rely on fixed similarity matching and cannot adapt to diverse user intents. We introduce INSPIRE, the first benchmark for instruction-aware speech retrieval, in which natural-language instructions dynamically specify relevance criteria, including semantic content, speaker identity, speaking style, environmental sounds, and their combinations. We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models. Our results reveal that no current method robustly handles all retrieval intents. Text-based approaches perform relatively better at semantic retrieval but struggle with paralinguistic attributes, while speech-based models are moderately better at capturing acoustic properties but falter at following instructions. These findings highlight the need for unified architectures capable of instruction-aware speech retrieval.

Figures & tables

Figure 1: Illustrating instruction-aware speech retrieval across diverse user intents such as content, speaker, style, background, and their combinations.

Instruction Type	Instruction Example
Continuation	Search for the follow-up lines to the query speech.
Same speaker	Find all documents said by the same talker as the query.
Same speaking style	Return documents with a similar way of speaking to the query.
Same speaker + Same style	Get documents where the voice matches and the speaking style matches.
Same speaker + Sadness style + Same env.	Get sad-style recordings from the same speaker in the same sound environment.
Same speaker + Footsteps env.	Retrieve clips from the same speaker that include footsteps in the background.

Table 1: Example instructions in INSPIRE. Instructions range from single-attribute criteria to complex multi-attribute combinations.

Attribute

\backslash

Subset

DailyTalk

VCTK

Expresso

Synthetic

Semantic

–

Speaker

–

Speaking Style

–

Environmental Sound

–

Table 2: Each column represents a subtask in subsets, where attribute constraints are defined as follows: S indicates the same value as the query, T indicates a specific target value, and – indicates no constraint.

Subset	$\|Q\|$	$\|Q_{Z}\|$	$\|D\|$	$\|D^{+}\|/\|Q_{Z}\|$
DailyTalk	200	200	4,882	1.000
Expresso	200	800	3,861	2.000
VCTK	80	80	3,082	2.000
Synthetic	200	3,000	5,400	1.946
Total / Avg.	680	4,080	17,225	1.910

Table 3: Statistics of INSPIRE.

|Q|

denotes the number of unique spoken queries,

|Q_{Z}|

the number of query-instruction pairs,

|D|

the number of documents,

|D^{+}|/|Q_{Z}|

the average number of positive documents per query-instruction pair.

Random Baseline
	Size (B)	DailyTalk			VCTK			Expresso			Synthetic
	Size (B)	R@10	R@50	R@100	R@10	R@50	R@100	R@10	R@50	R@100	R@10	R@50	R@100
Random	-	0.24	1.06	2.10	0.35	1.84	3.63	0.30	1.50	2.96	0.20	1.00	1.98
LALMs Embeddings
Audio-Flamingo-3 [ 57 ]	8	30.00	57.50	69.50	1.25	6.25	7.50	2.19	7.31	10.94	3.82	6.31	7.72
Qwen2.5-Omni [ 58 ]	3	26.50	53.00	64.00	1.88	3.13	3.13	0.75	3.75	6.69	3.14	5.20	6.88

Table 4: Main results on INSPIRE. We report Recall@10/50/100 across four subsets. Model size is reported in billions of parameters, and the best results per subset are shown in bold.

Panel of Figure 2 — Figure 2: Performance across attribute constraints shown in radar plots for LALMs (left), cascaded pipelines (middle), and self-supervised/contrastive models (right). We report Recall@50. Sem refers to semantic, Spk to speaker, Sty to speaking style, and Env to environmental sound. Attributes marked with * specify a distinct target value rather than matching the query. Scores are normalized by the best-performing model per attribute, with a shared scale across all subfigures.

	DailyTalk		VCTK		Expresso		Synthetic
Model	N@10	N@50	N@10	N@50	N@10	N@50	N@10	N@50
LALMs Embeddings Rerank with LALMs
Qwen2.5-Omni-7B	21.47	27.45	1.10	1.52	0.46	1.18	2.02	2.47
+ Qwen2.5-Omni-3B	38.50	41.69	0.77	1.34	1.05	1.53	3.29	3.47
+ Voxtral-Mini-3B	46.32	48.32	1.01	1.79	0.18	0.77	3.14	3.43
Voxtral-Small	26.60	32.55	1.53	2.17	0.12	0.41	2.59	3.08

Table 5: Reranking performance measured by NDCG@10/50 on the top 100 documents from first-stage retrievers.

Figure 3: Impact of instruction usage on retrieval performance. Solid bars show results with instructions; hatched bars without. Models A to F are LALMs; G to J are cascaded pipelines.

Figure 4: Comparison of captioning models for cascaded pipelines using detailed and instruction-aware prompts.

	DailyTalk		VCTK		Expresso		Synthetic
Model	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
BM25	36.50	59.50	1.25	5.63	0.56	3.06	4.06	6.89
+ Oracle	\textcolor red36.00	\textcolor red49.00	\textcolor darkgreen53.13	\textcolor darkgreen95.00	\textcolor darkgreen0.63	\textcolor red2.69	\textcolor darkgreen5.18	\textcolor darkgreen7.57
SentenceBERT	32.50	50.50	0.63	3.75	0.44	2.63	4.16	6.28
+ Oracle	\textcolor red14.00	\textcolor red25.50	0.63	\textcolor red1.25	\textcolor darkgreen0.50	\textcolor darkgreen3.50	\textcolor red1.53	\textcolor red3.86
E5-Mistral	55.50	78.00	1.25	5.63	1.63	5.25	6.50	8.32

Table 6: Effect of oracle metadata on cascaded text retrieval. \textcolor darkgreenGreen indicates improvement and \textcolor redred indicates degradation compared to the caption baseline.

	DailyTalk		VCTK		Expresso		Synthetic
Model	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
LALMs Embeddings
Qwen3-Omni	51.50	74.50	1.25	3.13	0.94	3.56	4.88	6.38
Voxtral-Small	41.00	67.50	1.25	3.75	0.31	1.44	5.17	7.23
Cascaded Pipelines with Open-source Models
E5-Mistral	55.50	78.00	1.25	5.63	1.63	5.25	6.50	8.32

Table 7: Comparison with proprietary models. Results are shown for representative open-source models and proprietary APIs from Google and OpenAI.

	DailyTalk		VCTK		Expresso		Synthetic
Model	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
Audio-Flamingo-3	30.00	57.50	1.25	6.25	2.19	7.31	3.82	6.31
+ Mean Pooling	34.00	56.50	3.75	8.13	1.94	6.25	0.57	2.08
Qwen2.5-Omni-7B	36.00	62.50	1.25	3.13	0.63	3.38	4.18	6.00
+ Mean Pooling	27.00	44.50	0.63	2.50	1.50	4.25	1.22	3.68
Qwen3-Omni	51.50	74.50	1.25	3.13	0.94	3.56	4.88	6.38

Table 8: LALMs pooling strategy comparison. We compare the prompt-based last token embedding approach versus mean pooling without prompts.

Panel of Figure 5 — Figure 5: Layer-wise analysis of self-supervised speech models.

	DailyTalk		VCTK		Expresso		Synthetic
Modality	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
A $\rightarrow$ A	5.00	13.00	16.88	31.88	13.88	20.94	3.28	8.15
T $\rightarrow$ A	0.00	1.00	1.25	4.38	3.13	6.44	0.45	1.70
A $\rightarrow$ T	0.00	1.50	1.88	4.38	1.94	5.63	0.80	2.22
T $\rightarrow$ T	7.00	17.00	0.63	5.00	0.81	5.50	0.72	2.70

Table 9: Modality analysis for a contrastive audio-language model.

A

and

T

denote audio and text, respectively.

X\rightarrow Y

indicates that the query is encoded with the

X

encoder and the document with the

Y

encoder.

Explore similar work

VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval
Sep 16, 2026Aaron Yee, Fengjie Lu, Jiarui Hai +5Audio-Text Retrieval Hybrid Retrieval
Zhejiang University · Johns Hopkins University · independent researcher
SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data
Oct 13, 2026Sangkwon Park, Dongwook Lee, Youngho Cho +3Audio-Text Retrieval
ECE, Seoul National University · IPAI, Seoul National University · University of Seoul +1
Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Mar 10, 2026Maike Züfle, Sara Papi, Fabian Retkowski +5Audio-Language Model Evaluation Speech Language Models
Karlsruhe Institute of Technology, Germany · Fondazione Bruno Kessler, Italy · ACC Cyfronet AGH, Poland +2

Sep 16, 2026J/K move · Enter open · S save

VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval

Aaron Yee, Fengjie Lu, Jiarui Hai, Chenang Jiang+4

Zhejiang University · Johns Hopkins University · independent researcher

Audio-Text Retrieval Hybrid Retrieval

Open paper AskPDF arXiv

Speech retrieval has become increasingly important as spoken content continues to grow across meetings, lectures, podcasts, and videos. Existing benchmarks and models have advanced semantic search over spoken content, but largely focus on what is said while overlooking who says it. In many real-world scenarios, however, users need to retrieve speech based jointly on semantic content and a target speaker, where the speaker may be specified naturally through a reference speech utterance rather than a predefined identity. To address this gap, we introduce VoiceTrace-Bench, a benchmark for hybrid speech retrieval in which each query combines text specifying what to retrieve with reference speech specifying who to retrieve. This setting requires models to integrate complementary semantic and speaker information directly from heterogeneous query inputs. Motivated by the joint audio-text modeling capabilities of audio-language models (ALMs), we develop VoiceTrace, a two-stage retrieval framework consisting of VoiceTrace-Emb, an embedding model that learns unified representations for efficient large-scale retrieval, and VoiceTrace-Reranker, a reranking model that jointly examines each query--candidate pair for fine-grained relevance estimation. Experiments show that VoiceTrace achieves state-of-the-art performance on established semantic speech retrieval benchmarks, while substantially outperforming cascade-based approaches on VoiceTrace-Bench, demonstrating its effectiveness for both conventional semantic retrieval and the new hybrid retrieval setting.

Abstract

Explore similar work

VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval

SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data

Do What I Say: A Spoken Prompt Dataset for Instruction-Following