A Reranker for Orchestrating Heterogeneous Speech and Text Retrievers

Inho Kim; Sumyeong Ahn

cs.CLAug 24, 2026

A Reranker for Orchestrating Heterogeneous Speech and Text Retrievers

Authors: Inho Kim, Sumyeong Ahn

Organizations: Korea Institute of Energy Technology, South Korea

Abstract

Retrieval-Augmented Generation (RAG) systems have attracted significant interest for their ability to mitigate hallucinations in Large Language Models (LLMs). Although knowledge databases for RAG are increasingly diversifying to include various modalities such as speech and text, research on handling such multi-modal database scenarios remains limited. In this paper, we propose STeReO (Speech and Text Reranking Orchestrator), a reranker based on speech and text retrievers that aggregates disparate modality databases. To address the lack of specialized training data, we first curate a dataset comprising queries, mixed-modality evidence, and their corresponding relevance ranks. We then train the reranker and evaluate its effectiveness in both single-modality and mixed-modality scenarios. Our results demonstrate that the proposed algorithm excels at selecting the most relevant evidence, thereby significantly improving downstream question-answering performance.

Figures & tables

Figure 1: Comparison of three RAG systems: (a) Text-based, (b) Speech-based, and (c) Text+Speech-based RAG.

Figure 2: Overview of the proposed framework STeReO . (a) construction dataset for mixed modality reranker, (b) training reranker, and (c) inference procedure based on the proposed reranker model.

Method	Objective Formulation
Pointwise	$\displaystyle\mathcal{L}_{\text{point}}=-\frac{1}{\|\mathcal{C}^{\star}\|}\sum_{i}\bigl[y_{i}\log\sigma(s_{i})+(1{-}y_{i})\log(1{-}\sigma(s_{i}))\bigr]$
Pairwise	$\displaystyle\begin{aligned} \mathcal{L}_{\text{pair}}=&\frac{1}{\|\mathcal{P}\|}\sum_{(i,j)\in\mathcal{P}(q)}\log\left(1+\exp\left(-\delta_{ij}\cdot(s_{i}-s_{j})\right)\right)\\ &\text{where}\quad\mathcal{P}=\{(i,j):c_{i},c_{j}\in\mathcal{C}^{\star},y_{i}\neq y_{j}\},\quad\delta_{ij}=\operatorname{sign}(y_{i}-y_{j})\end{aligned}$
Listwise	$\displaystyle\begin{aligned} \mathcal{L}_{\text{list}}=1-\frac{1}{\mathrm{IDCG}}\sum_{i}\frac{2^{y_{i}}-1}{\log_{2}(1+\hat{r}_{i})}\quad\text{where}\quad\hat{r}_{i}=1+\sum_{j\neq i}\sigma\left(\tau(s_{j}-s_{i})\right)\end{aligned}$

Table 1: Training objectives for reranker optimization.

\sigma(\cdot)

denotes the sigmoid, and

\tau

is a hyperparameter set to

10

		Single-domain						Mixed-domain
		Spoken SQuAD			MS MARCO			Spoken SQuAD			MS MARCO
Backbone	Obj.	Hit@1	MRR	NDCG@5	Hit@1	MRR	NDCG@5	Hit@1	MRR	NDCG@5	Hit@1	MRR	NDCG@5
Baseline	$Z$ -score	0.6927	0.7498	0.7678	0.6864	0.7694	0.7987	0.5273	0.6408	0.6857	0.6621	0.7510	0.7846
Ultravox	Pointwise	0.7834	0.8008	0.8057	0.7432	0.8060	0.8262	0.7630	0.7892	0.7971	0.6977	0.7755	0.8032
	Pairwise	0.4855	0.6288	0.6775	0.4394	0.6195	0.6865	0.3602	0.5434	0.6131	0.3336	0.5349	0.6220
	Listwise	0.4430	0.6017	0.6572	0.3356	0.5487	0.6332	0.4430	0.6017	0.6572	0.1185	0.3737	0.4996

Table 2: Reranking performance under Single- and Mixed-domain settings. All results use Max pooling for audio window size.

Category	Metric	Value	Gain ( $\Delta$ )
Label Quality	F1 Score	0.700	-
Label Quality	MCC	0.614	-
Ranking (Hit@1)	Z-score (Baseline)	0.578	-
	Foundation ALM (gpt-4o)	0.652	+0.074
	Oracle (GT)	0.825	+0.247

Table 3: Evaluation of the foundation ALM. Label quality is measured on

5

K samples, and ranking performance is assessed via Hit@1 on

1

K, respectively.

Spoken SQuAD				MS MARCO
Generator	Scoring Model	Single	Mixed	Generator	Scoring Model	Single	Mixed
Ultravox	$Z$ -score Retrieval	0.4555	0.3565	Ultravox	$Z$ -score Retrieval	0.3637	0.3531
	Qwen2-Audio	0.5046	0.4762		Qwen2-Audio	0.3834	0.3729
	Qwen-Audio-Chat	0.4986	0.4659		Qwen-Audio-Chat	0.3795	0.3666
	Ultravox	0.5088	0.4796		Ultravox	0.3880	0.3719
Qwen2-Audio	$Z$ -score Retrieval	0.3335	0.2646	Qwen2-Audio	$Z$ -score Retrieval	0.3885	0.3774

Table 4: Downstream QA performance (EM) comparison. GPT evaluation is conducted on a representative subset of

1

K samples.

Query Set	w/o Z-score (%)	w/ Z-score (%)
Spoken SQuAD	4.0	45.2
MS MARCO	0.1	18.3

Table 5: The ratio of speech evidence in top-5 candidates.

Backbone	MS MARCO		Spoken SQuAD
	Mean	Max	Mean	Max
Ultravox	0.7918	0.7755	0.7460	0.7892
Qwen-Audio-Chat	0.7821	0.7720	0.7286	0.7728
Qwen2-Audio	0.7885	0.7777	0.7468	0.7872

Table 6: MRR under inference-time Mean/Max pooling.

	Spoken SQuAD			MS MARCO
Window	Hit@1	MRR	NDCG@5	Hit@1	MRR	NDCG@5
120s ${\times}$ 1	0.7509	0.7822	0.7919	0.7092	0.7834	0.8091
60s ${\times}$ 2	0.7515	0.7827	0.7922	0.7138	0.7862	0.8112
30s ${\times}$ 4	0.7584	0.7872	0.7957	0.6997	0.7777	0.8049

Table 7: Window-length analysis on Qwen2-Audio (Mixed).

Multimodal Reranking

Multimodal reranking reorders retrieved candidates using joint visual, textual, and other modality-specific evidence.

See all 35 papers

Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation
Oct 28, 2025Alexander Martin, William Walden, Reno Kriz +5Multimodal RAG RAG Evaluation
Johns Hopkins University · Human Language Technology Center of Excellence
Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present?
Jun 14, 2026Sadanand Singh, Allam Reddy, Manan ChopraCross-Encoder Reranking Learning to Rank
Cascade Research
MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation
Jun 24, 2026Xiaochen Wang, Bao Hoang, Han Liu +2Multimodal RAG Retrieval-Augmented Generation
The Pennsylvania State University University Park, Pennsylvania, USA · Michigan State University East Lansing, Michigan, USA · Dalian University of Technology Dalian, Liaoning, China +1

Oct 28, 2025J/K move · Enter open · S save

Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation

Alexander Martin, William Walden, Reno Kriz, Dengjia Zhang+4

Johns Hopkins University · Human Language Technology Center of Excellence

Multimodal RAG RAG Evaluation

Open paper AskPDF arXiv

We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a more prevalent source of information online, RAG systems must integrate such media into generation. Yet, existing evaluation methods for RAG are largely text-centric and do not readily transfer to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE and compare it to multimodal variants of three prominent text-centric RAG metrics---ALCE, ARGUE, and RAGAS---finding that MiRAGE outperforms all three on text while being the only one to generalize to multimodal sources. We release open-source implementations and outline evaluation methods for multimodal RAG.

Abstract

Multimodal Reranking

Explore similar work

Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation

Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present?

MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation