Reusing Latent Speech Representations for Query-Conditioned Topic Localization in Transcripts

Steffen Freisinger; Philipp Seeberger; Thomas Ranzenberger; Tobias Bocklet; Korbinian Riedhammer

cs.CLSep 18, 2026

Reusing Latent Speech Representations for Query-Conditioned Topic Localization in Transcripts

Authors: Steffen Freisinger, Philipp Seeberger, Thomas Ranzenberger, Tobias Bocklet, Korbinian Riedhammer

Organizations: Technische Hochschule Nürnberg Georg Simon Ohm

Abstract

Long transcripts are costly inputs for downstream NLP systems and often contain irrelevant context. We study query-conditioned topic localization: predicting the sentence span in a transcript that best addresses a topic-title query. To improve span localization, we reuse ASR encoder states as sentence-level representations and fuse them with textual embeddings. This lets lightweight span locators exploit speech information without running a separate audio encoder. Experiments on two public datasets show consistent gains over text-only baselines, especially under strict boundary-matching criteria. Cross-dataset experiments further indicate that the benefits are strongest for structured or semi-structured speech, while gains on spontaneous speech are limited and mixed.

Figures & tables

Figure 1: Visualization of the topic localization task and our audio-augmented framework.

Figure 2: Architecture overview of our audio-augmented topic localization setup. means that model weights are updated during training, indicates frozen weights, indicates shared model weights.

Method	Input			Euronews				YTSeg
		FLOPs		EM	R@1, IoU $\geq$			EM	R@1, IoU $\geq$
		Offline	Online	EM	0.7	0.5	0.3	EM	0.7	0.5	0.3
VSLNet	T	126M	37.2M	45.26	62.51	72.92	81.95	13.88	32.39	44.69	56.85
	A	203M		58.28	62.15	67.76	72.25	5.97	10.83	16.13	24.22
	T+A	220M		69.11	74.09	79.66	84.19	20.56	37.45	48.63	60.50
QMSum- Pointer	T	126M	21.2M	35.34	51.15	63.18	74.81	7.63	17.33	25.81	37.47

Table 1: Main results on Euronews and YTSeg . T=Text embeddings, A=ASR-derived embeddings, T+A=gated fusion. FLOPs are averaged over Euronews test samples and cover localization-stage computation, excluding shared ASR transcription and text-embedding extraction. Bold marks the best score for each model and dataset.

Encoder	Pooling	Fusion	EM	R@1
Whisper	-	-	45.26	72.92
	mean ${}^{\text{ext}}$	gated	69.11	79.66
		concat	68.52	78.09
		sum	67.89	79.07
	mean	gated	68.21	77.73
	mean+std	gated	67.80	78.40

Table 2: Ablations with VSLNet on Euronews . The first row for each ASR model is the text-only baseline. The last line shows results on Whisper transcriptions and a dedicated audio encoder. R@1 at IoU

\geq 0.5

Method	Text encoder	Euronews				YTSeg
		EM		R@1		EM		R@1
		T	$\boldsymbol{\Delta}_{\text{T+A}}$	T	$\boldsymbol{\Delta}_{\text{T+A}}$	T	$\boldsymbol{\Delta}_{\text{T+A}}$	T	$\boldsymbol{\Delta}_{\text{T+A}}$
VSLNet	MiniLM	45.26	+23.85	72.92	+6.74	13.88	+6.68	44.69	+3.94
	E5-large	55.86	+14.01	78.67	-0.40	19.07	+3.88	50.42	-2.01
	Qwen3-4B	57.16	+16.21	79.84	+3.19	19.68	+5.99	53.25	+2.14
QMSum-Pointer	MiniLM	35.34	+26.63	63.18	+11.31	7.63	+5.47	25.81	+2.90

Table 3: Text-encoder ablations. T denotes text-only performance and

\Delta_{\mathrm{T+A}}

the absolute change when adding ASR-derived representations. R@1 is reported at IoU

\geq 0.5

Figure 3: Performance of VSLNet, using ASR encoder states of different layers. Trained on Euronews with Whisper transcriptions.

Method	Segments	Retriever	Text encoder	AMI		Videoaula		Euronews		YTSeg
Method	Segments	Retriever	Text encoder	EM	R@1	EM	R@1	EM	R@1	EM	R@1
Retrieve	Fixed-size	BM25	–	0.00	6.44	0.13	9.81	1.62	40.59	0.28	19.10
	Predicted (T+A)	BM25	–	0.35	7.50	8.49	17.64	72.07	79.16	19.36	40.36
	Fixed-size	Dense	MiniLM	0.05	6.54	0.27	10.88	1.62	38.08	0.27	13.52
	Predicted (T+A)	Dense	MiniLM	0.41	8.26	5.57	13.93	67.76	74.41	16.93	34.23
	Fixed-size	Dense	E5-large	0.05	8.06	0.27	12.47	1.93	46.48	0.29	19.20

Table 4: Cross-dataset transfer results, with retrieval baselines and our topic-localization setup. All supervised models (segmenters and localizers) are trained on the combined train sets of Euronews and YTSeg . AMI and Videoaula are only used for evaluation. R@1 is reported at IoU

\geq 0.5

. T=Text embeddings, A=ASR-derived embeddings, and T+A=gated fusion. For retrieval over predicted segments, T+A indicates that the supervised segmenter uses text and audio input; retrieval itself is text-based.

Figure 4: Visualization of sentence roles of first 100 Euronews dev recordings. Plots compare text embeddings and ASR-derived audio embeddings, both before projection. Dimensionality reduction using PCA lowrank.

		Euronews		YTSeg		AMI		Videoaula
Task	Input	AUC	F1	AUC	F1	AUC	F1	AUC	F1
Start	T	82.90	35.63	74.06	13.75	68.34	10.91	80.48	28.61
Start	A	97.76	70.68	81.42	18.47	70.15	11.84	94.42	48.73
End	T	62.76	21.73	66.14	10.82	61.03	7.98	65.55	16.08
End	A	96.33	59.36	75.67	14.50	62.46	8.69	94.07	44.96
Boundary	T	71.69	42.23	68.63	20.96	65.76	16.74	69.18	33.62

Table 5: Embedding probe results (in %) for topic-boundary classification.

Cue type	Aspect	Eur	YTS	Vid	AMI
End prosody	Pres.	$\bullet$ $\bullet$ $\bullet$	$\bullet$ $\bullet$	$\bullet$ $\bullet$ $\bullet$ $\bullet$	$\bullet$
End prosody	Dist.	$\bullet$ $\bullet$ $\bullet$	$\bullet$ $\bullet$	$\bullet$ $\bullet$	$\bullet$
Start prosody	Pres.	$\bullet$ $\bullet$	$\bullet$ $\bullet$	$\bullet$ $\bullet$ $\bullet$ $\bullet$	$\bullet$
Start prosody	Dist.	$\bullet$ $\bullet$	$\bullet$ $\bullet$	$\bullet$ $\bullet$	$\bullet$
Discourse markers	Pres.	$\bullet$	$\bullet$ $\bullet$	$\bullet$ $\bullet$ $\bullet$	$\bullet$
Discourse markers	Dist.	$\bullet$	$\bullet$ $\bullet$ $\bullet$	$\bullet$ $\bullet$ $\bullet$ $\bullet$	$\bullet$

Table 6: Qualitative single-author assessment of cues observed around topic boundaries. Presence (Pres.) indicates how frequently a cue was observed, while distinctiveness (Dist.) indicates how clearly it differentiated topic boundaries from regular sentence transitions. Scores range from 0–5, shown as – to

\bullet

\bullet

\bullet

\bullet

\bullet

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Lang	Domain	Partition	# Rec	Duration mm:ss		Topics/Rec
					Recording	Topic
Euronews	multi	news	train	4964	10:56 $\pm$ 5:07	1:29 $\pm$ 1:06	7.17 $\pm$ 2.46
			dev	276	10:46 $\pm$ 5:01	1:22 $\pm$ 0:45	7.81 $\pm$ 3.25
			test	277	11:34 $\pm$ 4:23	1:25 $\pm$ 0:49	10.65 $\pm$ 6.65
YTSeg	en	mixed	train	16403	20:16 $\pm$ 25:15	2:17 $\pm$ 3:20	8.75 $\pm$ 5.36
			dev	1443	20:43 $\pm$ 25:09	2:18 $\pm$ 3:20	8.88 $\pm$ 5.67

Appendix

Table 7: Dataset statistics; reported

mean\pm stddev

; multi={en, de, es, fr, pt, it}.

Stage	Module	Input
Stage	Module	T	A	T+A
Offline	ASR pooling	-	57,086,045	57,086,045
	Text emb	274,575,236,920	274,575,236,920	274,575,236,920
	Index	8,502,409	28,341,363	45,346,180
	Context enc	117,749,276	117,749,276	117,749,276
	TOTAL	274,701,488,605	274,778,413,604	274,795,418,421

Appendix

Table 8: FLOPs by stage, module, localizer and input modalities. Values are averaged over Euronews test samples.

Dataset	Speech/content style	Typical observed cues	Interpretation
Euronews	Structured broadcast news.	Frequent transition sounds, background-noise changes, speaker changes, pauses, occasional language changes or foreign-language excerpts, falling intonation at topic ends.	Cues are highly distinct and often production-driven. This is consistent with the strong audio-only and text+audio results.
YTSeg	Heterogeneous YouTube content, including tutorials, lectures, reviews, interviews, news, and informative videos.	Pauses, pitch resets, emphatic starts, discourse markers such as "So", "Now", "Let’s", occasional swoosh effects, music changes, background changes, and explicit numbering.	Cues are present but vary strongly by format. Tutorials and reviews are often clearer than lectures or freer speech.
Videoaula	Portuguese computer-science lectures with guided, semi-spontaneous speech.	Pauses, falling intonation at topic ends, rising pitch or increased volume at topic starts, and markers such as "Bem" or "Bom". No production cues or speaker changes.	Boundaries are mainly vocal and discourse-structural. Yet, the cues are less explicit than in edited news.
AMI	Spontaneous multi-speaker meetings.	Frequent speaker changes, pauses, hesitations, incomplete speech, laughter, and markers such as "Okay", "Right", "Well", or "So".	Many cues occur throughout the meeting and are not boundary-specific. Topic changes are often gradual and tied to interaction shifts.

Appendix

Table 9: Dataset-level observations from the manual acoustic cue analysis.

Cue type	Examples
End prosody	Falling intonation, reduced energy, lower pitch, slower speech rate, lowered voice at topic end.
Start prosody	Higher pitch, emphatic or energetic start, increased volume, reset of voice, faster speech rate.
Discourse markers	Initial words such as "So", "Now", "Okay", "Right", "Well", Portuguese "Bem"/"Bom", or explicit numbering such as "Number 3".
Pauses	Short or medium silence between topic end and next topic start.
Speaker/language	Speaker change/alternation, language change, foreign-language excerpt.
Production cues	Swoosh or transition sound, jingle, background music, music swell, change in background noise, inserted recording excerpt, scene or sound-setting change.

Appendix

Table 10: Cue categories used in the manual acoustic cue analysis.

Query	Input	Euronews		YTSeg
		( $\Delta$ )EM	( $\Delta$ )R@1	( $\Delta$ )EM	( $\Delta$ )R@1
Title	T	45.26	72.92	13.88	44.69
Title	T+A	69.11	79.66	20.56	48.63
Short	T	-4.58	-5.83	-1.77	-5.75
Short	T+A	-8.94	-9.66	-2.30	-5.08
Long	T	-5.25	-7.63	-2.49	-8.19

Appendix

Table 11: Query perturbation results for VSLNet. Title rows show absolute EM and R@1 scores at IoU

\geq 0.5

. All other rows show changes relative to the Title setting. Negative values indicate performance drops.

SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data
Oct 13, 2026Sangkwon Park, Dongwook Lee, Youngho Cho +3Audio-Text Retrieval
ECE, Seoul National University · IPAI, Seoul National University · University of Seoul +1
CLAR: CIF-Localized Alignment for Retrieval-Augmented Speech LLM-Based Contextual ASR
Mar 26, 2026Shangkun Huang, Huan Shen, Wei Zou +1Contextual ASR Automatic Speech Recognition
BRVoice Team, Bairong, Inc., China
Topic Matching in the Wild: Benchmark and Lessons from Real-World ASR Transcripts
Aug 27, 2026Saman Rahbar, Xiliang Zhu, Irvin Cardoza +1Semantic Textual Similarity
Dialpad Inc.

Oct 13, 2026J/K move · Enter open · S save

SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data

Sangkwon Park, Dongwook Lee, Youngho Cho, Donghun Kang+2

ECE, Seoul National University · IPAI, Seoul National University · University of Seoul+1

Audio-Text Retrieval

Open paper AskPDF arXiv

End-to-end speech-to-text retrieval is becoming an important capability for voice agents, allowing spoken queries to retrieve relevant documents without intermediate transcription. Training these retrievers typically depends on large collections of spoken query-document pairs that are costly to acquire and difficult to extend across domains and languages. We introduce \ours, which replaces such task-specific supervision with speech-transcript pairs from standard ASR corpora. A lightweight adapter projects the output of a frozen speech encoder into the retrieval space of a frozen pretrained text embedding model. \ours is competitive with cascaded ASR-then-retrieve systems while reducing inference latency, and outperforms existing publicly available speech embedding models. It also transfers to text-to-speech and speech-to-speech retrieval without additional training. Controlled analyses validate the speech-transcript alignment paradigm, demonstrating robustness to reduced training data and consistent gains from larger corpora. Further ablations of loss functions and trainable modules establish practical guidelines for the framework. Model weights and codes are to be released.

Abstract

Explore similar work

SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data

CLAR: CIF-Localized Alignment for Retrieval-Augmented Speech LLM-Based Contextual ASR

Topic Matching in the Wild: Benchmark and Lessons from Real-World ASR Transcripts