Topic Matching in the Wild: Benchmark and Lessons from Real-World ASR Transcripts

Saman Rahbar; Xiliang Zhu; Irvin Cardoza; David Rossouw

cs.CLAug 27, 2026

Topic Matching in the Wild: Benchmark and Lessons from Real-World ASR Transcripts

Authors: Saman Rahbar, Xiliang Zhu, Irvin Cardoza, David Rossouw

Organizations: Dialpad Inc.

Abstract

In contact centers, real-time agent-assist tools determine, for each of many predefined topics, whether a live customer utterance is relevant and display a coaching card to the agent when it is. The input is noisy and challenging: ASR(Automatic Speech Recognition) transcripts of spontaneous phone conversations, which can be unclear, repetitive, and mostly lack punctuation. To systematically study this real-world task, we curate a human-annotated topic-utterance judgments dataset sourced from real call-center transcripts. We compare three types of matchers: a regex-based baseline, zero-shot sentence embedding encoders, and Gemini-based LLM matchers. In addition, two types of topic representations are studied in our benchmark:keyphrases and natural language description. Our empirical experiments highlight the superior performance of lightweight LLM matchers over embedding and regex models when equipped with natural language descriptions.

Figures & tables

Panel of Figure 1 — Figure 1: Our evaluation space. A live call yields a noisy utterance $u$ ; a predefined topic $t$ is expressed two ways: a keyphrase list or a natural-language description . Every matcher scores the utterance against a topic representation: a regex baseline (keyphrases only), a zero-shot embedding (cosine similarity), or an LLM prompted for a YES/NO decision. A positive match surfaces a coaching card to the agent. We evaluate every applicable (utterance, representation) pairing; the best ( $\star$ , blue) is an LLM reading a description.

Panel of Figure 2 — Figure 2: Main results: $F_{1}$ on the human-gold benchmark. Error bars are $95\%$ bootstrap CIs for the native-decision methods; the best embedding bar is a 5-fold cross-validated point estimate (fold $\sigma\!=\!0.027$ ; Table 1 ) and is shown without a bootstrap bar as it is not a native-decision method. LLM matchers (blue) lead the regex baseline (orange) and the best of six zero-shot embedding encoders (green); the two lightweight Flash-tier matchers top the board and the LLM–regex gap is significant ( $p<10^{-6}$ for all four matchers).

Keyphrases	P	R	$F_{1}$
Regex (baseline)	0.850	0.625	0.721
all-MiniLM-L6-v2	0.689	0.731	0.708
e5-base-v2	0.673	0.743	0.706
bge-base-en-v1.5	0.669	0.748	0.706
bge-small-en-v1.5	0.657	0.762	0.705
mpnet-base-v2	0.750	0.637	0.689

Table 1: All matchers on the keyphrase representation. Embeddings are 5-fold cross-validated; regex and LLMs use their native decision. Best

F_{1}

in bold.

Natural-language description	P	R	$F_{1}$
e5-base-v2	0.687	0.635	0.660
all-MiniLM-L6-v2	0.660	0.640	0.649
bge-small-en-v1.5	0.646	0.640	0.641
bge-base-en-v1.5	0.615	0.657	0.635
e5-small-v2	0.646	0.616	0.627
mpnet-base-v2	0.604	0.626	0.613

Table 2: All matchers on the natural-language description representation. Regex is omitted as it matches keyphrase lists only. Embeddings are 5-fold cross-validated. Best

F_{1}

in bold.

Matcher	Latency (ms)		Tokens		$/1k
	p50	p95	in	out
Gemini 3 Flash	1742	5018	100	165	0.55
2.5 Flash-Lite	346	528	100	1	0.010
2.5 Pro	5580	8365	100	525	5.38
2.5 Flash	461	771	100	1	0.033
e5-base-v2	29	32	–	–	–

Table 3: Per-decision latency and API cost on 200 sampled topic–utterance pairs. LLMs score the natural-language description; embedding latency is the per-utterance encode and regex the keyphrase match. Gemini $/1k is the marginal API cost (prices accessed 2026-08-27; output includes billed thinking tokens). Embeddings and regex are self-hosted and incur no API cost. Latency is measured under identical conditions from one client to the Google Gemini API; embeddings and regex are timed locally on a single CPU core.

Semantic Textual Similarity

Semantic Textual Similarity measures how closely two texts match in meaning, supporting comparison, retrieval, clustering, and evaluation.

See all 44 papers

Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
mu-bench: A Multilingual Utterance Transcription Benchmark
Sep 25, 2026Andrea Li, Soham RayConversational Speech Recognition ASR Evaluation
UC Berkeley · Sierra AI
agentic-ger: terminology recovery in long-form speech using global context
Sep 24, 2026Yanqiao Zhu, Wupeng Wang, Zhifu Gao +2Automatic Speech Recognition Long-Form Speech Recognition
X-LANCE Lab, Shanghai Jiao Tong University · Shanghai Innovation Institute · Alibaba Token Foundry

Sep 30, 2026J/K move · Enter open · S save

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang

Department of Electronic Engineering, Tsinghua University · University of Cambridge

ASR Evaluation Voice Agent Evaluation Computer-Use Agents LLM Agent Evaluation Automatic Speech Recognition Computer-Use Agent Evaluation+2

Open paper AskPDF arXiv

Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.

Abstract

Semantic Textual Similarity

Explore similar work

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

mu-bench: A Multilingual Utterance Transcription Benchmark

agentic-ger: terminology recovery in long-form speech using global context