DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agents

Puneet Mathur; Nedim Lipka; Zeyu Jin; Dinesh Manocha

cs.CLSep 29, 2026

DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agents

Authors: Puneet Mathur, Nedim Lipka, Zeyu Jin, Dinesh Manocha

Organizations: Adobe Research, San Jose, USA · University of Maryland College Park, USA

Abstract

Voice agents enable low-latency, natural interaction, yet their ability to faithfully ground responses in external documents remains underexplored. We introduce DuplexSpeechBench-Document Grounding (DSB-DG), a benchmark for evaluating document grounding in voice agents across five professional domains. DSB-DG targets three failure modes: Context Saturation, which measures grounding under increasing document length; Grounding Decay, which measures retention of document facts across multi-turn dialogue; and Proactive Grounding, which evaluates whether context re-injection mitigates conversational drift. The benchmark contains 1,636 adversarially verified QA pairs from 50 documents covering five professional domains, and supports fully automatic evaluation of grounding accuracy, hallucination, and response latency. Across systems spanning cascaded, proprietary full-duplex and real-time, and open-weight speech2speech architectures, we find substantial differences in effective grounding capacity. While cascaded pipeline (ASR-LLM-TTS) achieves the highest grounding accuracy, Gemini-Live and GPT-Realtime closely trail behind. Open-weight systems exhibit distinct failure modes, most notably an abrupt context-capacity collapse and multi-turn grounding decay. More broadly, grounding fidelity degrades with context and conversational load, and failures frequently manifest as unsupported generations rather than abstention. We show that contextual grounding as a key unresolved challenge for reliable full-duplex voice agents.

Figures & tables

Figure 1: DuplexSpeechBench–Document Grounding (DSB-DG) benchmark provides document context and spoken queries to a full-duplex speech model, and evaluates grounding accuracy, multi-turn retention, and real-time behavior through three complementary protocols.

Benchmark	Full-	Real-	Long	Long-	Multi-	Document-	Halluc.	Latency
	Duplex	Time	Document	Context	Turn	Factuality	Eval.	Eval.
SpokenSQuAD ( Li et al., 2018 )	✗	✗	✗	✗	✗	✓	✗	✗
LongBench ( Bai et al., 2024 )	✗	✗	✓	✓	✗	✓	✗	✗
Full-Duplex-Bench ( Lin et al., 2025 ; Lin et al., 2026c )	✓	✓	✗	✗	✗	✗	✗	✓
FD-Bench ( Peng et al., 2025 )	✓	✓	✗	✗	✗	✗	✗	✓
Full-Duplex-Bench-v2 ( Lin et al., 2026b )	✓	✓	✗	✗	✓	✗	✗	✓

Table 1: Comparison of DSB-DG with spoken-dialogue benchmarks. Existing voice agent benchmarks primarily evaluate conversational dynamics, multi-turn interaction, or task completion, whereas long-context benchmarks evaluate document understanding without real-time spoken interaction. DSB-DG jointly evaluates document factuality, multi-turn grounding retention, hallucination, and latency in full-duplex voice agents.

Figure 2: DuplexSpeechBench–Document Grounding (DSB-DG) . We inject document context into a full-duplex speech language model that answers spoken user queries in real time. The benchmark comprises three complementary protocols: Context Saturation , evaluating grounding under increasing document lengths; Grounding Decay , evaluating retention of document facts across multi-turn conversations using repeated factual probes separated by distractor dialogue; and Proactive Grounding , evaluating the effectiveness of periodic document re-injection. Model responses are automatically scored using Grounding Accuracy (GA), Hallucination Rate (HR), Grounding Decay Score (GDS), and First-Token Emission Delay (FTED).

Domain	# Docs	# QA	Avg. Qs / Doc	Document Characteristics
Clinical Trials	10	391	39.1	Structured protocols, eligibility criteria, clinical endpoints
Earnings Calls	10	314	31.4	Financial reporting, analyst Q&A, dense numerical content
FDA 510(k)	10	292	29.2	Regulatory filings, device specifications, standardized structure
Commercial Leases	10	358	35.8	Contract clauses, cross-references, legal reasoning
Municipal Minutes	10	281	28.1	Procedural narratives, motions, amendments, voting records
Total	50	1,636	32.7	Multi-turn spoken conversations (20) grounded in documents

Table 2: Domain composition and dataset statistics of DuplexSpeechBench-Document Grounding : 1,636 unique, adversarially verified QA pairs across five distinct professional domains along with complex multi-turn dialogue scripts.

Model	T1 ( $\approx$ 500)	T2 ( $\approx$ 1k)	T3 ( $\approx$ 2k)	T4 ( $\approx$ 4k)	T5 ( $\approx$ 8k)
Cascaded	88.7	89.8	90.3	90.4	88.5
Gemini-Live	85.9	86.9	87.4	90.2	86.9
GPT-Realtime	88.0	86.6	87.7	88.2	85.8
MiniCPM-o	85.9	87.6	88.0	87.2	74.0
UV-32B	87.0	85.4	85.5	80.2	75.4
SALMONN	77.4	76.3	70.3	37.2	25.4

Table 3: Context Saturation, measured via Grounding Accuracy (%), across token-length tiers. Cascaded, Gemini-Live, and GPT-Realtime stay robust throughout, while MiniCPM-o shows a sharp T5 drop; SALMONN collapses at T4 nears context capacity. Moshi never grounds.

Panel of Figure 3 — Figure 3: Hallucination Rate (HR) across context tiers. SALMONN’s T3 $\to$ T4 collapse coincides with rising hallucination, while MiniCPM-o shows a similar failure at T5. Context-capacity limits manifest as unsupported generation.

Domain	Cascaded	GPT-RT	Gemini-L	MiniCPM-o	UV-32B	SALMONN
Clinical	83.9	78.9	81.4	80.7	82.5	62.5
Earnings	93.8	89.4	88.2	91.9	91.9	70.8
FDA 510(k)	82.1	84.6	78.6	78.6	84.6	57.3
Leases	98.6	98.6	96.4	95.7	97.8	86.2
Municipal	96.1	93.3	95.0	96.1	95.0	78.2

Table 4: Grounding Accuracy (%) at the 2k-token tier (T3) across professional domains. Darker red cells reflect relative domain difficulty within each model family; bold is best model for each domain. Clinical protocols and FDA 510(k) filings are most challenging domains, while leases and municipal records are most tractable.

Model	GA A	GA B	GA ${}_{A^{\prime}}$	GDS
Gemini-Live	86.7	85.1	90.1	$\mathbf{+3.3}$
Cascaded	86.7	86.7	86.7	$+$ 0.0
GPT-Realtime	86.7	85.0	87.0	$+$ 0.3
MiniCPM-o	83.3	86.7	81.7	$-$ 1.7
UV-32B	80.0	78.3	75.0	$\mathbf{-5.0}$
SALMONN	76.7	65.0	72.1	$-$ 4.6

Table 5: Grounding Decay results across controlled multi-turn conversations. GA (%) is reported for early probes (

A

), intermediate probes (

B

), and repeated late probes (

A^{\prime}

), with GDS

\,=\,

(A^{\prime})-

(A)

. Gemini-Live is most robust. UV-32B exhibits most conversational forgetting (

-

ve GDS).

	Grounding Accuracy (%)			Injection Benefit (%)
Model	$\mathbf{C_{0}}$	$\mathbf{C_{1}}$	$\mathbf{C_{2}}$	$\boldsymbol{\Delta}_{\text{light}}$	$\boldsymbol{\Delta}_{\text{heavy}}$
	No refresh	1 refresh	Multiple	$(C_{1}-C_{0})$	$(C_{2}-C_{0})$
GPT-Realtime	79.6	94.2	88.1	$+$ 14.6	$+$ 8.5
Cascaded	87.8	94.2	94.9	$+$ 6.4	$+$ 7.1
Gemini-Live	83.89	84.44	83.89	$+$ 0.56	0.0
MiniCPM-o	90.6	89.4	89.4	$-$ 1.1	$-$ 1.1

Table 6: Grounding Accuracy (GA) under Proactive Grounding protocol: no refresh (

C_{0}

), a single refresh (

C_{1}

), and multiple refreshes (

C_{2}

), with improvement relative to

C_{0}

. Cascaded and GPT-Realtime benefit substantially from context re-injection, whereas repeated refreshes degrade UV-32B, MiniCPM-o, and SALMONN. Bold indicates the best GA.

Model	p50 (ms) $\downarrow$	mean (ms) $\downarrow$	p95 (ms) $\downarrow$	QAL (ms) $\downarrow$
Cascaded	1469	8118	61563	143
GPT-Realtime	1214	1311	1761	150
Moshi	229	251	737	228
MiniCPM-o	2317	4566	11975	278
Gemini-Live	6806	9410	6949	856
SALMONN	3763	7665	30349	1118

Table 7: First-Token Emission Delay (FTED) statistics and Quality-Adjusted Latency (QAL

=

FTED

{}_{\text{p50}}\times(1-\mathrm{GA@T3})

) across systems. GPT-Realtime and the cascaded pipeline dominate the frontier. Moshi attains has lowest latency but near-zero grounding. MiniCPM-o, UV-32B, and SALMONN exhibit severe latency tails at higher context tiers.

Panel of Figure 4 — Figure 4: Pareto frontier graph of the quality–latency trade-off in DSB-DG.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	# Docs	Source	Mean Tokens	Token Range
Clinical Trials	10	ClinicalTrials.gov	2,528	1,244–8,038
Earnings Calls	10	SEC EDGAR	9,403	1,370–22,524
FDA 510(k)	10	openFDA	4,946	2,788–7,301
Commercial Leases	10	Synthetic	6,779	6,671–6,875
Municipal Minutes	10	Legistar	3,092	1,335–7,660

Appendix

Table 8: Document provenance and length statistics for the five DSB-DG domains. Mean/range are full-document token counts ( source_tokens , before tiering). Token counts are computed using the cl100k_base encoding ( tiktoken ).

Domain	Grounded	Unanswerable	Total
Clinical Trials	331	60	391
Earnings Calls	254	60	314
FDA 510(k)	233	59	292
Commercial Leases	302	56	358
Municipal Minutes	221	60	281
Total	1,341	295	1,636

Appendix

Table 9: Distribution of QA question types across DSB-DG domains

Domain	Type	Question	Reference Answer
Clinical Trials	grounded / direct_lookup	How many participants were enrolled in this study?	20 participants
Earnings Calls	grounded / direct_lookup	What were Southern Company’s operating revenues for the fourth quarter of 2025?	$7.0 billion
FDA 510(k)	grounded / direct_lookup	What is the regulation number associated with the Francisella tularensis Real-time PCR Assay?	21 CFR 866.4000
Commercial Leases	grounded / multi_span	Who are the landlord and tenant in this lease agreement?	Landlord: Greenfield Realty Partners, LP; Tenant: David Martin
Municipal Minutes	grounded / direct_lookup	Who made the motion to approve the February 2025 meeting minutes?	Alderman James Rodriguez

Appendix

Table 10: Representative DSB-DG question–answer pairs.

Domain	Candidate Question	Decision	Reason
FDA 510(k)	What is the intended use of FMD Peripheral Guidewires?	Rejected	Insufficient document evidence
FDA 510(k)	Who is the regulatory contact for submissions related to the FMD Peripheral Guide Wire devices?	Rejected	Multiple plausible answers
FDA 510(k)	What is the exact submission date of the 510(k) premarket notification?	Rejected	“Unanswerable” but actually answerable
Earnings	What were Southern Company’s operating revenues for the fourth quarter of 2025?	Retained	Unique answer explicitly supported by context

Appendix

Table 11: Representative retained and rejected QA candidates from the verification pipeline.

Field	Operational Definition
Factual Accuracy	Binary judgment indicating whether the response is factually supported by the supplied document.
Hallucination Type	None ; Intrinsic , when the response contradicts information explicitly stated in the document; or Extrinsic , when the response introduces unsupported information not contained in the document.
No Response	The model fails to produce a scoreable answer, including silence or non-answer conversational behavior.
False Refusal	The model refuses to answer despite the required information being explicitly available in the supplied context.
Partial Credit	Integer partial_credit $\in\{0,1,2\}$ : $2$ = fully correct; $1$ = partially correct (captures the main point but misses a detail or hedges unnecessarily); $0$ = wrong, or refused when the question is answerable. For multi_span items a floor of $1$ is enforced when the response addresses the question and the judge’s explanation indicates at least one span was correct.
Question Addressed	Whether the response substantively attempts to answer the user’s question.

Appendix

Table 12: Structured output schema used by the automatic grounding judge.

System	Checkpoint	Backbone	Context Interface	Precision	Hardware/API
Cascaded	whisper-1 / gpt-4o-2024-11-20 / tts-1	GPT-4o	System prompt	–	hosted OpenAI API
GPT-Realtime	gpt-realtime-2025-08-28	Not Public	System item	–	hosted OpenAI Realtime API
MiniCPM-o	openbmb/MiniCPM-o-4_5	MiniCPM-o 4.5	System prompt	bf16	1 $\times$ A100-80GB ( $\sim$ 35 GB)
SALMONN	tsinghua-ee/SALMONN-7B	Vicuna-7B-v1.5	System prompt	fp16	1 $\times$ A100-80GB ( $\sim$ 27 GB)
UV-32B	fixie-ai/ultravox-v0_6-qwen-3-32b	Qwen3-32B	System prompt	bf16	1 $\times$ A100-80GB ( $\sim$ 62 GB)
Moshi	kyutai/moshiko-pytorch-bf16	Moshi/Helium-7B LM + Mimi	KV-cache prefill	bf16	1 $\times$ A100-80GB ( $\sim$ 46 GB)

Appendix

Table 13: Model, inference, and execution configurations used in DSB-DG.

System	Temperature	Top- $p$	Max Output	Seed
Cascaded	0.1	default	300	–
GPT-Realtime	0.6	N/A	provider default	–
MiniCPM-o	greedy	–	256	–
SALMONN	1.0	0.9	150 (max_length)	–
UV-32B	greedy	–	2048	–
Moshi	not exposed	not exposed	N/A	–

Appendix

Table 14: Decoding hyperparameters. For parameters not exposed by an API or architecture, we report “N/A” or “provider default”.

Turn(s)	Content
2–4	Early document probes ( $A$ )
5–6	Domain-relevant distractors
7–9	Intermediate probes ( $B$ )
10–12	Domain-relevant distractors
13–15	Repeated probes ( $A^{\prime}$ )

Appendix

Table 15: Turn structure of the Grounding Decay protocol (0-indexed turn numbers). Turns 0–1 are un-probed opening turns and turn 16 is a closing turn; the probe/distractor layout is identical across all 20 scripts. Each conversation is 17 turns.

Turn	Type	User Utterance
2	Probe $A$	What is the primary purpose of the study?
4	Probe $A$	What imaging technique is considered the gold standard in this study?
6	Distractor	Do you think telemedicine is going to completely replace in-person visits someday?
7	Probe $B$	What is the significance of an AUC value of 0.5?
11	Distractor	What are your thoughts on alternative medicine? Do you think it can really work alongside clinical treatments?
13	Probe $A^{\prime}$	What are the key takeaways from the study?

Appendix

Table 16: Representative probe–distractor–repeat conversation used in Experiment B (abbreviated from clinical_001 ; 0-indexed turn numbers). The turn-13

A^{\prime}

probe repeats the turn-2

A

probe semantically.

Condition	Initial Context	Refresh Turn(s)
$C_{0}$	T3 document (at start)	None
$C_{1}$	T3 document (at start)	one, before the late ( $A^{\prime}$ ) probe group
$C_{2}$	T3 document (at start)	two, before the mid ( $B$ ) and late ( $A^{\prime}$ ) probe groups

Appendix

Table 17: Context schedule for the Proactive Grounding protocol. All three conditions provide the T3 document at the start; they differ in the number of mid-conversation document re-injections (refreshes):

C_{0}

none,

C_{1}

one,

C_{2}

two.

Model	T1	T2	T3	T4	T5
Cascaded	12.2	10.6	9.5	9.5	12.3
GPT-Realtime	11.4	14.2	11.8	12.1	14.8
Gemini-Live	13.3	13.4	12.7	10.5	13.7
MiniCPM-o	13.9	12.7	12.3	13.1	25.0
UV-32B	13.0	11.6	11.4	10.5	12.7
SALMONN	22.6	24.4	30.1	29.8	36.5

Appendix

Table 18: Hallucination Rate (%) across context tiers (intrinsic

+

extrinsic, over scoreable turns). MiniCPM-o’s HR nearly doubles at T5 (12.3

\to

25.0), mirroring its T5 accuracy drop; SALMONN’s HR rises steadily with context. Gemini-Live’s HR stays flat and low across tiers (10.5–13.7%), with no saturation-driven rise. Moshi’s high HR reflects fluent but ungrounded speech.

Model	HR	Intr.	Extr.
Cascaded	10.8	8.2	2.5
MiniCPM-o	16.7	12.6	4.0
GPT-Realtime	13.1	11.4	1.7
Gemini-Live	12.6	10.0	2.6
UV-32B	11.6	10.0	1.6
SALMONN	29.5	27.7	1.8

Appendix

Table 19: Hallucination decomposition (%), pooled over tiers T1–T5 (grounded/ answered turns). Intr. / Extr. are intrinsic/extrinsic hallucination rates; Errors are overwhelmingly intrinsic (misreading the document) rather than extrinsic (inventing facts); Moshi is degenerate (GA

=

3.6%, fluent but ungrounded).

Model	NRR (%) $\uparrow$	FCR (%) $\downarrow$
GPT-Realtime	86.6	12.3
Gemini-Live	85.6	13.2
Cascaded	84.6	14.2
UV-32B	80.3	19.0
MiniCPM-o	77.2	21.2
SALMONN	61.7	31.4

Appendix

Table 20: Behaviour on unanswerable questions: NRR = appropriate-refusal rate (higher is better), FCR = false-confidence rate (lower is better). NRR/FCR are computed from the judge fields appropriate_refusal and false_confidence over all unanswerable turns, pooled across tiers T1–T5.

	Difficulty			Evidence position
Model	Easy	Med.	Hard	Early	Mid	Late
Cascaded	91.1	87.4	81.9	89.5	85.5	91.2
GPT-Realtime	89.7	86.7	68.1	84.2	87.0	88.6
Gemini-Live	87.6	89.1	82.6	83.4	88.0	90.4
MiniCPM-o	86.3	80.4	73.5	85.0	80.7	84.3
UV-32B	81.5	76.4	72.7	79.1	73.0	83.2

Appendix

Table 21: Grounding accuracy (%) by annotated question difficulty and by the position of the supporting evidence in the document. difficulty is a QA-metadata field; evidence position is bucketed by the character offset of the first supporting sentence in the one-third splits.

Model	Median	Mean	p95
Cascaded	14	17.1	43
GPT-Realtime	18	18.7	33
Gemini-Live	13	14.4	30
MiniCPM-o	14	28.4	75
UV-32B	14	17.5	43
SALMONN	15	24.5	94

Appendix

Table 22: Response length in whitespace tokens (transcripts;

\langle

think

\rangle

blocks stripped and /no_think used for UV-32B). With reasoning suppressed, UV-32B is now as concise as the other systems (median 14); Gemini-Live is the most concise (median 13); Moshi is the most verbose (median 29) despite not grounding, and MiniCPM-o/SALMONN have the longest tails.

Model	Tier	p50 (ms)	Mean (ms)	p95 (ms)
Moshi	T1	229	241	768
Moshi	T2	229	254	781
Moshi	T3	229	267	895
Moshi	T4	229	236	612
Moshi	T5	229	255	712
GPT-Realtime	T1	1210	1338	1851

Appendix

Table 23: FTED statistics by model and context tier. Per-tier p50/mean/p95 recomputed from the per-instance fted_ms fields (Exp A). Latency grows with context for MiniCPM-o, SALMONN, and UV-32B, while Moshi and GPT-Realtime are essentially context-invariant; SALMONN’s high T4 p95 is driven by large-context stalls, and the cascaded mean/p95 are inflated by a long tail from occasional LLM/API stalls. Gemini-Live has a high but nearly context-invariant latency floor (p50 6.4–7.0 s). UV-32B is not real-time capable.

Voice Agent Evaluation

Voice agent evaluation measures how reliably spoken AI systems understand, respond, take actions, and recover in real interactions.

See all 45 papers

$τ$ -Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
Mar 14, 2026Soham Ray, Keshav Dhandhania, Victor Barres +1Voice Agent Evaluation AI Agent Benchmarks
Sierra.ai, San Francisco, USA · Princeton Language and Intelligence, Princeton University
Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency
Apr 6, 2026Guan-Ting Lin, Chen Chen, Zhehuai Chen +1Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
National Taiwan University · NVIDIA
DuplexWorld: Can voice agents help you get through the day?
Aug 11, 2026Aryan Vijay Bhosale, Harshit Rajgarhia, Akhil Pothanapalli +3Voice Agent Evaluation Spoken Dialogue Systems
Centific Global Solutions Inc. · University of Maryland · Work done during an internship at Centific.

Mar 14, 2026J/K move · Enter open · S save

$τ$ -Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan

Sierra.ai, San Francisco, USA · Princeton Language and Intelligence, Princeton University

Voice Agent Evaluation AI Agent Benchmarks Full-Duplex Spoken Dialogue Systems Multi-Turn Dialogue Evaluation

Open paper AskPDF arXiv

Full-duplex voice agents--systems that listen and speak simultaneously--are rapidly moving from research to production. However, existing evaluations address conversational dynamics and task completion in isolation. We introduce

τ

-voice, a benchmark for evaluating voice agents on grounded tasks with real-world complexity: agents must navigate complex multi-turn conversations, adhere to domain policies, and interact with the environment. The framework extends

τ^2

-bench into a novel voice agent benchmark combining verifiable completion of complex grounded tasks, full-duplex interaction, and realistic audio--enabling direct comparison between voice and text performance. A controllable and realistic voice user simulator provides diverse accents, realistic audio environments, and rich turn-taking dynamics; by decoupling simulation from wall-clock time, the user simulator can use the most capable LLM without real-time constraints. We evaluate task completion (pass@1) and voice interaction quality across 278 tasks: while GPT-5 (reasoning) achieves 85%, voice agents reach only 31--51% under clean conditions and 26--38% under realistic conditions with noise and diverse accents--retaining only 30--45% of text capability; qualitative analysis confirms 79--90% of failures stem from agent behavior, suggesting that observed failures primarily reflect agent behavior under our evaluation setup.

τ

-voice provides a reproducible testbed for measuring progress toward voice agents that are natural, conversational, and reliable.

Abstract

Voice Agent Evaluation

Explore similar work

τττ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency

DuplexWorld: Can voice agents help you get through the day?

$τ$ -Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains