Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency

Guan-Ting Lin; Chen Chen; Zhehuai Chen; Hung-yi Lee

eess.ASApr 6, 2026

Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency

Authors: Guan-Ting Lin, Chen Chen, Zhehuai Chen, Hung-yi Lee

Organizations: National Taiwan University · NVIDIA

Abstract

We introduce Full-Duplex-Bench-v3 (FDB-v3), a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use. Unlike prior work, our dataset consists entirely of real human audio annotated for five disfluency categories, paired with scenarios requiring chained API calls across four task domains. We evaluate six model configurations -- GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a traditional Cascaded pipeline (Whisper $\rightarrow$ GPT-4o $\rightarrow$ TTS) -- across accuracy, latency, and turn-taking dimensions. GPT-Realtime leads on Pass@1 (0.600) and interruption avoidance (13.5%); Gemini Live 3.1 achieves the fastest latency (4.25~~s) but the lowest turn-take rate (78.0%); and the Cascaded baseline, despite a perfect turn-take rate, incurs the highest latency (10.12~~s). Across all systems, self-correction handling and multi-step reasoning under hard scenarios remain the most consistent failure modes.

Figures & tables

Figure 1: The Full-Duplex-Bench-v3 framework for evaluating real-time voice agents. The diagram illustrates a full-duplex interaction where the agent continuously processes user speech containing natural dysfluencies, such as fillers and self-corrections. Full-Duplex-Bench-v3 comprehensively assesses the agent across three primary dimensions: (1) Tool-use Performance (accuracy of API selection and execution), (2) Turn-Taking Dynamic (handling interruptions and conversational flow), and (3) Latency Breakdown (timing from listening to tool execution and speech generation).

Domain	Mock API Functions
Travel & Identity	search_flights(destination, date)
	book_ticket(passenger_name, flight_id)
	update_travel_profile(document_type, document_number)
Finance & Billing	query_card_benefits(card_last_4, category)
	calculate_currency_exchange(amount, from_currency, to_currency)
	modify_autopay_source(new_account_id)

Table 1: Overview of Domains and Mock API Functions

	Tool Use				Turn-Taking Dynamics
	Tool Sel $\uparrow$	Arg Acc $\uparrow$	Resp Qual $\uparrow$	Pass@1 $\uparrow$	Take-turn $\uparrow$	Latency $\downarrow$	Interrupt $\downarrow$	Filler $\downarrow$
GPT-Realtime	0.876	0.680	0.792	0.600	96.0%	6.89s	13.5%	16.9%
Gemini Live 2.5	0.786	0.593	0.554	0.490	92.0%	7.26s	14.1%	8.9%
Gemini Live 3.1	0.817	0.588	0.718	0.540	78.0%	4.25s	19.2%	31.7%
Grok	0.797	0.542	0.617	0.430	94.0%	6.65s	25.5%	44.3%
Ultravox	0.794	0.513	0.510	0.410	96.0%	8.40s	47.9%	88.0%

Table 2: Overall Performance and Turn-Taking Metrics. Best value in each column is in bold . Lower latency and lower filler rate are better. Filler denotes the percentage of responses containing filler sentences before the key information.

Model	Filler	Pause	Hesitation	False Start	Self-Corr
GPT-Realtime	0.621	0.556	0.700	0.667	0.588
Gemini Live 2.5	0.621	0.444	0.600	0.417	0.471
Gemini Live 3.1	0.586	0.500	0.600	0.583	0.353
Grok	0.483	0.333	0.500	0.583	0.294
Ultravox	0.414	0.333	0.500	0.250	0.353
Cascaded	0.448	0.444	0.600	0.500	0.176

Table 3: Robustness to disfluency features (Pass@1 per category).

Model	Easy	Medium	Hard
GPT-Realtime	0.750	0.588	0.433
Gemini Live 2.5	0.667	0.500	0.267
Gemini Live 3.1	0.694	0.588	0.300
Grok	0.583	0.471	0.200
Ultravox	0.556	0.382	0.267
Cascaded	0.639	0.441	0.233

Table 4: Pass@1 Performance by scenario difficulty.

Model	Ecommerce	Finance	Housing	Travel
GPT-Realtime	0.552	0.960	0.308	0.600
Gemini Live 2.5	0.483	0.760	0.231	0.500
Gemini Live 3.1	0.448	0.920	0.269	0.550
Grok	0.414	0.760	0.115	0.450
Ultravox	0.345	0.680	0.192	0.450
Cascaded	0.414	0.800	0.192	0.400

Table 5: Pass@1 Performance breakdown by task domain. Best value in each column is shown in bold .

Model	First Word	Tool Call	Task Compl.
GPT-Realtime	6.36	3.89	6.89
Gemini Live 2.5	7.03	4.61	7.26
Gemini Live 3.1	3.95	2.21	4.25
Grok	5.97	0.63	6.65
Ultravox	3.88	6.01	8.40
Cascaded	8.78	3.15	10.12

Table 6: Mean latency breakdown in seconds. Lower is better.

Model	Tool	Arg	Resp	1 st Resp	Tool Call	Compl.
	Sel	Acc	Qual	(s)	(s)	(s)
GPT-RT	1.00	1.00	1.00	8.48	6.40	9.20
Gem 2.5	1.00	1.00	1.00	6.16	3.60	6.16
Gem 3.1	1.00	1.00	1.00	3.36	2.43	3.92
Grok	1.00	1.00	1.00	5.20	2.39	5.92
Ultravox	1.00	0.67	0.00	2.64	6.31	8.16

Table 7: Case study: finance_18 (hard, 3 tools, no disfluency). All four end-to-end models achieve perfect accuracy; Gemini 3.1 completes in 3.92 s vs. GPT-Realtime’s 9.20 s (2.3

\times

faster).

Model	Tool	Arg	Resp	1 st Resp	Tool Call	Compl.
	Sel	Acc	Qual	(s)	(s)	(s)
GPT-RT	1.00	1.00	1.00	4.40	2.55	4.40
Gem 2.5	1.00	1.00	0.00	5.28	4.23	5.28
Gem 3.1	1.00	0.00	0.00	2.56	$-$ 2.27	2.56
Grok	0.50	0.00	1.00	—	$-$ 2.47	—
Ultravox	1.00	1.00	0.00	$-$ 1.36	$-$ 0.18	3.04

Table 8: Case study: travel_19 (hard, self-correction, state rollback). Only GPT-Realtime correctly applies both corrections (Rome

\rightarrow

Milan, June 1

\rightarrow

3). Gemini 3.1’s pre-emptive tool call (

-

2.27 s) locks in the stale destination.

Voice Agent Evaluation

Voice agent evaluation measures how reliably spoken AI systems understand, respond, take actions, and recover in real interactions.

See all 44 papers

Full-duplex Spoken Language Modeling with Discrete Diffusion
Oct 13, 2026Chen Chen, Pin-Jui Ku, Ankita Pasad +3Speech Language Models Full-Duplex Spoken Dialogue Systems
NVIDIA
Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue
Sep 25, 2026Chengqian Ma, Wenhao Feng, Weixuan Jin +7Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
Peking University · Renmin University of China · Tsinghua University +3
$τ$ -Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
Mar 14, 2026Soham Ray, Keshav Dhandhania, Victor Barres +1Voice Agent Evaluation AI Agent Benchmarks
Sierra.ai, San Francisco, USA · Princeton Language and Intelligence, Princeton University

Oct 13, 2026J/K move · Enter open · S save

Full-duplex Spoken Language Modeling with Discrete Diffusion

Chen Chen, Pin-Jui Ku, Ankita Pasad, Ante Jukić+2

NVIDIA

Speech Language Models Full-Duplex Spoken Dialogue Systems Diffusion Language Models Efficient Language Model Inference Diffusion Language Model Decoding

Open paper AskPDF arXiv

We introduce Diffuplex, a full-duplex spoken language model based on discrete diffusion that achieves an unprecedented balance between linguistic intelligence and response efficiency. With thinking mode, Diffuplex delivers 13.0 tokens per forward pass, reaching an average of 1872 tokens per second and substantially surpassing the frontier autoregressive (AR) models in decoding speed. Without thinking, Diffuplex matches the first-audio latency of existing full-duplex models while achieving markedly stronger linguistic intelligence. This capability is enabled by separating when to say from what to say at the training level, rather than compressing both into a single autoregressive timeline. During inference, Diffuplex continuously tracks the user speech through an AR interaction policy, while independently generating and refining response content through parallel diffusion. Consequently, semantic computation is no longer bounded by the perception audio frame rate, allowing intelligence and responsiveness to improve simultaneously. Taken together, these results establish a new paradigm for spoken agents that can sustain real-time dialogue while devoting parallel computation to intelligent response generation. Training code and checkpoints will be avaliable soon.

Abstract

Voice Agent Evaluation

Explore similar work

Full-duplex Spoken Language Modeling with Discrete Diffusion

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

τττ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

$τ$ -Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains