$τ$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

Soham Ray; Keshav Dhandhania; Victor Barres; Karthik Narasimhan

cs.SDMar 14, 2026

$τ$ -Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

Authors: Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan

Organizations: Sierra.ai, San Francisco, USA · Princeton Language and Intelligence, Princeton University

Abstract

Full-duplex voice agents--systems that listen and speak simultaneously--are rapidly moving from research to production. However, existing evaluations address conversational dynamics and task completion in isolation. We introduce $τ$ -voice, a benchmark for evaluating voice agents on grounded tasks with real-world complexity: agents must navigate complex multi-turn conversations, adhere to domain policies, and interact with the environment. The framework extends $τ^2$ -bench into a novel voice agent benchmark combining verifiable completion of complex grounded tasks, full-duplex interaction, and realistic audio--enabling direct comparison between voice and text performance. A controllable and realistic voice user simulator provides diverse accents, realistic audio environments, and rich turn-taking dynamics; by decoupling simulation from wall-clock time, the user simulator can use the most capable LLM without real-time constraints. We evaluate task completion (pass@1) and voice interaction quality across 278 tasks: while GPT-5 (reasoning) achieves 85%, voice agents reach only 31--51% under clean conditions and 26--38% under realistic conditions with noise and diverse accents--retaining only 30--45% of text capability; qualitative analysis confirms 79--90% of failures stem from agent behavior, suggesting that observed failures primarily reflect agent behavior under our evaluation setup. $τ$ -voice provides a reproducible testbed for measuring progress toward voice agents that are natural, conversational, and reliable.

Figures & tables

Figure 1: Task completion (pass@1) averaged across all domains. GPT-5 (reasoning) achieves 85%. Voice agents show two levels of degradation: under Clean conditions (clean audio, no interruptions), performance drops to 31–51% (

-

34 to

-

54pp); under Realistic conditions (realistic audio, interruptions), it falls further to 26–38% (retaining only 30–45% of text capability).

	Task Completion	Full- Duplex	Realistic Audio Env.
Task-Oriented (Text)
$\tau$ -bench ( Yao et al., 2024 )	✓
$\tau^{2}$ -bench ( Barres et al., 2025 )	✓
Conversational Dynamics
Full-Duplex-Bench ( Lin et al., 2025b )		✓
Full-Duplex-Bench-V2 ( Lin et al., 2025a )	$\sim$	✓

Table 1: Comparison of evaluation dimensions across benchmarks. Prior work advances individual dimensions;

\tau

-Voice combines all three.

Figure 2:

\tau

-Voice extends

\tau^{2}

-bench (gray) with voice-specific components (green): a voice user simulator with configurable personas, audio environment, and turn-taking policy; a full-duplex audio streaming channel discretized into simulation ticks; and a provider adapter for adding new voice APIs. Task infrastructure (instructions, tools, databases, domain policies) is inherited.

Figure 3: Voice user simulator pipeline. Each tick, the simulator generates text, synthesizes speech with a persona, mixes in environmental audio, and applies telephony degradation to produce realistic caller audio.

Time	User	Agent	Event
5–8s	Hi, I have two prob-
8s	-lems. First, I ordered	Hello!	agent int.
…
60–67s		…Which would you like
67–68s		to do first?
68–69s	Jigsaw first.		user int., yield

Table 2: Key moments from the Task 41 trajectory (Figure 4 ). At 8s, the agent interrupts; at 68s the user interrupts and the agent yields but fails to respond for 5 seconds; at 82s the agent incorrectly responds to non-agent-directed speech [in brackets]; at 113s the user interrupts but the agent does not yield; at 121s the agent correctly continues through a backchannel.

Figure 4: Speech activity timeline from a Retail domain simulation with Gemini Live. A customer calls about exchanging a jigsaw puzzle and correcting their address. The legend distinguishes observations (User Int. = user interruption, Non-Agent Dir. = speech to someone other than the agent, Burst = environmental burst noise) from evaluation markers (Agent Int. = agent interruption, BC Issue = incorrect backchannel handling, Voc. Tic Error / Non-Agent Dir. Error = agent incorrectly yielding or responding to these stimuli).

Provider	Model	Release
OpenAI	gpt-realtime-1.5	Feb 2026
Google	gemini-live-2.5-flash-native-audio	Dec 2025
xAI	grok-voice-agent	Dec 2025

Table 3: Audio-native models evaluated.

Category	Setting	Clean	Realistic
Accents	Personas	American	Diverse accents
Audio/Channel	Background noise	None	Indoor/outdoor
	Burst noise	None	$\sim$ 1/min
	Frame drops	None	$\sim$ 2.0% (G-E model)
	Telephony	G.711 $\mu$ -law 8kHz	G.711 $\mu$ -law 8kHz
	Muffling	None	Dynamic

Table 4: Speech complexity conditions: Clean vs Realistic.

Category	Setting	Cln	+N	+A	+T	Real
Accents	Personas			✓		✓
Audio/Channel	Background noise		✓			✓
	Burst noise		✓			✓
	Frame drops		✓			✓
	Telephony	✓	✓	✓	✓	✓
	Muffling		✓			✓

Table 5: Speech complexity conditions by ablation (single factors). Columns: Cln=Clean, +N=Noise, +A=Accents, +T=Turn-taking, Real=Realistic (all effects).

			Voice
Domain	Provider	Text	Clean	Realistic
All	Google	85% (54%)	31% (-54, -63.4%)	26% (-59, -69.5%)
	OpenAI		49% (-36, -42.1%)	35% (-49, -58.4%)
	xAI		51% (-34, -40.1%)	38% (-46, -54.7%)
Retail	Google	81% (76%)	45% (-36, -44.8%)	30% (-51, -63.2%)
	OpenAI		71% (-10, -12.3%)	45% (-36, -44.8%)

Table 6: Text vs Voice comparison (pass@1). Text shows GPT-5 (reasoning) and GPT-4.1 (non-reasoning). Voice evaluated under Clean and Realistic conditions. Deltas show gap from GPT-5.

Condition	Google	OpenAI	xAI	All
Clean	45%	71%	48%	55%
+ Noise	40% (-4, -9.8%)	67% (-4, -6.2%)	46% (-2, -3.6%)	51% (-4, -6.4%)
+ Accents	44% (-1, -2.0%)	60% (-11, -16.0%)	30% (-18, -38.2%)	44% (-10, -18.7%)
+ Turn-taking	33% (-11, -25.5%)	57% (-14, -19.8%)	52% (+4, +7.3%)	47% (-7, -13.4%)
Realistic	30% (-15, -33.3%)	45% (-26, -37.0%)	39% (-10, -20.0%)	38% (-17, -31.0%)

Table 7: Ablation: impact of individual acoustic factors on pass@1 (Retail domain).

Provider	Latency $\downarrow$	Responsiveness $\uparrow$	Interrupt $\downarrow$	Selectivity $\uparrow$
Google	1.14s	69%	21%	54%
OpenAI	0.90s	100%	14%	6%
xAI	1.15s	83%	84%	57%

Table 8: Voice interaction quality (Realistic condition, aggregated across domains). Bold indicates best. Full breakdown in Appendix H.1 .

Source	Error Type	Voice- Fragile	Noise- Fragile
Agent	Logical	13	16
	Transcription	10	16
	Hallucination	6	6
	VAD/Unresponsive	1	4
	Timeout	4	1
	Total	34 (79%)	43 (90%)

Table 9: Error analysis: distribution of error types by source. Agent errors dominate in both cohorts (79% and 90%).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Default	Description
Wait-to-respond (other)	1.0s	Min silence from agent before user responds
Wait-to-respond (self)	5.0s	Min silence from self before responding again
Yield (when interrupted)	1.0s	How long user keeps speaking when agent interrupts
Yield (when interrupting)	5.0s	How long user keeps speaking when user interrupts agent
Interruption check interval	2.0s	Interval for LLM interruption checks
Backchannel check interval	2.0s	Interval for LLM backchannel checks

Appendix

Table 10: Turn-taking thresholds controlling conversation flow.

Case	Condition	Action
No overlap	Segments don’t touch	Chronological order
Partial overlap	Segments cross, neither contained	Order by start time
Containment	X fully inside Y	Split Y at X’s end, insert X there

Appendix

Table 11: Linearization rules for converting overlapping speech to sequential messages.

Environment	Background Noise	Burst Noise
Indoor	People Talking, TV News	Ringing Phone, Dog Bark
Outdoor	Busy Street, Street & Metro	Car Horn, Engine Idling, Siren

Appendix

Table 12: Environment presets define which audio files are used for background and burst noise generation.

Effect	Scheduling	Rate (Realistic Preset)
Burst noise	Poisson process	1.0 events/min
Out-of-turn speech (phrases, vocal tics)	Poisson process	0.7 events/min
Frame drops	Gilbert-Elliott model	2% avg loss rate, 100ms burst
Dynamic muffling	Per-utterance probability	20% of utterances

Appendix

Table 13: Effect scheduling parameters for the Realistic complexity preset.

Error Type	Agent State	Trigger	Incorrect Behavior	Window
Turn-Taking
No-Response	Silent	User turn ends	No response	—
No-Yield	Speaking	User interrupts	Keep speaking	2.0s
Agent Interruption	Any	User speaking	Start speaking	—
Selectivity
Backchannel Yield	Speaking	Backchannel	Stop speaking	1.0s

Appendix

Table 14: Agent error definitions. Turn-taking errors affect

R_{R}

R_{Y}

, and

I_{A}

. Selectivity errors affect

S_{BC}

S_{VT}

, and

S_{ND}

		Latency $\downarrow$			Responsiveness $\uparrow$			Interrupt $\downarrow$		Selectivity $\uparrow$
		$L_{R}$		$L_{Y}$	$R_{R}$		$R_{Y}$	$I_{A}$		$S_{BC}$	$S_{VT}$	$S_{ND}$
Domain	Provider	C	R	$L_{Y}$	C	R	$R_{Y}$	C	R	$S_{BC}$	$S_{VT}$	$S_{ND}$
All	Google	1.40s	1.43s	0.86s	96%	81%	56%	7%	21%	85%	34%	45%
	OpenAI	1.69s	1.39s	0.42s	99%	100%	100%	1%	14%	2%	5%	10%
	xAI	1.05s	1.15s	1.15s	92%	91%	75%	44%	84%	93%	58%	21%
Retail	Google	1.50s	1.45s	0.86s	96%	79%	61%	6%	17%	91%	28%	45%

Appendix

Table 15: Voice interaction quality metrics—full breakdown (Realistic condition). Bold indicates best per domain.

\uparrow

= higher is better,

\downarrow

= lower is better.

Comparison	Provider	Rate A	Rate B	$\Delta$ (pp)	$p$ (adj)
Text (NR) $\to$ Clean	Google	76.3%	42.1%	$-$ 34.2	$<$ 0.001
	OpenAI	76.3%	67.5%	$-$ 8.8	0.032
	xAI	76.3%	50.0%	$-$ 26.3	$<$ 0.001
Text (R) $\to$ Clean	Google	81.6%	42.1%	$-$ 39.5	$<$ 0.001
	OpenAI	81.6%	67.5%	$-$ 14.0	$<$ 0.001
	xAI	81.6%	50.0%	$-$ 31.6	$<$ 0.001

Appendix

Table 17: Pairwise statistical significance for Retail domain (2 runs, n=114 tasks each). Text (NR) = GPT-4.1; Text (R) = GPT-5. All

p

-values are Holm-Bonferroni corrected paired permutation tests.

Property	Value
Domain	Retail
Agent	Gemini Live 2.5 Flash Native Audio
User Persona	wei_lin (Chinese woman from Sichuan)
Complexity	Realistic (all audio effects enabled)
Background Noise	Busy street (outdoor environment)
Duration	179 seconds (3 minutes)

Appendix

Table 18: Task 41 configuration.

Step	Tool Call	Key Arguments
1	find_user_id_by_name_zip	first_name: Mei, last_name: Patel, zip: 76165
2	get_user_details	user_id: mei_patel_7272
3	get_order_details	order_id: #W9583042
4	get_order_details	order_id: #W4082615
5	modify_pending_order_address	order_id: #W9583042, address: 445 Maple Drive…
6	modify_pending_order_address	order_id: #W4082615, address: 445 Maple Drive…

Appendix

Table 19: Example tool call sequence for Task 41. Read calls (steps 1–4, 8–9) gather information; write calls (steps 5–7, 10) modify the database. Only the final database state is checked for reward.

Required Action	Completed?	Notes
modify_pending_order_address (#W9583042)	No	Never called
modify_pending_order_address (#W4082615)	No	Agent claimed done but didn’t call
modify_user_address	Yes	Successfully updated profile
modify_pending_order_items	No	Exchange never completed

Appendix

Table 20: Write action completion status for Task 41.

Ticks (Time)	Agent	User	Tool Calls	Events
0–25 0:00–0:05	Hi! How can I help you today?	—	—	wait: Silence (x26)
26–111 0:05–0:22	Hello! I can help	Hi, I have two problems. First, I ordered a 1000-piece intermediate jigsaw, but I think it’s too hard for my kid—can I switch it to the easiest one with the fewest pieces? Second, I might have typed my address wrong. I want to check and maybe fix the address.	—	generate_message keep_talking (x70) [frame-drop tick 30] [agent-int 8.0s, 18.8s]
112 0:22	—	—	find_user_id_by_email Error: User not found	keep_talking
113–119 0:22–0:23	I’d be happy to help wi	—	—	keep_talking (x7) [frame-drop tick 117] [agent-int 23.0s]
120 0:24	th	—	—	stop_talking
121–168 0:24–0:33	both of those issues. First, I need to authenticate you. I tried using [email protected], but it wasn’t found. Can you provide me with your email address spelled out, or your f	[sneezes] (vocal tic)	—	wait: Listening (x48) [frame-drop tick 149] [vocal-tic tick 165]

Appendix

Table 24

Event Type	Count	Notes
User utterances	17
Agent utterances	15
User interruptions	8	Callback decided to interrupt
Agent interruptions	2	“Hello!” and “I can help” during user opening
Backchannels	1	“mm-hmm” at tick 609
Frame drops	12	150ms each (ticks 30, 117, 149, 257, 325, 339, 460, 656, 678, 739, 771, 809)

Appendix

Table 22: Event summary for Task 41 conversation.

Property	Value
Total duration	179.6 seconds (898 ticks at 200ms each)
Simulation ID	39ee01bf-37ff-4330-90c2-d15f9a940de0
Voice persona	wei_lin
Environment	outdoor (busy_street_iphone_mic.wav)
Burst noise files	car_horn.wav, engine_idling.wav, siren.wav
Telephony	G.711 $\mu$ -law 8kHz

Appendix

Table 23: Technical parameters for the Task 41 simulation.

Voice Agent Evaluation

Voice agent evaluation measures how reliably spoken AI systems understand, respond, take actions, and recover in real interactions.

See all 44 papers

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages
Sep 21, 2026Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
Sierra · Princeton University · Mercor
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
May 13, 2026Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agent Evaluation Spoken Dialogue Systems
ServiceNow AI Research
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge

Sep 21, 2026J/K move · Enter open · S save

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages

Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela, Karthik Narasimhan+2

Sierra · Princeton University · Mercor

Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems Speech Generation Evaluation

Open paper AskPDF arXiv

English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce

τ

-Multilingual, extending

τ

-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and evaluation of generated language and spoken output. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese, and Hindi remain within 3.2 task-completion points of English, but Korean and Mandarin fall by 14.7 and 8.4 points. The failure modes also vary: Korean systems miss more responses, Mandarin systems interrupt more often, and both struggle with tools and entities. Grok leads task completion but scores lowest on generation quality, motivating separate task, interaction, and generation reporting. We release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.

Abstract

Voice Agent Evaluation

Explore similar work

τττ-Multilingual: Benchmarking Voice Agents Across Languages

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages