$τ$-Multilingual: Benchmarking Voice Agents Across Languages

Soham Ray; Edgard dos Santos Paiva; Ruben Valenzuela; Karthik Narasimhan; Keshav Dhandhania; Victor Barres

cs.CLSep 21, 2026

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages

Authors: Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela, Karthik Narasimhan, Keshav Dhandhania, Victor Barres

Organizations: Sierra · Princeton University · Mercor

Abstract

English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce $τ$ -Multilingual, extending $τ$ -Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and evaluation of generated language and spoken output. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese, and Hindi remain within 3.2 task-completion points of English, but Korean and Mandarin fall by 14.7 and 8.4 points. The failure modes also vary: Korean systems miss more responses, Mandarin systems interrupt more often, and both struggle with tools and entities. Grok leads task completion but scores lowest on generation quality, motivating separate task, interaction, and generation reporting. We release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.

Figures & tables

Figure 1: Fixed-system means with worst-to-best language ranges. Task completion and Interaction average six languages; Generation averages the five localized languages.

Language component	Hindi (India)
Personas	Rishika uses fast Hinglish; Imran uses patient, Urdu-inflected Hindi.
Speech rules	Spell with “R for Rajdhani”; say at and dot in email addresses.
Task arms and entity localization	Allison Reeves $\rightarrow$ Neha Gupta; phone: 99400 97181.
Language-quality factors	Customer address ( aap vs. tum/tu ) and agreement with aap .
Review and freeze	Pin the Rishika and Imran voices; freeze preset multilingual_v1_hindi .

Table 2: LanguageFactory components illustrated with the Hindi pack. Entity localization maps the same source identity to locale-specific backend values without changing the task.

Figure 2: Headline outcomes by system and language. Higher is better; color scales normalize within panel, outlines mark language-best systems, Overall averages the five configurations, and Generation excludes English.

Figure 3: Main-metric breakdown across languages by system; lower is better. Bold numbers mark row minima. Subscripts c/u/t denote call/utterance/turn; units are in parentheses.

		Entity		Entity + DB
Lang.	System	Localized	Source	Romanized	Native
Hindi	GPT xhigh	63.3	63.3	63.3	36.7
	Gemini high	60.0	66.7	60.0	60.0
Mandarin	GPT xhigh	36.7	20.0	36.7	53.3
	Gemini high	53.3	53.3	53.3	53.3

Table 3: Retail entity ablations: task completion (%) on 30 tasks.

Explore similar work

Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents
Sep 21, 2026Chenxu Xiong, Dongming Shen, Yuzhi Tang +3Audio-Language Model Evaluation Voice Agent Evaluation
Boson AI Santa Clara, CA 95054
IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian Languages
Sep 25, 2026Rajarshi Roy, Shobhit Banga, Jonathan Raiman +12Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
NVIDIA · VoiceArena

Sep 30, 2026J/K move · Enter open · S save

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang

Department of Electronic Engineering, Tsinghua University · University of Cambridge

ASR Evaluation Voice Agent Evaluation Computer-Use Agents LLM Agent Evaluation Automatic Speech Recognition

Open paper AskPDF arXiv

Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.

Abstract

Explore similar work

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents

IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian Languages