Voice Agent Evaluation

Follow this topic Ask about this topic

Voice agent evaluation studies whether spoken AI systems can understand people and complete tasks reliably, not just produce fluent replies. Benchmarks use real or simulated speech, tool-connected tasks, and stateful environments to test task completion, turn-taking, interruption recovery, grounding, and performance across accents, languages, and noise conditions. Common approaches combine controlled audio perturbations and conversation simulations with backend checks, trace analysis, human ratings, and automated judges. A central challenge is that strong text or component scores do not guarantee reliable end-to-end behavior, while automated judges and aggregate metrics may miss failures tied to context, timing, or speaker cues.

Latest papers

45 papers11 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents
Jul 8, 2026A. Sayyad, J. Emmons, S. Jones +2Audio-Language Model Evaluation Voice Agent Evaluation
Salesforce Applied AI Research, eVerse team
Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems
Jun 30, 2026Ashish Hallur, Thomas Thebaud, Georgi Tinchev +2Voice Agent Evaluation Speech Generation Evaluation
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · Amazon Inc., Seattle, WA, USA
Real-Time Voice AI Hears but Does Not Listen
Jun 24, 2026Martijn Bartelds, Federico Bianchi, James ZouAudio-Language Model Evaluation Voice Agent Evaluation
Together AI · Stanford University
SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
Jun 24, 2026Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu +2Audio-Language Model Evaluation Voice Agent Evaluation
New York University · NVIDIA · Carnegie Mellon University +1
From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
May 14, 2026Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo +3Audio-Language Model Evaluation Voice Agent Evaluation
Dialpad Inc.
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
May 13, 2026Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agent Evaluation Spoken Dialogue Systems
ServiceNow AI Research
WASIL: In-the-Wild Arabic Spoken Interactions with LLMs
May 9, 2026Zien Sheikh Ali, Hamdy Mubarak, Soon-Gyo Jung +3Voice Agent Evaluation Arabic NLP
Qatar Computing Research Institute, Qatar
PersonaKit (PK): A Plug-and-Play Platform for User Testing Diverse Roles in Full-Duplex Dialogue
May 7, 2026Hyunbae Jeon, Jinho D. ChoiVoice Agent Evaluation Spoken Dialogue Systems
Department of Computer Science Emory University Atlanta, GA, USA
Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use
Apr 17, 2026Ramit Pahwa, Apoorva Beedu, Parivesh Priye +4Audio-Language Model Evaluation Voice Agent Evaluation
Rivian and Volkswagen Group Technologies
EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions
Apr 8, 2026Smit Nautambhai Modi, Gandharv Mahajan, Marc Wetter +1State Tracking Voice Agent Evaluation
Applied Machine Learning Research
Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency
Apr 6, 2026Guan-Ting Lin, Chen Chen, Zhehuai Chen +1Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
National Taiwan University · NVIDIA
Evaluating a Multi-Agent Voice-Enabled Smart Speaker for Care Homes: A Safety-Focused Framework
Mar 24, 2026Zeinab Dehghani, Rameez Raja Kureshi, Koorosh Aslansefat +6Voice Agent Evaluation Conversational Agents
University of Hull, UK · University of Southampton, UK · Connexin, Hull, UK +1
$τ$ -Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
Mar 14, 2026Soham Ray, Keshav Dhandhania, Victor Barres +1Voice Agent Evaluation AI Agent Benchmarks
Sierra.ai, San Francisco, USA · Princeton Language and Intelligence, Princeton University
Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction
Feb 27, 2026Xiang Li, Jiabao Gao, Sipei Lin +5Voice Agent Evaluation Spoken Dialogue Systems
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen +2
VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents
Oct 13, 2025Jiliang Hu, Wenfu Wang, Zuchao Li +6Audio-Language Model Evaluation Voice Agent Evaluation
School of Artificial Intelligence, Wuhan University, Wuhan, China · Tencent AI Lab, Beijing, China

Jul 8, 2026J/K move · Enter open · S save

A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

A. Sayyad, J. Emmons, S. Jones, T. Lin+1

Salesforce Applied AI Research, eVerse team

Audio-Language Model Evaluation Voice Agent Evaluation Inter-Rater Reliability LLM-as-a-Judge

Open paper AskPDF arXiv

We report the empirical reliability of Gemini models as audio judges that score full-duplex agent conversations directly from the raw stereo waveform, tested across three models in the Gemini family: 2.5 Flash, 3.5 Flash, and 3.1 Pro. Our primary evidence base uses Gemini 2.5 Flash as the ground-truth model, validated against three calibrated human raters on 209 stereo sessions, scored on 8 production dimensions: 152 full-duplex conversations across 13 accent-and-condition strata, together with 57 adversarial defect-injected clips. The evidence for Gemini 2.5 Flash is consistent across three tests. (i) On 5 of 8 dimensions the LALM-human Spearman rho departs from the pairwise human-human rho by at most 0.07, and on 7 of 8 dimensions the two quantities 95 percent bootstrap confidence intervals overlap. (ii) The LALM agrees with the three-rater human mean within 1 point on 60 to 92 percent of sessions on 6 of 8 dimensions. (iii) On 45 of 48 (defect, dimension) cells the LALM is as sensitive as humans or better under Newcombe-Wilson 95 percent confidence intervals, though most of these are underpowered nulls rather than demonstrated parity. Rank-ordering ability transfers across the Gemini family: 3.5 Flash improves simple agreement to 8 of 8 dimensions, while 3.1 Pro rates several dimensions markedly lower than humans despite comparable rank correlation. A model swap should be re-validated on calibration specifically, not assumed from rank-correlation alone. We identify four areas where deployment requires care, and we estimate that human rating alone for our current evaluation cadence costs roughly two orders of magnitude more than the equivalent LALM workload. The data presented here provides a defensible empirical basis for deploying the LALM as a substitute or fourth rater on the dimensions where the evidence supports it.