Overview

Voice agent evaluation studies whether spoken AI systems can understand people and complete tasks reliably, not just produce fluent replies. Benchmarks use real or simulated speech, tool-connected tasks, and stateful environments to test task completion, turn-taking, interruption recovery, grounding, and performance across accents, languages, and noise conditions. Common approaches combine controlled audio perturbations and conversation simulations with backend checks, trace analysis, human ratings, and automated judges. A central challenge is that strong text or component scores do not guarantee reliable end-to-end behavior, while automated judges and aggregate metrics may miss failures tied to context, timing, or speaker cues.

Latest papers

CardsList

51 papers6 papers in the last four weeks, down 65% on the four weeks before. 0.0% of all new papers.

  1. Does the Voice Matter? Evaluating Frustration Detection on Deployed Clinical Voice-Agent Calls

    Nov 16, 2026Annika Michael, Zachary Ellis, Yuxiang Zhou +4Voice Agent EvaluationClinical Speech Processing

  2. Latency, Speech Recognition and Turn-Taking in Voice Agents for Spanish Phone Negotiation: An Externally Measured Comparison of Pipecat, LiveKit and GPT-Live

    Nov 14, 2026Jovan Popoca Moran, Eduardo de Avila-Armenta, Raúl Alejandro Aquino-Ibarra +2Voice Agent EvaluationSpoken Dialogue Systems

  3. Human-Likeness and Conversational Preference Diverge for Synthesized Agent Speech in Customer-Service Calls

    Nov 11, 2026Jeffrey M. Girard, Antony D'Avirro, Jason Z. Zheng +2Voice Agent EvaluationTTS Evaluation

  4. "It Would Make a Geordie Feel at Home": Regional Accent Perceptions and Speech Accommodation in Voice AI

    Oct 27, 2026Dana Serditova, Kevin Tang, Jochen SteffensVoice Agent EvaluationDemographic Bias in ASR

  5. DuplexLedger: Benchmarking Transactional Memory in Full-Duplex Speech Agents

    Oct 20, 2026Yuxuan Li, Xiangchen Guan, Kunrong Li +7Memory Management for LLM AgentsMemory Agent Benchmarks

  6. When Does an ASR Error Matter? Task-Role Diagnostics for Tool-Using Voice Agents

    Oct 13, 2026Xiaolong Wang, Shijie Su, Naqi XiongVoice Agent EvaluationASR Evaluation

  7. Talk2Agent: Benchmarking Voice Interfaces for Text Agents

    Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR EvaluationVoice Agent Evaluation

  8. DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agents

    Sep 29, 2026Puneet Mathur, Nedim Lipka, Zeyu Jin +1Voice Agent EvaluationEvidence-Grounded Generation

  9. Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents

    Sep 26, 2026Yanjie Zhang, Nanchen Hu, Yushi SunAudio-Language ModelsVoice Agent Evaluation

  10. Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

    Sep 25, 2026Chengqian Ma, Wenhao Feng, Weixuan Jin +7Voice Agent EvaluationFull-Duplex Spoken Dialogue Systems

  11. Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes

    Sep 25, 2026Shivam Negi, Arpit Rawat, Rashi JainVoice Agent EvaluationAgent Evaluation

  12. ττ-Multilingual: Benchmarking Voice Agents Across Languages

    Sep 21, 2026Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3Voice Agent EvaluationFull-Duplex Spoken Dialogue Systems

  13. Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants

    Sep 20, 2026Vaishnav Negi, Lev Sorokin, Soroosh Tayebi Arasteh +1Multi-Turn Dialogue EvaluationVoice Agent Evaluation

  14. When Patients Cut In: Extending Clinical Conversational AI Safety to Interruptions

    Aug 29, 2026Zachary Ellis, Spencer Hazel, Adam Brandt +3Voice Agent EvaluationClinical Language Model Evaluation

  15. Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight

    Aug 25, 2026Anupam Purwar, Shashank Singh, Kritika SrivastavaVoice Agent EvaluationLLM-as-a-Judge