Voice Agent Evaluation

Follow this topic Ask about this topic

Voice agent evaluation studies whether spoken AI systems can understand people and complete tasks reliably, not just produce fluent replies. Benchmarks use real or simulated speech, tool-connected tasks, and stateful environments to test task completion, turn-taking, interruption recovery, grounding, and performance across accents, languages, and noise conditions. Common approaches combine controlled audio perturbations and conversation simulations with backend checks, trace analysis, human ratings, and automated judges. A central challenge is that strong text or component scores do not guarantee reliable end-to-end behavior, while automated judges and aggregate metrics may miss failures tied to context, timing, or speaker cues.

Latest papers

45 papers11 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

DuplexLedger: Benchmarking Transactional Memory in Full-Duplex Speech Agents
Oct 20, 2026Yuxuan Li, Xiangchen Guan, Kunrong Li +7Memory Management for LLM Agents Memory Agent Benchmarks
Nanyang Technological University · Peking University · Beijing University of Technology +3
VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model
Oct 15, 2026Yang XiaoTurn-Taking Prediction Speech Language Models
The University of Melbourne
SEA-HELM Audio: Evaluating Multilingual ASR-LLM Pipeline Performance Across SEA-HELM Subsets
Oct 14, 2026Aulia Adila, Yixian Tan, Jonathan Heng +3Voice Agent Evaluation Multilingual Language Model Evaluation
AI Singapore, Singapore
When Does an ASR Error Matter? Task-Role Diagnostics for Tool-Using Voice Agents
Oct 13, 2026Xiaolong Wang, Shijie Su, Naqi XiongVoice Agent Evaluation ASR Evaluation
Leibniz University Hannover, Hannover, Germany
Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators
Oct 7, 2026Riqiang Wang, Elena Khasanova, Harsh Saini +4User Simulation Voice Agent Evaluation
Dialpad Inc.
Mind the Accent Gap: British Accent Robustness in Speech-Driven Financial Voice Assistants
Oct 5, 2026Aadam Haq, Oggi Rudovic, Malcolm Chadwick +5ASR Evaluation Voice Agent Evaluation
NatWest, UK
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation
Sep 29, 2026Joshua Meyer, Sahar Shayegan, Ritiz Tambi +5Voice Agent Evaluation LLM Agent Evaluation
Veris AI
DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agents
Sep 29, 2026Puneet Mathur, Nedim Lipka, Zeyu Jin +1Voice Agent Evaluation Evidence-Grounded Generation
Adobe Research, San Jose, USA · University of Maryland College Park, USA
When Does a Spoken Agent Have Enough Evidence to Act? The PACT-SLM Contract Test
Sep 28, 2026Mengzhe GengVoice Agent Evaluation Speech Language Models
National Research Council Canada
APEX-Voice: Can Voice Agents Complete Professional Workflows Through Full-Duplex Interaction
Sep 28, 2026Puneet Mathur, Dinesh ManochaVoice Agent Evaluation Tool-Using Agents
University of Maryland College Park, USA
Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents
Sep 26, 2026Yanjie Zhang, Nanchen Hu, Yushi SunAudio-Language Models Voice Agent Evaluation
HKUST, Hong Kong SAR, China · LIGHTSPEED, Shenzhen, China
IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian Languages
Sep 25, 2026Rajarshi Roy, Shobhit Banga, Jonathan Raiman +12Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
NVIDIA · VoiceArena
Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue
Sep 25, 2026Chengqian Ma, Wenhao Feng, Weixuan Jin +7Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
Peking University · Renmin University of China · Tsinghua University +3
Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes
Sep 25, 2026Shivam Negi, Arpit Rawat, Rashi JainVoice Agent Evaluation Agent Evaluation
Northeastern University, Boston, MA, USA · Walmart · Sinch Mailgun +1
Inquesto Score: A reliability Protocol For Voice Agents
Sep 24, 2026Massa Baali, Bhiksha RajVoice Agent Evaluation AI Agent Reliability
Carnegie Mellon University
$τ$ -Multilingual: Benchmarking Voice Agents Across Languages
Sep 21, 2026Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
Sierra · Princeton University · Mercor
MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents
Sep 21, 2026Chenxu Xiong, Dongming Shen, Yuzhi Tang +3Audio-Language Model Evaluation Voice Agent Evaluation
Boson AI Santa Clara, CA 95054
Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants
Sep 20, 2026Vaishnav Negi, Lev Sorokin, Soroosh Tayebi Arasteh +1Multi-Turn Dialogue Evaluation Voice Agent Evaluation
Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · BMW Group, Germany · Technical University of Munich, Munich, Germany +2
Voice-Light: A Full-Duplex Cascaded Voice Agent with Causal Turn-Taking and Speculative Generation
Sep 17, 2026Bertil BraunSpoken Dialogue Systems Full-Duplex Spoken Dialogue Systems
MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents
Sep 17, 2026Pritish Mishra, Ishaan Kumar, Akshat Mandoli +1Voice Agent Evaluation Multilingual Language Model Evaluation
Smallest AI
$τ$ -Elicitation: Benchmarking multi-turn entity extraction in voice agents
Sep 11, 2026Soham Ray, Victor BarresVoice Agent Evaluation Spoken Language Understanding
Sierra · Mercor
When Patients Cut In: Extending Clinical Conversational AI Safety to Interruptions
Aug 29, 2026Zachary Ellis, Spencer Hazel, Adam Brandt +3Voice Agent Evaluation Clinical Language Model Evaluation
Ufonia Limited · Newcastle University
Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight
Aug 25, 2026Anupam Purwar, Shashank Singh, Kritika SrivastavaVoice Agent Evaluation LLM-as-a-Judge
Sprinklr AI Gurugram, India · Sprinklr AI Bengaluru, India
Does Listening Matter? Backchanneling and Nodding in AI Clone
Aug 20, 2026Koji Inoue, Kazushi Kato, Tatsuya Kawahara +1Voice Agent Evaluation
Kyoto University Kyoto, Japan · Sony Computer Science Laboratories Tokyo, Japan
Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does
Aug 20, 2026Xinyi Liu, Hooshang Nayyeri, Dilek Hakkani-Tur +5Task-Oriented Dialogue Systems Voice Agent Evaluation
Amazon · University of Illinois Urbana-Champaign · University College London
DuplexWorld: Can voice agents help you get through the day?
Aug 11, 2026Aryan Vijay Bhosale, Harshit Rajgarhia, Akhil Pothanapalli +3Voice Agent Evaluation Spoken Dialogue Systems
Centific Global Solutions Inc. · University of Maryland · Work done during an internship at Centific.
FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India
Aug 6, 2026Aman Dalmia, Sanskriti Midha, Jigar DoshiVoice Agent Evaluation Healthcare
Artpark, Indian Institute of Science, Bengaluru, Karnataka, India
VAmoS Bench: Voice Agent Simulation Bench
Jul 29, 2026Joshua Meyer, Sahar Shayegan, Ritiz Tambi +5AI Agent Benchmarks Customer Support Automation
Veris AI
RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems
Jul 16, 2026David Ayllon, Alice Baird, Jeffrey Brooks +11ASR Evaluation Voice Agent Evaluation
Hume AI Research

Oct 20, 2026J/K move · Enter open · S save

DuplexLedger: Benchmarking Transactional Memory in Full-Duplex Speech Agents

Yuxuan Li, Xiangchen Guan, Kunrong Li, Jin Wang+6

Nanyang Technological University · Peking University · Beijing University of Technology+3

Memory Management for LLM Agents Memory Agent Benchmarks Full-Duplex Spoken Dialogue Systems Voice Agent Evaluation

Open paper AskPDF arXiv

Full-duplex speech agents must update persistent memory while listening and speaking, yet it remains unclear which events should be stored, revised, or excluded. Existing benchmarks isolate interaction timing from memory outcomes, obscuring the state transitions behind a correct answer. We introduce DuplexLedger, a bilingual benchmark of 1,094 persistent sessions aligning four full-duplex scenarios with event-level Write/Update/Ignore/Retain supervision, explicit pre/post memory states, and 6,564 same-session questions and checks. Evaluating five full-duplex models reveals an interaction-memory gap: no system combines strong boundary control with coherent final-state memory, and the best final-state fact coverage is 2.01%. We propose ACT-Mem, an 8.92M-parameter sidecar comprising an event controller, a versioned transactional ledger, and a typed retriever coupled with a frozen MiniCPM-o 4.5 backbone. On 219 held-out sessions, ACT-Mem increases latest-value accuracy from 4.11% to 8.68% while reducing stale assertions and distractor adoption. It improves interaction boundaries but slows interruption stopping. These results motivate selective, auditable full-duplex memory rather than indiscriminate transcript retention. Code is available at https://github.com/lyx-researcher/DuplexLedger.