Inquesto Score: A reliability Protocol For Voice Agents

Massa Baali; Bhiksha Raj

cs.CLSep 24, 2026

Inquesto Score: A reliability Protocol For Voice Agents

Authors: Massa Baali, Bhiksha Raj

Organizations: Carnegie Mellon University

Abstract

Voice agents are increasingly deployed in workflows where failed interactions can affect transactions, access, and other consequential outcomes, creating a need for reproducible and interpretable evaluation. We introduce Inquesto Score (IS), a protocol for measuring voice-agent reliability as the percentage of calls in a fixed, versioned evaluation population that achieve the caller's goal without a functional failure or worse. Rather than combining heterogeneous metrics, IS defines explicit failure events and severity levels and evaluates the deployed voice pipeline. Timing failures, including talk-over and delayed responses, are measured directly from audio, while semantic and state-dependent failures are evaluated using scenario predicates, tool traces, and a pinned open-model judge. Diagnostic views of behavior, acoustic robustness, identity handling, and speaker groups accompany the score without being combined into it. Inquesto Score v0.1 evaluates 30 scenarios, three acoustic conditions, four speaker groups, and 306 calls per agent across 13 configurations of a reference voice-agent system. Our evaluation shows that reliable measurement requires evidence beyond transcripts, explicit treatment of deployment conditions, and validation of the evaluators used to determine outcomes. We release the protocol, reference implementation, and evaluation records.

Figures & tables

Event	Observable definition	Source	Sev.
verbose	mean agent turn $>$ 60 words	transcript	S1
slow median	median reply latency $>$ 1.5 s	audio	S2
talk-over	agent audio starts $\geq$ 0.5 s before caller audio ends	audio	S2
slow turn	a reply $>$ 3 s after the caller stops, or no reply	audio	S3
repeated talk-over	talk-over on $\geq$ 1/3 of agent turns ( $\geq$ 2)	audio	S3
context loss	corrected detail later used in its original form	judge	S3

Table 1: Failure events in protocol v0.1. S3 and above fail the call.

Agent (LLM / endpointing)	IS	95 % CI	B	R	I	F	US-F	US-M	UK-F	UK-M	S3/4/5	tx-only	audio-only
Qwen2.5-7B / 1100 ms	45	[39, 50]	42	46	39	41	42	41	53	43	80/75/11	58	39
Qwen2.5-7B / 700 ms	39	[34, 44]	46	36	44	32	41	32	37	45	104/78/16	56	51
Llama-3.1-8B / 700 ms	36	[31, 42]	46	36	39	27	42	33	27	41	122/18/54	58	68
Qwen2.5-32B / 700 ms	33	[28, 39]	38	30	33	27	27	29	35	43	94/12/80	56	70
Qwen2.5-14B / 700 ms	33	[28, 38]	38	32	39	29	35	29	31	36	103/22/89	54	64
Claude-Haiku-4.5 h / 700 ms	26	[21, 31]	17	27	22	24	24	25	28	27	263/7/0	66	122

Table 2: Inquesto v0.1 for every configuration of the reference agent: score, 95 % interval, diagnostic views B/R/I/F, pass rate per speaker group (US/UK

\times

female/male; F is their minimum), events by severity (S3/S4/S5), the score with audio-derived events removed (tx-only) and the number of failed calls whose only S3+ events were audio-derived. h = language model behind a hosted gateway, everything else local.

Agent	S2+	S3+ (v0.1)	S4+	S5
Qwen2.5-7B / 1100 ms	44	45	58	58
Qwen2.5-7B / 700 ms	34	39	56	56
Llama-3.1-8B / 700 ms	18	36	58	60
Qwen2.5-32B / 700 ms	27	33	56	57
Qwen2.5-14B / 700 ms	22	33	54	55
Claude-Haiku-4.5 h / 700 ms	3	26	66	67

Table 3: Score under each clean-success threshold: a call fails on any event of the given severity or worse. v0.1 uses S3 (functional failure).

Voice Agent Evaluation

Voice agent evaluation measures how reliably spoken AI systems understand, respond, take actions, and recover in real interactions.

See all 46 papers

VAmoS Bench: Voice Agent Simulation Bench
Jul 29, 2026Joshua Meyer, Sahar Shayegan, Ritiz Tambi +5AI Agent Benchmarks Customer Support Automation
Veris AI
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
May 13, 2026Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agent Evaluation Spoken Dialogue Systems
ServiceNow AI Research

Jul 29, 2026J/K move · Enter open · S save

VAmoS Bench: Voice Agent Simulation Bench

Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan+4

Veris AI

AI Agent Benchmarks Customer Support Automation LLM Agent Evaluation Voice Agent Evaluation Tool-Using Agents Conversational Agents+1

Open paper AskPDF arXiv

Production voice agents span cascaded, speech-to-speech, and hybrid architectures. Voice-agent benchmarks typically measure component quality and conversational properties such as word error rate, latency, naturalness, and turn-taking. Fewer measure whether the agent handled a phone call correctly on its own. Contact centers refer to this as ``containment'': the share of phone calls the automated system resolves without handing off to a human. On some phone calls the right outcome is refusal or a redirect. To address this gap, we introduce VAmoS Bench, the Voice Agent Simulation Bench. It measures complete voice-agent systems end to end in a stateful customer-support task. The agent is Riley, a credit-card support representative for a fictional bank who can freeze, cancel, replace, or activate a card. Each of 100 scenarios supplies a simulated caller with a private goal and a seeded PostgreSQL backend. The platform uses each scenario to populate and activate an isolated simulation in which the caller reaches Riley over audio; roughly one-third apply adversarial pressure. The agent can use five tools that execute real SQL against the backend. Each scenario also defines binary assertions. A grader evaluates them against the complete trace of what the caller and agent said and what the agent did, including tool invocations, arguments, and returned rows. This catches an agent that claims to have changed a card without updating the database, as well as one that makes the right database change while disclosing protected information. This first benchmark version focuses on financial services. Its evaluation protocol supports an evolving leaderboard: additional voice agents can be evaluated on the same version, while later versions can expand the tasks and scenarios.

Abstract

Voice Agent Evaluation

Explore similar work

VAmoS Bench: Voice Agent Simulation Bench

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents