VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation

Joshua Meyer; Sahar Shayegan; Ritiz Tambi; Ali Khan; Sun Kim; Victor Shih; Mehdi Jamei; Andi Partovi

cs.AISep 29, 2026

VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation

Authors: Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan, Sun Kim, Victor Shih, Mehdi Jamei, Andi Partovi

Organizations: Veris AI

Abstract

Voice agents in production must handle several requests, background speech, and customers who lose patience. We introduce VAmoS Energy, a benchmark that combines these challenges in 100 calls about utility billing and payment assistance. Each caller makes two to four requests. The agent has sixteen tools backed by a stateful Stripe billing twin and the Apache Fineract loan engine, with account access blocked until caller verification succeeds. The tasks use public household electricity data and a policy based on Pennsylvania's residential billing rules. An LLM-as-a-verifier checks the agent's actions and spoken figures against explicit requirements. On a calibration run, it agrees with a code verifier on 99.1% of checks. Across fourteen voice stacks and three repeats per task, completion ranges from 17.3% to 44.7%. Grok Voice leads, and Gemini 3.8 Live and GPT-Live follow at about the same cost per call. Background television reduces pooled completion from 38.7% to 8.6%. The simulated caller often accepts an incorrect result because it hears the agent's words but cannot inspect its actions. These findings show why voice agents need evaluation across the whole call, including what they say, what they change, and how they handle competing speech.

Figures & tables

	VAmoS Bench	VAmoS Energy
Backend	Seeded PostgreSQL	Stripe twin + Apache Fineract
Agent tools	5	16
Verification	Card digits, name, address/phone	Account number, name, second factor
Requests per call	One, sometimes multi-step	2–4
Data	Generated	ResStock usage; Pennsylvania policy
Caller personas	Per-scenario style	Calm, angry

Table 1: The two VAmoS benchmarks. Observed completion depends on both the benchmark and the evaluated configurations.

Panel of Figure 1 — Figure 1: Task construction and selection. The loop checks each added request against the account state left by earlier requests. Development runs inform the final selection; not every generated combination was run.

Stack	Complete (%)	95% interval	$n$	Latency (s)	Cost ($)
Grok Voice	44.7	[38.8, 50.7]	264	2.64	0.210
Gemini 3.8 Live	40.1	[34.6, 45.9]	284	1.66	0.193
OpenAI GPT-Live 1	39.2	[33.5, 45.3]	260	1.62	0.192
OpenAI Realtime 2.1	37.9	[32.3, 43.8]	269	2.12	0.633
ElevenLabs	36.4	[30.9, 42.2]	275	2.18	0.272
OpenAI Realtime	33.6	[28.2, 39.5]	265	2.32	0.607

Table 2: Completion, median response latency, and mean cost per call (Appendix D ). Intervals are Wilson 95%;

n

is the number of counted attempts. Latency and cost use calls with verdicts. Configurations are in Appendix F .

Figure 2: (a) Completion by background condition, with 46–75 counted attempts per stack and condition. (b) Transfers to a human by persona, among calls with verdicts and traces. All exam tasks are intended to be resolved by the agent.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Reads	Checks	terra agrees	luna agrees
Exact world effects, in order	tool results	97	96	96
No account tool before verification	tool calls	97	97	97
Required tool called	tool calls	20	20	20
Forbidden tool not called	tool calls	34	34	34
Quote, caller turn, then create	tool calls + turns	34	32	33
No secret spoken before verification	speech	97	95	95

Appendix

Table 3: Agreement between the code verifier and two LLM verifiers, check by check, on 97 completed attempts from a calibration run of this exam. “Reads” is the evidence a check needs; the speech rows are the ones where the code extracts values with patterns.

	Requests			Persona		Use case
Stack	2	3	4	Calm	Angry	High bill	Payment
Grok Voice	81	46	40	39	50	40	48
Gemini 3.8 Live	56	49	34	34	46	31	45
OpenAI GPT-Live 1	65	45	34	44	34	38	40
OpenAI Realtime 2.1	67	51	28	34	41	26	45
ElevenLabs	71	38	32	31	41	35	37

Appendix

Table 4: Completion (%) by number of requests, caller persona, and use case of the last request. Before exclusions, each stack has 18/90/192 attempts with 2/3/4 requests, 150 per persona, and 111/189 per use case; counted denominators are smaller.

Figure 3: Completion against median response latency and mean cost per call. Vertical bars are Wilson 95% intervals; the cost axis is logarithmic.

Stack	All (%)	Rank	Counted (%)	Rank	Excluded	Caller	Verifier	Lost	Passes	Soundex
Grok Voice	41.0	1	44.7	1	36	25	11	0	5	43
Gemini 3.8 Live	38.0	2	40.1	2	16	11	2	3	0	25
OpenAI GPT-Live 1	37.3	3	39.2	3	40	30	10	0	10	26
OpenAI Realtime 2.1	35.0	4	37.9	4	31	23	6	2	3	28
ElevenLabs	34.3	5	36.4	5	25	13	4	8	3	28
OpenAI Realtime	30.3	6	33.6	6	35	31	4	0	2	25

Appendix

Table 5: Completion over all attempts and over counted attempts, with the attempts excluded and why: the simulated caller broke its script (“caller”), the verifier misjudged the call (“verifier”), or the platform lost the session (“lost”). “Passes” is the number of excluded attempts that had passed. “Soundex” is the number of passes, over all attempts, whose caller was verified only through the sound-alike name match (described below the table). Ordered by counted completion.

Stack	Class	Models (as run)
Grok Voice	Speech-to-speech	grok-voice-think-fast-2.0 , voice eve
Gemini 3.8 Live	Speech-to-speech	gemini-3.8-live , same bridge as Gemini 3.1 Live
OpenAI GPT-Live 1	Speech-to-speech	gpt-live-1 , tool use delegated to gpt-5.6-terra , voice marin
OpenAI Realtime 2.1	Speech-to-speech	gpt-realtime-2.1 , same bridge as OpenAI Realtime
ElevenLabs	Bundled platform	Conversational AI: ElevenLabs ASR $\to$ gpt-4.1-mini $\to$ eleven_flash_v2
OpenAI Realtime	Speech-to-speech	gpt-realtime-2 , voice alloy , server VAD (threshold 0.5)

Appendix

Table 6: The fourteen stacks in this run, in order of observed completion. Every stack runs the same prompt, tool declarations, and dispatcher from the shared core. Models are as recorded in the candidate logs and the implementations’ configuration.

Request	Tasks
Enable paperless after confirming the email	73
Report the current balance and delinquency status	50
Reject a one-month arrangement	25
Record the caller’s exact partial instalment	25
Record a payment made today	25
Quote before creating an arrangement	22

Appendix

Table 7: The requests the 100 tasks draw on, with the number of tasks that include each.

Explore similar work

VAmoS Bench: Voice Agent Simulation Bench
Jul 29, 2026Joshua Meyer, Sahar Shayegan, Ritiz Tambi +5Voice Agents Dialogue Benchmarks
Veris AI
From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
May 14, 2026Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo +3Voice Agents Seed-Tts-Eval Benchmark
Dialpad Inc.
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
Date pendingTara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agents Evaluation Benchmarks
ServiceNow AI Research

Jul 29, 2026, cs.AIJ/K move · Enter open · S save

VAmoS Bench: Voice Agent Simulation Bench

Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan+4

Veris AI

Voice Agents Dialogue Benchmarks

Open paper AskPDF arXiv

Production voice agents span cascaded, speech-to-speech, and hybrid architectures. Voice-agent benchmarks typically measure component quality and conversational properties such as word error rate, latency, naturalness, and turn-taking. Fewer measure whether the agent handled a phone call correctly on its own. Contact centers refer to this as ``containment'': the share of phone calls the automated system resolves without handing off to a human. On some phone calls the right outcome is refusal or a redirect. To address this gap, we introduce VAmoS Bench, the Voice Agent Simulation Bench. It measures complete voice-agent systems end to end in a stateful customer-support task. The agent is Riley, a credit-card support representative for a fictional bank who can freeze, cancel, replace, or activate a card. Each of 100 scenarios supplies a simulated caller with a private goal and a seeded PostgreSQL backend. The platform uses each scenario to populate and activate an isolated simulation in which the caller reaches Riley over audio; roughly one-third apply adversarial pressure. The agent can use five tools that execute real SQL against the backend. Each scenario also defines binary assertions. A grader evaluates them against the complete trace of what the caller and agent said and what the agent did, including tool invocations, arguments, and returned rows. This catches an agent that claims to have changed a card without updating the database, as well as one that makes the right database change while disclosing protected information. This first benchmark version focuses on financial services. Its evaluation protocol supports an evolving leaderboard: additional voice agents can be evaluated on the same version, while later versions can expand the tasks and scenarios.

Abstract

Explore similar work

VAmoS Bench: Voice Agent Simulation Bench

From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents