cs.AISep 11, 2026

ττ-Elicitation: Benchmarking multi-turn entity extraction in voice agents

Authors: Soham Ray, Victor Barres

Organizations: Sierra · Mercor

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Voice agents often need to collect names, addresses, identifiers, dates, and times exactly, yet end-to-end benchmarks obscure where capture fails. We introduce ττ-Elicitation, a 200-task voice benchmark spanning 10 entity types, controlled difficulty, caller realisms, and three environments. A matched text agent passes all tasks, but four voice configurations achieve robust exact success from 0.14 to 0.41. Agents increase verification for hard and unfamiliar entities and sometimes for incorrect captures, but not for their weakest caller voice; only 24 to 37 percent of verified errors are repaired. A scaffold that prescribes spelling, read-back, correction, and confirmation raises robust Pass3^3 by 14 to 31 points, at a cost of 21 to 28 seconds per call. Realisms such as spelling variations and restarts do not detectably affect exact success; mispronunciation increases repair effort. These results identify strategy selection and successful recovery as the central bottlenecks in exact spoken entity collection.

Figures & tables

Figure 1: \uptau\uptau -Elicitation selects values from entity banks, builds a controlled task, and runs it through the \uptau\uptau -Voice interaction loop. Pass3\mathrm{Pass}^{3} requires success in all three environment realizations. The scale marks fidelity-judged agent speech.
WorkMultiTaskExactNNRepair
Voice agents [ 12 , 4 , 3 , 9 ]✓\checkmark✓\checkmarkD–D
Spoken dialogue [ 14 , 21 ]✓\checkmark–✓\checkmark––
Speech NER [ 13 , 20 , 1 ]––✓\checkmark––
\uptau\uptau -Elicitation✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark
Table 1: Coverage of related benchmarks. Task: executed tool task; Exact: exact entity score; NN : controlled entity count; D: diagnostic or partial coverage.
Figure 2: Task success over the combined task set. Each pair shows agent-directed and scaffolded runs; light full bars show Pass@1 , and dark inset bars show robust Pass3\mathrm{Pass}^{3} .
Figure 3: Mean verification effort (read-backs plus spelling requests). All systems increase effort for hard and unfamiliar entities, but not for their own lowest- Pass@1 caller voice; only Grok increases effort under noise.
Figure 4: Caller-realism effects in agent-directed (solid circles) and scaffolded (dashed squares) runs. Bars show task-clustered 95% Hájek intervals versus eligible clean assignments; none survives Holm correction.
BankConstruction / groundingPass3\mathrm{Pass}^{3} ↓\downarrow
DatesCalendar and clock.60
PhonesRegulator-reserved ranges.47
TimesCalendar and clock.43
Person namesSSA + Census [ 17 , 16 ].30
CodesISO 3779 VINs; mod-97 IDs.25
PropertiesRegistry-screened synthetic.18
Table 2: Value-bank construction and robust agent-directed results. Every bank has 80 values and contributes 20 tasks (10 easy, 10 hard). Pass3\mathrm{Pass}^{3} pools 60 system–task triples per bank.
WorkflowTask Pass@1Field Pass@1
Batch submit; no field validation.64±.06.64\pm.06.78±.03.78\pm.03
Validate each field; retry errors.82±.05.82\pm.05.86±.03.86\pm.03
Table 3: Effect of field validation and retry under the scaffolded prompt. Task Pass@1 requires every field to be correct; field Pass@1 scores individual values.
Figure 5: Task success declines as fields accumulate, roughly following the light-gray reference expected if every field independently succeeds 77% of the time (77%, 59%, and 46%). Error bars show 95% confidence intervals.
 GPT minimalGPT xhighGemini highGrok
Severity ≥2\geq 22.5%2.1%3.6%4.1%
Table 4: Share of scored agent utterances with a clear speech-fidelity defect by system (lower is better).
More in this topic

Voice Agent Evaluation

Voice agent evaluation measures how reliably spoken AI systems understand, respond, take actions, and recover in real interactions.

See all 45 papers

Explore similar work

CardsList

  1. ττ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

    Mar 14, 2026Soham Ray, Keshav Dhandhania, Victor Barres +1Voice Agent EvaluationAI Agent Benchmarks

    Sierra.ai, San Francisco, USA · Princeton Language and Intelligence, Princeton University

  2. Talk2Agent: Benchmarking Voice Interfaces for Text Agents

    Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR EvaluationVoice Agent Evaluation

    Department of Electronic Engineering, Tsinghua University · University of Cambridge

  3. ττ-Multilingual: Benchmarking Voice Agents Across Languages

    Sep 21, 2026Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3Voice Agent EvaluationFull-Duplex Spoken Dialogue Systems

    Sierra · Princeton University · Mercor