Joi-Bench-Task: Benchmarking Task Coordination in Full-duplex Voice Agents through Interactive Simulation
Organizations: Platform and Content Group, Tencent
Abstract
Full-duplex voice agents enable natural, hands-free interaction, allowing users to refine requests, redirect ongoing work, and seek clarification while the agent is speaking. However, existing voice-agent benchmarks insufficiently isolate coordination between task execution and conversation. We introduce a benchmark that assesses this coordination by examining whether agents track what is being executed, what has been completed, what has been communicated, and what remains to be reported as the conversation evolves. We propose an interactive user simulator that introduces configurable interventions at designated timing in tool execution and spoken result delivery, supporting controllable and reproducible evaluation conditions. We evaluate five leading commercial voice agents along three dimensions: execution responsiveness, coordination under request changes, and coordination under interruptions. Agent evaluations reveal persistent coordination failures. Under request changes, the highest result completion rate is only 55%, indicating difficulty for current voice agents adapting to new requests while preserving earlier obligations. After user interruption, four of the five agents exhibit average information omission rates of at least 23%. These findings highlight task coordination as a central challenge for current voice agents in complex, evolving conversations.
Figures & tables
| Scope / behavior | Full-Duplex- Bench v1 | Full-Duplex- Bench v1.5 | Full-Duplex- Bench v2 | Full-Duplex- Bench v3 | -Voice | EVA-Bench | VAmoS Bench | EchoChain | IHBench | APEX- Voice | VoiceAgent Bench | Our benchmark |
| Evaluation scope | CD | CD | CD | Both | Both | Both | Both | CD | CD | Both | AP | Both |
| Interruption | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ ∗ | ✓ | ✓ | ✓ | ✗ | ✓ |
| Resume | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ ∗∗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ |
| Priority change | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Task addition | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Task withdraw | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Group | Task | Prescribed user behavior | Successful agent behavior |
| Coordination during task execution | Status Inquiry | Issue one request then inquire about the execution status twice during task execution. | Answer the status inquiries and report the result. |
| Baseline with no request changes | Issue 5 requests with no further interventions. | Execute tasks concurrently and report all required results. | |
| Priority change | Issue 5 requests then ask for priority feedback on the result of a designated task during task execution. | Prioritize that result in subsequent reporting and report the remaining results. | |
| Task withdraw | Issue 5 requests then withdraw one task’s request during task execution. | Omit the withdrawn task and report the remaining results. | |
| Task addition | Issue 5 requests then add one more task’s request during task execution. | Execute the added task and report all required results. | |
| Coordination during spoken result delivery | Pause-resume | Interrupt the agent with a pause request during spoken result delivery. | Pause appropriately and wait; once the user returns, resume and complete the result report. |
| Dimension | Metric name | Definition |
| Execution responsiveness | Tool start latency (s) | The time elapsed from the end of the user request or the completion of the latest relevant clarification associated with a tool, to when the agent invokes that tool. |
| Announcement latency (s) | The time elapsed from successful tool completion to the start of the first substantive spoken announcement grounded in the returned tool result. | |
| Case–level first–attempt success rate (pass@1, %) | Percentage of cases in which every expected tool is selected with correct arguments on its first attempt. Missing tools or incorrect first attempts cause the case to fail. | |
| Status Inquiry Response Rate (%) | Percentage of cases in which the agent replies to the user’s status check inquiry. | |
| Concurrent invocation rate (concurrency met, %) | Percentage of cases in which the agent invokes tools concurrently when handling requests that require multiple tools. | |
| Coordination under request changes | Request change success rate (%) | Percentage of cases in which the agent satisfies the changed request: announcing the prioritized result first after a priority change, cancelling the withdrawn result, or reporting the added result. |
| Execution responsiveness | Coordination under request changes | Coordination under interruptions | ||||||
| Model | Tool start | |||||||
| latency S/M | Announcement | |||||||
| latency S/M | Status inquiry | |||||||
| response | Request change | |||||||
| success | Result | |||||||
| completion | Information |
| Model | Concurrency met | Pass@1 S/M |
| GPT Live 1.0 | 100.0% | 100.00% / 88.75% |
| Realtime 1.5 | 0.0% | 100.00% / 63.75% |
| Realtime 2 | 100.0% | 98.33% / 86.25% |
| Grok Voice 3.0 | 100.0% | 96.67% / 76.25% |
| Gemini 3.8 Live | 100.0% | 96.67% / 81.25% |
| Result completion rate | Request change success rate | ||||||
| Model | Baseline | Priority change | Task withdraw | Task addition | Priority change | Task withdraw | Task addition |
| GPT Live 1.0 | 90.0% | 85.0% | 90.0% | 55.0% | 80.0% | 40.0% | 75.0% |
| Realtime 1.5 | 85.0% | 80.0% | 80.0% | 75.0% | 95.0% | 90.0% | 100.0% |
| Realtime 2 | 85.0% | 75.0% | 85.0% | 65.0% | 10.0% | 95.0% | 95.0% |
| Grok Voice 3.0 | 80.0% | 60.0% | 65.0% | 50.0% | 35.0% | 85.0% | 100.0% |
| Gemini 3.8 Live | 60.0% | 40.0% | 50.0% | 50.0% | 75.0% | 90.0% | 100.0% |
| Model | Resumption rate | Information omission rate |
| Pause- / Inquiry-resume | Pause- / Inquiry-resume | |
| GPT Live 1.0 | 100.0% / 90.0% | 0.0% / 7.0% |
| Realtime 1.5 | 80.0% / 30.0% | 8.0% / 61.0% |
| Realtime 2 | 5.0% / 0.0% | 78.0% / 80.0% |
| Grok Voice 3.0 | 95.0% / 55.0% | 6.0% / 40.0% |
| Gemini 3.8 Live | 90.0% / 70.0% | 43.0% / 66.0% |
Voice Agent Evaluation
Voice agent evaluation measures how reliably spoken AI systems understand, respond, take actions, and recover in real interactions.
See all 51 papers