InteractionBench: Measuring Timing, Task Accuracy, Whisper and Volume, and Grounding in Real-Time Voice Systems
Organizations: AveraLabs
Abstract
Most benchmarks for real-time voice systems focus on one aspect, such as turn-taking, instruction following, paralinguistic control, or conversational memory, each with its own data and set of systems. We present InteractionBench, a benchmark and evaluation harness that measures four dimensions on the same systems from the same recorded user audio: conversational timing, spoken-task accuracy, whisper and volume, and groundedness in conversations of more than twenty turns. The benchmark contains 221 items per system in thirteen scenarios. Before any system runs, we fix the expected floor-control actions for spoken tasks by rule and take the expected answers from scripted dialogues. We score whispering from the audio signal. We evaluate GPT-Live, GPT-Realtime, Gemini Live, Moshi, PersonaPlex, Freeze-Omni, and a turn-based cascade in 1,519 graded runs. Among the six full-duplex systems, timing and spoken-task accuracy are rank-correlated (), while the cascade has the highest spoken-task score (38/50) and the second-lowest timing score. No system corrects a user's grammar within the required window. Only Gemini Live whispers on most requests (8/10). Hosted systems pass 67-88% of groundedness probes, compared with 14-35% for open full-duplex models. Timing is measured where the client hears each system, so the timing of hosted systems includes network delay: for GPT-Live, whose server reports its own clock, removing a delay of about 1.5 s lowers its timing score from 86% to 56%. We also document where the protocol differs across systems, including the text instructions.
Figures & tables
| Benchmark | Setting | User speech | Timing and floor | Answers and tasks | Conversation state | Voice output |
|---|---|---|---|---|---|---|
| Full-Duplex-Bench [ 15 ] | live | both | ||||
| Full-Duplex-Bench v1.5 [ 14 ] | live | synthetic | ||||
| FD-Bench [ 26 ] | live | synthetic | ||||
| Talking Turns [ 1 ] | live | recorded | ||||
| Instruct-FD [ 31 ] | live | synthetic | ||||
| TurnBench [ 9 ] | static | recorded |
| Dimension | Scenario | Items | A run passes when … |
| Timing | Backchannel | 20 | during an extended user turn, the system gives at least one short acknowledgment and no segment longer than 3 s or 10 words |
| Pause | 30 | no system word begins between the two halves of the user’s sentence, which are separated by 1.5 s of inserted silence, and the system speaks after the pause; any vocalization in the pause, including a backchannel, counts as taking the floor | |
| Turn-taking | 30 | the system starts speaking between 50 ms before and 2.5 s after the end of the user’s last word; a reply that begins earlier counts as a false start | |
| User backchannel | 30 | short user backchannels (usually ok and mhm ) injected into its answer do not derail it | |
| Interruption | 30 | it is speaking when the user cuts in with an unrelated request (a word that ended less than 1 s earlier counts), and what it says next addresses that request | |
| Spoken task | Logic | 10 | the spoken answer states the target value |
| System | Model | Access | Timing reference |
|---|---|---|---|
| GPT-Live | gpt-live-1 ; deep reasoning delegated to gpt-5.5 | OpenAI Live API, WebSocket, PCM | arrival, rebuilt (§ 4.2 ) |
| GPT-Realtime | gpt-realtime-2.1 | OpenAI Realtime API, WebSocket, PCM; server VAD with 500 ms silence | arrival |
| Gemini Live | gemini-3.1-flash-live-preview | Gemini Live API, PCM; automatic activity detection with 500 ms silence and high sensitivity | arrival |
| Moshi | Moshi [ 3 ] , kyutai/moshika-pytorch-bf16 at revision a49141e , served locally | WebSocket, Opus | arrival |
| PersonaPlex | PersonaPlex [ 28 ] , nvidia/personaplex-7b-v1 at revision fdaf409 , served locally; voice prompt NATF0 | WebSocket, Opus | arrival |
| Freeze-Omni | Freeze-Omni [ 34 ] , VITA-MLLM/Freeze-Omni at revision c8b1918 with Qwen2-7B-Instruct at revision f2826a0 , served locally | socket.io, PCM | arrival |
| System | Spoken task | Timing | Whisper & volume | Grounded | Coherence |
|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (0–100) | |
| GPT-Live | 68 (34/50) | 86 (121/140) | 55 (11/20) | 85 (44/52) | 82 |
| GPT-Realtime | 70 (35/50) | 70 (97/138) | 35 (7/20) | 67 (35/52) | 76 |
| Gemini Live | 72 (36/50) | 77 (102/132) | 75 (15/20) | 88 (46/52) | 86 |
| Moshi | 6 (3/50) | 41 (54/132) | 15 (3/20) | 14 (7/50) | 16 |
| PersonaPlex | 18 (9/50) | 68 (90/132) | 35 (7/20) | 30 (15/50) | 28 |
| System | Backchannel | Pause | Turn-taking | User backch. | Interruption | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| pass | floor-take | pass | barge-in | pass | lat. (ms) | pass | derail | pass | yielded | |
| GPT-Live | 0.60 | 0.00 | 0.77 | 0.23 | 0.97 | 1210 | 0.97 | 0.03 | 0.93 | 1.00 |
| GPT-Realtime | 0.10 | 0.05 | 0.87 | 0.13 | 0.93 | 1460 | 0.50 | 0.50 | 0.90 | 0.97 |
| Gemini Live | 0.50 | 0.11 | 0.46 | 0.54 | 0.96 | 1115 | 0.83 | 0.17 | 1.00 | 1.00 |
| Moshi | 0.78 | 0.22 | 0.29 | 0.61 | 0.04 | 190 | 0.70 | 0.30 | 0.36 | 0.57 |
| PersonaPlex | 0.83 | 0.11 | 0.75 | 0.21 | 0.18 | 3305 | 0.77 | 0.23 | 0.93 | 1.00 |
| System | Logic | Countdown | Grammar | Keyword-wait | Stay-quiet | Total |
|---|---|---|---|---|---|---|
| GPT-Live | 10/10 | 9/10 | 0/10 | 8/10 | 7/10 | 34/50 |
| GPT-Realtime | 10/10 | 8/10 | 0/10 | 9/10 | 8/10 | 35/50 |
| Gemini Live | 10/10 | 9/10 | 0/10 | 10/10 | 7/10 | 36/50 |
| Moshi | 3/10 | 0/10 | 0/10 | 0/10 | 0/10 | 3/50 |
| PersonaPlex | 3/10 | 5/10 | 0/10 | 0/10 | 1/10 | 9/50 |
| Freeze-Omni | 4/10 | 0/10 | 0/10 | 0/10 | 0/10 | 4/50 |
| System | Whisper produced | Volume understood | Volume adapted |
|---|---|---|---|
| (/10) | (/10) | (/5 whispered) | |
| GPT-Live | 2/10 | 9/10 | 0/5 |
| GPT-Realtime | 2/10 | 5/10 | 0/5 |
| Gemini Live | 8/10 | 7/10 | 1/5 |
| Moshi | 0/10 | 3/10 | 0/5 |
| PersonaPlex | 0/10 | 7/10 | 0/5 |
| System | Probes passed | Rate (%) | All 52 probes | Coherence (0–100) |
|---|---|---|---|---|
| GPT-Live | 44/52 | 85 | 44/52 | 82 |
| GPT-Realtime | 35/52 | 67 | 35/52 | 76 |
| Gemini Live | 46/52 | 88 | 46/52 | 86 |
| Moshi | 7/50 | 14 | 7/52 | 16 |
| PersonaPlex | 15/50 | 30 | 15/52 | 28 |
| Freeze-Omni | 18/51 | 35 | 19/52 | 39 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| System | Instruction |
|---|---|
| GPT-Live, GPT-Realtime, Gemini Live | “You are a friendly, concise English voice conversation partner. Keep replies short and natural, one or two sentences. If the user asks you to correct their grammar, jump in and correct mistakes as you hear them.” |
| Freeze-Omni | “You are an English-only voice assistant. Respond ONLY in English. Never speak Chinese or any other language. Say every number as an English word (one, two, three, …), never in Chinese. Keep replies short.” |
| PersonaPlex | “You are a friendly assistant. Always reply in English.” (voice prompt NATF0 ) |
| Moshi | none; the served model takes no text instruction |
| Cascade | “You are a friendly voice assistant in a spoken conversation. Answer directly and briefly, in one or two short spoken sentences. Always reply in English.” |
| Category | Expected action | Overlap | Onset window (ms) |
|---|---|---|---|
| grammar_correction | interrupt | required | [50, 300] |
| keyword_wait | interrupt | required ∗ | [50, 300] ∗ |
| stay_quiet_until_help | take_floor | forbidden | [200, 600] ∗ |
| Type | The probe asks | A grounded reply | Failure label | |
|---|---|---|---|---|
| nonsense | 10 | an incoherent question (“What color is my dinner party?”) | flags it | NONSENSE_NOT_FLAGGED |
| state update | 8 | a value after it has been changed | the latest value | STATE_UPDATE_FAILURE |
| unknown | 6 | something the user never stated | says it was not given | UNKNOWN_INFO_HALLUCINATION |
| recall | 6 | what the user said earlier | repeats it | HISTORY_HALLUCINATION |
| false accusation | 6 | why the system said something it never said | pushes back | FALSE_PREMISE_ACCEPTANCE |
| constraint | 5 | a request an earlier constraint must shape | respects it | CONSTRAINT_DROP |
| Interruption | User backchannels | |||
|---|---|---|---|---|
| System | after reply onset | after question end | first | second |
| GPT-Live | 2.88 | 4.13 | 4.02 | 8.82 |
| GPT-Realtime | 0.48 | 1.94 | 3.84 | 8.77 |
| Gemini Live | 0.84 | 1.81 | 3.76 | 8.64 |
| Moshi | 2.73 | 2.77 | 2.05 | 6.62 |
| PersonaPlex | 2.44 | 5.77 | 4.17 | 9.28 |
| Rule | GPT- Live | GPT- Realtime | Gemini Live | Moshi | PersonaPlex | Freeze- Omni | Cascade |
| Turn-taking, start 50 ms early (rule) | 29/30 | 28/30 | 27/28 | 1/28 | 5/28 | 29/30 | 30/30 |
| 150 ms early | 29/30 | 28/30 | 27/28 | 3/28 | 5/28 | 29/30 | 30/30 |
| 300 ms early | 29/30 | 28/30 | 27/28 | 3/28 | 5/28 | 29/30 | 30/30 |
| Pause, any vocalization fails (rule) | 23/30 | 26/30 | 13/28 | 8/28 | 21/28 | 15/30 | 0/30 |
| backchannels allowed | 25/30 | 26/30 | 13/28 | 10/28 | 25/28 | 15/30 | 0/30 |
| Whisper, HNR 4 dB, voiced 0.3 | 1/10 | 1/10 | 5/10 | 0/10 | 0/10 | 0/10 | 0/10 |
| Scenario | Session clock | At the client |
|---|---|---|
| Backchannel | 12/20 | 12/20 |
| Pause | 2/30 | 23/30 |
| Turn-taking | 8/30 | 29/30 |
| User backch. | 29/30 | 29/30 |
| Interruption | 27/30 | 28/30 |
| Timing aggregate | 56% (78/140) | 86% (121/140) |
Voice Agent Evaluation
Voice agent evaluation measures how reliably spoken AI systems understand, respond, take actions, and recover in real interactions.
See all 51 papers