Real-time voice assistants must revise task state when users interrupt mid-response, but existing spoken-dialog benchmarks largely evaluate turn-based interaction and miss this failure mode. We introduce EchoChain, a controlled benchmark for evaluating full-duplex state-update reasoning under mid-speech interruptions. EchoChain identifies three recurring failure patterns in post-interruption continuations: contextual inertia, interruption amnesia, and objective displacement. The benchmark generates scenario-driven conversations and injects interruptions at a standardized point relative to assistant speech onset, enabling controlled cross-model comparison. In a paired half-duplex control, total failures drop by 40.2% relative to interrupted runs, indicating that many errors are driven by state-update reasoning under interruption rather than task difficulty alone. Across evaluated real-time voice models, no system exceeds a 50% pass rate, showing substantial room for improvement in mid-generation state revision. EchoChain provides a reproducible benchmark for diagnosing state-update reasoning failures in full-duplex voice interaction.
Figures & tables
Figure 1: Illustrative examples of the three post-interruption failure modes in full-duplex state-update reasoning. Left: ( Contextual Inertia ) The assistant acknowledges the interruption but fails to correctly revise its prior reasoning. Center: ( Interruption Amnesia ) Previously established constraints are forgotten after the interruption. Right: ( Objective Displacement ) The assistant responds only to the interruption and fails to complete the original request.
Figure 2: Architecture of EchoChain. Scenario seeds drive a planner–simulator loop that generates context-dependent user utterances and barge-ins, which are synthesized into speech and streamed through a full-duplex orchestrator to a realtime responder model. The orchestrator injects interruptions at a controlled point during assistant speech, and the resulting transcripts and audio are evaluated automatically and, when flagged, reviewed by humans.
Figure 3: Observed preference proportions for cloned versus human samples, shown alongside the expected proportions under the symmetry null hypothesis.
Model
Contextual Inertia
Interruption Amnesia
Objective Displacement
Other
Total Failed Conversations
MPR (%)
MCP (%)
Amazon Nova Sonic 2
73
53
16
5
147
26.5
78.1
Gemini Live-2.5-flash-native-audio
38
65
64
0
167
16.5
64.8
GPT-realtime-2025-08-28
29
45
35
1
110
45.0
81.1
Grok Voice Agent
42
39
16
6
103
48.5
85.5
Table 1: EchoChain results across 200 interrupted conversations. Lower is better for failure counts; higher is better for MPR and MCP. Bold indicates the best value in each column.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Topic
Subtopic
Description
Audio Support
Troubleshooting Call
Guide users through problems via voice calls, providing simple short step-by-step audio instructions to resolve software or hardware issues. (Do not include headphone issues).
Setup Instructions Call
Provide simple step-by-step audio instructions for setting up devices or software, guiding users through installation and configuration processes via voice.
Voice Communication Training
Storytelling Practice
Develop spoken storytelling skills through voice practice, focusing on narrative delivery or character voices in oral storytelling.
Content Brainstorming
Brainstorm and develop content ideas through verbal discussion, focusing on topic exploration.
Speech Practice
Practice delivering speeches and presentations through voice coaching, focusing on verbal pacing or tone.
Appendix
Table 5
Rubric Criterion
Gemini Live
GPT Realtime
Grok Live
Nova Sonic 2
Excludes any drink recommendations containing caffeine.
Pass
Pass
Fail
Pass
Prioritizes the preference for a savory or smoky snack recommendation.
Pass
Pass
Pass
Pass
Prioritizes the preference for a crunchy snack recommendation.
Pass
Pass
Fail
Fail
Identifies the specific pairing of BBQ chips and ginger beer as the top choice.
Pass
Pass
Fail
Fail
Explains the snack’s sensory profile and the rationale for the drink pairing.
Pass
Pass
Pass
Pass
Appendix
Table 3: Rubric evaluation for the Contextual Intertia golden example.
Rubric Criterion
Gemini Live
GPT Realtime
Grok Live
Nova Sonic 2
Starts the spoken agenda with the Beacon update.
Pass
Pass
Pass
Pass
Includes the Atlas update with a deadline of Friday with both time zones, 5 PM PT and 8 PM ET.
Fail
Pass
Pass
Pass
Includes the Cirrus update with a deadline tomorrow with both time zones, 10 AM PT and 1 PM ET.
Pass
Fail
Pass
Pass
Includes repo steps with the Cirrus update.
Pass
Pass
Pass
Pass
Includes a spoken agenda as a script.
Pass
Pass
Pass
Pass
Appendix
Table 4: Rubric evaluation for the Interruption Amnesia golden example.
Rubric Criterion
Gemini Live
GPT Realtime
Grok Live
Nova Sonic 2
Provides a spoken-word pitch script.
Fail
Pass
Pass
Pass
Includes the tagline, “Get total control of your local supply chain with us.”
Fail
Pass
Pass
Pass
Specifies strategic pause locations within the script text.
Fail
Pass
Pass
Pass
Identifies specific words for vocal emphasis to increase the pitch’s impact.
Fail
Pass
Pass
Pass
Includes a distinct follow-up sentence to continue the conversation.
Pass
Pass
Pass
Pass
Appendix
Table 5: Rubric evaluation for the Objective Displacement golden example.
State tracking studies how AI systems maintain and update information about entities, tasks, or environments across sequential observations and actions.