Autonomous voice agents now conduct routine clinical follow-up calls, so some automated triage is unavoidable. A patient becoming frustrated is one thing that triage should catch. These systems typically read only the transcript the speech-to-text stage produces, not the voice, and whether that is enough is largely untested on real clinical speech. We study per-turn frustration detection on real patient calls from a deployed clinical voice agent: 150 turns from six calls labelled by three annotators, with a length-matched acted corpus as a control, and 714 turns from 30 calls. Patient turns are overwhelmingly short: across the deployed calls, roughly 59% are one or two words, against 13% in the full acted corpus, and a corpus accuracy gap remains even after the acted set is length-matched. On one-word turns, transcript-only detection does not exclude chance. Across five audio-capable language models, the transcript flags 3 or 4 of the 39 frustrated one-word turns in the 30 calls, and a classifier given only the word count reaches 57.3%. Whether audio helps depends on the model and on the instruction. Asked to attend to delivery, two Gemini 2.5 models gain 3.0 pp over the transcript across 30 calls (-1.0 to 7.5 pp). Of the turns they confidently clear, 6.9% were in fact frustrated for audio against 13.0% for the transcript, a difference of 6.1 pp (2.2 to 10.4 pp). Across the 30 calls, given one prompt identical for both inputs, no model's audio-text interval excludes zero. Under the input-specific prompts, a newer Gemini model and two smaller open models do worse with audio than with the transcript (-3.5, -8.6 and -10.6 pp). A WavLM probe trained on the clinical calls matches the best prompted model overall (71.6%) but misses every frustrated one-word turn in the 24 added calls. When a patient's turn is short, the transcript alone is not enough to rule out frustration.
Figures & tables
Figure 1: Two real calls with similar questions and responses. The patient’s response “Some of the time” is identical in wording but annotators labelled it frustrated in Call A and not frustrated in Call B. Transcript excerpts are verbatim.
Clinical
Acted
Raw
κ
Raw
κ
Annotators 1–2
89.1%
0.78
88.3%
0.76
Annotators 1–3
83.4%
0.67
85.9%
0.72
Annotators 2–3
86.2%
0.73
89.2%
0.78
Unanimous
78.5%
82.0%
Unanimous, by utterance length
Table 1: Inter-annotator agreement on the clinical and acted evaluation sets. Pairwise raw agreement and Cohen’s κ are reported alongside overall unanimity and unanimity by utterance length. Inconclusive judgements are excluded from the relevant calculation rather than counted as disagreement.
Figure 2: Distribution of word counts across over 2000 clinical conversations. Each row is one conversation, ordered by mean utterance length; the horizontal axis runs from the start of the conversation to its end. Clinical calls are dominated by 1-2 patient turns throughout. The acted corpus contains proportionally under a quarter as many.
Audio
Text
Turn length
n
Clinical
Acted
Clinical
Acted
1 word
47 / 41
60.0%
62.9%
54.8%
51.2%
2–3 words
24 / 20
71.9%
59.4%
56.2%
68.8%
4–5 words
10 / 11
60.0%
71.7%
90.0%
61.7%
6–10 words
34 / 28
73.5%
85.7%
64.7%
85.7%
11–20 words
22 / 24
65.6%
81.4%
70.8%
77.9%
Table 2: Best model’s balanced accuracy by turn length, clinical against acted, 150 turns each with matched length profiles. The clinical set balanced 75 / 75 and the acted set 76 / 74 . n gives the clinical and acted turns in each bucket, and Figure 15 gives each bucket’s interval.
Figure 3: Predicted-frustration rate against turn length for all three models, against the true rate in black. Every transcript condition starts far below the true rate on the shortest turns and climbs more steeply than it.
Frustrated turns
Silent failures
Turn length
n
share
n
share
Missed
1–2 words
26
34.7%
19
57.6%
73.1%
3–10 words
25
33.3%
10
30.3%
40.0%
11+ words
24
32.0%
4
12.1%
16.7%
All
75
100%
33
100%
44.0%
Table 3: Where the transcript condition’s silent failures fall. A silent failure is a confidently cleared turn that was labelled as frustrated by the human annotators (a falsely-cleared turn). Short turns are a third of the frustrated turns and well over half of the silent failures, and the miss rate falls as turns lengthen. Figure 14 plots the two shares.
Figure 4: Turns each input modality confidently clears, and how many of them were in fact frustrated. The transcript clears 87 of 150 turns and 37.9% of those were frustrated; audio clears 45 and 17.8% were. The audio rate holds only when its prompt directs the model to delivery (Section 5.5 ).
Figure 5: Comparison of audio against the transcript, as differences with 95% intervals from a 10000 -draw bootstrap over turns. A filled marker marks an interval that excludes zero. No frontier model’s audio gain excludes zero on its own. Pooled, the frontier pair’s gain excludes zero on the acted corpus and on both corpora together, though not on the real calls alone, and adding the open model removes it. The open model is worse with audio on the real calls, by an interval that excludes zero.
Figure 6: Balanced accuracy on the six original calls, the 24 added calls and all 30, with 95% bootstrap intervals over calls. Audio is solid and filled, the transcript dotted and hollow. The last panel shows the two WavLM probes trained on the 30 calls (Section 5.6 ). The bar and band mark the word-count-only baseline and its interval.
Figure 7: Share of the frontier pair’s confidently cleared turns that were frustrated, on the six calls ( 33 of 87 for the transcript, 8 of 45 for audio), the 24 ( 42 of 491 , 15 of 288 ) and all 30 ( 75 of 578 , 23 of 333 ), with 95% bootstrap intervals over calls. Each part’s frustrated base rate is marked: 50% , 11.5% and 19.6% .
Figure 8: Predicted-frustration rate by turn length on the 24 added calls ( 564 turns), against the true rate in black, for the five models and the two WavLM probes trained on the 30 calls. Audio is solid, the transcript dotted.
One-word recall
Model
Prompts
Audio
Text
Audio − text
audio / text
Gemini 2.5 Flash
Input-specific
71.3
66.7
+4.7 [ −2.6 , +10.9 ]
5 / 2
Identical
66.0
67.3
−1.3 [ −7.7 , +4.7 ]
0 / 2
Gemini 2.5 Pro
Input-specific
67.3
64.0
+3.3 [ −6.2 , +11.9 ]
13 / 2
Identical
67.3
66.7
+0.7 [ −7.0 , +6.8 ]
1 / 2
Gemini 3.8 Flash
Input-specific
64.0
68.7
−4.7 [ −8.7 , −0.5 ]
1 / 2
Table 4: Input-specific prompts against one prompt identical for both inputs, on the six calls, for all five models. Balanced accuracy, the paired audio-minus-text difference in percentage points with its 95% bootstrap interval over calls, and recall on the 21 frustrated one-word turns.
Figure 9: The four WavLM probes on the six, the 24 and all 30 calls: trained on IEMOCAP with native and band-matched audio, scored at the acted cut-off (filled) and with a threshold chosen leaving one call out (hollow), and trained on the 30 calls with a logistic regression and an SVM. Left: balanced accuracy with 95% bootstrap intervals over calls. Right: frustrated one-word turns flagged, of 21 , 18 and 39 ; the band-matched counts average five folds. Reference lines: Gemini 2.5 Flash audio and word count only.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Balanced accuracy of each of the three runs, scored on its own against the human labels; the marker shows the majority vote of the three. Standard deviations range from 0.4pp4.4pp .
Model
Identifier
Access
Weights
Sampling
Gemini 2.5 Flash
gemini-2.5-flash
Vertex AI
–
temperature 1.0
Gemini 2.5 Pro
gemini-2.5-pro
Vertex AI
–
temperature 1.0
Gemini 3.8 Flash
gemini-3.8-flash
Vertex AI
–
temperature 1.0
Qwen2.5-Omni 7B
ggml-org/Qwen2.5-Omni-7B-GGUF
llama.cpp
Q8_0
temperature 1.0
Voxtral Mini 3B
mistralai/Voxtral-Mini-3B-2507
Transformers
bf16
temperature 1.0
Appendix
Table 5: How each model is run.
Turns
Inconclusive judgements per turn
None
183
One
28
Two
9
Three
4
Final label
Appendix
Table 6: Inconclusive judgements on the six clinical calls, each patient turn labelled by all three annotators.
Clinical, 150 turns
IEMOCAP, 9980 turns
Median turn length
4 words
9 words
Turns of 3 words or fewer
47.3%
18.7%
Single-word turns
31.3%
6.5%
A bare yes / no / okay
20.0%
1.4%
Frustrated turns of 3 words or fewer
38.7%
15.0%
Appendix
Table 7: The 150 clinical evaluation turns against the whole of IEMOCAP (all 10039 utterances except the 59 with no words, which are breathing or laughter only: 9980 turns), on the same measures. The two are different regimes, not two samples of one.
Figure 11: Comparison of real against acted, as differences with 95% intervals. Left of zero means the real calls are harder. Rows below the rule pool across models and are the primary comparison; a green interval excludes zero.
Figure 12: The acted corpus as recorded and through the full telephony chain, on 90 of the 150 acted turns ( 45 frustrated, 45 not frustrated). The effect is small and changes sign between models, so the clinical–acted gap cannot be attributed to channel bandwidth.
Estimate
Over turns
Over calls
Audio − text, Gemini 2.5 Flash
+4.7
[ −2.0 , +11.3 ]
[ −2.6 , +10.9 ]
Audio − text, Gemini 2.5 Pro
+3.3
[ −5.3 , +12.0 ]
[ −6.2 , +11.9 ]
Audio − text, Qwen2.5-Omni 7B
−12.7
[ −22.0 , −3.3 ]
[ −19.5 , −7.0 ]
Audio − text, frontier pair
+4.0
[ −2.3 , +10.3 ]
[ −2.4 , +9.3 ]
Cleared but frustrated, text − audio
+20.2
—
[ +6.3 , +30.7 ]
One-word recall, Gemini 2.5 Pro audio
61.9
—
[0.0, 84.6]
Appendix
Table 8: Intervals that resample turns against intervals that resample whole calls, on the clinical set. Differences are audio minus text in percentage points; the clearance row is the transcript’s share of cleared turns that were frustrated minus audio’s, for the frontier pair. 95% bootstrap intervals, 10000 draws each.
Figure 13: One-word turns: not one transcript-only condition beats a coin flip. Every model and input, with 95% Wilson intervals against the chance line. Six transcript conditions, none excluding 50% .
Figure 14: Silent failures concentrate in short turns for the transcript condition. Turns wrongly cleared, by turn length. The transcript condition’s failures concentrate in the shortest turns.
Corpus
Model
Input
Bal. acc.
95% CI
Recall (count)
Excludes chance
Clinical
Gemini 2.5 Flash
Text
54.8
[47.5, 64.5]
9.5 (2/21)
Clinical
Gemini 2.5 Pro
Text
54.8
[47.5, 64.5]
9.5 (2/21)
Clinical
Qwen2.5-Omni 7B
Text
47.1
[37.4, 57.8]
9.5 (2/21)
Acted
Gemini 2.5 Flash
Text
51.2
[41.3, 62.5]
11.1 (2/18)
Acted
Gemini 2.5 Pro
Text
53.4
[44.1, 64.4]
11.1 (2/18)
Acted
Qwen2.5-Omni 7B
Text
52.4
[40.5, 65.0]
22.2 (4/18)
Appendix
Table 9: Every condition on one-word turns, the bucket the short-turn claims rest on. Balanced accuracy with its 95% Wilson interval, and recall as the share of frustrated one-word turns the condition flagged ( 21 such turns on the clinical set, 18 on the acted one). A tick marks the one condition whose interval excludes chance. The audio range quoted in Section 5.2 spans both models and corpora: its floor is the open model on clinical audio, which sits below chance, and its ceiling is Gemini 2.5 Pro on acted audio.
Clinical
Acted
Turn length
n
frustrated / not
n
frustrated / not
1 word
47
21 / 26
41
18 / 23
2–3
24
8 / 16
20
4 / 16
4–5
10
5 / 5
11
5 / 6
6–10
34
17 / 17
28
14 / 14
11–20
22
16 / 6
24
14 / 10
Appendix
Table 10: Turn counts by utterance length for both evaluation sets, with the frustrated and not-frustrated split in each bucket.
Input
Model
Clinical
Acted
True rate
39.4
60.8
Text
Gemini 2.5 Flash
72.7
74.9
Text
Gemini 2.5 Pro
88.1
77.3
Text
Qwen2.5-Omni 7B
50.9
70.8
Audio
Gemini 2.5 Flash
66.9
78.5
Audio
Gemini 2.5 Pro
32.9
58.5
Appendix
Table 11: Spread of the predicted-frustration rate across the six turn-length buckets, in percentage points, against the spread of the true rate. Larger than the true spread means a condition varies with length more than frustration itself does.
Figure 15: Balanced accuracy by turn length, with 95% Wilson intervals, for every model and input. Tick labels give the frustrated and not-frustrated counts in each bucket: the scarcer of the two sets the interval width, which is why some larger buckets are less precisely measured than smaller ones.
Figure 16: Text errors are almost all misses on short turns, split into missed frustration and false alarms. On one-word clinical turns the three transcript conditions produce 57 misses ( 19 each) and 4 false alarms, all from the open model: it is not judging those turns and getting them wrong, it is declining to judge them, and in a binary decision declining reads as calm.
Figure 17: Direction of error on the 55 short turns. Four of six model–modality conditions err almost exclusively by missing real frustration. Gemini 2.5 Pro and the open model on audio err in both directions about equally. Neither failure mode is usable for monitoring.
Figure 18: The clearing gap is neither a threshold artefact nor the product of one call. Left : sweeping the clearing threshold τ leaves text between 37.5% and 40.0% wrong, so loosening τ makes text clear more turns without making it safer; audio rises from 17.8% to 31.7% , closing much of the gap as the criterion weakens. Right : holding out each call in turn moves neither condition far from its full-sample value, shown dashed.
Figure 19: Balanced accuracy for every model and input on both corpora, with 95% Wilson intervals and the number of turns on each bar. Interval widths differ because a proportion’s variance peaks at one half, so cells nearer chance are measured less precisely.
Figure 20: The paired evidence: turns exactly one input answered correctly, with the count both answered correctly beneath. The two inputs score identical turns, so this is the comparison the pooled difference summarises.
Figure 21: Left : of the 19 turns only audio gets right, 11 are frustrated; of the 15 turns only text gets right, 9 are calm. Right : a simple vote across the models and inputs reaches 66.0% , against an oracle ceiling of 78.0% from picking the better input for each turn.
Run
Model
Input
Bal. acc. [95% CI]
Change [95% CI]
Frustrated
Not frustrated
One-word
One-word, other calls
Patient turn only
Gemini 2.5 Flash
Text
66.7 [59.4, 73.1]
–
36/75
64/75
2/21
0/9
Gemini 2.5 Pro
Text
64.0 [56.4, 70.8]
–
37/75
59/75
2/21
0/9
Qwen2.5-Omni 7B
Text
65.3 [57.8, 71.9]
–
37/75
61/75
2/21
0/9
Gemini 2.5 Flash
Audio
71.3 [64.1, 77.4]
–
41/75
66/75
5/21
1/9
Gemini 2.5 Pro
Audio
67.3 [59.4, 74.1]
–
49/75
52/75
13/21
2/9
Qwen2.5-Omni 7B
Audio
52.7 [44.8, 60.3]
–
36/75
43/75
8/21
2/9
Appendix
Table 12: The agent’s preceding turn added as context, on the clinical set, beside the main results on the patient turn only. Balanced accuracy with its 95% Wilson interval; the paired change from the main results, with its paired-bootstrap interval; frustrated and not-frustrated turns classified correctly, of 75 each; and recall on the 21 frustrated one-word turns, overall and on the 9 outside the call that holds the other 12 .
Model
Patient turn only
With agent’s preceding turn
Gemini 2.5 Flash
+4.7 [ −2.0 , +11.3 ]
+0.0 [ −6.0 , +6.0 ]
Gemini 2.5 Pro
+3.3 [ −5.3 , +12.0 ]
+6.7 [ −1.3 , +14.7 ]
Qwen2.5-Omni 7B
−12.7 [ −22.0 , −3.3 ]
−6.0 [ −14.0 , +2.0 ]
Appendix
Table 13: Audio minus transcript in balanced accuracy (pp) on the clinical set, on the patient turn only and with the agent’s preceding turn, with paired-bootstrap 95% intervals.
Figure 22: Supplying preceding patient audio degrades short-turn accuracy while leaving long turns intact; text moves in the opposite direction and gains little.
Figure 23: Does hearing the voice help? The modality comparisons that matter, as differences with 95% intervals. Solid bars exclude zero.
Figure 24: The probes and the audio-free baselines on both test sets, with 95% intervals and the number of turns on each bar. Left : training and test split by dialogue, the arrangement Table 14 reports. Right : split by speaker, which trains on more turns but tests the acted side on only 42 .
Figure 25: More training data turns the probes from no better than word count into clearly better. Probe advantage over counting words, trained on the labelled 150 against the full acted pool. The two 150 -turn rows use a deliberately conservative interval: that run did not save per-item predictions, so its probe and baseline cannot be compared on matched turns. The pool rows are properly paired.
Features
Classifier
Dialogue-disjoint
Session rotation
Acted
Clinical
Acted / Clinical
WavLM, band-matched
SVM
77.4
68.0
78.0 / 74.0 ± 0.9
WavLM, native
SVM
76.1
70.0
73.4 / 71.3 ± 2.7
WavLM, native
Logistic regression
71.3
60.0
66.6 / 64.8 ± 4.0
WavLM, band-matched
Logistic regression
66.6
60.0
64.0 / 62.7 ± 4.0
Word count
Logistic regression
62.3
58.0
62.3 / 57.9 ± 0.3
Appendix
Table 14: Generalisation from acted IEMOCAP speech to real clinical speech. All values are balanced accuracy (%), so chance is 50%. Models were trained on the larger IEMOCAP pool. Dialogue-disjoint evaluation uses a single held-out acted set and clinical set ( n=150 each; ntrain=1,810 ). Session-rotation evaluation pools predictions from five held-out-session folds for the acted set and reports the mean and standard deviation across the five models for the same clinical set.
Features
Classifier
n=150
n=1,810
Δ
WavLM, band-matched
SVM
56.7
68.0
+11.3
WavLM, native
SVM
66.7
70.0
+3.3
WavLM, native
Logistic regression
56.7
60.0
+3.3
WavLM, band-matched
Logistic regression
61.3
60.0
-1.3
Word count
Logistic regression
60.7
58.0
-2.7
Appendix
Table 15: Effect of acted training-set size on balanced accuracy for the same held-out clinical test set ( n=150 ). The smaller-data condition uses 150 acted training turns; the pooled condition uses 1,810 dialogue-disjoint acted turns. Δ denotes the change in percentage points.
Acted threshold
Held-out-call threshold
Bal. acc.
One-word
Bal. acc.
One-word
Band-matched, fold 1
74.0
11/21
72.0
9/21
Band-matched, fold 2
72.7
10/21
71.3
9/21
Band-matched, fold 3
74.0
14/21
72.0
15/21
Band-matched, fold 4
74.0
16/21
76.0
18/21
Band-matched, fold 5
75.3
19/21
78.7
19/21
Appendix
Table 16: WavLM SVM probes on the 150 clinical turns with the threshold set on acted speech or on the other clinical calls (leave one call out). Balanced accuracy and the frustrated one-word turns flagged, of 21 . Band-matched features use the five session-rotation folds; the native-band row is a single model.
Figure 26: Reliability of eGeMAPS parameters against the amount of speech the estimate is computed from. Lines are the IEMOCAP window sweep; squares mark the clinical calls at the median one-word duration, where the two corpora agree. Level parameters are reliable from half a second, variability parameters are not. Shaded band marks the one-word range, 0.32s1.18s . One-way random-effects ICC(1); 45 source utterances, 15 windows per utterance at each length.
Figure 27: Two ways frustration escapes the transcript, both mid-call and both from real calls. Call C : the patient answers “No.” to a question about pain and “No.” again eighteen turns later, and the two carry opposite labels, so the same single word is both. Call D : a longer answer whose wording reads as a mild dismissal is labelled frustrated, and both frontier models get it right from the audio while both call it not frustrated from the transcript. Agent prompts are elided where long; patient turns are complete and verbatim.
Figure 28: The first turn in each call that annotators labelled frustrated, and whether each condition detected it. Four of the six onsets are five words or fewer.
Figure 29: Detection timeline. Each row is one call, each block a patient turn coloured by its human label; the rule marks the first frustrated turn and the markers show where each condition first returns frustrated —audio above the row, text below. In the top call both audio conditions fire on the onset turn while both text conditions fire seven turns later. Gaps between blocks are turns outside the balanced sample, so a first-flag position is first among sampled turns rather than first in the call.
Figure 30: Frustrated one-word turns flagged, out of 39 in the 30 calls, for each model and input and for the two WavLM probes trained on the 30 calls, with 95% bootstrap intervals over calls. Beside each bar: the split between the six and the 24 calls, and false alarms among the 278 one-word turns that were not frustrated. The dashed line marks the three turns every transcript flags.
Figure 31: Audio minus transcript balanced accuracy for the five models on the six, the 24 and all 30 calls, under the input-specific prompts (filled) and the identical prompt (hollow), with 95% bootstrap intervals over calls.
Measure
Six calls, 3 models
30 calls, same 3 models
3.8 Flash ⋅ Voxtral, 30 calls
Share of turns of 1–2 words
36.7% of the balanced 150
53.4% ; 57.8% on the 24
—
One-word transcript balanced accuracy
47.1–54.8
49.9–53.8; lower bounds 44.6–50.0
53.8 ⋅ 53.7
Frustrated one-word turns flagged by the transcript
2 of 21 for each model, the same 2
3, 3 and 4 of 39
3 of 39 each; all five share the same 3
One-word audio balanced accuracy
Flash 60.0, Pro 61.7, Qwen 42.1
Flash 59.9 [53.3, 63.9], Pro 65.6 [51.9, 74.7], Qwen 49.8
54.6 [49.7, 59.8] ⋅ 50.0
Word count only
58.0 [50.5, 67.5]
57.3 [52.8, 62.0]; Qwen audio 55.1
Voxtral audio 54.4
Spread of the transcript’s predicted rate across lengths
50.9–88.1 pp against true 39.4
On the 24: 26.3–30.9 pp against true 21.2
17.3 ⋅ 40.4
Appendix
Table 17: Measures of Sections 5.1 – 5.3 on the six calls and the 30 calls. Balanced accuracy in percent unless stated; intervals are 95% bootstrap intervals over calls, so a few six-call values differ slightly from the main text. Clearing by length counts one-word turns, where Section 5.2 counts turns of one or two words. The prediction spread is shown on the 24 added calls alone.
Users convey frustration in what they say and in how they say it, and voice-native models, which hear speech instead of an ASR transcript, can register both. Existing evaluations read unchanged performance as robustness without checking perception. Following Brunswik's lens model, we measure on identical audio whether frustration is expressed, perceived, and utilized. We cross frustrated wording with frustrated voice in a within-item 2x2 design over 353 tasks and release the 1,412 clips as VoxAffect. Across 8 models, outputs look robust, with no detectable vocal effect on accuracy for 7. With neutral wording, all 8 label the frustrated voice frustrated in at most 3.1% of clips, although 7 of them detect acted anger in 73-100% of clips and human listeners classify the voice correctly in 65.3% of judgments. Yet linear probes decode the frustrated voice from the audio encoders of the 2 open-weight models, with paired accuracy 0.67 and 0.69, and in proprietary models too the voice raises perceived frustration once the words agree. The models thus hear the frustrated voice but do not listen to it, whereas their invariance to frustrated wording, perceived in 40-75% of clips, is genuine robustness. Our work is openly accessible at https://github.com/Astarojth/hear-not-listen.
Voice agents increasingly require reliable tool use from speech, whereas prominent tool-calling benchmarks remain text-based. We study whether verified text benchmarks can be converted into controlled audio-based tool calling evaluations without re-annotating the tool schema and gold labels. Our dataset-agnostic framework uses text-to-speech, speaker variation, and environmental noise to create paired text-audio instances while preserving the original dataset annotations. Based on extensive evaluation of 7 omni-modal models on audio-converted versions of Confetti and When2Call, our framework demonstrates that the performance is strongly model- and task-dependent: Gemini-3.1-Flash-Live obtains the highest Confetti score (70.4), whereas GPT-Realtime-1.5 performs best on When2Call (71.9). On Confetti, the text-to-voice gap ranges from 1.8 points for Qwen3-Omni to 4.8 points for GPT-Realtime-1.5. A targeted analysis of failure cases demonstrates that degradations most often reflect misunderstandings of argument values in the speech. Considering real-world deployment scenarios, we further report text-only results, an ambiguity-based reformulation stress test, and a reference-free LLM-as-judge protocol validated against human preferences. Notably, we find that open-source Qwen3 judges with at least 8B parameters exceed 80% agreement with proprietary judges, supporting privacy-preserving evaluation. Overall, our framework provides a verifiable and reproducible first-stage diagnostic that complements purpose-built audio corpora.
Clinical voice agents are now deployed in routine care, where real patients do not wait their turn: they interrupt. These systems typically use a cascaded architecture (speech-to-text -> LLM -> text-to-speech), so when a patient cuts the agent off mid-utterance, clinically required content can be lost even when the model handles cooperative transcripts well. Yet clinical conversational-AI benchmarks almost universally assume patients wait for the agent to finish, missing interruption-induced loss of required content. We present a transcript-based evaluation of interruption recovery, adapting conversation-analytic overlap categories into three operational types (recognitional, competitive, transitional sub-unit) and testing four deployment-oriented, non-reasoning LLM configurations across four cells spanning history-taking (information gathering) and FAQ (information provision), scored on whether the agent preserves the clinically required content. In the gathering cells, target-question failure varied across models; in the provision cells, where arms are directly comparable, failure rose for every model. Rankings differ across cells, and competitive FAQ interruption produced 30/30 provision-coverage failures for all four models (Wilson 95% CI: 88.6-100.0%; baseline 0/30 for three, 4/30 for Llama). A brief apology marker ("sorry to interrupt") shifts recovery by tens of percentage points, inconsistently across models, and for one it reduces recovery. Interruption robustness therefore cannot be a single score: evaluation must be content-grounded, reported per cell, and matched to the deployment's interruption profile.