In cascaded voice agents, ASR errors can have markedly different downstream consequences: a small transcription error may leave a tool call unchanged or alter its intent or arguments. Conventional word error rate (WER), however, does not capture these differences. We introduce SpeechEvalX, a paired diagnostic protocol that evaluates how task roles affected by ASR errors influence structured tool calls. Controlled edits are matched by source utterance, edit count, and operation, while task-role labels are compared with lexical severity, BERT cosine distance, Semantic-WER, and binary criticality. Across 594 source utterances, joint call accuracy falls from 82.3% with reference transcripts to 71.9-74.9% across three ASR front ends. Number and negation edits cause 21.9-25.7 percentage-point losses, versus 1.4-2.9 points for control edits. Detailed task-role labels improve source-grouped ROC-AUC from 0.73 to 0.78 over binary criticality, showing that downstream impact depends strongly on the task role affected.
Figures & tables
Role
Inclusion rule
Number
Quantity, time, date, ordinal, or identifier aligned to an action value
Entity
Named person, place, organization, product, or account aligned to an action value
Negation
Overt cue whose scope changes whether or how the action is executed
Intent cue
Predicate or phrase that selects the call or no-call intent
Other slot
Schema-grounded argument value not covered above
Function word
Closed-class token outside every task-bearing span
Table 1: Operational task-role taxonomy for span annotation.
Reference
Corrupted
Interpretation
1
1
Preserved success
1
0
Harmed pair (H)
0
1
Rescued pair (R)
0
0
Persistent failure
Table 2: Paired action outcomes. Both paths use the same original reference.
Input
WER
Tool
Arg-EM
No-call
JCA
Reference
–
91.3
89.5
88.9
82.3
Whisper large-v3
6.3
86.9
85.1
85.2
74.9
SenseVoice-Small
7.2
86.1
83.9
83.3
73.2
Paraformer
8.1
85.2
83.3
81.5
71.9
Table 3: ASR hypotheses on benchmark audio: 594 source utterances (%). Tool is accuracy on call-reference cases; Arg-EM is conditional on a recoverable correct tool.
Role
H
R
Lc
95% CI
Other non-critical
5
2
1.4
[-0.5, 3.8]
Function word
8
2
2.9
[0.5, 5.7]
Other slot value
24
4
9.5
[5.7, 13.8]
Intent cue
30
4
12.4
[8.1, 17.1]
Entity
40
4
17.1
[12.4, 22.4]
Number
51
5
21.9
[16.7, 27.6]
Table 4: Matched interventions on the same 210 sources. Lc is the net JCA loss in percentage points; intervals are source-bootstrap 95% CIs.
Model
Added information
ROC-AUC
AP
B0
Lexical controls
0.62
0.48
B1
+ BERT embedding cosine
0.68
0.55
B2
+ Semantic-WER
0.70
0.58
B3
+ Critical-span flag
0.73
0.62
B4
+ Detailed task roles
0.78
0.68
Primary contrast: B4–B3
+0.05
+0.06
Table 5: Diagnostic value in source-grouped five-fold failure prediction. Feature sets are cumulative; the primary contrast is B4 versus B3.
Role
Primary
Sensitivity
Other non-critical
1.4
1.0
Function word
2.9
3.3
Other slot value
9.5
8.6
Intent cue
12.4
10.5
Entity
17.1
15.7
Number
21.9
19.0
Table 6: Paired JCA loss across downstream agents (percentage points).
Category
Count
Share (%)
Call/no-call decision
17
7.9
Wrong tool
42
19.4
Argument structure
48
22.2
Argument value
103
47.7
Malformed output
6
2.8
Table 7: Final-output error categories among 216 harmed controlled pairs.