Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.
Figures & tables
Figure 1: Existing spoken dialogue benchmarks often allow models to rely on transcript shortcuts, under-utilizing acoustic evidence that carries crucial paralinguistic information.
Figure 2: ContraTalk Construction via Cross-Modal Disagreement and Human Verification.
Abbr.
Category
Description
# Conflict/Consistent
IB
Interaction Behavior
Turn-taking, hesitation, overlap, or interruption
68/34
ES
Emotion State
Affect such as anger, calmness, stress, or frustration
75/38
CI
Conversational Intent
Implied communicative goal or intention
67/34
DA
Dialogue Act
Utterance function, e.g., request or refusal
56/28
SS
Social Stance
Social attitude, e.g., politeness or authority
67/34
Ovr.
Overall
Total number of benchmark examples
333/168
Table 1: Benchmark composition. Number of examples in each discourse-level category.
Figure 3: Structured acoustic evidence used by the agentic-style reasoner. Audio Twin converts speech into time-aligned, LLM-compatible evidence entries that expose paralinguistic and interactional cues relevant to the question.
Accuracy ↑
Mislead Rate ↓
System
IB
ES
CI
DA
SS
Ovr.
IB
ES
CI
DA
SS
Ovr.
Text-only LLMs
DeepSeek V3.1
42.6
34.7
37.3
42.9
31.3
37.5
33.8
41.3
44.8
33.9
38.8
38.7
Nova 2 Lite
42.6
37.3
35.8
41.1
40.3
39.3
29.4
44.0
43.3
32.1
26.9
35.4
Haiku 4.5
36.8
26.7
28.4
44.6
31.3
33.0
47.1
53.3
50.7
37.5
34.3
45.0
Sonnet 4.5
57.4
40.0
47.8
50.0
44.8
47.7
29.4
41.3
38.8
32.1
29.9
34.5
Table 2: Shortcut diagnostic. We report category-wise accuracy and mislead rate across text-only LLMs, direct Audio-LLMs, and Audio Twin reasoning systems. Mislead rate measures the selection of the transcript-biased surface distractor.
Accuracy ↑
Accuracy ↑
System
IB
ES
CI
DA
SS
Ovr.
System
IB
ES
CI
DA
SS
Ovr.
Text-only LLMs
Text-only LLMs
DeepSeek V3.1
82.4
94.7
94.1
92.9
94.1
91.7
Haiku
88.2
94.7
97.1
96.4
88.2
92.9
Nova 2 Lite
85.3
92.1
94.1
89.3
97.1
91.7
Sonnet
91.2
97.4
100
100
94.1
96.4
Opus
97.1
97.4
100
100
97.1
98.2
Audio-LLMs
Audio Twin Reasoning
Table 3: Consistent-split performance. We report category-wise accuracy on examples where transcript-supported and audio-grounded answers are aligned. No mislead rate is reported because no transcript-biased distractor is defined for this split.
Text-only
Speech-grounded
System
Conf. Acc. ↑
Mis. ↓
Cons. Acc. ↑
Conf. Acc. ↑
Mis. ↓
Cons. Acc. ↑
Direct Audio-LLMs: Text vs. Audio
StepAudio-R1
42.3
34.5
91.7
46.8
32.1
89.9
StepAudio-2
34.2
37.8
83.9
39.3
29.7
69.6
Qwen3-Omni
42.9
40.5
91.1
40.5
34.2
83.3
KimiAudio
43.2
28.8
84.5
41.1
30.3
78.6
Table 4: Paired text-only and speech-grounded inference. Results from Tables 2 and 3 are reorganized into matched text-only and speech-grounded pairs, enabling direct comparison across conflict and consistent splits.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Overview of the ContraTalk benchmark construction pipeline. We first capture text-only surface interpretations, then identify disagreement regions, generate question-answer pairs, and perform multiple stages of quality assurance.
System
Conflict Acc.
Conflict Mislead
Consistent Acc.
Text-only LLMs
DeepSeek V3.1
37.5 [32.4, 42.7]
38.7 [33.9, 43.5]
91.7 [87.1, 95.7]
Nova 2 Lite
39.3 [33.5, 45.4]
35.4 [30.1, 40.7]
91.7 [87.3, 95.5]
Haiku 4.5
33.0 [27.8, 38.3]
45.0 [39.1, 50.9]
92.9 [89.0, 96.2]
Sonnet 4.5
47.7 [41.0, 54.1]
34.5 [28.1, 41.3]
96.4 [93.4, 98.8]
Opus 4.7
45.3 [39.4, 51.3]
36.9 [31.1, 43.0]
98.2 [96.0, 100.0]
Appendix
Table 5: Dialogue-level bootstrap 95% confidence intervals for aggregate model performance. Entries are point estimates with 95% percentile confidence intervals in brackets, reported as percentages. The conflict split contains 333 questions from 113 dialogues; the consistent split contains 168 questions from 75 dialogues. Split-specific dialogue counts do not sum to the total number of dialogues because a dialogue may contribute questions to both splits. Mislead rate is defined only for conflict cases. AT denotes Audio Twin.
Table 6: Human verification summary for ContraTalk . We manually review 230 conflict cases and 120 consistent cases, covering 350 of the 501 retained benchmark examples. Conflict cases are checked for answerability, speech-grounded evidence, and surface-distractor quality, while consistent cases are checked to ensure that transcript-supported and speech-grounded interpretations are aligned.
Model label
Checkpoint
Text
Audio
Audio Twin
Public size information
Haiku 4.5; Haiku 4.5 (agentic-style)
us.anthropic.claude-haiku-4-5-20251001-v1:0
Yes
No
Yes
Not publicly disclosed
Sonnet 4.5; Sonnet 4.5 (agentic-style)
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Yes
No
Yes
Not publicly disclosed
Kimi K2 Thinking
moonshot.kimi-k2-thinking
Yes
No
No
1T total, 32B active parameters
AudioFlamingoNext
nvidia/audio-flamingo-next-hf
No
Yes
No
8B parameters
MIMOAudio
XiaomiMiMo/MiMo-Audio-7B-Instruct
Yes
Yes
No
8B parameters
Qwen2.5-Omni
Qwen/Qwen2.5-Omni-7B
Yes
Yes
No
7B parameters
Appendix
Table 7: Model checkpoints and evaluation settings used in the experiments. The Model label column shows the shorthand names used in this work, while Checkpoint shows the models’ checkpoint identifier. Text denotes transcript-only evaluation with (T,q,C) , Audio denotes direct audio-input evaluation, and Audio Twin denotes evaluation with the agentic-style Audio Twin retrieval pipeline. Parameter counts are reported using public information when available. For mixture-of-experts models, total and active parameter counts are reported when available.
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.