Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.
Figures & tables
Figure 1: Existing spoken dialogue benchmarks often allow models to rely on transcript shortcuts, under-utilizing acoustic evidence that carries crucial paralinguistic information.
Figure 2: ContraTalk Construction via Cross-Modal Disagreement and Human Verification.
Abbr.
Category
Description
# Conflict/Consistent
IB
Interaction Behavior
Turn-taking, hesitation, overlap, or interruption
68/34
ES
Emotion State
Affect such as anger, calmness, stress, or frustration
75/38
CI
Conversational Intent
Implied communicative goal or intention
67/34
DA
Dialogue Act
Utterance function, e.g., request or refusal
56/28
SS
Social Stance
Social attitude, e.g., politeness or authority
67/34
Ovr.
Overall
Total number of benchmark examples
333/168
Table 1: Benchmark composition. Number of examples in each discourse-level category.
Figure 3: Structured acoustic evidence used by the agentic-style reasoner. Audio Twin converts speech into time-aligned, LLM-compatible evidence entries that expose paralinguistic and interactional cues relevant to the question.
Accuracy ↑
Mislead Rate ↓
System
IB
ES
CI
DA
SS
Ovr.
IB
ES
CI
DA
SS
Ovr.
Text-only LLMs
DeepSeek V3.1
42.6
34.7
37.3
42.9
31.3
37.5
33.8
41.3
44.8
33.9
38.8
38.7
Nova 2 Lite
42.6
37.3
35.8
41.1
40.3
39.3
29.4
44.0
43.3
32.1
26.9
35.4
Haiku 4.5
36.8
26.7
28.4
44.6
31.3
33.0
47.1
53.3
50.7
37.5
34.3
45.0
Sonnet 4.5
57.4
40.0
47.8
50.0
44.8
47.7
29.4
41.3
38.8
32.1
29.9
34.5
Table 2: Shortcut diagnostic. We report category-wise accuracy and mislead rate across text-only LLMs, direct Audio-LLMs, and Audio Twin reasoning systems. Mislead rate measures the selection of the transcript-biased surface distractor.
Accuracy ↑
Accuracy ↑
System
IB
ES
CI
DA
SS
Ovr.
System
IB
ES
CI
DA
SS
Ovr.
Text-only LLMs
Text-only LLMs
DeepSeek V3.1
82.4
94.7
94.1
92.9
94.1
91.7
Haiku
88.2
94.7
97.1
96.4
88.2
92.9
Nova 2 Lite
85.3
92.1
94.1
89.3
97.1
91.7
Sonnet
91.2
97.4
100
100
94.1
96.4
Opus
97.1
97.4
100
100
97.1
98.2
Audio-LLMs
Audio Twin Reasoning
Table 3: Consistent-split performance. We report category-wise accuracy on examples where transcript-supported and audio-grounded answers are aligned. No mislead rate is reported because no transcript-biased distractor is defined for this split.
Text-only
Speech-grounded
System
Conf. Acc. ↑
Mis. ↓
Cons. Acc. ↑
Conf. Acc. ↑
Mis. ↓
Cons. Acc. ↑
Direct Audio-LLMs: Text vs. Audio
StepAudio-R1
42.3
34.5
91.7
46.8
32.1
89.9
StepAudio-2
34.2
37.8
83.9
39.3
29.7
69.6
Qwen3-Omni
42.9
40.5
91.1
40.5
34.2
83.3
KimiAudio
43.2
28.8
84.5
41.1
30.3
78.6
Table 4: Paired text-only and speech-grounded inference. Results from Tables 2 and 3 are reorganized into matched text-only and speech-grounded pairs, enabling direct comparison across conflict and consistent splits.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Overview of the ContraTalk benchmark construction pipeline. We first capture text-only surface interpretations, then identify disagreement regions, generate question-answer pairs, and perform multiple stages of quality assurance.
System
Conflict Acc.
Conflict Mislead
Consistent Acc.
Text-only LLMs
DeepSeek V3.1
37.5 [32.4, 42.7]
38.7 [33.9, 43.5]
91.7 [87.1, 95.7]
Nova 2 Lite
39.3 [33.5, 45.4]
35.4 [30.1, 40.7]
91.7 [87.3, 95.5]
Haiku 4.5
33.0 [27.8, 38.3]
45.0 [39.1, 50.9]
92.9 [89.0, 96.2]
Sonnet 4.5
47.7 [41.0, 54.1]
34.5 [28.1, 41.3]
96.4 [93.4, 98.8]
Opus 4.7
45.3 [39.4, 51.3]
36.9 [31.1, 43.0]
98.2 [96.0, 100.0]
Appendix
Table 5: Dialogue-level bootstrap 95% confidence intervals for aggregate model performance. Entries are point estimates with 95% percentile confidence intervals in brackets, reported as percentages. The conflict split contains 333 questions from 113 dialogues; the consistent split contains 168 questions from 75 dialogues. Split-specific dialogue counts do not sum to the total number of dialogues because a dialogue may contribute questions to both splits. Mislead rate is defined only for conflict cases. AT denotes Audio Twin.
Table 6: Human verification summary for ContraTalk . We manually review 230 conflict cases and 120 consistent cases, covering 350 of the 501 retained benchmark examples. Conflict cases are checked for answerability, speech-grounded evidence, and surface-distractor quality, while consistent cases are checked to ensure that transcript-supported and speech-grounded interpretations are aligned.
Model label
Checkpoint
Text
Audio
Audio Twin
Public size information
Haiku 4.5; Haiku 4.5 (agentic-style)
us.anthropic.claude-haiku-4-5-20251001-v1:0
Yes
No
Yes
Not publicly disclosed
Sonnet 4.5; Sonnet 4.5 (agentic-style)
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Yes
No
Yes
Not publicly disclosed
Kimi K2 Thinking
moonshot.kimi-k2-thinking
Yes
No
No
1T total, 32B active parameters
AudioFlamingoNext
nvidia/audio-flamingo-next-hf
No
Yes
No
8B parameters
MIMOAudio
XiaomiMiMo/MiMo-Audio-7B-Instruct
Yes
Yes
No
8B parameters
Qwen2.5-Omni
Qwen/Qwen2.5-Omni-7B
Yes
Yes
No
7B parameters
Appendix
Table 7: Model checkpoints and evaluation settings used in the experiments. The Model label column shows the shorthand names used in this work, while Checkpoint shows the models’ checkpoint identifier. Text denotes transcript-only evaluation with (T,q,C) , Audio denotes direct audio-input evaluation, and Audio Twin denotes evaluation with the agentic-style Audio Twin retrieval pipeline. Parameter counts are reported using public information when available. For mixture-of-experts models, total and active parameter counts are reported when available.
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
Audio LLMs deployed in settings such as crisis hotlines and customer support are only useful beyond speech-to-text pipelines if they use acoustic cues rather than the transcript alone. We test whether they prioritise transcript content over paralinguistic cues (emotion, speaker identity, sarcasm) and trace the mechanism when they do. Across six audio LLMs under LISTEN-style conflict conditions, the mean text-dominance rate on mismatched text+audio items is 0.78 (0.86 excluding sarcasm). In the most text-dominant screened model, probing shows speaker identity remains linearly decodable at ceiling at the language model's input, so the failure is not perceptual. Back-patching, reported to repair lexical text dominance, has no measurable effect here; but it also fails on its own source task in our replication (n=300), so we treat its null as uninformative rather than as evidence of non-transfer. Attention knockout partially restores audio-grounded output by blocking the claim from propagating through intermediate tokens during open-ended generation, whereas blocking the answer position's direct connection to the conflicting claim has no effect. The claim's influence therefore travels through an intermediate relay rather than a single severable edge. The effect appears only in free-form generation and not in closed-set multiple choice, suggesting that benchmarks relying on that format may be systematically blind to interventions of this kind. Code and data are publicly available at https://github.com/AdvayRoongta/audio-llm-mech-interp.
Ankur Banerjee, Vishnu Guntakala, Danity Pike +3
Independent Researcher · Brown University · University of Birmingham
Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks, created with controlled speech synthesis to intentionally mismatch transcript claims and speaking style, enabling direct measurement of speech paralinguistic understanding. Evaluation of a diverse set of Audio LLMs reveals consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied (incorrect) answers. To understand the cause of this gap, we perform layer-wise probing and find that (i) paralinguistic cues can degrade in deeper encoder layers and at the encoder--LLM interface, and (ii) even when such cues are available in audio tokens, the language model frequently ignores them. To address these problems, we propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt, and pair it with Direct Preference Optimization (DPO) to explicitly prefer acoustically supported options over language-implied alternatives. These methods substantially improve Audio LLM paralinguistic understanding, improving Audio Flamingo 3 from 17.40% to 65.20% on VoxParadox, and from 37.74% to 54.78% on MMSU paralinguistic subset. Our project page is available at https://voxparadox.github.io/.
Jiacheng Pang, Ashutosh Chaubey, Mohammad Soleymani
Institute for Creative Technologies, University of Southern California, Los Angeles, USA
Spoken dialogue state tracking recovers slot-value pairs from speech, where ASR errors concentrate in entity values and persist across turns, making it both a generation and an editing problem. A strong per-turn text editor corrects much of this but, operating on the transcript alone, leaves three recoverable errors: a value predicted inconsistently across turns, an omitted slot, and a value the audio does not support. We present AVERT, which scores each candidate value by combining cross-turn agreement with a trained audio-conditioned verifier and resolves the three error types with three operators, vote, add, and swap, each restricted to the slots where its error is common. On SpokenWOZ, a base speech-LLM reaches 33.04 JGA, a text editor 38.34, and AVERT 40.13, without retraining either. This is in the range of a 1B end-to-end system that consumes the full spoken history (39.32), though AVERT uses two 1B decoders rather than one. The audio verifier contributes a statistically significant gain, and restricting each operator to a selected slot subset matters: removing it lets unrestricted voting overwrite correct categorical values and fall below the editor.