Speech Emotion Recognition (SER) models and Audio LLMs fail when vocal tone contradicts textual semantics. Extensive training on congruent data engenders a rigid semantic prior, causing "Semantic Dominance", a severe text bias during cross-modal conflicts. We introduce the ASPIRE benchmark of adversarial audio-text pairs across four conflict types, alongside two metrics: Semantic Overconfidence Penalty (SOP) and Latent Decoupling Degree (LDD). Unlike fusion methods that merely re-weight collapsed features, our Acoustic Conflict Resolution Network (ACR-Net) uses Cross-Modal Attention and Contrastive Decoupling Loss to disentangle contradictory representations into orthogonal latent spaces. Experiments show existing models fail under polarity conflicts due to extreme semantic overconfidence. Conversely, ACR-Net preserves acoustic fidelity, maintaining high LDD to drive SOP near zero, achieving superior Acoustic Accuracy (ACC) in both adversarial and standard scenarios.
Figures & tables
Figure 1: An illustration of Semantic Dominance in emotional conflict. Traditional multi-modal models blindly trust the semantic content, whereas our approach detects the acoustic-semantic contradiction.
Conflict Type
Retention
Characteristics &
(Example)
Rate
Synthesis Difficulty
Polarity Opposite (Sad + Happy)
45%
Hardest. Extreme polarity clash; text semantics heavily overpower acoustic prompts.
Arousal Opposite (Neutral + Angry)
84%
Easiest. Neutral acoustics are stable, but high-arousal acoustics easily collapse.
Bipolar. Simulates suppression. Easy for low-arousal text, hard to mask positive text.
Table 1: Structural Categorization of Conflict Pairings in ASPIRE.
Representation Strategy
ACC (%) ↑
SOP ↓
LDD ↑
Naive Concatenation
41.2
0.824
0.108
Advanced Fusion Mechanisms
+ Tensor Fusion [ 23 ]
43.8
0.776
0.115
+ MCR (Re-weighting) [ 16 ]
48.5
0.628
0.242
Explicit Decoupling Interventions (Ablation)
+ CMA (w/o Decouple Loss)
53.0
0.534
0.334
Table 2: Comparison of Representation Strategies on the ASPIRE Benchmark. ACR-Net explicitly disentangles features, achieving the highest ACC while minimizing SOP. The hyperparameter analysis shows the necessity of balanced decoupling.
Table 3: Acoustic Accuracy (ACC %) comparison on standard congruent datasets.
Conflict Type
Model
ACC ↑
SOP ↓
LDD ↑
Polarity Opposite (Sad + Happy)
MCR
35.2
0.612
0.205
FAS
42.5
0.548
0.288
ACR-Net (Ours)
68.4
0.185
0.812
Arousal Opposite (Neutral + Angry)
MCR
65.8
0.315
0.322
FAS
68.4
0.280
0.395
ACR-Net (Ours)
85.2
0.092
0.885
Table 4: Diagnostic Evaluation on the ASPIRE Benchmark. The table details the performance across four distinct structural conflict types. By maintaining a high Latent Decoupling Degree (LDD), ACR-Net effectively suppresses Semantic Overconfidence (SOP) and achieves state-of-the-art Acoustic Accuracy (ACC).
While Audio Large Language Models (Audio LLMs) excel at multimodal understanding, they suffer from text dominance, a bias where models blindly favor text over acoustic evidence, causing hallucinations. However, the internal mechanisms underlying how these models behave when audio and textual inputs contradict each other remain unexplored. In this work, we present the first mechanistic analysis of this phenomenon by tracing the propagation of internal representations across layers. Our investigation reveals three key findings: (i) text dominance is systematically and empirically across models; (ii) while text and audio rely on functionally distinct pathways, they ultimately converge into a shared semantic space in late layers; and (iii) the text pathway does not erase audio information, but rather actively suppresses intact audio representations. Building on these insights, we leverage back-patching, a training-free intervention that routes late-layer audio activations back into earlier layers. This amplifies the audio representations, enabling them to overcome textual suppression. Our evaluation shows that back-patching consistently reduces text dominance, paving the way for mechanistic multimodal alignment under conflict.
Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness), a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven Audio MLLMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding.
Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked. In particular, the effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood. We evaluate audio language models across five semantic and paralinguistic reasoning tasks: entailment, consistency, plausibility, accent drift, and accent restraint. Collectively, these tasks assess a model's ability to reason over spoken audio as the primary evidence source, including whether a textual hypothesis can be inferred, contradicted, or left undetermined by the audio, whether statements align or conflict with spoken content, whether claims are plausible given the discourse, and whether model predictions remain stable or appropriately constrained across accent variation. These findings highlight critical limitations in current audio reasoning evaluations and hope to provide guidance for more robust and equitable ALM design and assessment