cs.SDNov 11, 2026

Right Answers, Inconsistent Evidence: Source-Level Counterfactual Auditing of Large Audio-Language Models

Authors: Yi Zhang, Yi Li, Hongwei Du

Organizations: School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China · University of Windsor, Windsor, Canada

Abstract

A large audio-language model (LALM) can answer "is a cat present?" about a mixture correctly, and its confidence can drop when the cat is removed, yet this does not show that it can tell the cat's presence from its absence. We introduce TRACE. It deletes the queried source and each rival in turn and asks whether every view still containing that sound outscores the view with it removed. Across 8 LALMs and 3,456 controlled mixtures, 6.3% of correct answers fail this ordering: the target-absent view scores at or above some target-present view, a conflict no shared threshold repairs. A target-only score check misses 42.5% of them. The share rises from 4.9% to 8.4% as target SNR falls from +5 to -5 dB, even on units correct at every level, and survives loudness and re-pairing controls. Accuracy certifies the answer; TRACE audits whether it orders presence above absence. Code and data: https://github.com/yikise/TRACE.

Figures & tables

Explore similar work

CardsList
  1. TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence

    Oct 15, 2026Giovana Morais, Yujin Kim, Atilay Kucukoglu +1Audio-Language Model EvaluationAudio-Language Models

  2. Robustness assessment of large audio language models in multiple-choice evaluation

    Oct 6, 2025Fernando López, Santosh Kesiraju, Jordi LuqueAudio-Language Model EvaluationAudio QA