cs.AINov 9, 2026

AuDiR: Diagnosing Audio Reasoning with Rule and Percept Interventions

Authors: Chengwei Wei, Yi Ma, Nancy F. Chen

Organizations: Institute of Advanced Intelligence and Computing (IAIC) and the Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore. · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore.

Abstract

Answer accuracy alone does not distinguish failures in acoustic perception, rule knowledge, and rule application. AuDiR (Audio Diagnostic Reasoning) examines these distinctions through controlled input interventions. Each item contains an audio clip, a multiple-choice question, a task-general rule, and a textual description of the acoustic observations needed to answer. Three conditions progressively supply the rule and observations; a fourth removes the audio while retaining the question and rule. The first correct diagnostic condition assigns an operational response label, without identifying a unique internal cause. The current benchmark contains 201 items in 18 families drawn from 9 sources, with answer references documented separately from percept extraction. In a preliminary evaluation of Gemini-3.8-flash, accuracy is 29.3% without supplied information, 31.3% with the rule, and 98.5% with the observations also supplied. No-audio accuracy is 23.9%, with a confidence interval containing the 25.0% chance rate. A family-level knowledge probe associates incomplete rule recall with larger gains from rule provision. These results show the benefit of supplying acoustic observations as text, while incomplete validation, possible recognition shortcuts, and non-monotonic responses limit stronger failure-attribution claims.

Figures & tables

Explore similar work

CardsList
  1. Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

    Sep 9, 2026Yupei Li, Qiyang Sun, Mohamed Mady +4Audio-Language Model EvaluationAudio Reasoning

  2. AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA

    Apr 23, 2026Tasnim Kabir, Dmytro Kurdydyk, Aadi Palnitkar +3Audio QAAudio Reasoning

  3. AudioProcessBench: Benchmark for Identifying Process Errors in Audio-Grounded Reasoning

    Jun 7, 2026Xiangyu Zhao, Junyu Yan, Yaling Shen +7Audio-Language Model EvaluationReasoning Evaluation