AuDiR: Diagnosing Audio Reasoning with Rule and Percept Interventions
Authors: Chengwei Wei, Yi Ma, Nancy F. Chen
Organizations: Institute of Advanced Intelligence and Computing (IAIC) and the Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore. · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore.
Answer accuracy alone does not distinguish failures in acoustic perception, rule knowledge, and rule application. AuDiR (Audio Diagnostic Reasoning) examines these distinctions through controlled input interventions. Each item contains an audio clip, a multiple-choice question, a task-general rule, and a textual description of the acoustic observations needed to answer. Three conditions progressively supply the rule and observations; a fourth removes the audio while retaining the question and rule. The first correct diagnostic condition assigns an operational response label, without identifying a unique internal cause. The current benchmark contains 201 items in 18 families drawn from 9 sources, with answer references documented separately from percept extraction. In a preliminary evaluation of Gemini-3.8-flash, accuracy is 29.3% without supplied information, 31.3% with the rule, and 98.5% with the observations also supplied. No-audio accuracy is 23.9%, with a confidence interval containing the 25.0% chance rate. A family-level knowledge probe associates incomplete rule recall with larger gains from rule provision. These results show the benefit of supplying acoustic observations as text, while incomplete validation, possible recognition shortcuts, and non-monotonic responses limit stronger failure-attribution claims.
Figures & tables
Benchmark
Items
Failure attribution
Audio-necessity check
Answer ground truth
AIR-Bench [ 4 ]
∼ 21k
—
—
human annotation, model-judged
MMAU [ 2 ]
10k
—
noise substitution, per model, reported
human annotation
MMAR [ 1 ]
1,000
post-hoc, by reading 100 failed predictions from one model
noise substitution, per model, reported
curated human annotation
STAR-Bench [ 3 ]
2,353
from aggregate score patterns across task families
caption substitution, per benchmark, used as motivation
TABLE I: AuDiR against existing audio reasoning benchmarks. “Failure attribution” asks whether the benchmark says which step failed, and on what evidence. “Audio-necessity check” asks whether items are tested for being answerable without the audio, and what is done with those that are. Entries reflect what each paper reports; a dash means the paper reports none.
Fig. 1: An AuDiR item illustrated with vehicle speed. The model must recover the percept from the audio and apply the task-general rule to answer the question. The tone trajectory and frequencies are illustrative explanations, not text supplied in the base condition. The gold option is obtained separately by binning the cruise-control speed VS13 records for that pass; it is not estimated from the audio. This separation implements criterion C4 (Section IV-C ).
Condition
Full input
Intervention
① Base
A+Q
No supplied rule or percept
② + Knowledge
A+Q+K
Rule supplied
③ + Percept
A+Q+K+P
Observations also supplied
④ No audio
Q+K
Audio-necessity control
TABLE II: Complete inputs for the four conditions. Q includes the question, choices and fixed givens; P is the textual representation of the acoustic observations.
Source
Audio
Answer reference
License
Fam.
Items
VS13
single vehicles passing a roadside microphone at constant speed
speed held on the vehicle’s cruise control, logged per pass
IEEE, research use, no redistribution
2
24
Sound of Water
water poured into vessels of known geometry
ruler-measured net height and container geometry
MIT
2
20
GuitarSet
solo acoustic guitar, hexaphonic pickup
shipped chord and note transcription
CC BY 4.0
3
36
AudioTime
environmental scenes with several distinct sound events
shipped onset and offset timestamps
open
2
24
MAFAULDA
rotating machinery under controlled fault conditions
tachometer channel recorded alongside the audio
open (UFRJ)
2
24
MMAR
one retained clip, a stone dropped into a well
answer shipped with the benchmark
CC BY-NC 4.0
1
1
TABLE III: Sources currently in use. A source is admitted per pairing, not as a whole, so the same recordings support some target quantities and not others. Answer references are listed by their documented provenance; shipped annotations still require an independence audit. The MMAR item is retained as a flagged case study.
Family
Source
Percept (heard)
Rule (told)
Answer reference
n
Doppler speed
VS13
approach and recede engine-tone frequencies
f=f0c/(c∓v)
cruise-control speed
12
Doppler speed ratio
VS13
both cars’ tone shifts
ratio of two Doppler solutions
cruise-control speed
12
Container height
Sound of Water
empty-container resonant pitch
f=c/(4(H+βR))
ruler-measured net height
10
Taller vessel, fuller vessel
Sound of Water
both containers’ empty-start pitches
f→H , then V from H and the given R
ruler height and diameters
10
Event rate
AudioTime
number of distinct sound events
rate=60N/T
shipped event timestamps
12
Chord quality
GuitarSet
the chord’s interval pattern
stacked thirds → quality
GuitarSet transcription
12
TABLE IV: Current AuDiR composition. Families pair acoustic observations with a target quantity and a stated rule. The reference column records the documented answer source; independence of shipped annotations requires provenance verification. Counts include the flagged MMAR case study and remain subject to revision.
Category
n
Percept type
frequency shift
24
event count
24
rotation tone
24
pitch pattern
24
resonant pitch
20
TABLE V: Distribution over perceptual axes, question forms, and rule domains. Axes organize items by perceptual demand; most currently occur in only one source.
Fig. 2: AuDiR construction and diagnosis. (a) Candidate percept–target pairings are screened and materialized into four inputs. The audio-removed condition ④ probes text-only answerability; current control results are reported in aggregate. Rejection reasons are detailed in Table VI . (b) The first correct response among conditions ①–③ assigns the operational diagnosis; failure under all three is assigned to application. Condition ④ is a separate validity control, not a fourth step in this assignment sequence.
Source
Target quantity
Fails
Reason
Sound of Water
water level or fill fraction
C4
would be read back from the same pitch trajectory the model is given
Sound of Water
fill rate in mL/s
C4
no flow meter accompanies the recordings
Sound of Water
time remaining until full
C4
circular, and no vessel is filled to the top
dEchorate
source distance from echo delay
C1, C5
first-reflection delays are 2–4 ms, below what is audible
dEchorate
direction of arrival
C3
needs multichannel input and resolves in one step
dEchorate
room size
C3
the corpus has one room, so the answer is constant
TABLE VI: Pairings considered and not admitted. C4, an independent ground truth, is the single most common cause and the only one that cannot be repaired by listening harder or recording better. A rejection removes one target quantity, not the source: Sound of Water appears here three times and in Table III as an admitted source.
Family
Target quantity
Independent ground truth
Bouncing ball
coefficient of restitution, drop height
high-speed video and ruler
Free fall into a shaft
drop height
laser rangefinder
Pendulum
string length
ruler
Rotating wheel
surface speed, revolutions per second
laser tachometer and ruler
Struck rod
rod length, material
ruler; alloy known at capture
Plucked string
tension
hanging calibrated weights
TABLE VII: Families under construction. Each is recorded or synthesized so that the target quantity is fixed or measured by an instrument that never touches the audio, which is what C4 requires and what most existing corpora cannot supply.
Condition
Gemini-3.8-flash
①
29.3 [23, 36]
②
31.3 [25, 38]
③
98.5 [96, 99]
④
23.9 [19, 30]
Chance
25.0
TABLE VIII: Baseline check on 201 items. Brackets are 95% Wilson intervals; chance is 25.0%. The reported no-audio interval contains chance. This aggregate check does not establish the absence of text-only shortcuts in individual items.
Probe outcome
Fam.
n
①
②
Δ
③
states the same rule
10
107
25.2
25.2
+0.0
97.2
states it incompletely
5
58
20.7
32.8
+12.1
100.0
states a different rule
3
36
55.6
47.2
-8.3
100.0
TABLE IX: Family-level rule recall for Gemini-3.8-flash, graded by a separate call to the same model. Columns report accuracy (%) under each condition; Δ is the change from base to rule-supplied accuracy in percentage points. Groups describe probe responses, not verified internal knowledge states.
Gemini-3.8-flash
Diagnosis
n
%
① Capable
59
29.3
② Missing knowledge
20
10.0
③ Failed perception
122
60.7
① Failed application
0
0.0
Total
201
100
TABLE X: Operational attribution over all 201 current items, including the flagged case. An item is assigned to the first condition at which the model answers it correctly: ① nothing supplied, ② fixed by supplying the rule, ③ fixed by supplying the percept. An item incorrect under all three diagnostic conditions is assigned to application. Unaided accuracy is 29.3%; the labels summarize intervention responses.
Percept axis
n
①
②
③
③%
resonant pitch
20
5
0
15
75
cycle timing
12
3
0
9
75
rotation tone
24
4
3
17
71
frequency trajectory
12
1
3
8
67
level trend
12
3
1
8
67
beat rate
12
3
1
8
67
TABLE XI: First-success counts for Gemini-3.8-flash by perceptual axis. The last column is the percentage first solved with percept text. The single flagged well-depth item is excluded from this breakdown; it remains in the overall results.
Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.
Yupei Li, Qiyang Sun, Mohamed Mady +4
Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE +2
Existing audio question answering benchmarks largely emphasize sound event classification or caption-grounded queries, often enabling models to succeed through shortcut strategies, short-duration cues, lexical priors, dataset-specific biases, or even bypassing audio via metadata and captions rather than genuine reasoning Thus, we present AUDITA (Audio Understanding from Diverse Internet Trivia Authors), a large-scale, real-world benchmark to rigorously evaluate audio reasoning beyond surface-level acoustic recognition. AUDITA comprises carefully curated, human-authored trivia questions grounded in real-world audio, designed to stress robust auditory reasoning through challenging distractors and long-range temporal dependencies, using probing queries that cannot be answered from isolated text or sound cues alone. Human average accuracy of 32.13% shows both the challenge of the task while demonstrating meaningful comprehension of the audio. In stark contrast, state of-the-art audio question answering models perform poorly, with average accuracy below 8.86%. Beyond raw accuracy, we apply Item Response Theory (IRT) to estimate latent proficiency, question difficulty, and expose systematic deficiencies of the models and data.
Tasnim Kabir, Dmytro Kurdydyk, Aadi Palnitkar +3
University of Maryland · Davidson College · Columbia University +1
Large audio-language models (LALMs) increasingly use explicit reasoning traces for complex audio understanding, yet the evaluation of reasoning quality remains underexplored. Although process-level benchmarks for process reward models (PRMs) have advanced reasoning evaluation in text and multi-modal domains, comparable evaluation for audio reasoning remains limited. In this paper, we present AudioProcessBench, a comprehensive benchmark for step-level process error identification in audio reasoning. AudioProcessBench contains diverse reasoning traces generated by 6 audio and omni language models. Each trace is segmented into discrete reasoning steps and annotated with binary step correctness and fine-grained error types. Our benchmark evaluates models under three complementary paradigms: (1) step correctness identification, (2) error-type-conditioned detection for diagnosing audio-specific verifier capacities, and (3) chain-level aggregation, where verifiers select or aggregate among multiple reasoning traces for the same question. This design enables a systematic analysis of whether current models can detect process errors, whether their weaknesses differ across audio-specific error types, and whether process verification translates into improved answer selection. AudioProcessBench provides a testbed for future research on audio reasoning verifiers, process reward models, and reliable omni-modal reasoning.
Xiangyu Zhao, Junyu Yan, Yaling Shen +7
Monash University · Xi’an Jiaotong-Liverpool University · Orygen +1