AuDiR: Diagnosing Audio Reasoning with Rule and Percept Interventions
Organizations: Institute of Advanced Intelligence and Computing (IAIC) and the Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore. · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore.
Abstract
Answer accuracy alone does not distinguish failures in acoustic perception, rule knowledge, and rule application. AuDiR (Audio Diagnostic Reasoning) examines these distinctions through controlled input interventions. Each item contains an audio clip, a multiple-choice question, a task-general rule, and a textual description of the acoustic observations needed to answer. Three conditions progressively supply the rule and observations; a fourth removes the audio while retaining the question and rule. The first correct diagnostic condition assigns an operational response label, without identifying a unique internal cause. The current benchmark contains 201 items in 18 families drawn from 9 sources, with answer references documented separately from percept extraction. In a preliminary evaluation of Gemini-3.8-flash, accuracy is 29.3% without supplied information, 31.3% with the rule, and 98.5% with the observations also supplied. No-audio accuracy is 23.9%, with a confidence interval containing the 25.0% chance rate. A family-level knowledge probe associates incomplete rule recall with larger gains from rule provision. These results show the benefit of supplying acoustic observations as text, while incomplete validation, possible recognition shortcuts, and non-monotonic responses limit stronger failure-attribution claims.
Figures & tables
| Benchmark | Items | Failure attribution | Audio-necessity check | Answer ground truth |
|---|---|---|---|---|
| AIR-Bench [ 4 ] | 21k | — | — | human annotation, model-judged |
| MMAU [ 2 ] | 10k | — | noise substitution, per model, reported | human annotation |
| MMAR [ 1 ] | 1,000 | post-hoc, by reading 100 failed predictions from one model | noise substitution, per model, reported | curated human annotation |
| STAR-Bench [ 3 ] | 2,353 | from aggregate score patterns across task families | caption substitution, per benchmark, used as motivation | procedural synthesis and simulation |
| AuDiR (ours) | 201 | per-item intervention response patterns | no-audio accuracy, reported in aggregate | independent-reference design; provenance audit pending |
| Condition | Full input | Intervention |
|---|---|---|
| ① Base | No supplied rule or percept | |
| ② Knowledge | Rule supplied | |
| ③ Percept | Observations also supplied | |
| ④ No audio | Audio-necessity control |
| Source | Audio | Answer reference | License | Fam. | Items |
|---|---|---|---|---|---|
| VS13 | single vehicles passing a roadside microphone at constant speed | speed held on the vehicle’s cruise control, logged per pass | IEEE, research use, no redistribution | 2 | 24 |
| Sound of Water | water poured into vessels of known geometry | ruler-measured net height and container geometry | MIT | 2 | 20 |
| GuitarSet | solo acoustic guitar, hexaphonic pickup | shipped chord and note transcription | CC BY 4.0 | 3 | 36 |
| AudioTime | environmental scenes with several distinct sound events | shipped onset and offset timestamps | open | 2 | 24 |
| MAFAULDA | rotating machinery under controlled fault conditions | tachometer channel recorded alongside the audio | open (UFRJ) | 2 | 24 |
| MMAR | one retained clip, a stone dropped into a well | answer shipped with the benchmark | CC BY-NC 4.0 | 1 | 1 |
| Family | Source | Percept (heard) | Rule (told) | Answer reference | |
|---|---|---|---|---|---|
| Doppler speed | VS13 | approach and recede engine-tone frequencies | cruise-control speed | 12 | |
| Doppler speed ratio | VS13 | both cars’ tone shifts | ratio of two Doppler solutions | cruise-control speed | 12 |
| Container height | Sound of Water | empty-container resonant pitch | ruler-measured net height | 10 | |
| Taller vessel, fuller vessel | Sound of Water | both containers’ empty-start pitches | , then from and the given | ruler height and diameters | 10 |
| Event rate | AudioTime | number of distinct sound events | shipped event timestamps | 12 | |
| Chord quality | GuitarSet | the chord’s interval pattern | stacked thirds quality | GuitarSet transcription | 12 |
| Category | |
| Percept type | |
| frequency shift | 24 |
| event count | 24 |
| rotation tone | 24 |
| pitch pattern | 24 |
| resonant pitch | 20 |
| Source | Target quantity | Fails | Reason |
|---|---|---|---|
| Sound of Water | water level or fill fraction | C4 | would be read back from the same pitch trajectory the model is given |
| Sound of Water | fill rate in mL/s | C4 | no flow meter accompanies the recordings |
| Sound of Water | time remaining until full | C4 | circular, and no vessel is filled to the top |
| dEchorate | source distance from echo delay | C1, C5 | first-reflection delays are 2–4 ms, below what is audible |
| dEchorate | direction of arrival | C3 | needs multichannel input and resolves in one step |
| dEchorate | room size | C3 | the corpus has one room, so the answer is constant |
| Family | Target quantity | Independent ground truth |
|---|---|---|
| Bouncing ball | coefficient of restitution, drop height | high-speed video and ruler |
| Free fall into a shaft | drop height | laser rangefinder |
| Pendulum | string length | ruler |
| Rotating wheel | surface speed, revolutions per second | laser tachometer and ruler |
| Struck rod | rod length, material | ruler; alloy known at capture |
| Plucked string | tension | hanging calibrated weights |
| Condition | Gemini-3.8-flash |
|---|---|
| ① | 29.3 [23, 36] |
| ② | 31.3 [25, 38] |
| ③ | 98.5 [96, 99] |
| ④ | 23.9 [19, 30] |
| Chance | 25.0 |
| Probe outcome | Fam. | ① | ② | ③ | ||
|---|---|---|---|---|---|---|
| states the same rule | 10 | 107 | 25.2 | 25.2 | +0.0 | 97.2 |
| states it incompletely | 5 | 58 | 20.7 | 32.8 | +12.1 | 100.0 |
| states a different rule | 3 | 36 | 55.6 | 47.2 | -8.3 | 100.0 |
| Gemini-3.8-flash | ||
|---|---|---|
| Diagnosis | % | |
| ① Capable | 59 | 29.3 |
| ② Missing knowledge | 20 | 10.0 |
| ③ Failed perception | 122 | 60.7 |
| ① Failed application | 0 | 0.0 |
| Total | 201 | 100 |
| Percept axis | ① | ② | ③ | ③% | |
|---|---|---|---|---|---|
| resonant pitch | 20 | 5 | 0 | 15 | 75 |
| cycle timing | 12 | 3 | 0 | 9 | 75 |
| rotation tone | 24 | 4 | 3 | 17 | 71 |
| frequency trajectory | 12 | 1 | 3 | 8 | 67 |
| level trend | 12 | 3 | 1 | 8 | 67 |
| beat rate | 12 | 3 | 1 | 8 | 67 |
Audio Reasoning
Audio reasoning studies how AI systems infer events, relationships, intent, and context from sound, beyond recognizing words or matching labels.
See all 71 papers