PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

Yuanjian Chen; Yang Xiao; Han Yin; Xubo Liu; Jinjie Huang; Ting Dang

eess.ASMar 5, 2026

PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

Authors: Yuanjian Chen, Yang Xiao, Han Yin, Xubo Liu, Jinjie Huang, Ting Dang

Organizations: Harbin University of Science and Technology, China · The University of Melbourne, Australia · KAIST, South Korea · University of Surrey, UK

Abstract

Large Audio Language Models (LALMs) are increasingly capable of reasoning over audio, yet existing benchmarks offer limited coverage of reasoning in polyphonic audio, where multiple sound events co-occur and induce compositional structure. To address this gap, we introduce PolyBench, a benchmark designed to evaluate compositional reasoning in polyphonic audio, comprising five evaluation subsets that cover counting, classification, detection, concurrency, and duration estimation, all of which require reasoning over multiple concurrent events and their relations. Our evaluation of state-of-the-art LALMs reveals consistent performance degradation in polyphonic settings, indicating a fundamental bottleneck in current LALMs.

Figures & tables

Figure 1: PolyBench pipeline for benchmark construction and evaluation. The process includes (1) observing polyphony-induced failure patterns by contrasting monophonic and polyphonic audio; (2) curating real-world polyphonic clips and generating MCQAs for five task types via human–LLM collaboration with iterative quality control. The circles denote the five tasks, which are from basic perception to complex reasoning; and (3) evaluating LALMs by scoring their MCQA predictions against golden answers. The right panel shows representative examples of the five tasks (Counting, Duration, Concurrency, Classification, and Detection).

Figure 2: Visual analysis of PolyBench dataset characteristics. Left: distribution of PolyBench event tags across top-level AudioSet ontology categories—CE (Channel, environment and background), MU (Music), NS (Natural sounds), AS (Animal sounds), HS (Human sounds), and ST (Sounds of things). Numbers at the end of each bar indicate the event occurrence counts in the dataset, while the x-axis shows the proportion of tags within each source dataset. Right: proportion of audio clips by the number of overlapping sound sources (e.g., 2, 3, and 4 concurrent events).

Models	Counting		Duration		Concurrency		Classification		Detection
	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC
Qwen3-Omni-30B-A3B [ 30 ]	55.9	57.5	70.2	68.5	85.2	76.0	77.2	77.9	57.3	63.4
R1AQA [ 29 ]	34.5	30.1	42.1	43.2	93.7	90.4	29.5	33.1	22.4	26.2
Audio Flamingo 3 [ 28 ]	56.3	53.4	67.4	65.8	53.1	37.7	47.8	50.3	36.2	37.2
TimeAudio [ 24 ] +Qwen3-8B [ 31 ]	47.7	43.8	63.7	63.7	86.9	78.1	53.8	50.3	53.5	51.7
AUDSEMTHINKER-QA GRPO [ 32 ]	57.3	51.4	67.1	64.4	59.1	43.2	70.6	70.3	39.6	37.2

Table 1: Performance (%) of various LALMs on PolyBench compositional reasoning tasks under polyphonic audio (MCQA setting). Best results are in bold and second-best results are underlined for each metric column.

Model	TP	FP	TN	FN
Qwen3-Omni-30B-A3B [ 30 ]	18.2	8.1	8.8	64.9
R1AQA [ 29 ]	24.3	54.7	2.7	18.3
Audio Flamingo-3 [ 28 ]	10.1	16.2	16.9	56.8
TimeAudio [ 24 ] +Qwen3-8B [ 31 ]	20.2	41.9	6.8	31.1
AUDSEMTHINKER-QA GRPO [ 32 ]	10.1	29.8	16.9	43.2

Table 2: Performance (%) of various LALMs on the PolyBench Concurrency task. Best results are in bold and second-best results are underlined for each metric column.

Audio Reasoning

Audio reasoning studies how AI systems infer events, relationships, intent, and context from sound, beyond recognizing words or matching labels.

See all 64 papers

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models
Sep 9, 2026Yupei Li, Qiyang Sun, Mohamed Mady +4Audio-Language Model Evaluation Audio Reasoning
Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE +2
AudioProcessBench: Benchmark for Identifying Process Errors in Audio-Grounded Reasoning
Jun 7, 2026Xiangyu Zhao, Junyu Yan, Yaling Shen +7Audio-Language Model Evaluation Reasoning Evaluation
Monash University · Xi’an Jiaotong-Liverpool University · Orygen +1
MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models
Mar 10, 2026Chih-Kai Yang, Yun-Shao Tsai, Yu-Kai Guo +7Audio-Language Model Evaluation Audio Understanding
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

Sep 9, 2026J/K move · Enter open · S save

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Yupei Li, Qiyang Sun, Mohamed Mady, Chenxi Wang+3

Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE+2

Audio-Language Model Evaluation Audio Reasoning Audio-Language Models

Open paper AskPDF arXiv

Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.

Abstract

Audio Reasoning

Explore similar work

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

AudioProcessBench: Benchmark for Identifying Process Errors in Audio-Grounded Reasoning

MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models