MixCards: Evaluating Structured Descriptions of Multi-Source Sound Scenes

Sripathi Sridhar; Yujin Kim; Giovana Morais; Liang-Yuan Wu; Mehmet Atilay Kucukoglu; Prem Seetharaman; Lloyd May; Mark Cartwright

cs.SDOct 28, 2026

MixCards: Evaluating Structured Descriptions of Multi-Source Sound Scenes

Authors: Sripathi Sridhar, Yujin Kim, Giovana Morais, Liang-Yuan Wu, Mehmet Atilay Kucukoglu, Prem Seetharaman, Lloyd May, Mark Cartwright,

Organizations: New Jersey Institute of Technology · New York University · Adobe Research · Monash University

Abstract

Audio captioning benchmarks typically evaluate a single unstructured free-text description of an audio scene, providing limited insight into how a model fails to understand polyphonic audio. A plausible caption may omit a source, or conflate multiple sources into one, or assign an attribute of one source to another, and conventional captioning evaluation cannot locate these errors. We introduce MixCardBench, a benchmark for structured captioning of polyphonic audio, which evaluates models in two stages: first, identifying and temporally localizing the sources in a scene, and second, generating a structured MixCard describing the semantic, temporal, and acoustic properties of each source. The benchmark contains 4884 ecologically plausible synthetic soundscapes with up to nine overlapping speech, music and sound effects sources built from high-quality recordings and source-level annotations, where each source has a reference MixCard. We evaluate six state-of-the-art large audio language models and find substantial limitations in current models, which struggle considerably with speech and sound effects timing and detection, with the strongest model reaching only around 60% mean card-field score. Performance varies significantly across attributes, exposing capability differences that are hidden by aggregate caption scores. We assess the quality and ecological validity of MixCardBench through human validation, distribution matching and a listening study, and show that human corrections have little effect on model rankings, and our synthesized soundscapes are closer to in-the-wild YouTube videos than source guided but otherwise random synthesis. The benchmark dataset and code are publicly available, and demo examples are at https://mixcards-anon.github.io/mixcards-demo/

Figures & tables

Figure 1: We propose MixCardBench, a benchmark for multi-source structured captioning where a model detects sound sources and generates a structured description per source (i.e. MixCard). We depict an example from our benchmark and its corresponding speech, music, and sound effects ground truth MixCards. We include only a subset of MixCard fields here due to space constraints.

Figure 2: Our pipeline used to generate open, high-quality, ecologically-valid soundscapes containing speech, music, and sound effects sources. We use SAM Audio for source separation. We use Vevo-Timbre to clone separated speech into public domain voices. We use MULE embeddings to find similar music as the original AudioSet mixture. We use ontology mapping and sound event detection using PE-A-Frame and PE-AV to sample sources matching the target AudioSet event class.

Model	Orig.	Val.	$\Delta$	Orig.	Val.	$\Delta$	Orig.	Val.	$\Delta$
	Speech			Music			Sound effects
Qwen3-Omni	0.639	0.645	$+0.006$	0.595	0.575	$-0.020$	0.598	0.588	$-0.011$
Qwen2.5-Omni	0.475	0.462	$-0.013$	0.470	0.465	$-0.005$	0.509	0.508	$-0.001$
AF 3	0.031	0.030	$-0.001$	0.045	0.042	$-0.003$	0.128	0.139	$+0.010$
Music Flamingo	0.307	0.307	$+0.001$	0.305	0.293	$-0.013$	0.298	0.305	$+0.007$
AF Next	0.339	0.346	$+0.007$	0.307	0.307	$-0.001$	0.471	0.475	$+0.003$

Table 1: Mean card field score (all fields) on the 774 validated cards, scored against the original GT (Orig.) and the validated GT (Val.).

Dataset	KAD $\downarrow$	FAD $\downarrow$
Ours	3.26	1.28
Variant 1	4.00	1.74
Variant 2	4.48	1.86
Variant 3	6.29	2.16

Table 2: Synthetic soundscape distributions vs. AudioSet-SL.

Figure 3: Mean ecological validity.

Figure 4: Benchmark performance. Top: source detection and mean MixCard score by category. Bottom: category-specific MixCard field scores. Error bars show 95% confidence intervals.

Figure 5: Effect of polyphony.

Figure 6: P/R heatmap on segment-based F1.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Before and After annotations of AudioSet sample “KxzuJRh9GaQ_30000”. Speech is the parent class of Female speech, woman speaking , therefore it is dropped as they have clearly identical annotations. The same happens with Owl and Hoot .

Figure 8: Before and After annotations of AudioSet sample “xAyFvqkINEA_30000”.

Step	Files
AudioSet-SL eval	16,975
SAM Audio Judge, min. recall $\geq 4.0$	9110
Speech gates (Table 5 )	5478
Drop Narration, monologue , Conversation , Whispering , Speech synthesizer	5180
Drop segments with breathing + speech	4933
Used for generating MixCardBench	4933

Appendix

Table 3: Cascaded quality filters applied to AudioSet-SL eval.

Step	Speakers
LibriTTS-R (2456) + VCTK (110)	2566
Test pool: LibriTTS-R test-clean (39) + dev-clean (40) + VCTK (110)	189
Used for generating MixCardBench	186

Appendix

Table 4: Speaker source files.

Step	Segments
AudioSet-SL segments with speech	6962
No babble/hubbub labels	6546
No singing overlapping speech	6312
WER computation succeeds	5994
Cloned-voice language matches original	5435
$\geq 3$ reference words	4772

Appendix

Table 5: Cascaded quality filters applied to AudioSet-SL segments containing speech. Each row gives the number of segments remaining after the filter.

Step	Tracks
FMA large, test split	11,263
Distinct FMA tracks used	2024

Appendix

Table 6: Music source files.

Figure 9: Sound-effect sampling: ontology mapping and PE-A-Frame detection build candidate pools, RMS onset detection isolates individual events, and PE-AV text–audio similarity keeps the most semantically relevant candidates.

Step	Files
Our test split of PSE	324,493
Ambience / archived removed	278,364
Files that AudioSet-SL queries map to	146,153
PE-A-Frame event detection	143,847
RMS crops inside detected spans	132,184
Crop duration $\leq 10$ s	118,552

Appendix

Table 7: Cascaded quality filters applied to sound-effect source files.

Field	Description	Example
Common: signal features
(start, stop) tuples	Onset–offset for every instance of the event.	[00:00.000, 00:04.075]; [00:09.636, 00:09.996]
Loudness	Integrated LUFS per instance with a six-level label (very loud / loud / medium / quiet / very quiet / extremely quiet).	[-27.8 LUFS, Quiet]; [-26.8 LUFS, Medium]
Frequency profile	Energy across 10 bands (125 Hz to 10 kHz), each on a seven-level scale (very-low / low / medium-low / medium / medium-high / high / very-high).	[medium-high; medium-high; medium-high; medium-low; low; low; low; low; low; low]; [medium-high; medium-high; medium-high; medium-high; medium-low; low; low; low; medium-low; medium-low]
Common: free-text fields
Short caption	Brief caption of the event.	“Monotonous mid voice speaking” (speech); “Driving house music blasts” (music); “Rapid, sharp dog barks” (SFX)

Appendix

Table 8: MixCard field schema. Common fields are shared across all three card types, while category-specific fields exist only in MixCards for those categories. Examples are taken from dataset cards.

Figure 10: LUFS distribution and discretized bands per domain.

Figure 11: Audiometry frequency profile distribution and discretized levels per domain.

Event type	Scenes	Cards
Speech	1116	1116
Music	4009	4009
Sound effects	2269	4493
Total	4884 (13.6h)	9618

Appendix

Table 9: MixCardBench card count per audio category. In total, we have 9618 cards.

Panel of Figure 12 — Figure 12: Source and soundscape statistics in MixCardBench.

Figure 13: Distribution of AudioSet tags and UCS categories used in our data generation pipeline. Card classes shows the top 30 final classes in our MixCards. The plots are showing top 30.

Event type	1	2	3	4	5	6	7	8	9	Total
Music	2520	720	395	226	95	37	9	4	3	4009
Speech	34	302	380	230	115	35	16	2	2	1116
Sound effects	9	906	688	401	178	57	22	5	3	2269
All soundscapes	2563	967	688	401	178	57	22	5	3	4884

Appendix

Table 10: Number of Polyphony (sources) by domain.

Figure 14: Distribution of frequency profile bands in the final benchmark. This is computed at the segment level and not at the soundscape level.

field	tags (share of cards)
genre (FMA metadata) 135 distinct, top 15; 2.45 tags per card	Experimental (23%), Electronic (21%), Avant-Garde (10%), Ambient (10%), Folk (9%), Experimental Pop (7%), Rock (7%), Ambient Electronic (7%), Pop (6%), Lo-Fi (6%), Improv (5%), Jazz (5%), Electroacoustic (5%), Indie-Rock (4%), Hip-Hop (4%)
mood (GEMS-9) 9 distinct; 1.42 tags per card; 0.6% empty	tension (39%), calmness (27%), joyful activation (21%), power (15%), nostalgia (15%), tenderness (13%), sadness (7%), solemnity (5%), amazement (0%)
perceived audio quality 13 distinct; 3.12 tags per card	low-fidelity (48%), dry (44%), thin (43%), clear (36%), bass-heavy (27%), room reverb (22%), low background noise (21%), muffled (18%), distorted (15%), high-fidelity (13%), high background noise (11%), reverb-heavy (7%), harsh (7%)
instruments 53 distinct, top 20; 2.34 tags per card; 0.6% empty	drums (40%), synthesizer (35%), guitar (28%), bass guitar (26%), vocals (22%), electric guitar (14%), drum machine (13%), piano (9%), percussion (7%), strings (5%), violin (5%), flute (3%), accordion (3%), glockenspiel (3%), saxophone (2%), choir (2%), acoustic guitar (2%), trumpet (2%), cymbals (2%), electric piano (1%)
music descriptors 36 distinct, top 20; 4.11 tags per card; 0.0% empty	repetitive (34%), melodic (33%), rhythmic (27%), sparse (26%), layered (22%), raw (19%), synthetic (18%), percussive (18%), looping (17%), arpeggiated (16%), evolving (15%), sustained (15%), detuned (15%), delicate (14%), glitchy (13%), lively (10%), intricate (10%), groovy (10%), plucked (9%), frenetic (8%)
tempo label 6 distinct; 1.00 tags per card	very slow (7%), slow (28%), medium (15%), fast (16%), very fast (19%), no tempo (15%)

Appendix

Table 11: Music cards (4009 cards in the final set): tag distribution per field. Percentages are the share of cards carrying the tag (loudness: share of segments); closed fields list every tag of the vocabulary, open fields the most frequent ones.

field	tags (share of cards)
perceived audio quality 11 distinct; 3.29 tags per card	dry (79%), muffled (77%), low-fidelity (66%), low background noise (34%), clear (33%), thin (24%), room reverb (11%), bass-heavy (4%), high background noise (1%), harsh (0%), distorted (0%)
manner (open) 115 distinct, top 15; 2.28 tags per card	conversational (37%), monotonous (36%), slow (36%), steady (19%), breathy (11%), relaxed (11%), resonant (8%), raspy (6%), deliberate (5%), nasal (4%), hesitant (4%), flat (3%), casual (3%), soft (2%), calm (2%)
intelligibility words 5 distinct; 1.46 tags per card	highly intelligible (95%), clean channel (43%), mostly intelligible (5%), poor channel quality (3%), degraded intelligibility (0%)
emotion words 5 distinct; 1.01 tags per card	neutral (67%), excited/happy (16%), angry/stressed (12%), sad/subdued (3%), calm (2%)
language 1 distinct; 1.00 tags per card	en (100%)
loudness label 1.65 segments per card	extremely quiet (1%), very quiet (4%), quiet (14%), medium (33%), loud (36%), very loud (13%)

Appendix

Table 12: Speech cards (1116 cards in the final set): tag distribution per field. Percentages are the share of cards carrying the tag (loudness: share of segments); closed fields list every tag of the vocabulary, open fields the most frequent ones.

field	tags (share of cards)
class (source library) 147 distinct, top 20; 1.00 tags per card	Misc (23%), General (5%), Knock (4%), Impact (4%), Scream (4%), Human (3%), Tick (3%), Laugh (3%), Breath (2%), Songbird (2%), Applause (2%), Click (2%), Cheering (2%), Combustion (1%), Power (1%), Movement (1%), Hand (1%), Cheer (1%), Synthetic (1%), Insect (1%)
temporal descriptors 13 distinct; 2.02 tags per card	impact/abrupt (55%), single event (37%), sustained (28%), continuous (27%), intermittent (24%), rising (9%), periodic (7%), rhythmic (5%), fade out (5%), oscillating (2%), falling (2%), fade in (1%), sustained with occasional bursts (0%)
plurality 3 distinct; 1.00 tags per card	one (79%), multiple (18%), none (3%)
perceived audio quality 13 distinct; 3.29 tags per card	dry (73%), low background noise (53%), low-fidelity (47%), thin (43%), clear (43%), bass-heavy (23%), muffled (11%), harsh (9%), high-fidelity (9%), high background noise (8%), distorted (6%), room reverb (4%), reverb-heavy (1%)
sound descriptors (open) 451 distinct, top 15; 3.08 tags per card	sharp (43%), metallic (27%), high-pitched (22%), hollow (19%), rumbling (10%), low-pitched (10%), rapid (7%), deep (7%), piercing (6%), resonant (6%), percussive (5%), harsh (5%), mechanical (5%), steady (4%), scraping (4%)
sounding object (open) 1123 distinct, top 15; 1.09 tags per card; 3.5% empty	crowd (5%), human (3%), wind (2%), water (2%), hands (2%), door (2%), metal (2%), machine (1%), alarm clock (1%), door knocker (1%), female (1%), wood (1%), relay (1%), glass (1%), synth (1%)

Appendix

Table 13: Sound-effect cards (4493 cards in the final set): tag distribution per field. Percentages are the share of cards carrying the tag (loudness: share of segments); closed fields list every tag of the vocabulary, open fields the most frequent ones.

	Stage 1: sound-effect segment F1			Stage 2: judged-field score
Model	Llama-3-8B	Qwen2.5-7B	Qwen3-4B	Llama-3-8B	Qwen2.5-7B	Qwen3-4B
Qwen3-Omni	0.374 (1)	0.337 (1)	0.224 (1)	0.577 (1)	0.569 (1)	0.643 (1)
Qwen2.5-Omni	0.276 (2)	0.252 (2)	0.164 (2)	0.454 (2)	0.444 (2)	0.512 (2)
AF 3	0.259 (3)	0.239 (3)	0.164 (3)	0.060 (6)	0.052 (6)	0.059 (6)
Music Flamingo	0.084 (6)	0.069 (6)	0.054 (6)	0.327 (3)	0.310 (3)	0.349 (4) †
AF Next	0.208 (4)	0.184 (4)	0.145 (4)	0.291 (5)	0.262 (5)	0.310 (5)

Appendix

Table 14: Model scores and ranks (in parentheses) under the three LLM judges. † marks a rank that differs from Llama-3-8B’s.

Figure 18: Change in model performance when scored against the human-validated ground truth instead of the generated ground truth.

Figure 19: Frequency profile agreement per band, shown as each model’s deviation from the six-model mean.

	Segment F1			Event F1			Card field score
Model	Sp.	Mu.	SFX	Sp.	Mu.	SFX	Sp.	Mu.	SFX
Qwen3-Omni	0.639	0.932	0.374	0.510	0.858	0.239	0.648	0.631	0.612
Qwen2.5-Omni	0.599	0.886	0.276	0.185	0.785	0.103	0.466	0.456	0.517
AF 3	0.577	0.858	0.259	0.129	0.676	0.075	0.017	0.039	0.127
Music Flamingo	0.334	0.894	0.084	0.065	0.365	0.030	0.309	0.291	0.319
AF Next	0.487	0.937	0.208	0.072	0.674	0.066	0.298	0.324	0.443

Appendix

Table 15: Source inventory F1 (segment-based, 100 ms; event-based, 1 s collar) and mean card field score, per card type, on 4884 scenes. Best per column in bold.

		Qwen3-Omni	Qwen2.5-Omni	AF 3	Music Flamingo	AF Next	Kimi-Audio
loudness	Speech	0.822	0.455	0.001	0.652	0.619	0.620
	Music	0.803	0.559	0.064	0.688	0.738	0.440
	Sound effects	0.803	0.530	0.041	0.497	0.654	0.355
frequency profile	Speech	0.732	0.503	0.000	0.006	0.364	0.533
	Music	0.740	0.610	0.000	0.000	0.462	0.578
	Sound effects	0.734	0.593	0.011	0.002	0.501	0.595

Appendix

Table 16: Score on the fields every card type carries, per card type, on 4884 scenes. Best model per row in bold.

		Qwen3-Omni	Qwen2.5-Omni	AF 3	Music Flamingo	AF Next	Kimi-Audio
Speech	median f0 hz	0.849	0.451	0.000	0.006	0.441	0.512
	intelligibility	0.537	0.458	0.040	0.399	0.066	0.233
	emotion	0.709	0.670	0.036	0.534	0.237	0.458
	manner	0.498	0.447	0.025	0.376	0.315	0.433
Music	instruments	0.688	0.538	0.040	0.285	0.335	0.451
	genre	0.467	0.478	0.100	0.292	0.202	0.272

Appendix

Table 17: Score on the fields unique to each card type, on 4884 scenes. Best model per row in bold.

Panel of Figure 20 — Figure 20: Stage-1 precision (P) and recall (R) per card type.

Figure 21: Model performance on the five fields common to all card types, compared across speech, music and sound effects.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 208 papers

Feb 27, 2026

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: accuracy (semantic correctness), completeness (coverage of reference content), and hallucination (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini3Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.

Jielin Qiu, Jianguo Zhang, Zixiang Chen +13

Salesforce AI Research

Audio-Language Model Evaluation Audio Captioning

Ask PDF

Jul 29, 2026

MMAC: A Massive Multi-dimensional Benchmark for Audio Captioning

With the development of audio large language models (AudioLLMs), audio captioning needs to move from brief descriptions toward open-ended and fine-grained free-form descriptions. Existing evaluations often focus on generation quality or task performance, making it difficult to diagnose information coverage and description reliability. We propose MMAC, a Massive Multi-dimensional benchmark for Audio Captioning. MMAC contains 5,638 audio clips from more than 20 data sources, covering 6 capability categories and 15 evaluation dimensions. Given a model-generated caption, MMAC checks whether it mentions relevant information in the target dimension and whether the mentioned content is consistent with the reference label. We evaluate representative open-source and proprietary AudioLLMs. Results show clear differences across evaluation dimensions, information coverage, and description reliability. We will release the MMAC benchmark and evaluation code.

Weijie Wu, Junbo Li, Lin Li +2

School of Informatics, Xiamen University, China · DiDi Global Inc., Beijing, China · School of Electronic Science and Engineering, Xiamen University, China

Audio-Language Model Evaluation Audio Captioning

Ask PDF

Mar 5, 2026

PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

Large Audio Language Models (LALMs) are increasingly capable of reasoning over audio, yet existing benchmarks offer limited coverage of reasoning in polyphonic audio, where multiple sound events co-occur and induce compositional structure. To address this gap, we introduce PolyBench, a benchmark designed to evaluate compositional reasoning in polyphonic audio, comprising five evaluation subsets that cover counting, classification, detection, concurrency, and duration estimation, all of which require reasoning over multiple concurrent events and their relations. Our evaluation of state-of-the-art LALMs reveals consistent performance degradation in polyphonic settings, indicating a fundamental bottleneck in current LALMs.

Yuanjian Chen, Yang Xiao, Han Yin +3

Harbin University of Science and Technology, China · The University of Melbourne, Australia · KAIST, South Korea +1

Audio Reasoning Audio-Language Model Evaluation Compositional Reasoning Audio-Language Models

Ask PDF

Abstract

Audio-Language Model Evaluation

Explore similar work

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

MMAC: A Massive Multi-dimensional Benchmark for Audio Captioning

PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio