CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

Marcel Gibier; Thomas Thebaud; Olivier Boëffard; Jean-François Bonastre

cs.SDOct 7, 2026

CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

Authors: Marcel Gibier, Thomas Thebaud, Olivier Boëffard, Jean-François Bonastre

Organizations: Inria, Paris, France · AMIAD, France

Abstract

Automatic audio scene description turns a recording into a text account of a situation. One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all. Annotators disagree about this, making a ground truth hard to obtain. In this work, we first define the ground truth, then generate the data. We focus on audio events and define sound salience with a simple rule: a sound is salient when a speaker audibly reacts to it. For scale and variety, a controlled set of scenarios fixes the ground truth, and a language model writes the dialogues. The resulting corpus, CARES, contains 10,000 two-speaker scenes. We then benchmark six audio-language models on three tasks: identifying the scene, tagging the sounds present, and classifying reactions. We show that these models hear the sounds but miss how the speakers react to them.

Figures & tables

Figure 1: Four types of speaker reaction to an audio event. Each block shows the transcript (T, one box per word) and the speech signal (S) over time. The shaded box marks the audio event

[t_{\mathrm{on}},t_{\mathrm{off}}]

. Colored elements are altered by the event. Pivot : persistent change of T and S. Verbal : local verbal reaction (T and S). Behavioral : local change of S only. Ambient : no impact. The speech signal, the audio event (

-23

LUFS) and the background (

-32

LUFS) are summed (

\oplus

) to form the final mix.

Figure 2: The CARES construction pipeline: text generation and audio rendering. A deterministic allocator fixes the scene, relation, speaker genders, sound events, reaction registers, and composition. A language model fills only the conversation topic and the dialogue. The reaction-recovery filter sits between the two phases: rejected dialogues are regenerated until an independent evaluator recovers the intended reactions. utt. : utterances; ev. : acoustic events.

Model	Accuracy(%) $\uparrow$	Pivot	Verbal	Behav.	Ambient	Accuracy (%) $\uparrow$	Accuracy (%) $\uparrow$	Pivot	Verbal	Behav.	Ambient
	Reaction Classification					Acoustic Scene	Audio	Summary
	Balanced	Precision(%) / Recall(%)				Classification	Tagging	Recall (%) $\uparrow$
Audio Flamingo Next	$29.7_{\pm 2.0}$	50.0/0.3	30.6/65.0	50.0/0.7	41.1/52.7	$62.5_{\pm 3}$	$34.8_{\pm 2}$	$9.5_{\pm 3.7}$	$3.3_{\pm 1.7}$	$0.5_{\pm 1.0}$	$2.5_{\pm 1.5}$
Qwen3-Omni	$\mathbf{39.0}_{\pm 2.0}$	27.0/89.5	33.6/41.7	41.5/24.2	75.0/0.5	$\mathbf{89,6}_{\pm 2}$	$\mathbf{57.7}_{\pm 2}$	$\mathbf{22.2}_{\pm 4.9}$	$\mathbf{6.0}_{\pm 2.2}$	$\mathbf{1.6}_{\pm 1.4}$	$\mathbf{3.3}_{\pm 1.7}$
Kimi-Audio	$25.6_{\pm 2.0}$	27.8/1.6	29.6/1.3	22.2/5.4	31.2/94.1	$78.1_{\pm 2}$	$48.6_{\pm 2}$	$6.7_{\pm 3.3}$	$1.7_{\pm 1.4}$	$0.5_{\pm 1.0}$	$0.8_{\pm 1.0}$
MiMo-Audio	$32.8_{\pm 2.0}$	40.6/40.0	30.4/91.2	—/0.0	—/0.0	$81.7_{\pm 2}$	$46.5_{\pm 2}$	$12.0_{\pm 4.0}$	$4.0_{\pm 2.0}$	$1.0_{\pm 1.0}$	$2.0_{\pm 1.2}$

Table 1: Benchmark on the 1,000 test scenes. Whisper + LLM is the text-only control. All scores in %, chance is 25 for all accuracies. Balanced accuracy is the mean of the four per-class accuracies. Recall and precision are provided for each reaction class. Acoustic scene classification and audio tagging are four-way multiple choice. Summary recall refers to the proportion of annotated sounds of this type that are mentioned in the 200-words summary generated by the model. (

\pm

) give the 95 % confidence interval. Best per column in bold , except for per-class precision/recall. Results significantly higher than the text-only control are in blue . Classes with a Reaction Classification recall

\leq 7\%

are shown in red .

Explore similar work

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models
Jun 24, 2026Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif +6Large Audio Language Models
University of California Irvine · University of Illinois Chicago · Kennesaw State University
TAC: Timestamped Audio Captioning
Feb 17, 2026Sonal Kumar, Prem Seetharaman, Ke Chen +8
University of Maryland, College Park, USA · Adobe Research, USA · OpenAI, USA (work done while at Adobe).
Logbook: Extremely Long-form Audio Event Understanding
Oct 5, 2026Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk +7Audio Understanding
UT Austin, USA · Meta Reality Labs, USA

Jun 24, 2026, cs.SDJ/K move · Enter open · S save

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif, Yutong Song+5

University of California Irvine · University of Illinois Chicago · Kennesaw State University

Large Audio Language Models

Open paper AskPDF arXiv

Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including speech, sound, and music. However, existing benchmarks predominantly evaluate these layers in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world auditory scenes. Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU): the ability to comprehend the holistic scene by integrating sound layers. To evaluate this capability, we introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events (e.g., announcements), and background environments (e.g., traffic), and reason about the logical relationships between these layers. We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech. Building on this data, we design four tasks that probe scene understanding: contextual question answering, entity extraction from the scene, speaker role inference, and counterfactual reasoning where scene is manipulated. Experiments across multiple LALMs demonstrate that effective auditory scene understanding requires integration over all auditory layers, rather than reliance on speech or sound alone, underscoring the necessity of CASU for advancing complex audio understanding in LALMs.

Abstract

Explore similar work

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

TAC: Timestamped Audio Captioning

Logbook: Extremely Long-form Audio Event Understanding