Listen, Reason, and Segment: Aligning LALMs with Editorial Judgment for Media Chapterization

Tony Alex; Wish Suharitdamrong; Sara Atito; Armin Mustafa; Muhammad Awais; Philip J. B. Jackson; Jiankang Deng; Ismail Elezi

cs.SDAug 17, 2026

Listen, Reason, and Segment: Aligning LALMs with Editorial Judgment for Media Chapterization

Authors: Tony Alex, Wish Suharitdamrong, Sara Atito, Armin Mustafa, Muhammad Awais, Philip J. B. Jackson, Jiankang Deng, Ismail Elezi

Organizations: University of Surrey · Huawei Noah’s Ark Lab

Abstract

Large Audio Language Models (LALMs) have made rapid progress on standardized benchmarks, yet their deployment in practical media workflows, curation, archival indexing, and content distribution remains largely unrealized. We identify automated audio chapterization, the task of segmenting continuous audio streams into thematically coherent chapters, as a demanding and commercially consequential setting that exposes this gap. Chapterization is challenging because boundaries are defined less by objective acoustic events than by subjective editorial judgment, requiring models to reason sequentially over long acoustic contexts and approximate creator-authored boundary decisions. We present AudioChaps, a post-training framework for aligning end-to-end LALMs for this task via Group Relative Policy Optimization (GRPO) guided by Chain-of-Thought (CoT) reasoning. To support training and evaluation, we curate three datasets: AudioChaps-Alignment, derived from creator-annotated chapter boundaries on YouTube; AudioChaps-CoT, which provides structured supervision for well-formatted, high-quality, and evidence-grounded boundary reasoning; and AudioChaps-Eval, a held-out benchmark for audio chapterization. Applying GRPO directly without a Supervised Fine-Tuning (SFT) cold start, AudioChaps-R1-Zero already improves average F1 by 33 points over the state-of-the-art LALM Audio-Flamingo-3-Think. The AudioChaps framework produces our final aligned LALM, AudioChaps-R1, which improves average F1 by 49 points. These results demonstrate that GRPO-trained LALMs can reliably transform unstructured auditory streams into navigable, structured media. Our code, models, and dataset resources will be released upon acceptance at https://github.com/ta012/AudioChaps.

Figures & tables

Figure 1: Illustration of the audio chapterization task: the media’s audio is analysed in 60-s chunks via AudioChaps-R1-8B to determine the chapter boundaries, presented to the user as a timeline above the scrubber bar.

Figure 2: AudioChaps-CoT curation pipeline. A boundary-relevant pseudo-CoT is generated, sanitized into an acoustic perception log, and then converted into the final structured CoT target used for the SFT cold start.

	$\displaystyle J_{\mathrm{GRPO}}(\theta)$	$\displaystyle=\mathbb{E}_{q\sim P(Q),\;\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(O\mid q)}$		(1)
		$\displaystyle\left[\frac{1}{G}\sum_{i=1}^{G}\min\left(\frac{\pi_{\theta}(o_{i}\mid q)}{\pi_{\theta_{\mathrm{old}}}(o_{i}\mid q)}A_{i},\;\mathrm{clip}\left(\frac{\pi_{\theta}(o_{i}\mid q)}{\pi_{\theta_{\mathrm{old}}}(o_{i}\mid q)},1-\epsilon,1+\epsilon\right)A_{i}\right)-\beta D_{\mathrm{KL}}\left(\pi_{\theta}\parallel\pi_{\mathrm{ref}}\right)\right].$		(1)

Table 3

Model	sc	Acc	Pre	Rec	F1
	DM	60.2	55.3	62.7	58.8
Step-Audio-R1-32B	G	60.1	59.1	53.4	56.1
(Zero-Shot)	M	62.4	53.1	51.9	52.5
	SS	65.6	70.1	69.7	69.9
	Avg	62.1	59.4	59.4	59.3
	DM	55.8	47.5	23.7	31.6

Table 1: Performance comparison on AudioChaps-Eval across the four audio chapterization subcategories: Dynamic Media (DM), Gaming (G), Music (M), and Structured Speech (SS). Accuracy (Acc), precision (Pre), recall (Rec), and F1 are reported as percentages, together with the macro-average (Avg) over subcategories. Detailed comparison with additional multimodal and omni-modal LLMs is provided in Supplementary Table 7 .

Model	F1	Pre	Rec	dev-R2E
Fixed interval, 180 s	9.5	9.0	11.6	49.0 s
AF3-Think-8B	6.5	8.6	9.6	38.0 s
AudioChaps-R1-8B	37.6	34.1	45.7	10.0 s

Table 2: Full-length chapter-boundary detection on uncropped audio. Predictions are matched one-to-one with reference boundaries under a

\pm 10

-second tolerance. Precision, recall, and F1 are computed per recording, macro-averaged within each acoustic subtype, and then averaged equally across the four subtypes. dev-R2E denotes the pooled median temporal distance from each reference boundary to its nearest predicted boundary. Detailed results are provided in Supplementary Table 8 .

Model	DM	G	M	SS
AF3-Think-8B	31.6	49.9	6.0	27.0
AudioChaps-R1-Zero-8B	57.8	64.4	59.1	62.9
AudioChaps-SFT-8B	70.0	72.5	74.1	77.9
AudioChaps-R1-8B	73.4	75.5	84.6	77.8

Table 3: Ablation of the AudioChaps training stages. F1 scores are reported for Dynamic Media (DM), Gaming (G), Music (M), and Structured Speech (SS). Detailed results are provided in Supplementary Section B , Table 5 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: The unified prompt template used in both the SFT cold-start and GRPO stages. The model is asked a binary single-choice question, instructed to reason with natural reflective markers inside <think> tags, and to emit a single option letter inside <answer> tags.

Hyperparameter	SFT	GRPO
Epochs	2	1
Learning rate	$1\mathrm{e}{-6}$	$1\mathrm{e}{-6}$
Per-device batch size	1	1
Gradient accumulation	4	1
Max completion length	—	768
Generations per prompt ( $G$ )	—	8

Appendix

Table 4: Training hyperparameters for the SFT cold-start and GRPO alignment stages.

Model	sc	Acc	Pre	Rec	F1
	DM	55.8	47.5	23.7	31.6
AF3-Think-8B	G	50.1	45.9	54.6	49.9
(Base-LALM)	M	62.3	64.0	3.2	6.0
	SS	44.7	46.4	19.1	27.0
	Avg	53.2	51.0	25.2	28.6
	DM	61.6	55.0	61.0	57.8

Appendix

Table 5: Detailed ablation results corresponding to Table 3 in the main paper across four audio chapterization subcategories (sc): Dynamic Media (DM), Gaming (G), Music (M), and Structured Speech (SS). AF3-Think-8B is the unaligned base LALM, which lacks a sufficiently strong reasoning prior and does not reliably follow the target structured output format. AudioChaps-R1-Zero-8B applies GRPO directly to AF3-Think-8B without an SFT cold start. AudioChaps-SFT-8B uses supervised fine-tuning to establish the target reasoning behavior and structured output format, while AudioChaps-R1-8B subsequently applies GRPO to calibrate the model’s chapter-boundary decisions. This ablation isolates the contribution of the SFT cold start and the subsequent GRPO alignment stage. The macro average (Avg) is reported over the four subcategories.

Model	sc	Acc	Pre	Rec	F1
	DM	55.8	47.5	23.7	31.6
AF3-Think-8B	G	50.1	45.9	54.6	49.9
(Base-LALM)	M	62.3	64.0	3.2	6.0
	SS	44.7	46.4	19.1	27.0
	Avg	53.2	51.0	25.2	28.6
	DM	75.9	70.0	77.2	73.4

Appendix

Table 6: Backbone-generalization results across audio chapterization subcategories (sc): Dynamic Media (DM), Gaming (G), Music (M), and Structured Speech (SS). Grey rows report the AF3-based models for reference. Across both AF3 and MOSS, GRPO alignment improves precision (Pre), accuracy (Acc), and F1 while reducing the high-recall behaviour of the preceding model. Avg denotes the macro-average over subcategories.

Model	sc	Acc	Pre	Rec	F1
	DM	72.5	66.7	78.8	72.2
Gemini-2.5-Flash	G	73.0	67.9	82.5	74.5
(Zero-Shot)	M	81.0	74.9	81.4	78.0
	SS	78.3	77.3	88.0	82.3
	Avg	76.2	71.7	82.7	76.8
	DM	73.1	68.6	69.2	68.9

Appendix

Table 7: Comparison with additional multimodal and omni-modal LLMs on AudioChaps-Eval across four audio chapterization subcategories: Dynamic Media (DM), Gaming (G), Music (M), and Structured Speech (SS). Accuracy (Acc), precision (Pre), recall (Rec), and F1 are reported as percentages, together with the macro-average (Avg) over the four subcategories. Gemini-2.5-Flash, Qwen3-Omni-30B-A3B-Thinking, MOSS-Think-8B, Step-Audio-R1-32B, and AF3-Think-8B are evaluated in the zero-shot setting. The two AudioChaps-R1 variants apply our alignment framework to the AF3-Think and MOSS-Think backbones, respectively.

Model	sc	Pre	Rec	F1	dev-R2E $\downarrow$
	DM	18.9	13.5	10.7	–
AF3-Think-8B	G	3.4	10.5	4.6	–
(Window)	M	0.4	1.7	0.7	–
	SS	11.8	12.7	10.0	–
	Overall	8.6	9.6	6.5	38.0 s
	DM	32.6	38.6	34.5	–

Appendix

Table 8: Full-length audio chapter-boundary detection across four acoustic subcategories: Dynamic Media (DM), Gaming (G), Music (M), and Structured Speech (SS). Predictions are matched one-to-one with reference boundaries under a

\pm 10

-second tolerance. Precision (Pre), recall (Rec), and F1 are computed per recording, macro-averaged within each subcategory, and then averaged equally across the four subcategories. dev-R2E denotes the pooled median distance from each reference boundary to its nearest prediction and is reported only in the Overall row; lower values indicate better localization. The window-run decoder merges contiguous positive windows into a single recording-level boundary estimate.

Figure 4: Per-subtype clip counts for the training (AudioChaps-Alignment) and held-out test (AudioChaps-Eval) splits, split by positive (boundary-spanning) and negative (within-chapter) clips. Both splits follow the same four-way stratification, with no source-video overlap between the training set and the held-out test set.

Panel of Figure 5 — Figure 5: Instructions shown to human evaluators. Each sample pairs a 1-minute clip and the boundary question with two anonymized, randomly ordered model outputs (Output A and Output B), each rated on a 1–5 scale.

Figure 6: Qualitative comparison of AF3-Think-8B and AudioChaps-R1-8B on human-evaluation (HE).

Figure 7: Qualitative comparison of AF3-Think-8B and AudioChaps-R1-8B on human-evaluation.

Figure 8: Qualitative comparison of AF3-Think-8B and AudioChaps-R1-8B on human-evaluation.

Figure 9: Qualitative comparison of AF3-Think-8B and AudioChaps-R1-8B on human-evaluation.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Sep 9, 2026

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.

Yupei Li, Qiyang Sun, Mohamed Mady +4

Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE +2

Audio-Language Model Evaluation Audio Reasoning Audio-Language Models

Ask PDF

Oct 7, 2026

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.

Pooneh Mousavi, Mirco Ravanelli, Cem Subakan

Concordia University · Mila – Quebec AI Institute · Laval University

Audio-Language Models Audio Reasoning Audio QA

Ask PDF

Mar 10, 2026

ALARM: Audio-Language Alignment for Reasoning Models

Large audio language models (ALMs) extend LLMs with auditory understanding. A common approach freezes the LLM and trains only an adapter on self-generated targets. However, this fails for reasoning LLMs (RLMs) whose built-in chain-of-thought traces expose the textual surrogate input, yielding unnatural responses. We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment. We further fuse and compress multiple audio encoders for stronger representations. For training, we construct a 6M-instance multi-task corpus (2.5M unique prompts) spanning 19K hours of speech, music, and sound. Our 4B-parameter ALM outperforms similarly sized models and surpasses most larger ALMs on related audio-reasoning benchmarks, while preserving textual capabilities with a low training cost. Notably, we achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.

Petr Grinberg, Hassan Shahmohammadi

EPFL, Switzerland · Sony Europe Ltd., Germany

Audio-Language Models Audio Reasoning Audio Representation Learning Self-Training for Language Models

Ask PDF

Abstract

Audio-Language Models

Explore similar work

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

ALARM: Audio-Language Alignment for Reasoning Models