Long-paragraph fine-grained audio captioning requires models to recover diverse acoustic facts while avoiding omissions and unsupported details. However, prevailing captioners remain passive one-shot generators: once a detail is overlooked, they cannot identify the evidence gap, query the audio for targeted information, or decide when sufficient evidence has been collected. We formulate this task as active evidence acquisition and introduce Agentic Co-Evolution for Captioning (ACE-Cap). The framework uses multi-turn interaction between a Composer and an Instruct model to form a closed evidence-acquisition loop. A Captioner first produces an initial description. Conditioned on this description and the interaction history, a text-only Composer asks targeted questions about unresolved acoustic attributes, while an audio-conditioned Instruct model provides grounded answers. The Composer then decides when to terminate and synthesizes the accumulated evidence into a final caption. ACE-Cap trains these roles through a unified gold-to-prediction reward derived from fixed, gold-grounded multiple-choice questions and a frozen caption-only judge. For credit assignment in variable-length interactions, LOOP-GRPO replaces the trajectory-wide scalar advantage with span-aligned signals: leave-one-out contributions of individual questions to the accumulated evidence, a quality-cost utility for stopping, and an evidence-preservation utility for final synthesis. Role-wise warm-up followed by alternating Composer and Instruct optimization keeps each update a well-defined single-policy problem while allowing the roles to co-evolve. ACE-Cap thus turns captioning from passive one-shot generation into an adaptive process that learns what evidence to acquire, when to stop, and how to preserve it in a long-paragraph caption.
Figures & tables
Figure 1: Paradigm comparison for long-paragraph fine-grained audio captioning. Passive one-shot captioners produce a caption without actively resolving missing details (top). ACE-Cap instead uses a text-only Composer to query an audio-conditioned Instruct model, accumulates the resulting question–answer evidence as Accumulated Evidence Eg , and synthesizes a detailed final caption (bottom).
Figure 2: Gold-MCQ construction and G2P scoring. Colors link fixed MCQs to gold-caption evidence spans. Candidate questions are filtered offline; during GRPO, the frozen judge scores only the candidate caption against the retained MCQs, without audio or the gold caption.
Method
MMAU ↑
MMAR ↑
MMSU ↑
Proprietary models
GPT-4o Audio
62.4
59.3
56.4
Gemini2.0-Flash
58.6
50.6
51.0
Gemini2.5-Flash
65.6
58.2
58.1
Gemini2.5-Pro
70.0
64.1
71.8
Gemini3.1-Pro
71.3
65.7
69.5
Table 1: Caption-as-evidence accuracy (%) on the audio-only MMAU, MMAR, and MMSU evaluations. Qwen3.6-27B answers each question using only the generated caption as audio-derived evidence. All entries are reserved for results obtained under the unified protocol. Best and second-best results among open-source models are highlighted in bold and underlined, respectively.
Method
Omni-Cloze ↑
Proprietary models
GPT-4o Audio
35.8
Gemini2.0-Flash
20.0
Gemini2.5-Flash
42.6
Gemini2.5-Pro
48.0
Gemini3.1-Pro
64.1
Table 2: Audio-only results on the Omni-Cloze audio subset. The metric is official overall accuracy (%). All entries are reserved for results obtained under the same caption-only Qwen3.6-27B evaluation protocol.
Figure 3: Composer analysis on Omni-Cloze. (a) Query-category behavior: sector angle denotes the proportion of queries assigned to each category, while radius denotes the category-specific mean query turn. (b) Accuracy versus the maximum Composer query budget K . Dashed and solid lines evaluate the Accumulated Evidence Eg and final caption cgF , respectively. Blue denotes the frozen base Qwen3.6-27B Composer, which was not trained with LOOP-GRPO or alternating co-evolution. Teal denotes our Qwen3-8B Composer, which was trained with both.
Figure 4: Ablation of the ACE-Cap alternating training pipeline on MMAR. S1–S5 follow the ordered training stages in Algorithm 1 : S1 is the all-SFT initialization; S2 applies Captioner GRPO after S1; S3 further applies Instruct GRPO; S4 further applies Composer LOOP-GRPO warm-up; and S5 is the final ACE-Cap model after alternating Composer–Instruct co-evolution. Accuracy (%) is reported for the four MMAR capability layers and the full test set. The y-axis begins at 20%. The corresponding single- and mixed-modality results are reported in Appendix C.1 of the supplementary document.
Long-paragraph detailed audio captioning, which requires dense and transcript-faithful descriptions of fine-grained audio content, remains unsolved for current audio-visual multimodal language models. We attribute this failure to two structural problems. The first is data poverty, as no public corpus jointly provides long clips, paragraph captions, and verbatim-transcript fidelity. The second is generation-mode failure, evidenced by a 44.8 to 46.4 percentage-point gap between right-audio and shuffled-audio multiple-choice question (MCQ) accuracy. We address both within Self-Check Captioning (SCC), a unified framework that instantiates audio-grounded question answering as the verification primitive at every lifecycle stage. SCC yields three artifacts. Long-paragraph Audio Caption 50k (LACap-50k) is a 50,222-clip audio-visual corpus with 491.5-word captions and a post-hoc automatic speech recognition (ASR) audit. Layer-Curvature Supervised Fine-Tuning (LC-SFT) is the first on-policy supervised fine-tuning method to weight tokens by intermediate-layer evidence, motivated by our identification of Late-Layer Semantic-Entropy Collapse (SEC). SCC-Verifier arbitrates among caption rollouts via audio-grounded self-answering at inference. Across multiple benchmarks, our system attains state-of-the-art among open-source captioners and is competitive with proprietary baselines. We release LACap-50k to fill the resource gap for long-paragraph detailed audio captioning research.
Fengji Ma, Yan Rong, Xu Li +3
The Hong Kong University of Science and Technology (Guangzhou) · Kling Team, Kuaishou Technology
Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introduce Timestamped Audio Captioner (TAC), a model that produces temporally grounded audio descriptions at varying degrees of detail and resolution. TAC is trained with a synthetic data pipeline that constructs challenging and dynamic mixtures from real-world audio sources, enabling robust learning under realistic polyphonic conditions. Across event detection and dense captioning, TAC outperforms all competing methods, with a low hallucination rate and accurate temporal grounding. We also introduce TAC-V, an audio-visual pipeline to generate semantically rich audio-visual descriptions. We then show that TAC and TAC-V serves as a "semantic bridge" for a text-only reasoner: a simple TAC→LLM and TAC-V→LLM cascade achieves state-of-the-art scores on benchmarks for both audio (MMAU-Pro, MMSU, MMAR) and audio-visual (DailyOmni, VideoHolmes) understanding and reasoning respectively.
Sonal Kumar, Prem Seetharaman, Ke Chen +8
University of Maryland, College Park, USA · Adobe Research, USA · OpenAI, USA (work done while at Adobe).
We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: accuracy (semantic correctness), completeness (coverage of reference content), and hallucination (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini3Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.