Audio Captioning

Follow this topic Ask about this topic

Audio captioning studies how to describe environmental sounds, speech, and music in language that captures what is audible and, increasingly, when events occur. Systems use audio encoders paired with language models, trained on paired audio-text data with contrastive or caption-generation objectives; newer approaches add synthetic fine-grained data, temporal supervision, and preference-based reinforcement learning. Benchmarks evaluate caption quality, coverage, temporal grounding, and hallucination, while datasets such as AudioCaps and Clotho support training and comparison. A central tension is producing detailed, useful descriptions without adding claims unsupported by the audio, especially across different sound domains and in long-form captions.

Latest papers

23 papers2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

TAC2: Timestamped Audio Captioning as composable functions
Oct 27, 2026Swapnil Bhosale, Prem Seetharaman, Ke Chen +3Audio-Language Models Audio Captioning
University of Surrey, UK · Adobe Research, USA
A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models
Oct 15, 2026Shreeram Suresh Chandra, Ashwyn Sharma, Liang Lu +4Audio Captioning Audio-Language Model Evaluation
Center for Language and Speech Processing, Johns Hopkins University, USA · Otter AI · The University of Texas at Dallas, USA +1
CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound
Oct 7, 2026Marcel Gibier, Thomas Thebaud, Olivier Boëffard +1Audio Understanding Audio Captioning
Inria, Paris, France · AMIAD, France
The Internet Archive Music Dataset
Sep 15, 2026Paraskevas Stamatiadis, Bernardo V Miranda, Clémentine Berger +3Audio Captioning
S2A, LTCI, IDS · LTCI, Télécom Paris, Institut Polytechnique de Paris, France · LTCI, IP Paris, S2A, IDS, IMT +1
SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval
Sep 2, 2026Zineb Lahrichi, Marc Ferras, Gaël Richard +1Audio Captioning Audio-Language Models
Sony CTC, France · LTCI, Telecom Paris, Institut polytechnique de Paris, France
Closing the Verification Loop: Self-Check Captioning for Long-Paragraph Detailed Audio Captioning
Aug 31, 2026Fengji Ma, Yan Rong, Xu Li +3Audio-Language Models Audio Captioning
The Hong Kong University of Science and Technology (Guangzhou) · Kling Team, Kuaishou Technology
AllMusicCaps: Album Reviews as Complementary Supervision for Music CLAP
Aug 26, 2026Pablo Alonso-Jiménez, Xavier Lizarraga-Seijas, Xavier Serra +1Audio-Language Models Audio-Text Retrieval
Music Technology Group, Universitat Pompeu Fabra, Barcelona
ACE-Cap: Active Evidence Acquisition via Agentic Co-Evolution for Long-Paragraph Fine-Grained Audio Captioning
Aug 17, 2026Fengji Ma, Yan Rong, Xu Li +3Audio Captioning Audio Understanding
The Hong Kong University of Science and Technology (Guangzhou) · KlingAI Research
AudioMap: Cloze-and-Choice Reinforcement Learning for Time-Aware Dense Audio Captioning
Aug 10, 2026Yan Rong, Fengji Ma, Xu Li +3Audio Captioning
The Hong Kong University of Science and Technology (Guangzhou) · Kling Team, Kuaishou Technology
MMAC: A Massive Multi-dimensional Benchmark for Audio Captioning
Jul 29, 2026Weijie Wu, Junbo Li, Lin Li +2Audio-Language Model Evaluation Audio Captioning
School of Informatics, Xiamen University, China · DiDi Global Inc., Beijing, China · School of Electronic Science and Engineering, Xiamen University, China
CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning
Jul 6, 2026Ganesh Pavan Kartikeya Bharadwaj Kolluri, Yuchen Zhang, Michael Kampouridis +1Audio Representation Learning Audio-Language Models
School of Computer Science and Electronic Engineering, University of Essex · Institute for Analytics and Data Science, University of Essex
MOSS-Audio Technical Report
Jun 1, 2026Chen Yang, Chufan Yu, Hanfu Chen +27Audio Understanding Audio-Language Models
COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings
May 28, 2026Yonggang Zhu, Liting Gao, Aidong Men +1Audio Understanding Audio-Language Models
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Centre for Vision, Speech, and Signal Processing (CVSSP), University of Surrey, GU2 7XH Guildford, U.K.
Locality Matters for Training-Free Audio Token Compression in Audio-Language Models
May 24, 2026Jiale Luo, Xiaoyu Liang, Haoji HuToken Merging Audio Token Compression
1Zhejiang University
Noise-Aware In-Context Learning for Hallucination Mitigation in ALLMs
Apr 10, 2026Qixuan Huang, Khalid Zaman, Masashi UnokiAudio-Language Models Audio-Language Model Hallucination Detection
Graduate School of Advanced Science and Technology, Japan Advanced Institute of Science and Technology 1-1 Asahidai, Nomi, Ishikawa 923–1292 Japan
EvA: An Evidence-First Audio Understanding Paradigm for LALMs
Mar 29, 2026Xinyuan Xie, Shunian Chen, Zhiheng Liu +4Audio Understanding Audio-Language Models
The Chinese University of Hong Kong, Shenzhen · Didi Chuxing
ACAVCaps: Enabling large-scale training for fine-grained and diverse audio understanding
Mar 25, 2026Yadong Niu, Tianzi Wang, Heinrich Dinkel +6Audio Captioning Audio-Language Models
MiLM Plus, Xiaomi Inc, Beijing, China · The Chinese University of Hong Kong, Hong Kong, China
EmoSURA: Towards Accurate Evaluation of Detailed and Long-Context Emotional Speech Captions
Mar 10, 2026Xin Jing, Andreas Triantafyllopoulos, Jiadong Wang +3Audio Captioning
CHI – Chair of Health Informatics, TUM University Hospital, Munich, Germany · MCML - Munich Center for Machine Learning, Munich, Germany · Huawei, Netherlands +1
AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech
Feb 27, 2026Jielin Qiu, Jianguo Zhang, Zixiang Chen +13Audio-Language Model Evaluation Audio Captioning
Salesforce AI Research
TAC: Timestamped Audio Captioning
Feb 17, 2026Sonal Kumar, Prem Seetharaman, Ke Chen +8Audio Captioning Temporal Audio Grounding
University of Maryland, College Park, USA · Adobe Research, USA · OpenAI, USA (work done while at Adobe).
Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
Feb 5, 2026Jinchuan Tian, Haoran Wang, Bo-Hao Su +14Audio Understanding Audio-Language Models
1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model Evaluation Audio Representation Learning
University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1
Aligning Audio Captions with Human Preferences
Sep 18, 2025Kartik Hegde, Rehana Mahfuz, Yinyi Guo +1Reward Modeling Audio-Language Models
Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA

Oct 27, 2026J/K move · Enter open · S save

TAC2: Timestamped Audio Captioning as composable functions

Swapnil Bhosale, Prem Seetharaman, Ke Chen, Haonan Chen+2

University of Surrey, UK · Adobe Research, USA

Audio-Language Models Audio Captioning Temporal Audio Grounding Speech Processing

Open paper AskPDF arXiv

Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.