TAC2: Timestamped Audio Captioning as composable functions

Swapnil Bhosale; Prem Seetharaman; Ke Chen; Haonan Chen; Justin Salamon; Zeyu Jin

cs.SDOct 27, 2026

TAC2: Timestamped Audio Captioning as composable functions

Authors: Swapnil Bhosale, Prem Seetharaman, Ke Chen, Haonan Chen, Justin Salamon, Zeyu Jin

Organizations: University of Surrey, UK · Adobe Research, USA

Abstract

Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.

Figures & tables

Figure 1: Overview of the two-stage TAC2 / TAC2-S framework. (1) Perceptual Foundation ( TAC2 ): Predictions from weak teachers (SED, Diarization, Captioner) are merged via time-registered IoU matching to generate intermediate structured captions, which train a base adapter via the proposed Instruction-Input-Output (IIO) format. (2) Speech Elicitation ( TAC2-S ): Individual LoRA specialists (Stress, Emotion, Voice Profile, Acoustics) are trained on small gold corpora to elicit paralinguistic attributes from the pre-trained backbone. This cascade re-annotates the corpus, which is subsequently distilled into our unified student model, TAC2-S .

Figure 2: Programmatic data compilation via time-registered IoU matching. Predictions from three independent weak teachers (

m_{1}

: audio captioner;

m_{2}

: speaker diarizer;

m_{3}

: sound event detector) are aligned on a shared time axis using temporal IoU with overlap gate (Eq. 1 , 2 ). Their aligned predictions form the fused target timeline

\hat{S}

(Eq. 3 ), containing concurrent acoustic events with speaker attribution.

Figure 3: The IIO completion format. The fused target

\hat{S}

is decomposed into ( INSTRUCTION , INPUT , OUTPUT ) completion tasks. The INSTRUCTION specifies the task, INPUT provides optional teacher-forced context, and OUTPUT specifies the target attributes to generate. Structural tasks train TAC2 ( caption , diarize , and a single-decode comprehensive task). Appending elicited speech tasks ( stress , emotion , voice profile , acoustics ) seamlessly extends this format to TAC2-S without changing the training procedure. At inference, the comprehensive task supports single-pass generation, while sequencing attribute-specific tasks with preceding outputs yields a modular multi-pass cascade.

Model	DtR	MMSU			MMAR				MMAU				Avg.
Model	DtR	Perception	Reasoning	Overall	Speech	Sound	Music	Overall	Speech	Sound	Music	Overall	Avg.
Kimi-Audio	✗	44.8	75.7	59.8	58.5	49.7	33.0	48.0	62.2	75.7	66.8	68.2	58.7
Qwen2.5-Omni	✗	42.7	77.6	58.1	59.9	58.8	40.8	56.7	70.6	78.1	65.9	71.5	62.1
StepAudio2	✗	42.9	73.2	57.6	61.2	54.6	42.2	56.8	68.2	79.3	68.4	72.7	61.9
UAS-Audio	✗	55.7	77.4	66.2	66.0	58.8	45.2	60.1	67.0	70.0	71.3	69.4	65.2
Gemini-3.7 Flash	✓	68.7	84.4	76.3	82.7	51.5	56.8	69.5	80.5	66.1	67.1	71.2	72.3

Table 1: Spoken and general audio understanding. Models in DtR ✓ rows emit a text schema as the description, which is read by a frozen text-only reasoner. Each score reflects the fidelity and coverage of audio perception in its generated description.

Text-only reasoner		SP	MP
TAC2-S	$\rightarrow$ Gemini3	81.3	81.9
	$\rightarrow$ Qwen3-Next	78.8	79.7
	$\rightarrow$ Qwen3.8	80.5	80.9

Table 2: Inference and Reasoner Robustness. Left: TAC2-S schemas under single-pass (SP) vs. multi-pass (MP) comprehensive decodes across text-only reasoners (top), and the MMSU tasks where MP helps most (bottom). Right: Perception fidelity. TAC2-S uses describe-then-reason; OmniLLM baselines process raw audio directly (left radar) or act as describers (right radar). TAC2-S (ours), Gemini 3.7, Qwen3Omni

System	T1	T2	Avg
Kimi-Audio	41.0	47.0	43.0
Qwen2.5-Omni	19.0	21.0	19.0
StepAudio2	44.0	46.0	44.0
AudioFlamingo-3	40.0	38.0	39.0
MiMoAudio	52.0	64.0	56.0
Gemini-3.7 Flash	70.7	84.8	76.4

Table 3: Speaker-centric understanding on MSU-Bench. Evaluated on conversational multi-speaker scenarios in English/Mandarin. Left: Consistent with Table 1 , TAC2-S uses a text reasoner. Right: Tier 1 (Identification) task accuracy. (AI: accent identification, AR: age recognition, GI: gender identification, ER: emotion recognition, SP: speaker profiling, SR: speaker retrieval, SVS: speaker-specific viewpoint summarization, SC: speaker counting, SV: speaker verification) OmniLLM baselines process raw audio directly (left radar) or act as describers (right radar). TAC2-S ’s trace (blue) is superior across both, demonstrating robust speaker-centric perception. TAC2-S (ours), Gemini 3.7, Qwen3Omni

Figure 4: Impact of individual IIO tasks. (left) Building TAC2-S ’s description one attribute at a time, from a transcript-only base (who spoke, the words, and their timestamps). (right) For each attribute, the top-3 it lifts most, the gains land on the tasks that attribute governs and leave the transcript carried semantics tasks flat. Bar colour marks the added attribute (blue: stress and intonation, pink: emotion, orange: voice profile(demographics), green: pitch and loudness).

	TAC2-S -as-judge							Temporal grounding			Gemini-3-Pro judge
System	Gen.	Age	Acc.	Emo.	Ev.R.	Sp.R.	WER	PE	FLAM	PE-AV	vs. real	vs. cap.
	$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\downarrow$	$\downarrow$	$\downarrow$	$\downarrow$	$\uparrow$	$\uparrow$
PicoAudio2 ( Zheng et al., 2026 )	0.44	0.35	0.24	0.28	0.27	0.09	1.15	2.47 $\mid$ 2.77	3.16 $\mid$ 3.50	0.52 $\mid$ 0.51	1.39	1.61
MiDashengLM-Gen ( Sun et al., 2026 )	0.54	0.36	0.21	0.36	0.33	0.09	1.39	3.57 $\mid$ —	3.78 $\mid$ —	0.44 $\mid$ —	1.54	1.60
Seed Audio 1.0 ( ByteDance Seed, 2026 )	0.94	0.89	0.71	0.74	0.78	0.90	0.14	2.16 $\mid$ —	2.62 $\mid$ —	0.35 $\mid$ —	3.53	4.05
Ours ( TAC2 )	—	—	—	—	0.80	0.94	0.12	1.67 $\mid$ 2.47	2.13 $\mid$ 2.77	0.35 $\mid$ 0.35	3.38	4.01

Table 4: Structured captions generated by TAC2-S provide renderable supervision for audio synthesis. Temporal-grounding cells report with

\mid

without input timestamps. PE and FLAM measure mean localizer-peak offset in seconds; PE-AV is a text-free frame-wise semantic distance between original and reconstructed audio (real-vs.-real

=0

). Gemini-3-Pro judgments are on a

1

–

5

scale.

Figure 5: TAC2-S -conditioned reconstruction. We caption real audio (top) with TAC2-S (middle), then condition the trained text-to-audio generator on it to synthesize new audio (bottom). The reconstruction reproduces speech and non-speech events, including background elements like a censor beep , engine rev , camera click , at times aligned with the real audio.

Figure 6: Word-level emphasis as a renderable control channel. Each column synthesizes the same sentence while varying the emphasized word in the TAC2-S caption (yellow). The bottom two rows increase emphasis from mild ( <e> ) to strong ( <E> ) while the cyan curve shows the F0 contour. Bottom panels show WhiStress ( Yosha et al., 2025 ) stress probability over time, with $\star$ marking the commanded word. The detected stress peak follows the targeted word and scales with emphasis intensity, demonstrating zero-shot emphasis control without a dedicated stress-control model.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Trait (granularity)	Backbone peak	Early (base / spec)	Late (base $\to$ spec)	Lift
Relative loudness (segment)	0.89	0.79 / 0.81	0.87 $\to$ 0.99	$+0.12$
Relative pitch (segment)	0.90	0.80 / 0.82	0.89 $\to$ 0.99	$+0.10$
Word stress (word)	0.93	0.88 / 0.89	0.91 $\to$ 0.99	$+0.08$
Accent (speaker)	0.99	0.97 / 0.97	0.99 $\to$ 1.00	$+0.01$

Appendix

Table 5: The backbone (Qwen3Omni-captioner) possesses latent perception; the specialist isolates late-stage readouts. Early / Late : mean AUC over depths 0–24 / 36–48, backbone vs. specialist. Lift : the specialist’s late-stage gain.

Attribute (metric)	Frozen backbone	+ specialist
Word stress (top-word accuracy)	0.60	0.81
Accent (exact, 13 classes)	0.37	0.62
Accent (region)	0.61	0.91
Emotion (5 classes)	0.15	0.59

Appendix

Table 6: The specialist mitigates output mode collapse without altering early perception. Accuracy of the emitted textual attribute, frozen backbone vs. its specialist, on clips with ground-truth labels (word stress is scored as the top stressed word against Whistress ( Yosha et al., 2025 ) ).

Task	Input	Weight	Output
comprehensive	none (audio)	1.0	full description (all sections)
caption	none (audio)	0.5	overview + events with transcript
stress	transcript turns	0.5	per-word stress and intonation
emotion	transcript turns	0.5	per-turn emotion
voice profile	transcript turns	0.3	per-speaker gender, age, accent
acoustics	transcript turns	0.3	per-segment pitch and loudness

Appendix

Table 7: IIO tasks during training. Each training sample is formulated as a single completion task, with weights denoting their respective sampling probabilities. The comparative speech specialists are conditioned on the speaker turns and transcript as teacher-forced inputs.

Figure 7: Segment localization on conversational speech. The segments from the human annotation, TAC2 , the Qwen3-FA aligner, and the wav2vec2-based aligner share the time axis. TAC2 ’s segment onsets track the human pause boundaries closely, while being competitive with the dedicated aligners. (Dotted lines mark the human segment onsets.)

System	Onset (ms)	F1@100	F1@200	Count
Qwen3-FA aligner Shi et al. (2026)	23	0.784	0.823	n/a
wav2vec2 aligner Hwang et al. (2023)	72	0.608	0.752	n/a
TAC2 (ours)	49	0.774	0.929	84%

Appendix

Table 8: Segment- and pause-level localization on conversational speech. Median onset error (ms,

\downarrow

) and boundary F1 at 100/200 ms (

\uparrow

). Count : fraction of windows where TAC2 proposes the correct segment count.

Text-only Baselines
				Accuracy (%)				Rubric (%)				Chain (%)
Model	Size	Think	Limit	S	M	L	Avg	S	M	L	Avg	S	M	L	Avg
GPT-5.4 (question-only)	–	✓	–	41.1	41.1	40.6	40.9	–	–	–	–	8.9	11.9	10.3	10.4
GPT-5.4 (transcript)	–	✓	–	64.8	60.7	53.4	59.7	41.0	41.5	37.2	39.9	9.4	14.2	14.2	12.6
GPT-5.4 (caption)	–	✓	–	83.2	80.0	77.3	80.2	59.1	57.4	53.0	56.5	66.7	64.4	53.1	61.4
Open-source Models

Appendix

Table 9: Long-form audio understanding on LongAudioSpan ( Huang et al., 2026 ) . Accuracy, Rubric and Chain over duration tiers S (10 to 30 min), M (30 to 60 min) and L (over 60 min); Avg is the mean over tiers. Baseline rows are from Huang et al. (2026) ; gray cells use audio truncated to the model’s input Limit (minutes). For Chain, the caption baseline sees the edits written into its caption. Our row pairs TAC2 (Size) with a frozen text-only GPT-5.4 reasoner. Best in bold , second underlined , among audio models and ours.

	Native MLLM		Describe-then-reason
Benchmark	Model	Score	VLM + Qwen3	TAC-V + Qwen3	TAC-V + Gemini3	TAC2-S -V + Qwen3	TAC2-S -V + Gemini3
Daily-Omni	Qwen3-Omni	76.2	51.5	72.9	77.9	77.2	79.8
	Gemini 2.5 Flash	72.7
	OmniVinci	66.5
World-Sense	Gemini 2.5 Pro	65.1	37.4	45.7	58.6	55.8	59.0
	OmniVinci	48.2

Appendix

Table 10: Audio-visual reasoning performance. Our TAC2-S -V pipeline substitutes TAC2-S for the prior TAC audio describer, keeping describe-then-reason pipeline constant. Daily-Omni, World-Sense, and Video-Holmes report multiple-choice accuracy. AVHBench reports accuracy for AVH/VAH/AVM and METEOR for the AVC subtask. Best per row in bold.

Panel of Figure 8 — Figure 8: The MSU-Bench describe-then-reason prompt. {description} is TAC2-S ’s text description of the referenced segment(s). The output-format instruction is the official MSU-Bench harness prompt ( Wang et al., 2026c ) used verbatim.

Panel of Figure 9 — Figure 9: Gemini-3-Pro audio-native judge prompt for the vs. real rubric.

Panel of Figure 10 — Figure 10: Gemini-3-Pro audio-native judge prompt for the vs. caption rubric. Since every system is judged against the same TAC2-S caption, this yields a level-field cross-model comparison.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Feb 17, 2026

TAC: Timestamped Audio Captioning

Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introduce Timestamped Audio Captioner (TAC), a model that produces temporally grounded audio descriptions at varying degrees of detail and resolution. TAC is trained with a synthetic data pipeline that constructs challenging and dynamic mixtures from real-world audio sources, enabling robust learning under realistic polyphonic conditions. Across event detection and dense captioning, TAC outperforms all competing methods, with a low hallucination rate and accurate temporal grounding. We also introduce TAC-V, an audio-visual pipeline to generate semantically rich audio-visual descriptions. We then show that TAC and TAC-V serves as a "semantic bridge" for a text-only reasoner: a simple TAC

\rightarrow

LLM and TAC-V

\rightarrow

LLM cascade achieves state-of-the-art scores on benchmarks for both audio (MMAU-Pro, MMSU, MMAR) and audio-visual (DailyOmni, VideoHolmes) understanding and reasoning respectively.

Sonal Kumar, Prem Seetharaman, Ke Chen +8

University of Maryland, College Park, USA · Adobe Research, USA · OpenAI, USA (work done while at Adobe).

Audio Captioning Temporal Audio Grounding Audio-Visual Understanding Audio Reasoning

Ask PDF

Aug 30, 2026

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present TEMPO (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech, and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, TEMPO outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements.

Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh +3

University of Maryland, College Park, USA · CNH Industrial India

Audio Understanding Audio-Language Models Temporal Audio Grounding Language Model Post-Training

Ask PDF

Jul 11, 2026

GigaChat Audio: Time-aware Large Audio Language Model

Temporal grounding in long recordings remains challenging for audio-conditioned LLMs. We present a time-aware audio LLM that answers questions with explicit timestamps over up to 120 minutes of input. Our approach interleaves periodic time markers with continuous audio tokens using large-scale synthetic supervision from a cascaded pipeline. Our model achieves strong temporal-grounding accuracy on short and long benchmarks and supports time-anchored fragment descriptions and summaries. Extensive ablations examine how time representation, marker frequency, tokenization, and duration-mixture design affect accuracy and computational cost. We release model weights and datasets to support further research on time-aware audio understanding, available at https://huggingface.co/ai-sage/GigaChat3.1-Audio-10B-A1.8B.

Aleksandr Kutsakov, Mariia Sadovina, Georgii Gospodinov +4

SaluteDevices, Russia

Audio QA Audio Understanding Audio-Language Models Temporal Audio Grounding Temporal QA

Ask PDF

Abstract

Audio-Language Models

Explore similar work

TAC: Timestamped Audio Captioning

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

GigaChat Audio: Time-aware Large Audio Language Model