cs.SDOct 27, 2026
SaveTAC2: Timestamped Audio Captioning as composable functions
Organizations: University of Surrey, UK · Adobe Research, USA
Abstract
Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.
Figures & tables
Figure 1: Overview of the two-stage TAC2 / TAC2-S framework. (1) Perceptual Foundation ( TAC2 ): Predictions from weak teachers (SED, Diarization, Captioner) are merged via time-registered IoU matching to generate intermediate structured captions, which train a base adapter via the proposed Instruction-Input-Output (IIO) format. (2) Speech Elicitation ( TAC2-S ): Individual LoRA specialists (Stress, Emotion, Voice Profile, Acoustics) are trained on small gold corpora to elicit paralinguistic attributes from the pre-trained backbone. This cascade re-annotates the corpus, which is subsequently distilled into our unified student model, TAC2-S .
Figure 2: Programmatic data compilation via time-registered IoU matching. Predictions from three independent weak teachers ( : audio captioner; : speaker diarizer; : sound event detector) are aligned on a shared time axis using temporal IoU with overlap gate (Eq. 1 , 2 ). Their aligned predictions form the fused target timeline (Eq. 3 ), containing concurrent acoustic events with speaker attribution.
Figure 3: The IIO completion format. The fused target is decomposed into ( INSTRUCTION , INPUT , OUTPUT ) completion tasks. The INSTRUCTION specifies the task, INPUT provides optional teacher-forced context, and OUTPUT specifies the target attributes to generate. Structural tasks train TAC2 ( caption , diarize , and a single-decode comprehensive task). Appending elicited speech tasks ( stress , emotion , voice profile , acoustics ) seamlessly extends this format to TAC2-S without changing the training procedure. At inference, the comprehensive task supports single-pass generation, while sequencing attribute-specific tasks with preceding outputs yields a modular multi-pass cascade.
| Model | DtR | MMSU | MMAR | MMAU | Avg. | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Perception | Reasoning | Overall | Speech | Sound | Music | Overall | Speech | Sound | Music | Overall | |||
| Kimi-Audio | ✗ | 44.8 | 75.7 | 59.8 | 58.5 | 49.7 | 33.0 | 48.0 | 62.2 | 75.7 | 66.8 | 68.2 | 58.7 |
| Qwen2.5-Omni | ✗ | 42.7 | 77.6 | 58.1 | 59.9 | 58.8 | 40.8 | 56.7 | 70.6 | 78.1 | 65.9 | 71.5 | 62.1 |
| StepAudio2 | ✗ | 42.9 | 73.2 | 57.6 | 61.2 | 54.6 | 42.2 | 56.8 | 68.2 | 79.3 | 68.4 | 72.7 | 61.9 |
| UAS-Audio | ✗ | 55.7 | 77.4 | 66.2 | 66.0 | 58.8 | 45.2 | 60.1 | 67.0 | 70.0 | 71.3 | 69.4 | 65.2 |
| Gemini-3.7 Flash | ✓ | 68.7 | 84.4 | 76.3 | 82.7 | 51.5 | 56.8 | 69.5 | 80.5 | 66.1 | 67.1 | 71.2 | 72.3 |
Table 1: Spoken and general audio understanding. Models in DtR ✓ rows emit a text schema as the description, which is read by a frozen text-only reasoner. Each score reflects the fidelity and coverage of audio perception in its generated description.
| Text-only reasoner | SP | MP | |
|---|---|---|---|
| TAC2-S | Gemini3 | 81.3 | 81.9 |
| Qwen3-Next | 78.8 | 79.7 | |
| Qwen3.8 | 80.5 | 80.9 | |
Table 2: Inference and Reasoner Robustness. Left: TAC2-S schemas under single-pass (SP) vs. multi-pass (MP) comprehensive decodes across text-only reasoners (top), and the MMSU tasks where MP helps most (bottom). Right: Perception fidelity. TAC2-S uses describe-then-reason; OmniLLM baselines process raw audio directly (left radar) or act as describers (right radar). TAC2-S (ours), Gemini 3.7, Qwen3Omni
| System | T1 | T2 | Avg |
|---|---|---|---|
| Kimi-Audio | 41.0 | 47.0 | 43.0 |
| Qwen2.5-Omni | 19.0 | 21.0 | 19.0 |
| StepAudio2 | 44.0 | 46.0 | 44.0 |
| AudioFlamingo-3 | 40.0 | 38.0 | 39.0 |
| MiMoAudio | 52.0 | 64.0 | 56.0 |
| Gemini-3.7 Flash | 70.7 | 84.8 | 76.4 |
Table 3: Speaker-centric understanding on MSU-Bench. Evaluated on conversational multi-speaker scenarios in English/Mandarin. Left: Consistent with Table 1 , TAC2-S uses a text reasoner. Right: Tier 1 (Identification) task accuracy. (AI: accent identification, AR: age recognition, GI: gender identification, ER: emotion recognition, SP: speaker profiling, SR: speaker retrieval, SVS: speaker-specific viewpoint summarization, SC: speaker counting, SV: speaker verification) OmniLLM baselines process raw audio directly (left radar) or act as describers (right radar). TAC2-S ’s trace (blue) is superior across both, demonstrating robust speaker-centric perception. TAC2-S (ours), Gemini 3.7, Qwen3Omni
Figure 4: Impact of individual IIO tasks. (left) Building TAC2-S ’s description one attribute at a time, from a transcript-only base (who spoke, the words, and their timestamps). (right) For each attribute, the top-3 it lifts most, the gains land on the tasks that attribute governs and leave the transcript carried semantics tasks flat. Bar colour marks the added attribute (blue: stress and intonation, pink: emotion, orange: voice profile(demographics), green: pitch and loudness).
| TAC2-S -as-judge | Temporal grounding | Gemini-3-Pro judge | ||||||||||
| System | Gen. | Age | Acc. | Emo. | Ev.R. | Sp.R. | WER | PE | FLAM | PE-AV | vs. real | vs. cap. |
| PicoAudio2 ( Zheng et al., 2026 ) | 0.44 | 0.35 | 0.24 | 0.28 | 0.27 | 0.09 | 1.15 | 2.47 2.77 | 3.16 3.50 | 0.52 0.51 | 1.39 | 1.61 |
| MiDashengLM-Gen ( Sun et al., 2026 ) | 0.54 | 0.36 | 0.21 | 0.36 | 0.33 | 0.09 | 1.39 | 3.57 — | 3.78 — | 0.44 — | 1.54 | 1.60 |
| Seed Audio 1.0 ( ByteDance Seed, 2026 ) | 0.94 | 0.89 | 0.71 | 0.74 | 0.78 | 0.90 | 0.14 | 2.16 — | 2.62 — | 0.35 — | 3.53 | 4.05 |
| Ours ( TAC2 ) | — | — | — | — | 0.80 | 0.94 | 0.12 | 1.67 2.47 | 2.13 2.77 | 0.35 0.35 | 3.38 | 4.01 |
Table 4: Structured captions generated by TAC2-S provide renderable supervision for audio synthesis. Temporal-grounding cells report with without input timestamps. PE and FLAM measure mean localizer-peak offset in seconds; PE-AV is a text-free frame-wise semantic distance between original and reconstructed audio (real-vs.-real ). Gemini-3-Pro judgments are on a – scale.
Figure 5: TAC2-S -conditioned reconstruction. We caption real audio (top) with TAC2-S (middle), then condition the trained text-to-audio generator on it to synthesize new audio (bottom). The reconstruction reproduces speech and non-speech events, including background elements like a censor beep , engine rev , camera click , at times aligned with the real audio.
Figure 6: Word-level emphasis as a renderable control channel. Each column synthesizes the same sentence while varying the emphasized word in the TAC2-S caption (yellow). The bottom two rows increase emphasis from mild ( <e> ) to strong ( <E> ) while the cyan curve shows the F0 contour. Bottom panels show WhiStress ( Yosha et al., 2025 ) stress probability over time, with marking the commanded word. The detected stress peak follows the targeted word and scales with emphasis intensity, demonstrating zero-shot emphasis control without a dedicated stress-control model.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Trait (granularity) | Backbone peak | Early (base / spec) | Late (base spec) | Lift |
|---|---|---|---|---|
| Relative loudness (segment) | 0.89 | 0.79 / 0.81 | 0.87 0.99 | |
| Relative pitch (segment) | 0.90 | 0.80 / 0.82 | 0.89 0.99 | |
| Word stress (word) | 0.93 | 0.88 / 0.89 | 0.91 0.99 | |
| Accent (speaker) | 0.99 | 0.97 / 0.97 | 0.99 1.00 |
Table 5: The backbone (Qwen3Omni-captioner) possesses latent perception; the specialist isolates late-stage readouts. Early / Late : mean AUC over depths 0–24 / 36–48, backbone vs. specialist. Lift : the specialist’s late-stage gain.
| Attribute (metric) | Frozen backbone | + specialist |
|---|---|---|
| Word stress (top-word accuracy) | 0.60 | 0.81 |
| Accent (exact, 13 classes) | 0.37 | 0.62 |
| Accent (region) | 0.61 | 0.91 |
| Emotion (5 classes) | 0.15 | 0.59 |
Table 6: The specialist mitigates output mode collapse without altering early perception. Accuracy of the emitted textual attribute, frozen backbone vs. its specialist, on clips with ground-truth labels (word stress is scored as the top stressed word against Whistress ( Yosha et al., 2025 ) ).
| Task | Input | Weight | Output |
|---|---|---|---|
| comprehensive | none (audio) | 1.0 | full description (all sections) |
| caption | none (audio) | 0.5 | overview + events with transcript |
| stress | transcript turns | 0.5 | per-word stress and intonation |
| emotion | transcript turns | 0.5 | per-turn emotion |
| voice profile | transcript turns | 0.3 | per-speaker gender, age, accent |
| acoustics | transcript turns | 0.3 | per-segment pitch and loudness |
Table 7: IIO tasks during training. Each training sample is formulated as a single completion task, with weights denoting their respective sampling probabilities. The comparative speech specialists are conditioned on the speaker turns and transcript as teacher-forced inputs.
Figure 7: Segment localization on conversational speech. The segments from the human annotation, TAC2 , the Qwen3-FA aligner, and the wav2vec2-based aligner share the time axis. TAC2 ’s segment onsets track the human pause boundaries closely, while being competitive with the dedicated aligners. (Dotted lines mark the human segment onsets.)
| System | Onset (ms) | F1@100 | F1@200 | Count |
|---|---|---|---|---|
| Qwen3-FA aligner Shi et al. (2026) | 23 | 0.784 | 0.823 | n/a |
| wav2vec2 aligner Hwang et al. (2023) | 72 | 0.608 | 0.752 | n/a |
| TAC2 (ours) | 49 | 0.774 | 0.929 | 84% |
Table 8: Segment- and pause-level localization on conversational speech. Median onset error (ms, ) and boundary F1 at 100/200 ms ( ). Count : fraction of windows where TAC2 proposes the correct segment count.
| Accuracy (%) | Rubric (%) | Chain (%) | |||||||||||||
| Model | Size | Think | Limit | S | M | L | Avg | S | M | L | Avg | S | M | L | Avg |
| Text-only Baselines | |||||||||||||||
| GPT-5.4 (question-only) | – | ✓ | – | 41.1 | 41.1 | 40.6 | 40.9 | – | – | – | – | 8.9 | 11.9 | 10.3 | 10.4 |
| GPT-5.4 (transcript) | – | ✓ | – | 64.8 | 60.7 | 53.4 | 59.7 | 41.0 | 41.5 | 37.2 | 39.9 | 9.4 | 14.2 | 14.2 | 12.6 |
| GPT-5.4 (caption) | – | ✓ | – | 83.2 | 80.0 | 77.3 | 80.2 | 59.1 | 57.4 | 53.0 | 56.5 | 66.7 | 64.4 | 53.1 | 61.4 |
| Open-source Models | |||||||||||||||
Table 9: Long-form audio understanding on LongAudioSpan ( Huang et al., 2026 ) . Accuracy, Rubric and Chain over duration tiers S (10 to 30 min), M (30 to 60 min) and L (over 60 min); Avg is the mean over tiers. Baseline rows are from Huang et al. (2026) ; gray cells use audio truncated to the model’s input Limit (minutes). For Chain, the caption baseline sees the edits written into its caption. Our row pairs TAC2 (Size) with a frozen text-only GPT-5.4 reasoner. Best in bold , second underlined , among audio models and ours.
| Native MLLM | Describe-then-reason | ||||||
| Benchmark | Model | Score | VLM + Qwen3 | TAC-V + Qwen3 | TAC-V + Gemini3 | TAC2-S -V + Qwen3 | TAC2-S -V + Gemini3 |
| Daily-Omni | Qwen3-Omni | 76.2 | 51.5 | 72.9 | 77.9 | 77.2 | 79.8 |
| Gemini 2.5 Flash | 72.7 | ||||||
| OmniVinci | 66.5 | ||||||
| World-Sense | Gemini 2.5 Pro | 65.1 | 37.4 | 45.7 | 58.6 | 55.8 | 59.0 |
| OmniVinci | 48.2 | ||||||
Table 10: Audio-visual reasoning performance. Our TAC2-S -V pipeline substitutes TAC2-S for the prior TAC audio describer, keeping describe-then-reason pipeline constant. Daily-Omni, World-Sense, and Video-Holmes report multiple-choice accuracy. AVHBench reports accuracy for AVH/VAH/AVM and METEOR for the AVC subtask. Best per row in bold.
Figure 8: The MSU-Bench describe-then-reason prompt. {description} is TAC2-S ’s text description of the referenced segment(s). The output-format instruction is the official MSU-Bench harness prompt ( Wang et al., 2026c ) used verbatim.
Figure 9: Gemini-3-Pro audio-native judge prompt for the vs. real rubric.
Figure 10: Gemini-3-Pro audio-native judge prompt for the vs. caption rubric. Since every system is judged against the same TAC2-S caption, this yields a level-field cross-model comparison.
More in this topic
Audio-Language Models
Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.
See all 303 papers