LongCap: Hour-Scale Timestamped Audio Captioning via Chunk-wise Processing
Organizations: Language Technologies Institute, Carnegie Mellon University · Independent Researcher
Abstract
Describing hours of audio requires accurate event timing and sustained descriptive detail throughout the recording. We present LongCap, a framework for timestamped audio captioning (TAC), with the human-annotated LongCapBench and three diarization-inspired metrics: Timeline Error Rate (TER), Granularity Error Rate (GER), and Caption Similarity (tcpCapSim). LongCap interleaves audio chunks with structured event captions, using relative timestamps and retained context to support length extrapolation within a 1M-token inference context window. Event-level reannotation restores caption detail, while systematic studies examine model scaling, dependency supervision, and self-distillation. LongCap-Diar achieves strong benchmark results and extrapolates to 16 hours of concatenated audio despite training on recordings no longer than 40 minutes. Our TAC models are competitive with Qwen3.5-Omni-Plus on short clips, outperform evaluated baselines in caption similarity and timeline accuracy in segment-level comparisons on LongCapBench-TAC-Long, and caption full recordings approaching two hours. On LongCapBench-TAC, 9B and 35B-A3B offer favorable size-performance tradeoffs, while 122B-A10B substantially improves event-dependency recovery. These findings support long-horizon general audio understanding and provide a foundation for always-on voice agents.
Figures & tables
| Subset | Recordings | Volume (h) | Mean length (min) | Maximum length (min) | Utterances/events |
|---|---|---|---|---|---|
| Diar.-short | |||||
| Diar.-long | |||||
| TAC-short | |||||
| TAC-long |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Recording | Language | Min. | Source |
|---|---|---|---|
| A Star Is Born (1937) | English | 110.67 | link |
| Personal Tailor (2013) | Chinese | 117.05 | link |
| 92nd Oscars (2020) | English | 93.60 | link |
| Roast! (S3E05) | Chinese | 101.49 | link |
| Setting | Value |
|---|---|
| Optimizer | AdamW |
| Pre-training steps | 100k |
| Pre-training warmup | 2k steps |
| Pre-training peak learning rate | |
| Pre-training learning-rate decay | Cosine decay to |
| Tokens per GPU | 32k |
| Test set | MOSS | VibeVoice-ASR | Doubao ASR 2.0 | Qwen3.8- Omni-Flash | LongCap-Diar (ours) |
|---|---|---|---|---|---|
| AliMeeting | 9.27 / 18.30 15.49 / 15.68 | 34.73/35.99 38.28/38.62 | 28.63/24.65 32.13/34.57 | 18.60/24.20 21.92/22.26 | 9.49/20.46 16.30/16.65 |
| AISHELL-4 | 5.80 /13.44 13.15 / 13.43 | 13.57/18.19 22.29/22.70 | 24.66/ 12.59 34.46/41.48 | 10.19/14.58 15.40/15.73 | 8.37/14.26 16.94/17.44 |
| AMI-IHM | 25.17/24.61 20.92 / 21.58 | 27.18/22.50 24.35/25.13 | 47.15/31.81 49.29/56.40 | 47.84/40.57 37.01/39.32 | 21.98 / 21.03 21.70/22.46 |
| AMI-SDM | 27.97/29.87 32.95/34.09 | 31.60/27.58 38.75/39.73 | 50.37/42.91 57.41/66.56 | 44.69/35.56 35.83/41.14 | 15.46 / 22.73 21.86 / 23.48 |
| LongCapBench-Diar-Short | 11.41/5.98 17.34/18.84 | 16.96/6.28 28.24/29.48 | 17.00/ 5.52 25.49/27.51 | 23.18/12.31 31.10/35.48 | 9.94 /5.89 15.47 / 16.56 |
| LongCapBench-Diar-Long | 27.11/27.00 31.77/32.63 | 15.45/14.23 20.35/25.69 | 18.05/10.74 18.54/20.73 | 29.65/23.52 28.38/43.72 | 6.02 / 6.85 13.13 / 14.06 |
| Model | DER | WER | cpWER | tcpWER |
|---|---|---|---|---|
| MOSS | 99.81 | 99.81 | 99.81 | 99.97 |
| VibeVoice-ASR | 100.00 | 99.99 | 99.99 | 100.00 |
| Doubao ASR 2.0 | 69.90 | 53.77 | 69.48 | 77.04 |
| Qwen3.8-Omni-Flash | 79.07 | 69.25 | 71.35 | 75.59 |
| LongCap-Diar (ours) | 29.04 | 37.08 | 32.29 | 34.14 |
| Model | DER | WER | cpWER | tcpWER |
|---|---|---|---|---|
| holistic processing | 109.13 | 80.59 | 115.87 | 147.16 |
| chunk-wise processing | 20.30 | 12.74 | 29.01 | 34.80 |
| LongCap model | Baseline | Short (95% interval) | Long (10-min chunks) |
|---|---|---|---|
| LongCap-35B-A3B | AF-Next | 88.30 (86.80–89.63) | 97.22 |
| LAT | 98.97 (98.48–99.39) | 97.78 | |
| LongCap-122B-A10B | AF-Next | 87.75 (86.21–89.19) | 97.22 |
| LAT | 99.20 (98.78–99.54) | 98.89 |
| Model | Checkpoint | Short form | Full long form | ||||
|---|---|---|---|---|---|---|---|
| TER | GER | tcpCapSim | TER | GER | tcpCapSim | ||
| 35B | Before | 15.93 | 64.59 | ||||
| After | 12.42 | 41.09 | 10.83 | 51.24 | |||
| 122B | Before | 15.51 | 64.48 | 51.67 | |||
| After | 12.41 | 39.64 | 11.64 | ||||
| Benchmark | Condition | LS | |||
|---|---|---|---|---|---|
| DeepSeek | Gemini | DeepSeek | Gemini | ||
| Short form | 35B native | ||||
| 122B native | |||||
| Reference + GLM | |||||
| 35B + GLM | |||||
| 122B + GLM | |||||
Audio Captioning
Audio captioning turns sound into natural-language descriptions of events, attributes, speakers, music, and their timing.
See all 25 papers