Describing hours of audio requires accurate event timing and sustained descriptive detail throughout the recording. We present LongCap, a framework for timestamped audio captioning (TAC), with the human-annotated LongCapBench and three diarization-inspired metrics: Timeline Error Rate (TER), Granularity Error Rate (GER), and Caption Similarity (tcpCapSim). LongCap interleaves audio chunks with structured event captions, using relative timestamps and retained context to support length extrapolation within a 1M-token inference context window. Event-level reannotation restores caption detail, while systematic studies examine model scaling, dependency supervision, and self-distillation. LongCap-Diar achieves strong benchmark results and extrapolates to 16 hours of concatenated audio despite training on recordings no longer than 40 minutes. Our TAC models are competitive with Qwen3.5-Omni-Plus on short clips, outperform evaluated baselines in caption similarity and timeline accuracy in segment-level comparisons on LongCapBench-TAC-Long, and caption full recordings approaching two hours. On LongCapBench-TAC, 9B and 35B-A3B offer favorable size-performance tradeoffs, while 122B-A10B substantially improves event-dependency recovery. These findings support long-horizon general audio understanding and provide a foundation for always-on voice agents.
Figures & tables
Figure 1: Timestamped audio captioning (TAC) results on the LongCapBench-TAC-Long benchmark. LongCap produces stable and comprehensive audio captions up to 2 hours, while competitors show severe degradation within 10 minutes. (a) TER, (b) GER, and (c) tcpCapSim measure timeline mismatch, granularity mismatch, and caption semantic similarity, respectively.
Figure 2: TAC format. A schematic excerpt of long-form audio illustrates overlapping speech, music, and sound events with start/end timestamps and structured attribute dictionaries.
Figure 3: TAC scoring metrics. (a) TER measures timeline mismatch, (b) GER measures event granularity mismatch, and (c) tcpCapSim measures time-constrained caption similarity.
Subset
Recordings
Volume (h)
Mean length (min)
Maximum length (min)
Utterances/events
Diar.-short
442
53.01
7.20
11.10
44,860
Diar.-long
15
15.02
60.06
78.16
4,068
TAC-short
1,500
18.50
0.74
1.00
9,380
TAC-long
4
7.05
105.70
117.05
6,520
Table 1: Statistics of LongCapBench. Recordings are evaluation clips.
Figure 4: Holistic (top) versus chunk-wise (bottom) processing of a long recording.
Figure 5: Speech recognition and diarization: (a) DER, WER, cpWER, and tcpWER, macro-averaged over five benchmark conditions; (b–c) LongCapBench-Diar-Long errors in disjoint five-minute windows. Gray bars count contributing recordings. Benchmark-specific checkpoints and AMI overlap are described in App. D.1 .
Figure 6: Diarization design comparisons: (a) Relative vs. absolute timestamps. (b) Different chunking strategies. (c) With and without long-form text in the training mixture. Comparison protocols are detailed in App. D.1 .
Figure 7: (a) Short-form TAC results on LongCapBench-TAC-Short. (b–c) Continued-pretraining scale comparisons: (b) MMAU/MMAR accuracy and DER/WER averaged equally over English and Chinese evaluation sets; (c) WER and DER over long-form recordings.
Figure 8: TAC scaling and caption density. (a–b) LongCap-SFT-2 on LongCapBench-TAC-Short and LongCapBench-TAC-Long; 0.8B and 2B lack reliable full-recording outputs. (c) Filled attributes per event by position within the chunk.
Figure 9: LS and F1 (%) for 35B-A3B, 122B-A10B, and GLM-5.3-Flash on reference events.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Recording
Language
Min.
Source
A Star Is Born (1937)
English
110.67
link
Personal Tailor (2013)
Chinese
117.05
link
92nd Oscars (2020)
English
93.60
link
Roast! (S3E05)
Chinese
101.49
link
Appendix
Table 2: Sources and licensing documentation for LongCapBench-TAC-Long
Setting
Value
Optimizer
AdamW
Pre-training steps
100k
Pre-training warmup
2k steps
Pre-training peak learning rate
2×10−5
Pre-training learning-rate decay
Cosine decay to 1×10−5
Tokens per GPU
32k
Appendix
Table 3: Pre-training configuration. The batch size shown is for TAC training on 128 GPUs; diarization uses 32 GPUs.
Test set
MOSS
VibeVoice-ASR
Doubao ASR 2.0
Qwen3.8- Omni-Flash
LongCap-Diar (ours)
AliMeeting
9.27 / 18.30 15.49 / 15.68
34.73/35.99 38.28/38.62
28.63/24.65 32.13/34.57
18.60/24.20 21.92/22.26
9.49/20.46 16.30/16.65
AISHELL-4
5.80 /13.44 13.15 / 13.43
13.57/18.19 22.29/22.70
24.66/ 12.59 34.46/41.48
10.19/14.58 15.40/15.73
8.37/14.26 16.94/17.44
AMI-IHM
25.17/24.61 20.92 / 21.58
27.18/22.50 24.35/25.13
47.15/31.81 49.29/56.40
47.84/40.57 37.01/39.32
21.98 / 21.03 21.70/22.46
AMI-SDM
27.97/29.87 32.95/34.09
31.60/27.58 38.75/39.73
50.37/42.91 57.41/66.56
44.69/35.56 35.83/41.14
15.46 / 22.73 21.86 / 23.48
LongCapBench-Diar-Short
11.41/5.98 17.34/18.84
16.96/6.28 28.24/29.48
17.00/ 5.52 25.49/27.51
23.18/12.31 31.10/35.48
9.94 /5.89 15.47 / 16.56
LongCapBench-Diar-Long
27.11/27.00 31.77/32.63
15.45/14.23 20.35/25.69
18.05/10.74 18.54/20.73
29.65/23.52 28.38/43.72
6.02 / 6.85 13.13 / 14.06
Appendix
Table 4: Speech recognition and diarization results on each benchmark. Each cell reports DER / WER on the first line and cpWER / tcpWER on the second, in %; lower is better.
Model
DER
WER
cpWER
tcpWER
MOSS
99.81
99.81
99.81
99.97
VibeVoice-ASR
100.00
99.99
99.99
100.00
Doubao ASR 2.0
69.90
53.77
69.48
77.04
Qwen3.8-Omni-Flash
79.07
69.25
71.35
75.59
LongCap-Diar (ours)
29.04
37.08
32.29
34.14
Appendix
Table 5: CHiME-6 results on sessions S01, S02, S09, and S21, micro-averaged across sessions. The test set has multiple near-field recordings for each example. We infer them all and keep the best results of each example. The maximum audio length is 2 hours 40 minutes. Most competitors are not practically usable on this application.
Model
DER
WER
cpWER
tcpWER
holistic processing
109.13
80.59
115.87
147.16
chunk-wise processing
20.30
12.74
29.01
34.80
Appendix
Table 6: Comparison of 2B models with holistic processing and chunk-wise processing on LongCapBench-Diar-Long. All metrics are reported in %; lower is better. tcpWER uses a 5-second collar. Best values are bold.
Figure 10: Cumulative error rates up to 16 hours, averaged over three recording orders.
LongCap model
Baseline
Short (95% interval)
Long (10-min chunks)
LongCap-35B-A3B
AF-Next
88.30 (86.80–89.63)
97.22
LAT
98.97 (98.48–99.39)
97.78
LongCap-122B-A10B
AF-Next
87.75 (86.21–89.19)
97.22
LAT
99.20 (98.78–99.54)
98.89
Appendix
Table 7: Tie-adjusted LongCap preference (%), with 95% source-cluster bootstrap intervals for Short.
Model
Checkpoint
Short form
Full long form
TER ↓
GER ↓
tcpCapSim ↑
TER ↓
GER ↓
tcpCapSim ↑
35B
Before
13.01
15.93
64.59
44.84
11.46
50.85
After
12.42
16.13
63.75
41.09
10.83
51.24
122B
Before
12.47
15.51
64.48
42.35
12.15
51.67
After
12.41
15.94
63.84
39.64
11.64
51.19
Appendix
Table 8: Before and after self-distillation on all 1,500 Short clips and all four full Long recordings. All values are percentages; arrows indicate preferred directions. Bold marks the better value within each model-size pair.
Benchmark
Condition
LS
F1
DeepSeek
Gemini
DeepSeek
Gemini
Short form
35B native
54.40
55.71
62.34
64.39
122B native
58.54
59.95
67.20
68.98
Reference + GLM
70.27
72.29
69.60
71.26
35B + GLM
57.38
58.83
66.16
68.06
122B + GLM
58.16
59.56
66.43
68.12
Appendix
Table 9: Dependency agreement (%, higher is better). Native denotes the model’s original links; +GLM denotes GLM-5.3-Flash links on the indicated events. LS covers all relations; F1 covers same_entity .