Weakly-Supervised Dense Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries. Prior work introduced an implicit supervision paradigm based on Gaussian masking and complementary captioning. However, existing method focuses merely on generating non-overlapping masks without considering their semantic relationship to corresponding events, resulting in simplistic, uniformly distributed masks that fail to capture semantically meaningful regions. Moreover, relying solely on ground-truth captions leads to sub-optimal performance due to the inherent sparsity of existing datasets. In this work, we propose SAIL, which constructs semantically-aware masks through cross-modal alignment. Our similarity aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions. Furthermore, to guide more accurate mask generation under sparse annotation settings, we introduce an LLM-based augmentation strategy that generates synthetic captions to provide additional alignment signals. These synthetic captions are incorporated through an inter-mask mechanism, providing auxiliary guidance for precise temporal localization without degrading the main objective. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance on both captioning and localization metrics.
Figures & tables
Figure 1: (a) Previous work generates masks that simply cover different temporal regions without considering semantic alignment with corresponding events. (b) Our proposed method leverages cross-modal similarity to guide masks toward event-relevant regions and addresses caption sparsity through LLM augmentation.
Figure 2: (a) The fixed mask baseline applies Gaussian masks with equal width uniformly distributed according to the number of events. It performs comparably to the existing method [ 11 ] . (b) shows the dataset’s annotation sparsity example, where potential events are missed. (c) The majority of video samples have events spanning the entire duration but contain only a small number of events (red box).
Figure 3: SAIL Pipeline. Our method exploits cross-modal similarity to guide mask optimization toward increased alignment with event captions and further enriches supervision through LLM-generated synthetic captions.
Setting
Model
Features
Captioning
SODA_c
METEOR
CIDEr
ROUGE-L
BLEU-1
BLEU-2
BLEU-3
BLEU-4
Fully-Supervised
DCE [ 22 ]
C3D
–
5.69
12.43
–
10.81
4.57
1.90
0.71
DVC [ 25 ]
C3D
–
6.93
12.61
–
12.22
5.72
2.27
0.73
PDVC [ 36 ]
C3D
5.26
7.50
25.87
–
–
–
–
1.65
Vid2Seq [ 42 ]
CLIP
5.80
8.50
30.10
–
–
–
–
–
CM 2 [ 14 ]
CLIP
6.18
8.55
33.01
–
–
–
–
2.38
Table 1: Comparison with state-of-the-art methods on ActivityNet validation set. ∗ denotes results reported in [ 11 ] . Our method outperforms the previous state-of-the-art weakly-supervised approach [ 11 ] and remarkably surpasses several fully-supervised methods [ 42 , 14 , 39 ] . This demonstrates semantic alignment and caption augmentation can effectively replace temporal boundary supervision.
Setting
Model
Features
Captioning
Localization
SODA_c
METEOR
CIDEr
ROUGE-L
BLEU@N
R@AVG
P@AVG
F1
Weakly-Supervised
WSDEC ∗ [ 8 ]
C3D
2.11
1.47
8.43
–
–
–
–
–
PWS-DVC ∗ [ 7 ]
C3D
3.14
2.48
9.81
–
–
–
–
–
ILCACM † [ 11 ]
CLIP
3.60
3.41
13.49
4.75
2.59
17.76
18.01
17.88
SAIL (Ours)
CLIP
4.08
3.63
14.61
5.42
2.94
20.76
21.13
20.94
Table 2: Comparison with other weakly-supervised methods on YouCook2 validation set. † means our re-implementation results, as the original paper does not report localization performance on this dataset. Our approach also achieves the best captioning and localization performance among all weakly-supervised methods on this dataset. BLEU@N denotes the average of BLEU-1 through BLEU-4.
Setting
Model
Features
Localization
R@Avg
P@Avg
F1
Fully- Supervised
SDVC [ 27 ]
C3D
55.58
57.57
56.56
PDVC [ 36 ]
TSN
55.42
58.07
56.71
Vid2Seq [ 42 ]
CLIP
52.70
53.90
53.29
CM 2 [ 14 ]
CLIP
53.71
56.81
55.21
E 2 DVC [ 39 ]
CLIP
54.67
57.70
56.14
Table 3: Comparison of the event localization performance across various fully-supervised and weakly-supervised models on ActivityNet validation set. SAIL achieves state-of-the-art performance in most of the localization metrics.
Figure 4: Impact of caption density on model performance. Performance decreases consistently as annotation density reduces from 100% to 25% (left), highlighting annotation sparsity as a critical challenge. Densifying supervision through LLM-generated synthetic captions (right) improves performance.
Table 4: Ablation study on our key components on ActivityNet. Sim denotes similarity-aware mask, and Synth denotes Synthetic Captions with inter mask. S_c, M, and C denote SODA_c, METEOR, and CIDEr, respectively.
Method
Captioning
Localization
S_c
C
R
F1
0% (Baseline)
6.08
33.42
14.77
56.20
25%
6.08
33.76
15.02
56.38
50%
6.27
34.19
15.00
56.39
75%
6.23
34.66
15.09
56.91
100% (Ours)
6.29
35.38
15.29
57.00
Table 5: Ablation study on the amount of synthetic captions used. The results suggest that the benefits of synthetic captions are evident even at a small scale. R denotes ROUGE-L.
Method
Captioning
S_c
M
C
R
B@4
ILCACM [ 11 ]
6.08
8.48
33.42
14.77
2.26
Synth + HN
6.24
8.44
33.84
15.10
2.02
Synth + Inter
6.29
8.63
35.38
15.29
2.30
Table 6: Ablation study on synthetic caption utilization strategies. Leveraging synthetic captions indirectly via an inter-mask mechanism yields the best overall performance. B @ 4 denotes BLEU-4.
Mask Type
Model
Captioning
Localization
SODA_c
METEOR
CIDEr
ROUGE-L
R@Avg
P@Avg
F1
Gaussian Mask
ILCACM [ 11 ]
6.08
8.48
33.42
14.77
53.72
58.92
56.20
SAIL (Ours)
6.29
8.63
35.38
15.29
54.39
59.87
57.00
Hard Binary Mask
ILCACM † [ 11 ]
3.89
6.52
16.96
11.24
35.76
50.85
41.91
SAIL (Ours)
4.09
6.53
17.54
11.07
35.81
51.00
42.00
Cauchy Mask
ILCACM † [ 11 ]
6.02
8.32
32.22
14.50
53.06
58.87
55.81
Table 7: Ablation study of various mask designs. SAIL demonstrates consistent performance improvements over the baseline across all mask designs. † means our re-implementation localization results, as the original paper does not report localization performance.
Figure 6: Impact of hyper-parameter winter for inter masks on captioning and localization performance.
Method
Train time
Inference time
GPU usage
ILCACM
1H 38M 27S
7M 11S
33.06 GiB
SAIL (Ours)
1H 41M 59S
7M 01S
33.11 GiB
Table 8: Computational cost comparison against the baseline [ 11 ] .
Figure 7: A qualitative result from the ActivityNet validation set.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Impact of hyper-parameter αaug for scaling Laug on captioning and localization performance.
Figure 9: Another qualitative result from the ActivityNet validation set.
Table 9: Analysis of mask width diversity during training. We report the mean (Mean), minimum (Min), and maximum (Max) standard deviation of mask widths across all training videos.
Figure 11: We visualize the mask outputs from the mask generation module during training and annotate each mask with its corresponding width value. ILCACM maintains almost constant mask widths throughout training, whereas SAIL shows progressive adaptation: masks for shorter events (blue, green) decrease in width, while masks for longer events (red) increase, aligning with actual event durations.
Weakly-Supervised Dense Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration. To address these, we propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted. Leveraging a VLM, we generate frame-level narratives for the inter-event gaps and detect transitions from the semantic variation across them. For identified transitions, we then refine inter-event temporal masks by blending the temporal midpoint with the semantic change point and selecting the width that maximizes vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization.
Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.
Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, STaRC, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at https://github.com/ermitaju1/STaRC