Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, STaRC, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at https://github.com/ermitaju1/STaRC
Figures & tables
Figure 1: An example of caption retrieval and generation by existing methods (e.g., Sali4Vid [ 10 ] ) and ours. The misaligned video segments in existing methods lead to less relevant retrieval results. Our supervised saliency scores are well-aligned with the ground truth, enabling more accurate retrieval. This alignment also provides the decoder with more precise and contextually appropriate information.
Figure 2: Correlation analysis between segment-quality metrics and captioning performance . The x-axis shows three indicators of temporal localization quality - [email protected] , Mean IoU , and Matched Segments - which respectively measure the proportion of correctly detected events (IoU ≥0.5 ), the average overlap with ground truth segments, and the number of predictions aligned with the reference events. As these segment-quality metrics improve from Sali4Vid [ 10 ] to ours, downstream DVC performances (e.g., CIDEr, METEOR) also rise consistently, demonstrating a strong positive correlation between accurate event segmentation and caption generation performance.
Figure 3: Overview of our STaRC framework for DVC. A SWSA module refines video features, and a highlight detection module is supervised using binary highlight labels derived from existing DVC annotations to predict frame-level saliency scores. The SGSR module then performs OT-based clustering guided by these saliency scores to form coherent retrieval segments. In addition, the SaliP integrates saliency into the decoder’s attention for saliency-aware caption generation. STaRC unifies retrieval and caption generation using saliency signals learned in a supervised manner, ensuring consistent alignment between visual context and semantic description.
Method
PT
YouCook2 (val)
ViTT (test)
CIDEr
METEOR
SODA_c
BLEU_4
CIDEr
METEOR
SODA_c
BLEU_4
PDVC ICCV21
✗
29.69
5.56
4.92
1.40
-
-
-
-
CM 2 CVPR24
✗
31.66
6.08
5.34
1.63
-
-
-
-
E 2 DVC CVPR25
✗
34.26
6.11
5.39
1.68
-
-
-
-
CACMI AAAI26
✗
34.83
6.21
5.57
1.70
-
-
-
-
Streaming V2S CVPR24
✓
32.90
7.10
6.00
-
25.20
5.80
10.00
-
Table 1: Comparison with state-of-the-art methods on the YouCook2 validation set and the ViTT test set. PT indicates whether the model is pretrained. Bold and underlined values represent the best and second best results, respectively. “–” denotes unavailable results, and † marks results reproduced from official implementations. Our method achieves state-of-the-art performance across most evaluation metrics.
Method
PT
YouCook2 (val)
ViTT (test)
F1
Recall
Precision
F1
Recall
Precision
PDVC
✗
26.81
22.89
32.37
-
-
-
CM 2
✗
28.43
24.76
33.38
-
-
-
E 2 DVC CVPR25
✗
28.87
25.01
34.13
-
-
-
CACMI AAAI26
✗
29.34
25.54
34.63
-
-
-
Streaming V2S
✓
24.10
-
-
35.40
-
-
Table 2: Localization results on the YouCook2 validation set and the ViTT test set. Bold and underlined values indicate the best and second-best performances, respectively. “–” denotes unavailable results.
Method
SGSR 3.3
SaliP 3.3
CIDEr
METEOR
Baseline
66.29
12.41
(1)
✓
76.94
13.60
(2)
✓
78.74
13.75
STaRC
✓
✓
80.53
13.86
Table 3: Ablation study on the key components of STaRC . We evaluate the contributions of our main proposed modules, SGSR , and SaliP on YouCook2 validation set. The baseline corresponds to the performance of Vid2Seq.
Window sizes
YouCook2 (val)
CIDEr
METEOR
SODA_c
BLEU_4
w/o SWSA 3.2
75.82
13.23
10.33
6.49
6, 36, 66
78.46
13.66
10.57
6.66
8, 32, 64
80.53
13.86
10.73
6.75
10, 40, 70
70.52
13.26
9.72
5.92
8
78.03
13.69
10.62
6.65
Table 4: Ablation study on window sizes and the effect of removing SWSA module on the YouCook2 validation set.
Figure 4: (a) Effect of anchor count and top- k segment selection. CIDEr scores across different numbers of anchors ( K ) and top- k segment selections, where only the k highest-scoring segments are used for retrieval. Each line represents a different anchor configuration, and K corresponds to the number of segments produced by SGSR . (b-d) Impact of saliency-prompt corruption. These plots show the effect of replacing SaliP with Zero and Gaussian noise prompts. Both corrupted prompts still outperform the baseline ( STaRC without SaliP ), but Gaussian noise causes a larger drop, emphasizing the importance of accurate saliency cues.
Method / Variant
YouCook2 (val)
CIDEr
METEOR
SODA_c
BLEU_4
Clustering Type
k-means
75.63
13.34
10.46
6.29
Adaptive clustering [ 10 ]
78.19
13.69
10.55
6.57
SGSR (Ours)
80.53
13.86
10.73
6.75
Pooling Strategy (within SGSR)
Table 5: Ablation on segmentation and pooling strategies. The top part compares clustering methods, where our SGSR outperforms k-means and adaptive clustering [ 10 ] . The bottom part compares pooling strategies within SGSR , showing the benefit of saliency-weighted averaging over uniform averaging.
Num of Retrieval ( p )
YouCook2 (val)
CIDEr
METEOR
SODA_c
BLEU_4
3
79.65
13.79
10.61
6.79
5
79.12
13.73
10.70
6.67
10
80.53
13.86
10.73
6.75
15
78.00
13.81
10.56
6.57
Table 6: Ablation study on the number of top-ranked (top- p ) captions retrieved based on similarity.
Figure 5: A qualitative result from YouCook2 validation set. The arrows indicate the duration of each localized event, and the text below each arrow represents its corresponding caption.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : (a-b) CIDEr performance for different saliency-loss weights λ . Results on YouCook2 and ViTT showing how different values of λ influence the strength of supervised saliency learning and overall caption quality. (c) CIDEr performance in YouCook2 validation set for different temperature values τ . This figure visualizes model sensitivity to different τ values used in the loss formulation.
Figure A.2 : Visualization of predicted frame-level saliency scores. Black lines show predicted saliency scores overlaid with ground truth event boundaries (green regions). Saliency scores are higher within event regions than in non-event regions. This separation confirms that the model successfully learns to align saliency prediction with true event boundaries.
Figure A.3 : Performance comparison of SGSR under different OT configurations. We evaluate Balanced (B) and Unbalanced (UnB) OT for both frame-side and anchor-side matching. The configuration with UnB for frames and B for anchors achieves the best performance across all metrics.
Figure A.4 : CIDEr performance heatmap across different combinations of saliency weight μ and temporal-consistency weight γ . Each cell shows the CIDEr score obtained under a specific μ and γ configuration in the SGSR objective.
Datstore Type
YouCook2 (val)
CIDEr
METEOR
SODA_c
BLEU_4
CC3M
76.65
13.63
10.53
6.63
COCO
77.31
13.57
10.66
6.67
Hierarchical [ 13 ]
77.28
13.59
10.42
6.59
In-domain
80.53
13.86
10.73
6.75
Appendix
Table A.1 : Ablation study on different datstores used for retrieval.
Figure A.5 : Qualitative comparison of retrieved captions under different segmentation strategies. (a) Ground truth segments and captions. (b) Segments from uniform sampling (HiCM2 [ 13 ] ) and their retrieved captions. (c) Segments from adaptive sampling (Sali4Vid [ 10 ] ) and their retrieved captions. (d) Our SGSR produces segments that align more closely with true event boundaries and retrieves more relevant captions.
Figure A.6 : Visualization of the Semantic Prototypes.
Figure A.7 : Architecture of the SWSA module. SWSA refines X through local self-attention without linear projections over sliding windows, where overlapping regions are averaged and residually added to produce refined features X′ .
Figure A.8 : Frame transition sharpness comparison between original and refined features . For each video in the YouCook2 validation set, red spans indicate ground truth event regions. Refined features exhibit larger transition magnitudes within event intervals, indicating stronger semantic contrast and clearer event boundaries than original features.
Method
PT
CIDEr
SODA_c
F1
TimeChat CVPR24
✓
11.0
3.4
19.5
VTG-LLM AAAI25
✓
13.4
3.6
20.6
TRACE ICLR25
✓
35.5
6.7
31.8
TimeExpert ICCV25
✓
39.0
7.2
33.5
Ours
✓
80.53
10.73
34.34
Appendix
Table A.2 : Comparison with VLM methods on the YouCook2 validation set
Figure A.9 : Failure case of STaRC .
Figure A.10 : Additional qualitative results for STaRC on the YouCook2 validation set.
Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.
Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.
Zihan Lin, Songhe Deng, Shuwei He +6
ERNIE Team, Baidu · College of Artificial Intelligence, Inner Mongolia University
Weakly-Supervised Dense Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries. Prior work introduced an implicit supervision paradigm based on Gaussian masking and complementary captioning. However, existing method focuses merely on generating non-overlapping masks without considering their semantic relationship to corresponding events, resulting in simplistic, uniformly distributed masks that fail to capture semantically meaningful regions. Moreover, relying solely on ground-truth captions leads to sub-optimal performance due to the inherent sparsity of existing datasets. In this work, we propose SAIL, which constructs semantically-aware masks through cross-modal alignment. Our similarity aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions. Furthermore, to guide more accurate mask generation under sparse annotation settings, we introduce an LLM-based augmentation strategy that generates synthetic captions to provide additional alignment signals. These synthetic captions are incorporated through an inter-mask mechanism, providing auxiliary guidance for precise temporal localization without degrading the main objective. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance on both captioning and localization metrics.