Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning

Seung Hyup Baek; Jimin Lee; Hyeongkeun Lee; Jae Won Cho

cs.CVMar 12, 2026

Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning

Authors: Seung Hyup Baek, Jimin Lee, Hyeongkeun Lee, Jae Won Cho

Organizations: Konkuk University · Sejong University · KAIST

Abstract

Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.

Figures & tables

Figure 1: Comparison of our method with previous methods. (a) Captioning results. The baseline model captures overlapping events, causing redundant captions. In contrast, our model generates distinct non overlapping regions for a set caption. (b) Decoder query sampled attention weights. The baseline’s single query attends indiscriminately. Previous “decomposition” queries show similar attention distributions. In contrast, our method employs two specialized queries: localization queries attend broadly for boundaries and a caption queries attend densely on key frames.

Figure 2: An overview of our proposed ROS-DVC framework. The input video is first fed into the pretrained encoder, and a transformer encoder processes it to generate frame-level features. In the decoding stage, two types of queries are independently initialized and retrieve their role-specific information from the frame-level features. The output localization queries are trained with the Overlap Suppression Loss to minimize mutual overlap and are matched with ground truths via the Hungarian algorithm. Subsequently, the CTCA loss is employed to semantically align the caption queries with their corresponding localization queries. Finally, these processed queries are fed into respective heads to obtain the predictions for event-number, localize timestamps, event captions, and event-level concepts.

Figure 3: Mechanism of the Overlap Suppression Loss. This loss discourages overlap among queries. The suppression strength is inversely modulated by the query’s IoU with the ground truth (GT). Queries with high GT IoU receive weak suppression, while queries with low GT IoU receive strong suppression.

Method	#PT	CIDEr $\uparrow$	METEOR $\uparrow$	BLEU4 $\uparrow$	SODA $\_$ c $\uparrow$
Vid2Seq	✓	47.10	9.30	–	7.90
DIBS	✓	44.40	7.51	–	6.39
DDVC	✗	38.75	6.92	1.92	6.68
PDVC	✗	29.69	5.56	1.40	4.92
CM 2	✗	31.66	6.08	1.63	5.34
MCCL	✗	36.09	6.53	2.04	5.21

Table 1: YouCook2 Caption metrics. Bold represents best and underline represents second best performance. We do not directly compare to DDVC for fair comparison as they utilize GPT2 as their captioning head.

Method	#PT	CIDEr $\uparrow$	METEOR $\uparrow$	BLEU4 $\uparrow$	SODA $\_$ c $\uparrow$
Vid2Seq	✓	30.10	8.50	–	5.80
DIBS	✓	31.89	8.93	–	5.85
DDVC	✗	35.48	8.62	2.44	6.55
PDVC	✗	29.97	8.06	2.21	5.92
CM 2	✗	33.01	8.55	2.38	6.18
MCCL	✗	34.92	9.05	2.68	6.16

Table 2: ActivityNet Captions metrics. Bold represents best and underline represents second best performance. We do not directly compare to DDVC for fair comparison as they utilize GPT2 as their captioning head.

		YouCook2			ActivityNet
Method	#PT	Rec. $\uparrow$	Pre. $\uparrow$	F1 $\uparrow$	Rec. $\uparrow$	Pre. $\uparrow$	F1 $\uparrow$
Vid2Seq	✓	27.90	27.80	27.84	52.70	53.90	53.29
DIBS	✓	26.24	39.18	31.43	53.14	58.31	55.61
DDVC	✗	30.81	37.25	33.73	54.77	57.54	56.12
PDVC	✗	22.89	32.37	26.81	53.27	56.38	54.78
CM 2	✗	24.76	33.38	28.43	53.71	56.81	55.21

Table 3: Localization metrics on YouCook2 and ActivityNet. Bold represents best and underline represents second best performance. We do not directly compare to DDVC for fair comparison as they utilize GPT2 as their captioning head.

RSQI	CTCA	OSL	CG	C $\uparrow$	M $\uparrow$	B@4 $\uparrow$	S $\uparrow$	F1 $\uparrow$
✗	✗	✗	✗	$29.69$	$5.56$	$1.40$	$5.39$	$26.81$
✓	✗	✗	✗	$32.33$	$5.66$	$1.66$	$5.43$	$27.00$
✗	✗	✓	✗	$33.60$	$5.41$	$1.46$	$6.79$	$31.22$
✗	✗	✗	✓	$31.40$	$5.58$	$1.58$	$5.62$	$27.69$
✓	✓	✗	✗	$34.48$	$5.82$	$1.88$	$5.58$	$27.59$
✓	✗	✓	✗	$32.72$	$5.60$	$1.49$	$6.73$	$31.98$

Table 4: The ablation results of different components. RSQI denotes Role Specific Query Initialization , CTCA denotes Cross-Task Contrastive Loss , OSL denotes Overlap Suppression Loss , and CG denotes Concept Guider . Note that CTCA cannot be applied when RSQI is not used.

$\gamma$	CIDEr $\uparrow$	SODA $\_$ c $\uparrow$	Rec. $\uparrow$	Pre. $\uparrow$	F1 $\uparrow$
0.15	37.00	6.69	29.65	35.44	32.28
0.25	39.18	7.06	29.34	35.26	32.03
0.35	37.49	7.16	30.80	35.28	32.88
0.45	35.76	6.94	30.40	34.43	32.28

Table 5: Ablation results of varying

\gamma

on OSL.

\gamma

is a hyperparameter that adjusts the influence of IoU with respect to the ground truth.

$\alpha$	CIDEr $\uparrow$	SODA $\_$ c $\uparrow$	Rec. $\uparrow$	Pre. $\uparrow$	F1 $\uparrow$
W/o $\alpha$	34.81	7.20	31.01	33.82	32.35
W/ $\alpha$	39.18	7.06	29.34	35.26	32.03

Table 6: Ablation results with and without

\alpha

\alpha

controls whether the IoU is compared with the ground truth in OSL.

Figure 4: Qualitative results of dense video captioning on YouCook2. We compare the localization and captioning results with the ground truth, the baseline (PDVC), and ours. Each arrow represents localization boundaries and its corresponding caption is provided below. We find that while PDVC captures redundant events with the same captions, our model avoids this phenomenon.

$\mathrm{N}_{q}$	CIDEr $\uparrow$	METEOR $\uparrow$	BLEU4 $\uparrow$	SODA c $\uparrow$	F1 $\uparrow$
10	28.72	4.78	1.17	6.79	29.64
20	35.21	5.69	1.60	7.16	32.55
30	34.29	5.65	1.56	7.03	31.96
40	34.59	5.77	1.58	7.19	33.56
50	39.18	6.14	2.10	7.06	32.02
60	36.71	5.90	1.77	6.53	30.64

Table 7: Ablation results varying numbers of queries.

\mathrm{N}_{q}

denotes the number of queries.

Dense Video Captioning

See all 9 papers

Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning
Mar 12, 2026Seung hee Choi, MinJu Jeon, Hyunwoo Oh +2Video Saliency Prediction Dense Video Captioning
Hanyang University
SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning
Mar 5, 2026Ye-Chan Kim, SeungJu Cha, Si-Woo Kim +3Large Language Model-Based Data Augmentation Video Captioning
Hanyang University, South Korea
CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning
May 26, 2026Zihan Lin, Songhe Deng, Shuwei He +6Video Captioning Video-Language Models
ERNIE Team, Baidu · College of Artificial Intelligence, Inner Mongolia University

Mar 12, 2026J/K move · Enter open · S save

Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning

Seung hee Choi, MinJu Jeon, Hyunwoo Oh, Jihwan Lee+1

Hanyang University

Video Saliency Prediction Dense Video Captioning Video Captioning Temporal Video Understanding Retrieval-Augmented Generation

Open paper AskPDF arXiv

Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, STaRC, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at https://github.com/ermitaju1/STaRC

Abstract

Dense Video Captioning

Explore similar work

Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning

SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning

CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning