SAPO: Self-Supervised Attention Policy Optimization for Improving Temporal Video Grounding in Vision-Language Models

Zixuan Chen; Depeng Wang; Wei Chen; Ke Xu; Ya Guo; Huijia Zhu; Tanfeng Sun; Xinghao Jiang

cs.CVOct 19, 2026

SAPO: Self-Supervised Attention Policy Optimization for Improving Temporal Video Grounding in Vision-Language Models

Authors: Zixuan Chen, Depeng Wang, Wei Chen, Ke Xu, Ya Guo, Huijia Zhu, Tanfeng Sun, Xinghao Jiang

Organizations: Shanghai Jiao Tong University · Ant Group

Abstract

Video vision-language models (VLMs) have strong semantic understanding, yet remain unreliable at precise temporal grounding. Existing gains from explicitly aligning video frames with absolute time suggest that VLMs may already encode accurate temporal locations internally but lose precision when expressing them as timestamps. Our analysis across four VLMs and two grounding datasets confirms this hypothesis: attention intervals outperform predicted intervals on approximately 66% of valid examples. We term this discrepancy the temporal generation gap. Because precise temporal annotations are expensive, this finding motivates closing the gap to improve temporal grounding using only unlabeled video-query pairs. We propose Self-Supervised Attention Policy Optimization (SAPO), which uses internal temporal attention as self-supervision for Group Relative Policy Optimization and applies a ground-truth-free criterion to select high-value samples. SAPO consistently improves temporal grounding across multiple datasets and models spanning weak to strong initial grounding ability. Relative mIoU gains reach 33.7% on Qwen2.5-VL-3B and 5.4% on the strong Time-R1-7B. The improvements transfer across datasets while preserving general video understanding.

Figures & tables

Figure 1: Attention knows better than prediction. The attention-decoded interval $I_{a}$ can localize the queried event more accurately than the generated interval $I_{p}$ . SAPO uses attention as supervisor to improve native timestamp prediction without temporal-boundary annotations during training.

Model	Data	Attn. wins †	Attn. mIoU	Pred. mIoU
Qwen2.5-VL-3B	Charades	65.0	39.5	36.5
Qwen2.5-VL-3B	ActivityNet	71.0	32.8	24.6
Qwen2.5-VL-7B	Charades	62.0	46.1	43.6
Qwen2.5-VL-7B	ActivityNet	67.0	32.4	27.4
InternVL3-2B	Charades	79.0	39.2	15.2
InternVL3-2B	ActivityNet	64.0	23.9	16.9

Table 1: The temporal generation gap across models and datasets. Attn. wins † is the fraction of valid samples where

\mathrm{IoU}(I_{a},I_{g})>\mathrm{IoU}(I_{p},I_{g})

, excluding all-zero ties.All values are percentages.

Figure 2: Average temporal intervals visualize the generation gap. Each panel overlays the average ground-truth interval, the average predicted interval, and the average attention interval for one model–dataset pair. The attention interval is consistently closer to the ground-truth region than the explicit prediction.

Figure 3: Conventional TVG training and our annotation-free alternative. Top: Conventional TVG post-training relies on ground-truth temporal boundaries, using them either as supervised targets for fine-tuning or as rewards for reinforcement learning. Bottom: SAPO instead derives self-supervision from the VLM’s internal temporal attention to improve the model’s temporal grounding ability without ground-truth boundaries.

Figure 4: Attention lead rate increases monotonically with attention–prediction JS divergence. The rate is computed over all samples, with ties retained as non-leads.

			Charades-STA					ActivityNet-Captions
Base	Method	train split	[email protected]	[email protected]	[email protected]	mIoU	$\Delta$	[email protected]	[email protected]	[email protected]	mIoU	$\Delta$
Qwen2.5-VL-3B	Zero-shot	–	0.553	0.346	0.175	0.365	–	0.342	0.216	0.119	0.246	–
	+ SAPO	Charades	0.664	0.392	0.158	0.415	+13.7%	0.409	0.241	0.126	0.281	+14.2%
	+ SAPO	ActivityNet	0.629	0.368	0.165	0.398	+9.0%	0.489	0.285	0.133	0.327	+32.9%
	+ SAPO	Charades+ActivityNet	0.656	0.384	0.156	0.410	+12.3%	0.493	0.287	0.130	0.329	+33.7%
Qwen2.5-VL-7B	Zero-shot	–	0.641	0.422	0.213	0.436	–	0.381	0.238	0.130	0.274	–

Table 2: SAPO consistently improves temporal grounding across base models and training sources .

	Video-MME
Model	Acc.	$\Delta$
Qwen2.5-VL-3B-Instruct	58.0	–
+ SAPO trained on Charades	57.0	-1.7%
+ SAPO trained on ActivityNet	57.4	-1.0%
+ SAPO trained on Charades+ActivityNet	57.4	-1.0%
Time-R1-7B	60.1	–

Table 3: General video understanding on Video-MME .

	Charades-STA		ActivityNet-Captions
Setting	mIoU	$\Delta$	mIoU	$\Delta$
Base (zero-shot)	0.365	–	0.246	–
SAPO from train sets	0.410	+12.3%	0.329	+33.7%
SAPO from test sets	0.421	+15.3%	0.341	+38.6%

Table 4: Standard and transductive adaptation with SAPO.

	Predicted Interval: Charades-STA				Predicted Interval: ActivityNet				Attention Interval
Variant	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU	Char. mIoU	Act. mIoU
Base	0.553	0.346	0.175	0.365	0.342	0.216	0.119	0.246	0.395	0.328
Attention pseudo-label + SFT	0.647	0.371	0.149	0.395	0.480	0.247	0.102	0.315	0.394	0.322
SAPO (Full)	0.656	0.384	0.156	0.410	0.493	0.287	0.130	0.329	0.401	0.335
1 w/o attention-derived rewards	0.492	0.312	0.159	0.330	0.226	0.113	0.054	0.163	–	–
2 w/o temporal distribution reward	0.597	0.362	0.161	0.378	0.406	0.249	0.122	0.277	–	–

Table 5: Ablation of SAPO . The first eight columns evaluate predicted intervals, while the last two report the mIoU of attention intervals. Variants 1–5 are discussed in the same order in the ablation analysis. Best and second-best results are marked in bold and underlined, respectively.

Figure 5: SAPO moves the average predicted interval toward the ground-truth temporal region.

Figure 6: SAPO preferentially improves samples whose base attention is more accurate than their base prediction.

SAPO with different training-sample selectors
	Charades-STA			ActivityNet-Captions
Selector	[email protected]	mIoU	$\boldsymbol{\Delta}$	[email protected]	mIoU	$\boldsymbol{\Delta}$
Base (no training)	0.346	0.365	–	0.216	0.246	–
Random	0.369	0.396	+8.5%	0.271	0.312	+26.8%
JS divergence	0.384	0.410	+12.3%	0.287	0.329	+33.7%
GT oracle †	0.386	0.423	+15.9%	0.292	0.343	+39.4%

Table 6: Downstream TVG performance under different training-set selection strategies . The variants differ only in the selected training samples;

\Delta

denotes the relative mIoU improvement over the base model.

Figure 7: Comparison of selection metrics.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Dataset	Layer
Qwen2.5-VL-3B	Charades-STA	L28
Qwen2.5-VL-3B	ActivityNet	L28
Qwen2.5-VL-7B	Charades-STA	L22
Qwen2.5-VL-7B	ActivityNet	L22
Qwen3-VL-4B	Charades-STA	L24
Qwen3-VL-4B	ActivityNet	L24

Appendix

Table 7: Attention layers used for temporal interval decoding. A representative middle-to-late layer is specified without ground-truth layer selection and kept fixed across both datasets and all experiments.

		Attention Better		Prediction Better		Tie (Both Zero)
Model	Dataset	TAD	PTD	TAD	PTD	TAD	PTD
Qwen2.5-VL-3B	Charades-STA	0.515	0.268	0.298	0.663	0.000	0.000
InternVL3-2B	Charades-STA	0.489	0.077	0.227	0.511	0.000	0.000
Qwen2.5-VL-7B	Charades-STA	0.587	0.346	0.368	0.689	0.000	0.000
Qwen3-VL-4B	Charades-STA	0.617	0.359	0.360	0.660	0.000	0.000
Qwen2.5-VL-3B	ActivityNet-Captions	0.412	0.124	0.270	0.656	0.000	0.000

Appendix

Table 8: Pre-SAPO group-wise attention–prediction accuracy (mIoU). Queries are grouped by comparing the ground-truth IoU of TAD and PTD.

Dataset	Group	Samples	Share	TAD mIoU	PTD mIoU
Charades-STA	Attention better (TAD $>$ PTD)	1,682	45.2%	0.565	0.352
	Prediction better (PTD $>$ TAD)	1,745	46.9%	0.310	0.535
	Tie (TAD $=$ PTD)	293	7.9%	0.000	0.000
ActivityNet-Captions	Attention better (TAD $>$ PTD)	1,689	42.2%	0.454	0.250
	Prediction better (PTD $>$ TAD)	1,957	48.9%	0.293	0.457
	Tie (TAD $=$ PTD)	354	8.9%	0.000	0.000

Appendix

Table 9: Post-SAPO attention–prediction group statistics. Queries are grouped by comparing the ground-truth IoU of TAD and PTD. TAD mIoU and PTD mIoU are computed within each group.

Panel of Figure 8 — Figure 8: Diagnostic sensitivity analysis for Qwen2.5-VL-3B on Charades-STA, shown as a representative example. We report the attention lead rate : the fraction of examples on which the attention interval attains a higher IoU with the ground truth than the model’s own predicted interval, excluding ties. (a) The lead rate remains stable across the examined middle-to-late layers. (b) The same conclusion holds across a range of attention-mass thresholds around the fixed value $0.5$ .

Setting	Charades	ActivityNet
Qwen2.5-VL-3B + SAPO	40.9 $\pm$ 0.3	32.8 $\pm$ 0.4
Qwen2.5-VL-7B + SAPO	46.1 $\pm$ 0.2	33.3 $\pm$ 0.4
Time-R1-7B + SAPO	53.2 $\pm$ 0.2	46.7 $\pm$ 0.3
SAPO from test sets	42.0 $\pm$ 0.3	33.9 $\pm$ 0.4
Random selection	39.5 $\pm$ 0.3	31.1 $\pm$ 0.4
JS selection	40.9 $\pm$ 0.3	32.8 $\pm$ 0.4

Appendix

Table 10: Run-to-run stability (mIoU, mean

\pm

std.).

Model	Video-MME Acc.
Qwen2.5-VL-3B	58.0
+ SAPO	57.4 $\pm$ 0.3
Time-R1-7B	60.1
+ SAPO	60.3 $\pm$ 0.2

Appendix

Table 11: General-ability stability across SAPO runs (accuracy, mean

\pm

std.).

Model	Overall	Short	Medium	Long
Qwen2.5-VL-3B-Instruct	58.0	65.7	58.6	49.9
+ SAPO on Charades	57.0	65.2	57.2	48.7
+ SAPO on ActivityNet	57.4	65.7	57.8	48.7
+ SAPO on Charades+ActivityNet	57.4	66.2	57.2	48.7
Time-R1-7B	60.1	69.7	60.3	50.2
+ SAPO on Charades	60.4	69.9	60.4	51.0

Appendix

Table 12: Full Video-MME results (accuracy, %).

Temporal Video Grounding

Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.

See all 169 papers

TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action
May 2, 2025Jen-Hao Cheng, Yi-Hao Peng, Huapeng Zhou +11Temporal Action Segmentation Temporal Video Grounding
University of Washington · Carnegie Mellon University · National Yang Ming Chiao Tung University +1
Temporal-Aware Reasoning Optimization for Video Temporal Grounding
Jun 8, 2026Minghang Zheng, Zihao Yin, Yi Yang +2Temporal Video Grounding Temporal Reasoning
Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University.
Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification
Jun 27, 2026Tianshu Zhang, Yan Wang, Ji Qi +1Temporal Video Grounding Efficient Multimodal Inference
Tsinghua University

May 2, 2025J/K move · Enter open · S save

TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action

Jen-Hao Cheng, Yi-Hao Peng, Huapeng Zhou, Vivian Wang+10

University of Washington · Carnegie Mellon University · National Yang Ming Chiao Tung University+1

Temporal Action Segmentation Temporal Video Grounding Video Captioning Video Reasoning Causal Reasoning in Videos Sequence Modeling+1

Open paper AskPDF arXiv

Understanding causal event relationships and achieving fine-grained temporal grounding in videos remain challenging for vision-language models (VLMs). We propose TEMPURA (Temporal Event Masked Prediction and Understanding for Reasoning in Action), a two-stage training framework that enhances the video temporal understanding of VLMs. Inspired by infilling techniques in language modeling, TEMPURA first performs masked event prediction, learning to reconstruct missing events and generate step-by-step causal explanations from dense event annotations. It then learns video segmentation and dense captioning, decomposing videos into non-overlapping events with detailed, timestamp-aligned descriptions. We train TEMPURA on VER, our large-scale dataset of 500K videos annotated with temporally aligned event descriptions and structured reasoning steps. Experiments on video temporal grounding and highlight detection benchmarks show that TEMPURA substantially improves strong base VLMs across model families and scales, confirming that combining event-level reasoning with fine-grained temporal segmentation is an effective recipe for video temporal understanding.

Abstract

Temporal Video Grounding

Explore similar work

TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action

Temporal-Aware Reasoning Optimization for Video Temporal Grounding

Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification