CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding

Yiduo Jia; Muzhi Zhu; Jinchuan Shi; Hao Zhong; Yuling Xi; Ke Liu; Hao Chen

cs.CVSep 30, 2026

CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding

Authors: Yiduo Jia, Muzhi Zhu, Jinchuan Shi, Hao Zhong, Yuling Xi, Ke Liu, Hao Chen

Organizations: Zhejiang University, State Key Lab of CAD & CG

Abstract

Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy-tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.

Figures & tables

Figure 1: Policy–tool coevolution framework for ultra-long video temporal grounding. A frozen VLM executes grounding tasks with an external skill comprising high-level policies and executable media tools. Across evolution rounds, an external skill updater jointly refines the policies and evolves the tools from execution trajectories, synthesizing code to upgrade existing tools or create new ones.

Figure 2: Policy-guided agentic inference with the evolved skill. Guided by the evolved policy, the VLM autonomously orchestrates media tools, using accumulated evidence to decide on subsequent observations without relying on a separate, stronger planning model. The illustrated trajectory combines image-based global search and candidate refinement with selective video verification to localize the queried event.

		VUE-LVTR				ExtremeWhenBench		CoMET-Bench
		30.1–105.5 min [-1pt](mean 50.9 min)				45.0–542.5 min [-1pt](mean 75.8 min)		30.0–123.7 min [-1pt](mean 50.5 min)
Method	Setting	Precision [-1pt] AUC $\uparrow$	Recall [-1pt] AUC $\uparrow$	IoU [-1pt] AUC $\uparrow$	IoU [-1pt] @0.5 $\uparrow$	Mean [-1pt] IoU $\uparrow$	Recall [-1pt] @0.5 $\uparrow$	Mean [-1pt] IoU $\uparrow$	Recall [-1pt] @0.5 $\uparrow$	F1 [-1pt] @0.5 $\uparrow$	Rejection [-1pt] F1 $\uparrow$
Qwen3.5-27B	768 frames	0.2645	0.2657	0.1904	0.1889	0.0392	0.0233	0.0683	0.0708	0.0455	55.05
InternVL3.5-8B	128 frames	0.1452	0.1649	0.0804	0.0619	0.0048	0.0018	0.0098	0.0037	0.0018	57.18
TimeLens-7B	384 frames	0.3665	0.3873	0.2758	0.2866	0.1024	0.0612	0.0277	0.0129	0.0126	72.17
Gemini 2.5 Flash	128 frames	0.1670	0.1815	0.0907	0.0684	0.0089	0.0057	0.0498	0.0369	0.0088	57.41

Table 1: Gains from policy–tool coevolution on ultra-long video temporal grounding. Bold : best; blue : gains over the base skill, with

\uparrow

indicating relative improvements.

		Token Efficiency -1pt			Interaction Efficiency -1pt

Benchmark	Method	Visual [-1pt] Tokens $\downarrow$	Image [-1pt] Tokens	Video [-1pt] Tokens	Model [-1pt] Calls	Local Tool [-1pt] Calls	Visual [-1pt] Obs.	Image [-1pt] Obs.	Video [-1pt] Obs.
VUE-LVTR [2pt]30.1–105.5 min [-1pt](mean 50.9 min)	Qwen3.5-27B + Base Skill	202.58	162.01	40.57	17.03	8.16	8.11	7.35	0.76
	Qwen3.5-27B + Evolved Skill	141.89	141.37	0.52	9.63	4.79	3.86	3.82	0.04
	$\Delta$ (Evolved $-$ Base)	$-$ 60.69	$-$ 20.64	$-$ 40.05	$-$ 7.40	$-$ 3.37	$-$ 4.25	$-$ 3.53	$-$ 0.72
		$\downarrow$ 29.96%	$\downarrow$ 12.74%	$\downarrow$ 98.72%	$\downarrow$ 43.5%	$\downarrow$ 41.3%	$\downarrow$ 52.4%	$\downarrow$ 48.0%	$\downarrow$ 94.7%

Table 2: Inference efficiency with the base and evolved skills. Green (

\downarrow

) and purple (

\uparrow

) indicate decreases and increases relative to the base skill.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Value
Video Duration (Range / Mean)	30.35–118.61 min / 52.90 min
Keyword / Phrase / Sentence	18 / 34 / 48
Single-Interval / Multi-Interval	56 / 44
Sparse-Target Queries	80
Boundary-Sensitive Queries	88
Video-Edge Queries	30

Appendix

Table 5: Composition and difficulty characteristics of the evolution set. The 100 queries come from 100 distinct videos. Sparse-target queries have a total target duration of at most 30 seconds, or at most 1% of the video duration. Boundary-sensitive queries contain at least one target interval no longer than 10 seconds. Video-edge queries have targets within the first or last 30 seconds of the video.

Tool	Role	Function
probe_media	Media Preparation	Read metadata such as video duration, resolution, and frame rate
extract_frames_at	Media Preparation	Extract frames at specified source-video timestamps
extract_video_clip	Media Preparation	Extract a video clip without audio from a specified time window
inspect_images	Image Observation	Add images to the VLM’s visual context
inspect_video	Video Observation	Add video to the VLM’s visual context
final_answer	Prediction	Submit the final answer in the format required by the current task

Appendix

Table 6: Tools in the base skill and their functions. Image and video observations are performed by the same VLM that executes the task. Local media tools prepare the inputs without calling additional visual models.

Benchmark	Total queries	Evolution queries	Evaluation queries
VUE-LVTR	407	100	307
ExtremeWhenBench	2,273	0	2,273
CoMET-Bench	2,789	0	1,599

Appendix

Table 7: Query counts and data splits for evolution and evaluation on the three temporal grounding benchmarks.

Parameter	Setting
Serving Engine	vLLM
Numerical Precision	bfloat16
Maximum Context Length	262,144 tokens
Thinking Mode	Enabled
Decoding	Greedy
Temperature / Top-p / Top-k	0 / 1.0 / 0

Appendix

Table 8: Model configuration for Qwen3.5-27B. The base and evolved skills use identical runtime settings.

VLM	Method	Precision AUC $\uparrow$	Recall AUC $\uparrow$	IoU AUC $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$
Qwen3.5-9B	+ Base Skill	0.1759	0.1961	0.1142	0.1600	0.1300	0.0500
	+ Evolved Skill	0.3065	0.3545	0.2024	0.3000	0.2000	0.1100
	Evolution Gain	+0.1306	+0.1584	+0.0882	+0.1400	+0.0700	+0.0600
		$\uparrow$ 74.2%	$\uparrow$ 80.8%	$\uparrow$ 77.2%	$\uparrow$ 87.5%	$\uparrow$ 53.8%	$\uparrow$ 120.0%
Qwen3.5-27B	+ Base Skill	0.3620	0.3047	0.2556	0.3400	0.2800	0.1700
	+ Evolved Skill	0.4285	0.3916	0.3207	0.4300	0.3500	0.2400

Appendix

Table 9: Grounding gains from policy–tool coevolution on the evolution set. Results are obtained by evaluating the fixed skills after evolution. Bold : better results; blue : gains over the base skill. Evolution Gain reports absolute changes, with arrows indicating relative changes.

Benchmark	Method	Visual Tokens $\downarrow$	Image Tokens	Video Tokens
VUE-LVTR held-out	Base Skill	241.21	214.98	26.23
	Evolved Skill	139.36	134.10	5.26
	$\Delta$ (Evolved $-$ Base)	$-$ 101.85	$-$ 80.88	$-$ 20.97
		$\downarrow$ 42.2%	$\downarrow$ 37.6%	$\downarrow$ 79.9%
ExtremeWhenBench	Base Skill	220.29	195.24	25.05
	Evolved Skill	158.33	152.64	5.69

Appendix

Table 10: Visual token cost of the base and evolved skills on Qwen3.6-27B. Bold : lower cost. Green (

\downarrow

) and purple (

\uparrow

) indicate decreases and increases relative to the base skill.

\Delta

denotes evolved cost minus base cost, with arrows indicating relative changes. Tokens: k/query.

VLM	Method	Action	Environment	Object	Reaction	Scene
Qwen3.5-9B	+ Base Skill	0.0543	0.0986	0.0877	0.0439	0.0570
	+ Evolved Skill	0.1092	0.1485	0.1518	0.1218	0.1246
	Evolution Gain	+0.0549	+0.0499	+0.0641	+0.0779	+0.0676
		$\uparrow$ 101.1%	$\uparrow$ 50.6%	$\uparrow$ 73.1%	$\uparrow$ 177.4%	$\uparrow$ 118.6%
Qwen3.5-27B	+ Base Skill	0.1446	0.2040	0.1862	0.1167	0.1377
	+ Evolved Skill	0.2459	0.2884	0.3427	0.1872	0.2664

Appendix

Table 11: Grounding results by query category on ExtremeWhenBench. Each column reports mIoU (

\uparrow

) for the corresponding category. Bold : better results; blue : gains over the base skill. Evolution Gain reports absolute changes, with arrows indicating relative changes.

VLM	Method	MAE $\downarrow$	OBO $\uparrow$	Pearson $\uparrow$
Qwen3.5-9B	+ Base Skill	3.7104	0.5378	0.3305
	+ Evolved Skill	3.6929	0.5516	0.6614
	Evolution Gain	$-$ 0.0175	+0.0138	+0.3309
		$\downarrow$ 0.5%	$\uparrow$ 2.6%	$\uparrow$ 100.1%
Qwen3.5-27B	+ Base Skill	3.6717	0.5760	0.0775
	+ Evolved Skill	3.4290	0.6116	0.1258

Appendix

Table 12: Event counting results on CoMET-Bench. Bold : better results; blue : gains over the base skill. Evolution Gain reports absolute changes, with arrows indicating relative changes.

VLM	Method	Rejection-F1 $\uparrow$	FPR $\downarrow$	PosCoverage $\uparrow$
Qwen3.5-9B	+ Base Skill	67.51	0.0989	0.5397
	+ Evolved Skill	68.80	0.0968	0.5556
	Evolution Gain	+1.29	$-$ 0.0021	+0.0159
		$\uparrow$ 1.9%	$\downarrow$ 2.1%	$\uparrow$ 2.9%
Qwen3.5-27B	+ Base Skill	66.13	0.1183	0.5291
	+ Evolved Skill	77.06	0.1032	0.6755

Appendix

Table 13: Rejection and positive-query coverage results on CoMET-Bench. Rejection-F1 is reported on a 0–100 scale, while FPR and PosCoverage use a 0–1 scale. Bold : better results; blue : gains over the base skill. Evolution Gain reports absolute changes, with arrows indicating relative changes.

VLM	Method	1 event $\uparrow$	2–3 $\uparrow$	4–7 $\uparrow$	8–15 $\uparrow$	$\geq$ 16 $\uparrow$
Qwen3.5-9B	+ Base Skill	0.1055	0.0673	0.0460	0.0376	0.0124
	+ Evolved Skill	0.1314	0.1296	0.0574	0.0622	0.0239
	Evolution Gain	+0.0259	+0.0623	+0.0114	+0.0246	+0.0115
		$\uparrow$ 24.5%	$\uparrow$ 92.6%	$\uparrow$ 24.8%	$\uparrow$ 65.4%	$\uparrow$ 92.7%
Qwen3.5-27B	+ Base Skill	0.2058	0.1846	0.1257	0.0860	0.0226
	+ Evolved Skill	0.2615	0.2101	0.1498	0.0939	0.0529

Appendix

Table 14: Grounding results by target-event count on CoMET-Bench. Groups are defined by the ground-truth event count, and each column reports macro-averaged [email protected] over the corresponding subset of positive queries. Bold : better results; blue : gains over the base skill. Evolution Gain reports absolute changes, with arrows indicating relative changes.

Figure 6: Candidate localization and motion verification. The base skill repeatedly inspects an incorrect candidate, whereas the evolved skill combines global contact-sheet search with local frame inspection and a single short video to verify the complete action, precisely localizing the target interval while reducing visual token cost by 51.27% .

Figure 7: Long-range search and boundary refinement. Sparse sampling misses the brief target, leading the base skill to mistakenly focus on a visually similar scene. The evolved skill instead combines global and local contact sheets with boundary frames to precisely localize the target interval, reducing visual token cost by 45.78% .

Explore similar work

VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding
Oct 7, 2026Yongchao Xu, Bowen Ye, Jiefeng Gan +5Long-Video Understanding Memory-Augmented Video Understanding
Alibaba Group, Hangzhou, China · University of Science and Technology of China, Hefei, China · Shanghai Jiao Tong University, Shanghai, China +2
MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understanding
Aug 5, 2026Benlei Cui, Ruize Wang, Junjie Li +9Video QA Long-Video Understanding
Yuvion Team, Alibaba Group · Zhejiang University · Beihang University +1
VideoEvolve: Evolving Agent Harnesses for Video Temporal Grounding
Oct 1, 2026Bingjun Luo, Yuhuan Fan, Jialin Guo +1Temporal Video Grounding Evolutionary Optimization
Tsinghua University · Columbia University · Harbin Engineering University

Oct 7, 2026, cs.CVJ/K move · Enter open · S save

VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding

Yongchao Xu, Bowen Ye, Jiefeng Gan, Junkai Ma+4

Alibaba Group, Hangzhou, China · University of Science and Technology of China, Hefei, China · Shanghai Jiao Tong University, Shanghai, China+2

Long-Video Understanding Memory-Augmented Video Understanding Video Retrieval RL for VLMs

Open paper AskPDF arXiv

Long video understanding increasingly relies on external memory to organize massive visual streams into compact representations. However, most memory-based methods dynamically adapt how information is retrieved for different questions, while largely fixing what is remembered. This mismatch makes missing details costly to recover, whereas stored information is valuable only when it can be reliably retrieved. To address this issue, we propose VideoEvolve, a novel self-evolving framework that jointly evolves memory and retrieval for long video understanding. Specifically, starting from a coarse low-frame-rate overview, VideoEvolve couples a Memory Evolver for selective memory augmentation with a Retrieval Evolver for adaptive retrieval over the evolving memory. We then co-evolve the two Evolvers through alternating agentic reinforcement learning (Agentic RL), updating one while freezing the other. To steer this alternating evolution, Bottleneck-Aware Evolution Feedback (BEF) identifies whether the current bottleneck lies in memory or retrieval and directs optimization toward the more limiting side. Furthermore, VideoEvolve introduces Capability-Aware Evolution Feedback (CEF) to alleviate downstream feedback from over-specializing memory to a fixed set of training questions, shifting training toward underdeveloped yet learnable video capabilities. By integrating Agentic RL with BEF and CEF, VideoEvolve transforms downstream reasoning experience into transferable capability updates, providing a concrete path from static long-video systems toward experience-driven, self-improving multimodal intelligence. Extensive experiments on multiple long video understanding benchmarks demonstrate the effectiveness of VideoEvolve.

Abstract

Explore similar work

VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding

MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understanding

VideoEvolve: Evolving Agent Harnesses for Video Temporal Grounding