VideoEvolve: Evolving Agent Harnesses for Video Temporal Grounding

Bingjun Luo; Yuhuan Fan; Jialin Guo; Siqi Li

cs.AIOct 1, 2026

VideoEvolve: Evolving Agent Harnesses for Video Temporal Grounding

Authors: Bingjun Luo, Yuhuan Fan, Jialin Guo, Siqi Li

Organizations: Tsinghua University · Columbia University · Harbin Engineering University

Abstract

Video temporal grounding aims to localize events in videos from natural-language queries. For agents built around frozen video-language models, the harness determines how queries guide temporal predictions and how those predictions are refined. Manually refining these harnesses requires diagnosing grounding failures and coordinating changes to both agent workflows and instructions. We introduce VideoEvolve, a framework that automatically evolves agent harnesses for video temporal grounding. VideoEvolve uses a Cloze-Structured Harness Representation that preserves stage interfaces while leaving agent workflows and instructions open to evolution. Branch-Guided Harness Evolution preserves promising code branches for continued refinement, using execution feedback to guide local edits and validation to determine which improvements are carried forward. Experiments demonstrate improved grounding performance across multiple benchmarks. Component analyses identify instruction refinement as a consistent source of gains, while the benefits of evolved code vary across evaluation settings. Together, these results support automated harness evolution as an effective approach to improving video temporal grounding. Code is available at https://github.com/bingjunluo/VideoEvolve .

Figures & tables

Figure 1: VideoEvolve improves the harness around a frozen model. (a) Workflows and instructions evolve within stable stage interfaces, guided by execution feedback and validation (schematic). (b) Evolving the harness improves macro

S

on TimeLens-Bench (vertical axis starts at 30).

Figure 2: VideoEvolve. (a) Fixed stage interfaces contain editable workflows, with instructions evolving alongside code. (b) Retained code branches are refined off the accepted-harness mainline. Development comparison and validation gate promotion, while eligible validation rejections provide a repair path.

Input: frozen model $M$ , tools $U$ , limits $B$ , seed $h_{\mathrm{seed}}$ ,
	development and validation data
$h\leftarrow h_{\mathrm{seed}}$ ; initialize code-branch and repair state
while stopping condition is not met do
	Execute $h$ on the current development batch; construct evidence $E$
	Select parents from $h$ or the retained code branch; add eligible near misses as repair context
	Propose local edits and retain candidates passing interface checks

Algorithm 1 Branch-Guided Harness Evolution

Training-based Methods
	Charades-STA		ActivityNet		QVHighlights		ReXTime		TimeLens-Bench
Method	R@.5	mIoU	R@.5	mIoU	R@.5	mIoU	R@.5	mIoU	R@.5	mIoU
MH-DETR	56.37	–	47.15	–	60.84	–	–	–	–	–
TimeChat	32.20	–	–	–	–	–	7.61	11.65	–	–
VTimeLLM	27.50	31.20	27.80	30.40	–	–	17.41	20.14	–	–
UVCOM	59.25	–	–	–	65.10	–	–	–	–	–

Table 1: Video temporal grounding results on five benchmarks, comparing VideoEvolve with training-based and agentic methods. R@.5 and mIoU are reported as percentages; higher is better. Bold indicates the best result in each column.

	Charades-STA		ActivityNet		QVHighlights		ReXTime		TimeLens-Bench
Search variant	R@.5	mIoU	R@.5	mIoU	R@.5	mIoU	R@.5	mIoU	R@.5	mIoU
No-Branch	66.59	58.82	41.11	44.24	67.81	64.22	56.89	53.38	51.56	49.64
Full	70.81	61.90	41.99	44.04	72.32	67.24	58.96	54.56	53.72	51.05

Table 2: Branch-retention ablation; endpoints come from a separate search pair.

Dataset	Harness configuration	R@.5 $\uparrow$	mIoU $\uparrow$	Calls $\downarrow$	Input tokens (k) $\downarrow$
ReXTime	Seed	51.14	48.71	0.996	3.24
	Evolved instructions only	55.70	52.00	0.996	3.97
	Evolved code only	52.99	49.85	1.963	6.53
	Full VideoEvolve	59.17	53.93	1.948	7.89
TimeLens-Bench	Seed	46.86	45.68	1.000	3.73
	Evolved instructions only	53.02	50.35	1.000	4.45

Table 3: Grounding quality and per-query inference cost of harness components. TimeLens-Bench quality is macro-averaged.

Figure 3: Grounding refinement and failure. Final predictions from the seed (

C_{0}T_{0}

), evolved instructions alone (

C_{0}T_{1}

), and full harness (

C_{1}T_{1}

). Shading marks the ground-truth interval; calls count solver invocations.

Explore similar work

EvoGround: Self-Evolving Video Agents for Video Temporal Grounding
May 13, 2026Minjoon Jung, Byoung-Tak Zhang, Lorenzo TorresaniVideo Temporal Grounding Temporal Grounding
Seoul National University · Northeastern University
MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understanding
Aug 5, 2026Benlei Cui, Ruize Wang, Junjie Li +9Video Agent Self-Evolving Agents
Yuvion Team, Alibaba Group · Zhejiang University · Beihang University +1
Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding
May 21, 2026Zelin Zheng, Xinyan Liu, Ruixin Li +4Video Temporal Grounding Temporal Grounding
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Beijing Key Laboratory of Embodied Intelligence Computing, Beijing, China · Faculty of Computing, Harbin Institute of Technology, Weihai, China +1

May 13, 2026, cs.CVJ/K move · Enter open · S save

EvoGround: Self-Evolving Video Agents for Video Temporal Grounding

Minjoon Jung, Byoung-Tak Zhang, Lorenzo Torresani

Seoul National University · Northeastern University

Video Temporal Grounding Temporal Grounding Vision-Language Model Grounding Self-Evolving Agents

Open paper AskPDF arXiv

Video temporal grounding (VTG) takes an untrimmed video and a natural-language query as input and localizes the temporal moment that best matches the query. Existing methods rely on large, task-specific datasets requiring costly manual annotation. We introduce EvoGround, a framework of two coupled self-evolving agents, a proposer and a solver, that learn temporal grounding from raw videos without any human-labeled data. The proposer generates query--moment pairs from raw videos, while the solver learns to ground them and feeds back signals that improve the proposer in return. Through this self-reinforcing reinforcement-learning loop, the two agents are initialized from the same backbone and mutually improve across iterations. Trained on 2.5K unlabeled videos, EvoGround matches or surpasses fully supervised models across multiple VTG benchmarks, while emerging as a state-of-the-art fine-grained video captioner without manual labels.

Abstract

Explore similar work

EvoGround: Self-Evolving Video Agents for Video Temporal Grounding

MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understanding

Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding