VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness

Zihao Lin; Haoyu Dong; Haobo Xu; Chi-Chih Chang; Xuying Ning; Yinglong Xia; Shuang Yang; Zhuokai Zhao

cs.CVOct 14, 2026

VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness

Authors: Zihao Lin, Haoyu Dong, Haobo Xu, Chi-Chih Chang, Xuying Ning, Yinglong Xia, Shuang Yang, Zhuokai Zhao,

Organizations: University of California, Davis · Meta MRS Research · Duke University · University of Illinois at Urbana-Champaign ·

Abstract

LLM system capabilities depend not only on model parameters but also on the executable harnesses that organize information acquisition, tool use, reasoning, and memory. Recursive harness evolution can automate their design, but how it can reliably discover generalizable systems from limited, noisy, and repeatedly reused feedback remains unclear. We study this question in agentic video understanding, where rich multimodal evidence and coupled mechanisms make harness optimization challenging. We first introduce VHarness-BRSI, a recursive self-improvement framework that evolves video harnesses around a frozen vision-language model. Analyzing over 4,000 evolved candidates, we find it competitive with hand-engineered systems but identify three interconnected failure modes: validation overfitting, search-budget dilution, and the early-experience trap. We trace all three to a common root: the same imperfect validation evidence drives both harness revisions and the proposer's search strategy. We therefore propose VHarness-SRSI, an evidence-governed framework in which every search decision rests on evidence that is transferable, sufficient, and gathered before the search commits. It restricts the proposer to content-free feedback, informs exploration with instrumented warm-up experiments, and develops mechanisms in isolated probe-race branches before factorial composition and joint refinement. Across four video understanding benchmarks, VideoHarness-SRSI improves the equally weighted mean held-out score by 6.2% relative to VideoHarness-BRSI and reduces the mean validation-test gap by 37.9%. Reliable harness evolution thus depends on how search is organized and informed, not only on how well it optimizes validation performance.

Figures & tables

Figure 1: Overview of VHarness-BRSI . (a) The evolution loop: a coding proposer inspects the validation videos and questions, revises the harness around a frozen VLM, and archives each candidate’s validation scores and execution traces, which later iterations reuse. (b) In a retrieval-only run on Video-MME, the evolved harness matches the held-out accuracy of the strongest hand-designed AKS variant with

38.7\%

fewer frames (Section 3.1 ). Bottom: the same validation verdict both selects the harness and steers the next search; it mixes the true gain with validation fit, noise, and champion inflation, which gives rise to the three failure modes of Section 3.2 .

Figure 2: Evolving the retrieval strategy with VHarness-BRSI on Video-MME. (a) Best validation accuracy found so far over 50 iterations (line) and the range of the candidates proposed in each iteration (shaded), starting from the clip_topk seed (dotted). Stars mark the strategies on the Pareto frontier; the red labels are their names, chosen autonomously by VHarness-BRSI . (b) Held-out accuracy on 2,250 questions never seen during evolution against the mean number of frames per question, for the deployed evolved harness, three variants of the hand-designed AKS method, and the seed.

Figure 3: Three failure modes of VHarness-BRSI : what happens (top) and why (bottom). (a) Validation minus held-out accuracy of hand-written baselines and of harnesses evolved under the three feedback arms on LongVideoBench; each dot is a harness, bars show means with 95% intervals, and diamonds mark each arm’s validation-selected winner. (b) Best validation gain over the seeds on Video-MME when evolving retrieval only (one family) or the full harness (six families); lines are means and bands span the three runs. (c) Champion frame count of retrieval runs on Video-MME. Thick lines are the two runs discussed in the text, with their frame-raising probes circled (

+10

and

-10

validation questions); the labels give the mean held-out accuracy of runs that expand the frame budget and of runs that stay near 16 frames. (d) Shares of the spread in validation scores due to true gain (

\rho_{\Delta}

), validation fit (

\rho_{S}

), and noise (

\rho_{\varepsilon}

) in each feedback arm. (e) Each run placed by its evaluations per mechanism combination

r

and verdict noise

\sigma

; small and large dots are runs and search-space means, and contours mark equal power to detect a 1-point gain, shaded below

0.2

. (f) Against the verdict

z

(in standard errors), the probability that later proposals build on a candidate (black) and that the candidate beats the champion on held-out data (blue dots, with the model curve); shading marks abandoned candidates that were likely better (red) and continued ones that were likely worse (gray).

Figure 4: Overview of VHarness-SRSI . Top: under information isolation, the harness executes on the validation videos and questions, but the proposer receives only scores and content-free traces, never the videos, question text, options, or answers. Bottom: evolution proceeds in three stages under this policy. Stage I (instrumented warm-up) adds a removal and an intensity change to the free proposals of each round, summarizes the evidence in a per-mechanism report, and selects a seed pool and a backbone

H_{0}

. Stage II (isolated branch evolution) evolves each mechanism axis in its own branch from

H_{0}

; every axis is probed with three candidates, the top half race on to fifteen, and at most three axes are kept. Stage III (factorial composition) distills each kept axis into a fixed modification, evaluates all combinations against

H_{0}

to measure transfer and interactions, and then jointly refines them into the final harness

H^{\star}

Fixed Baselines
Method	Video-MME		LVB		OVO-Bench		MomentSeeker
Method	Val	Test	Val	Test	Val	Test	Val	Test
uniform	$60.44$	$61.64$	$58.97$	$59.15$	$52.76$	$52.20$	$11.94$	$11.43$
clip_topk	$66.21$	$63.40$	$62.33$	$62.07$	$51.94$	$53.32$	$16.58$	$17.98$
tool_survey	$65.78$	$62.27$	$59.42$	$60.49$	$50.51$	$51.28$	$16.72$	$17.66$
reflect_loop	$60.44$	$60.04$	$62.11$	$60.61$	$51.33$	$55.36$	$16.23$	$17.66$

Table 1: Main results. Evolved harnesses report mean

\pm

sample standard deviation over three seeds; fixed baselines are deterministic.

Benchmark	Method	Test	Frames/q	VLM calls/q	Tool calls/q
Video-MME	VHarness-SRSI	70.1	106.1	2.44	19.3
Video-MME	VHarness-BRSI	66.5	55.1	2.43	1.6
LVB	VHarness-SRSI	67.3	64.8	2.85	17.5
LVB	VHarness-BRSI	66.6	69.2	2.77	5.0
OVO-Bench	VHarness-SRSI	72.5	47.8	4.91	2.6
OVO-Bench	VHarness-BRSI	67.2	27.3	3.57	1.5

Table 2: Cost of the deployed harnesses. Held-out score and per-question cost, averaged over three seeds.

Benchmark	Budget	Test	Frames/q	Calls/q	Test	Frames/q	Calls/q	$\Delta$
		VHarness-SRSI (budget-matched)			VHarness-BRSI (deployed)
Video-MME	$\leq$ 55.1	68.93	52.5	2.04	66.50	55.1	2.43	$+$ 2.43
LVB	$\leq$ 69.2	67.19	57.8	2.86	66.60	69.2	2.77	$+$ 0.59
OVO-Bench	$\leq$ 27.3	68.42	25.8	3.64	67.20	27.3	3.57	$+$ 1.22
MomentSeeker	$\leq$ 60.3	21.41	56.0	2.48	19.00	60.3	2.52	$+$ 2.41

Table 3: Budget-matched comparison. VHarness-SRSI restricted to the frame budget of the deployed VHarness-BRSI harness and selected by validation accuracy, averaged over three seeds.

\Delta

: held-out gain of VHarness-SRSI .

Figure 5: Held-out score versus frames per question on each benchmark. Deployed harnesses of VHarness-BRSI (squares) and VHarness-SRSI (circles), budget-matched VHarness-SRSI harnesses (stars), and fixed baselines (gray dots), each averaged over three seeds. The dotted line marks the budget, i.e., the frames per question of the deployed VHarness-BRSI harness. On every benchmark, the budget-matched VHarness-SRSI harness lies above VHarness-BRSI without using more frames.

Setting	Val	Held-out
VHarness-BRSI feedback (Multimodal)	$71.00_{\pm 3.21}$	$65.43_{\pm 1.18}$
VHarness-BRSI + isolation (Blindfolded)	$71.22_{\pm 0.34}$	$66.93_{\pm 0.81}$
VHarness-SRSI	$71.1_{\pm 2.0}$	$\mathbf{67.3}_{\pm 1.0}$

Table 4: Information-isolation ablation on LongVideoBench. Accuracy (%) of each run’s validation argmax, mean

\pm

std over three seeds.

Proposer	Val	Test
Opus 5	$75.0_{\pm 1.9}$	$70.1_{\pm 1.0}$
Sonnet 5	$73.3_{\pm 0.6}$	$67.7_{\pm 0.4}$
Opus 4.8	$71.4_{\pm 0.8}$	$65.8_{\pm 0.1}$

Table 5: Proposer ablation on Video-MME. Mean

\pm

std over three seeds.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Benchmarks	Factor varied	Runs	Budget	Candidates	Where
Diagnosis of VHarness-BRSI (Section 3 )
Effectiveness	VM	– (retrieval only)	1	$50\times 3$	150	Fig. 2
Validation overfitting	LVB	proposer feedback; validation fraction	27	$30\times 3$	2,430	App. E
Search-budget dilution	VM	number of mechanism families (1, 2, 3, 6)	11	varies a	1,820	App. F
Early-experience trap	VM, LVB	– (re-analysis of existing runs)	44	varies b	4,806	App. G
Simple remedies (Section 3.2 )

Appendix

Table 6: Index of experiments. All experiments use the frozen Qwen3-VL-8B-Instruct executor and, except for the proposer ablation of Section 5.4 , Claude Opus 5 as the proposer. Budget : iterations

\times

candidates per iteration of one run. Candidates : scheduled total over the study’s runs; for the early-experience trap, the candidates actually proposed (the “over 4,000” of the abstract). Studies share runs, so the column does not sum. BRSI: VHarness-BRSI ; SRSI: VHarness-SRSI . VM: Video-MME; LVB: LongVideoBench; OVO: OVO-Bench; MS: MomentSeeker.

Benchmark	Pool	Validation	Held-out	Split unit	Metric
Video-MME	2,700	450	2,250	question	accuracy
LongVideoBench	1,337	446	891	question	accuracy
OVO-Bench	1,468	489	979	question	accuracy
MomentSeeker	1,000	261	739	video	mIoU

Appendix

Table 7: Benchmark splits. Seeded random splits; the split unit is the question, or the source video for MomentSeeker.

Default frames per pass	16	32
	Multiple choice	Temporal grounding
Default pixels per frame	230,400	151,200
Default output tokens	16	64
Maximum frames per pass	768	256
Executor calls per question	12
Tool calls per question	60

Appendix

Table 8: Default inference settings and per-question limits. The frame and pixel values are defaults that a harness may override per call; the limits are enforced by the interface.

Harness (code name)	Procedure	Frames
Uniform ( uniform )	Samples frames uniformly over the video and answers in one grounded call.	16
Retrieval ( clip_topk )	Ranks pool frames by CLIP similarity to the question and options, backfills the selection with uniform frames to the budget, and answers in one call.	16
Self-reflection ( reflect_loop )	Answers on the retrieved frames, confronts the executor with its own answer on the same frames, and takes the revision (at most two rounds).	32
Tool-gated Selection ( tool_survey )	Asks the executor which entities would resolve the question, detects them in a retrieved candidate pool, and answers on the frames that contain them.	16
Role Debate ( debate_pair )	A proponent argues for the leading option, a skeptic for the strongest rival, and an arbiter decides after looking at the frames again with both arguments.	80
Memory Survey ( memo_multiturn )	Splits the video into shots, writes one observation per shot keyframe into working memory, answers with the memory digest, and re-retrieves once on a cue named by the executor if the answer remains uncertain.	56

Appendix

Table 9: Seeded harnesses. Frames: mean frames shown to the executor per Video-MME question, summed over calls.

Arm	What the proposer observes about the validation set
Blindfolded	Per-question predicted letter and correctness, and about ten behavioral counters (model and tool calls, frames, rendering, memory). No question text, options, or frames.
Trace	Blindfolded feedback plus each question, its options, the full interaction trace, and the first raw answer.
Multimodal	Trace feedback plus on-demand rendered validation frames, and a written anti-overfitting protocol.

Appendix

Table 10: Information arms (three seeds

\times

three ladders per arm).

	$n$	Mean gap, candidates	Val $>$ held-out	Winner val / held-out (%)	Winner gap
Hand-written	18	$-0.16$ $[-1.07,0.82]$	39%	–	–
Blindfolded	89	$2.02$ $[1.62,2.39]$ $[1.11,2.95]$	84%	$70.2$ / $66.5$ $[65.6,67.4]$	$3.69$
Trace	103	$2.91$ $[2.43,3.39]$ $[2.13,4.00]$	89%	$71.0$ / $66.3$ $[64.4,67.8]$	$4.76$
Multimodal	89	$3.21$ $[2.73,3.71]$ $[2.16,4.22]$	90%	$71.1$ / $65.7$ $[64.8,66.5]$	$5.46$

Appendix

Table 11: Validation–held-out gaps (pp). Candidate CIs are 95% bootstrap intervals over candidates, with run-clustered intervals in brackets. “Winner” is each run’s validation argmax, averaged over the nine runs of an arm.

Arm	Candidates	Transfers ( $\beta$ )	Instance-specific	Noise	Reliability
Blindfolded	752	$0.54$ $[0.33,0.74]$	$0.31$ $[0.13,0.50]$	$0.16$ $[0.14,0.17]$	$0.84$
Trace	774	$0.35$ $[0.24,0.44]$	$0.51$ $[0.42,0.62]$	$0.14$ $[0.11,0.18]$	$0.86$
Multimodal	763	$0.30$ $[0.15,0.41]$	$0.56$ $[0.44,0.70]$	$0.14$ $[0.11,0.19]$	$0.86$

Appendix

Table 12: Reliability versus validity of the validation score. Shares of across-candidate validation variance, with 95% run-cluster bootstrap CIs; reliability is one minus the noise share.

Figure 8: Transfer of validation gains. (a) Held-out versus validation gain over the run’s best hand-written baseline for every tested candidate. Lines have slope

\hat{\beta}

through the arm’s mean; stars mark the mean winners. Selection moves right along a line, so a flatter line converts the same validation advantage into less held-out gain. (b)

\beta

under alternative estimators and per ladder, with 95% CIs.

Arm	Matched questions	Validation	Expected	Ratio	$p$
Blindfolded	50	26	27.1	0.96	0.68
Trace	233	168	108.3	1.55	$<10^{-4}$
Multimodal	69	49	38.7	1.27	0.007

Appendix

Table 13: Verbatim validation-question 5-grams in evolved code (33% and 50% ladders, six runs per arm).

p

: one-sided permutation test over the split.

Space	$K$	$P$	Runs	Iterations $\times$ candidates	Champion found at iteration
Retrieval only	1	8	3	$50\times 3$	21, 46, 50
+ frame selection	2	8	3	$50\times 3$	28, 45, 39
+ perception tools	3	12	2	$50\times 3$ ; $100\times 5$	49, 99
Full harness	6	33	3	$30\times 3$	28, 29, 21

Appendix

Table 14: The search-space ladder on Video-MME.

K

: chartered mechanism families;

P

: callable primitives. The deployed champion is the validation-selected harness at the end of each run.

Figure 9: All four spaces. (a) Best-so-far validation gain against candidates evaluated (mean and min–max over runs), on a budget-matched axis. (b) Validation (hollow) and held-out (filled) accuracy of each run’s deployed champion; bars are means. (c) Distinct mechanism combinations visited against candidates evaluated; the dashed line is the limit in which every candidate is a new mechanism combination. (d) Evidence landscape of Figure 3 (e) with the three-family space, which has the breadth of the full harness but the footprint of retrieval-only evolution.

Space	Val. gain	Val.	Held-out	Combos	$r$	$d$	$\sigma$	$\hat{\delta}$	0.5	1	2
	Outcome (%)			Search behavior					Power for $\Delta$ (pp)
Retrieval only (1)	6.4	72.7	66.8	26.0	3.54	25.3	1.03	$-1.10$	0.15	0.45	0.95
+ frame selection (2)	5.1	70.5	65.8	19.0	4.61	31.8	1.19	$-1.42$	0.15	0.44	0.95
+ perception tools (3)	7.6	77.2	66.0	45.0	1.93	23.6	1.08	$-0.76$	0.10	0.26	0.73
Full harness (6)	5.1	71.8	65.7	51.3	1.73	51.3	1.54	$-3.14$	0.07	0.14	0.41

Appendix

Table 15: Outcome and evidence by space. Validation gain is the best-so-far gain over the run’s best seed harness after 87 candidates; accuracies are those of the deployed champions. Evidence columns use the first 87 candidates of each run: distinct mechanism combinations, evaluations per mechanism combination

r

, mean footprint

d

, mean verdict noise

\sigma

(pp), mean verdict

\hat{\delta}

(pp), and the noncentrality

\lambda

and power of Proposition F.1 for an advantage

\Delta

. All entries are means over runs.

Figure 10: The behavioral footprint of edits. (a) Distribution of the footprint

d

, the number of validation answers whose correctness an edit flips relative to its incumbent. (b) Mean verdict by footprint bin (95% intervals); the line is Proposition F.2 with the pooled fix probability

q=0.37

. Bins above 80 are dominated by crashes and omitted. (c) Standard deviation of verdicts within each bin against the paired noise scale

100\sqrt{d}/n

Benchmark	Setting	Runs
Video-MME	retrieval only (16-frame seeds; one from 32-frame AKS)	6
	retrieval + frame selection	4
	retrieval + perception tools (one with $100\times 5$ )	2
	self-reflection	2
LongVideoBench	Blindfolded / Trace / Multimodal feedback	$9+9+9$
	clade-UCB parent suggestions (33% ladder)	3

Appendix

Table 16: The 44 VHarness-BRSI runs analyzed in Section 3.2.3 . Video-MME runs search one or a few mechanism families for 50 iterations of three candidates (one run: 100 iterations of five); the 12 retrieval-family runs (all but self-reflection) form the frame-budget contrast of Figure 3 (c). LongVideoBench runs search the full harness for 30 iterations under the feedback arms of Section 3.2.1 , three seeds at each of three validation fractions.

Subset	Pairs	$\widehat{\tau}$	$\widehat{\sigma}$	$\widehat{\omega}$	Reliability
Video-MME, all	387	0.49 [0.31, 0.58]	1.04 [0.90, 1.23]	1.02 [0.64, 1.39]	0.18 [0.09, 0.24]
iterations 1–10	86	0.45 [0.18, 0.66]	1.49 [1.40, 1.59]	1.01 [0.69, 1.32]	0.08
iterations $>10$	301	0.48	0.87	1.02	0.24
fully tested runs	296	0.46	1.00	1.11	0.18
LongVideoBench, all	158	1.01 [0.72, 1.22]	1.06 [0.94, 1.16]	0.44 [0.21, 0.70]	0.48 [0.32, 0.57]

Appendix

Table 17: Verdict-model estimates (Eq. ( 15 ); pp). Reliability is

\tau^{2}/(\tau^{2}+\sigma^{2})

, i.e.,

\kappa

of Proposition G.1 with

s^{2}=0

. Brackets: 95% run-bootstrap intervals.

Figure 11: Theory of the trap. (a) LongVideoBench counterpart of Figure 3 (f): the continuation policy is a step at

z=0

, while the held-out win rate of the same verdicts changes gradually; the curve is Proposition G.1 with the fitted parameters. (b) Probability that a beneficial direction is lost after one trial (Proposition G.2 ) at the Video-MME calibration, without incumbent optimism, with a policy that reconsiders half of all rejections, and with four times as many validation questions (which halves

\sigma

and, being selection noise,

\omega

). Points mark the frame-budget direction; the square is the observed fraction of retrieval-family runs that never expand the budget (Wilson 95% interval).

Figure 12: Over-commitment to an early winner (Video-MME, retrieval-only, seed 1; the run of Figure 3 (c)). (a) Validation accuracy of every candidate, the validation champion (solid), and the champion’s held-out accuracy (dashed); iterations 6–50 are shaded. (b) Held-out accuracy of the 115 tested candidates against their validation verdict at proposal. The horizontal line marks the deployed champion’s held-out accuracy.

Figure 13: Parent selection. (a) Validation (hollow) and held-out (filled) accuracy of each run’s deployed harness; bars mark arm means. (b) Best-so-far validation gain over the best seeded harness (mean and seed range). (c) Share of candidates bred from the most-used parent against held-out gain over the best baseline; large outlined markers are arm means.

Figure 14: Stronger initialization (Video-MME, retrieval only; blue: three runs from simple seeds, orange: the AKS-seeded run). (a) Best-so-far validation accuracy. (b) Held-out accuracy of tested running champions; stars mark deployed harnesses. (c) Held-out accuracy of the starting point (hollow) and deployed harness (filled); shading spans the simple-seed endpoints. (d) Iteration at which each validation level is first reached; hollow markers are levels not reached in 50 iterations.

Setting	Value	Setting	Value
Warm-up slots A / B / C per round	3 / 1 / 1	Probe survivors (tie slack)	$k_{\mathrm{r}}$ (+1)
Warm-up chunk; min. / max. rounds; patience	2; 8 / 10; 4	Gate $z_{0}$ / floor (pp) / min. clusters	1.64 / 0.5 / 3
Class-1 / free-form templates; min. kept	6 / 1; 4	Composed axes (cells)	$\leq 3$ ( $\leq 7$ )
Branch candidates per round; $r_{1}$ ; $r_{2}$	3; 3; 15	MERGE rounds $\times$ candidates	$10\times 3$

Appendix

Table 18: Settings of VHarness-SRSI . The free-only warm-up control (Section 5.4 ) differs only in the warm-up slots (5 / 0 / 0). Here

k_{\mathrm{r}}=\min\bigl(k,\max(2,\lceil k/2\rceil)\bigr)

for

k

probed axes. The warm-up runs in chunks of two rounds and stops once the champion has not changed for four rounds, after at least 8 and at most 10 rounds.

Run	A	B	C	Imp.	Und.	Val	Test	Fr.	Val	Test	Fr.
	Slots			Families		Backbone			Deployed
Full-1	27	11	12	6	2	73.56	69.87	50	77.11	71.07	110
Full-2	26	9	15	3	1	71.11	68.40	55	73.33	70.00	141
Full-3	23	15	12	5	2	70.44	68.58	38	74.67	69.16	66
Mean	25.3	11.7	13.0	4.7	1.7	71.70	68.95	48	75.04	70.07	106
Free-1	38	2	10	2	4	73.11	66.58	44	74.00	66.31	63

Appendix

Table 19: Per-run results of the warm-up ablation (Video-MME). Slots: behavioral counts of warm-up candidates. Families with an importance estimate (Imp.) or an undefined coverage increment (Und.). Backbone: warm-up champion; deployed: validation argmax of joint refinement. Val, Test in %; Fr.: frames per validation question. † Stopped after round 8 (plateau rule).

Figure 15: Warm-up ablation on Video-MME (thin lines, markers: runs; thick lines, bars: arm means). (a) Best-so-far validation accuracy (round 0: best seeded harness). (b) Warm-up backbones. (c) Held-out accuracy vs. frames per question, backbone (hollow) to deployed system (filled).

Comparison	Harness	All held-out	Unseen videos	$\Delta$ (unseen)
Information isolation (LongVideoBench, Table 4 )	VHarness-BRSI feedback (Multimodal)	65.43	67.12	$+1.64$ $[+0.11,+3.28]$
Information isolation (LongVideoBench, Table 4 )	VHarness-BRSI + isolation (Blindfolded)	66.93	68.76	$+1.64$ $[+0.11,+3.28]$
Evolved vs. AKS (Video-MME, Figure 2 )	Evolved medium_augspread (38 frames)	68.36	68.02	–
	AKS-BLIP (62 frames)	68.40	68.73	$-0.71$ $[-2.45,+1.16]$
	AKS-CLIP (62 frames)	68.09	68.22	$-0.19$ $[-2.20,+1.74]$

Appendix

Table 20: Held-out accuracy (%) on all held-out questions and on unseen-video questions.

\Delta

: difference on unseen-video questions, with its 95% bootstrap interval over videos.

	Stage	$r\uparrow$	$\sigma\downarrow$	Power $\uparrow$
BRSI	Full harness	1.55	1.55	0.13
SRSI	I: warm-up	1.59	1.62	0.12
	II: branch	9.80	1.61	0.50
	III: merge	1.68	1.12	0.22
	Whole run	2.31	1.48	0.18

Appendix

Table 21: Evidence per decision on Video-MME: evaluations per configuration

r

, verdict noise

\sigma

, and power to detect a 1-point gain. The VHarness-BRSI row uses the first 57 candidates of each run.

	VM	LVB	OVO	MS	Total
VHarness-BRSI	1 / 3	0 / 3	1 / 3	2 / 3	4 / 12
VHarness-SRSI	0 / 3	0 / 3	0 / 3	0 / 3	0 / 12

Appendix

Table 22: Early-experience trap. Runs whose champion never exceeds

1.2\times

the seeds’ frame budget (16 frames; 32 on MS), three seeds each; VHarness-BRSI searches the full harness with Multimodal feedback. VM: Video-MME; OVO: OVO-Bench; MS: MomentSeeker.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 387 papers

Sep 29, 2026

Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution

Video understanding agents acquire evidence through an executable harness that controls what they observe and how they use those observations. However, execution traces contain only the evidence acquired by the current harness, leaving competing explanations for failure unresolved and limiting the basis for self-improvement. We introduce Video-RSI, a framework for recursive self-improvement in which a video understanding agent uses its own language model to revise its harness. Through active video investigation, the model revisits the original training videos to test competing failure explanations with additional observations, grounding proposed changes in evidence beyond the existing trace. Cost-aware harness evolution turns these diagnoses into reusable revisions and determines which revisions to retain by considering both answer accuracy and visual cost. Across our evaluation settings on video understanding benchmarks, the evolved agent improves accuracy while processing fewer frames and achieves competitive accuracy-efficiency trade-offs against existing video understanding agents. These results demonstrate the potential for video understanding agents to improve their own evidence acquisition and use through harness evolution. Code is available at https://github.com/bingjunluo/Video-RSI .

Bingjun Luo, Jialin Guo, Siqi Li

Tsinghua University · Harbin Engineering University

Video Understanding Agent Harness Optimization LLM Agent Self-Improvement Self-Improving Agents

Ask PDF

Aug 25, 2026

VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models

Long-video understanding depends not only on the capability of a vision-language model (VLM), but also on how its limited context is constructed from a much longer video. Existing systems typically introduce hand-designed sampling, retrieval, memory, or agentic control strategies, making the context-construction program itself difficult to study as an independent optimization target. We introduce VideoHarness-RSI, a controlled framework that recursively searches executable context constructors around a frozen VLM while keeping the answering model and interface fixed. We study this baseline under complementary weak- and strong-initialization regimes. From a weak uniform constructor, recursive search progressively discovers more structured context-construction programs; from a stronger AKS harness, the same process further advances an already competitive hand-crafted frontier. The resulting harness retains its advantage under a matched cumulative visual-token control and transfers directly to additional long-video benchmarks without further search. Together, these results establish executable context construction as a distinct optimization layer and provide an auditable baseline for studying harness discovery, transfer, and efficiency around frozen VLMs.

Guoyang Xu, Hao Chen

Tencent

Program Synthesis Long-Video Understanding Long-Context VLMs Temporal Video Understanding

Ask PDF

Sep 29, 2026

VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding

Vision-language models (VLMs) can answer questions about hour-long videos, but processing every frame is prohibitively expensive, even though the evidence for a question usually spans only a few seconds. Video agents, i.e., harness programs wrapped around a frozen VLM, address this by observing the video selectively, yet existing harnesses are hand-crafted by experts through slow build-and-test cycles. We propose VidHarness, a framework that automates harness design for cost-efficient long video understanding, in which a harness proposer iteratively evolves harnesses based on execution feedback from an evolution environment. To escape the local optima of greedy refinement, we organize the evolution as Monte Carlo tree search (MCTS), and to reduce the evaluation cost, we integrate uncertainty-aware multi-fidelity validation, which screens new harnesses on a few questions and promotes only the promising ones. Since the best harness varies with the frame budget, we further introduce a mixture-of-harness that routes each question to a harness specialized for its budget. VidHarness sets new state-of-the-art results on LongVideoBench, Video-MME, and Video-Holmes, outperforms the strongest hand-crafted video agent by up to

11.2

points, and generalizes to the knowledge-intensive benchmarks Video-MMMU and MMVU with fewer than half of the frames of uniform sampling.

Susan Liang, Jianmin Wu, Daxiang Dong

Baidu, Inc.

Long-Video Understanding LLM Agent Harnesses Long-Context VLMs

Ask PDF

Abstract

LLM Agent Harnesses

Explore similar work

Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution

VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models

VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding