TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding

Pengyiang Liu; Junbo Niu; Xiaoyang Hu; Zhongyue Shi; Zitian Wang; Linjiang Huang; Si Liu

cs.CVAug 23, 2026

TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding

Authors: Pengyiang Liu, Junbo Niu, Xiaoyang Hu, Zhongyue Shi, Zitian Wang, Linjiang Huang, Si Liu

Organizations: Colab, Beihang University · Shanghai AI Laboratory · Peking University

Abstract

A long-video answer is evidence-supported only when the frames decoded from the video cover every event the answer depends on. Existing evaluations score final-answer correctness or predicted evidence intervals, but the frames a method decodes before answering are rarely audited, so correct answers can still rest on incomplete observation. We introduce VES-Bench, a 600-question benchmark of Temporal Ordering and Event Counting items over 348 public long videos. Each item carries a jointly necessary set of evidence intervals, letting us audit at three strictness levels whether a method's decoded frames cover every one of them. We also propose TRACE, a training-free agent that grounds answers in raw visual clips, builds an evidence bundle round by round, and stops only when the answer stabilises as the bundle grows and a final pass over the same clips returns the same answer. Under a same-backbone audit, TRACE answers 50.7% of questions correctly with at least two decoded frames inside every evidence interval, at 98.7 frames per question: over 10 points above uniform decoding at 128 frames (40.2%), and within 2.6 points of uniform decoding at 256 frames at 0.39x its frame cost, while reaching the highest answer accuracy in the audit (63.5%). TRACE also stays competitive on Video-MME (86.1), LVBench (75.6), and LongVideoBench (75.1).

Figures & tables

Figure 2: TRACE workflow. The system converts the full video into visual anchors, constructs prefix bundles to form an evidence growth trajectory, and uses the Trajectory Reconciler to stop, drop, refine, or expand the observed evidence.

Figure 3: VES-Bench duration and evidence-layout statistics.

Method	Video-MME	LVBench	LongVideoBench
Closed-source VLMs
Gemini-2.5-Flash	74.2	62.2	66.2
Gemini-2.5-Pro	82.4	67.4	69.8
Open-source VLMs
Qwen3-VL-30B-A3B	79.2	67.7	–
AdaReTaKe-72B	73.5	53.3	67.0

Table 1: Answer-only long-video benchmark accuracy (Video-MME no-subtitle; LVBench/LongVideoBench validation). Backbones in parentheses. The VCA (Gemini-2.5-Pro) rows are our reruns; all other rows reproduce the numbers reported by their sources ( Comanici et al. 2025 ; Bai et al. 2025a ; Wang et al. 2025b ; Bai et al. 2025b ; Zuo et al. 2025 ; Guo et al. 2025 ; Zhang et al. 2025 ; OpenAI 2025 ; Yang et al. 2025 ; Hurst et al. 2024 ; Wang et al. 2025d ) . Evidence-supported claims are on VES-Bench (Table 2 ).

	Fr	Acc $\uparrow$	EP $\uparrow$	AR $\uparrow$	Cov@ $k$ $\uparrow$	ECA@ $k$ $\uparrow$
Method		(%)	(%)		$k\!=\!1\,/\,2\,/\,3$	$k\!=\!1\,/\,2\,/\,3$
Non-visual baselines
Random	0	25.0	–	–	–	0.0 / 0.0 / 0.0
Blind GPT-4-Turbo (text-only)	0	25.3	–	–	–	0.0 / 0.0 / 0.0
Same-backbone uniform sweep (Gemini-3-Flash)
Gemini-3-Flash, 16	16	35.3	13.7	1.00	19.8 / 4.0 / 1.7	9.2 / 2.5 / 1.3

Table 2: VES-Bench audit. Random and Blind GPT-4-Turbo (text-only) are non-visual baselines; ECA@

k

0

by construction. Uniform-

k

rows report sampler statistics (Fr, EP, AR, Cov@

k

) computed on the canonical centered-uniform schedule

t_{j}=(j+\tfrac{1}{2})T_{i}/n

; these are dataset-defined and identical across backbones, so the Direct VLM block at Uniform-

64

shares them with the Gemini-3-Flash,

64

row above and varies only in Acc and ECA@

k

Figure 4: Frame budget against evidence-supported correctness, plotted from Table 2 on a log frame axis. The grey curve is the same-backbone uniform sweep; each point is labelled with its ECA@

2

. TRACE (

98.7

frames) and AVP (

101.3

) sit on the vertical rule,

2.6

frames apart.

Figure 5: Every prediction at

k\!=\!2

split into Cov-Corr (correct, Cov@

2

passing), Uncov-Corr (correct without Cov@

2

), and Wrong. Cov-Corr is ECA@

2

and Cov-Corr

+

Uncov-Corr is Acc, both in Table 2 , up to independent rounding of each value.

	Temporal Ordering		Event Counting
Method	Acc $\uparrow$	ECA@ $k$ $\uparrow$	Acc $\uparrow$	ECA@ $k$ $\uparrow$
	(%)	$1\,/\,2\,/\,3$	(%)	$1\,/\,2\,/\,3$
Gemini-3-Flash, 64	49.7	37.0 / 6.3 / 1.0	38.0	36.7 / 24.7 / 15.3
Gemini-3-Flash, 128	75.0	72.0 / 45.3 / 14.7	40.0	40.0 / 35.0 / 30.0
Gemini-3-Flash, 256	74.3	74.0 / 67.7 / 52.7	39.0	39.0 / 39.0 / 36.3
VCA	48.0	2.7 / 0.0 / 0.0	27.7	14.0 / 6.7 / 3.0

Table 3: Per-family breakdown. ECA@

k

k\!=\!1,2,3

. Temporal Ordering has the largest spread across

k

(short per-event intervals); Event Counting Acc plateaus near

40\%

across the Gemini-3-Flash budget points, reflecting a recognition bottleneck.

Variant	Acc $\uparrow$	EP $\uparrow$	Cov@2 $\uparrow$	ECA@2 $\uparrow$
TRACE	63.5	29.1	78.5	50.7
w/o final answer replay	59.7	28.2	77.3	42.0
w/o growth trajectory	55.0	19.4	56.3	27.5
w/o EXPAND	53.0	26.4	49.0	25.5
fixed-round stopping	52.3	25.7	47.2	24.0

Table 4: Ablation of TRACE components on VES-Bench (Gemini-3-Flash). Headline densities Cov@

2

, ECA@

2

; full

k\!=\!1,2,3

in Appendix Table 7 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Action	Trigger	Execution
STOP	A minimal stable prefix exists: later anchors change neither its answer nor its supported facts, and it has no unmet evidence need.	Select the prefix bundle for final answering.
DROP	Some anchors introduce noise, conflict, or misleading local evidence.	Remove target anchors and reconstruct the trajectory.
REFINE	An observed interval is relevant but lacks temporal or spatial detail.	Reobserve the candidate interval with a more suitable observation strategy.
EXPAND	The current evidence bundle lacks events, states, or identity cues that may affect the answer.	Introduce new candidate intervals from unexplored segments.

Appendix

Table 5: Control actions emitted by the Trajectory Reconciler.

Method	Acc	EP	Cov@2	ECA@2
G3F, $64$	$43.8_{\pm 3.8}$	$13.7_{\pm 1.1}$	$34.3_{\pm 3.8}$	$15.5_{\pm 2.8}$
G3F, $128$	$57.5_{\pm 3.9}$	$13.8_{\pm 1.1}$	$74.7_{\pm 3.2}$	$40.2_{\pm 3.9}$
G3F, $256$	$56.7_{\pm 4.1}$	$13.7_{\pm 1.1}$	$95.5_{\pm 1.7}$	$53.3_{\pm 4.2}$
VCA	$37.8_{\pm 3.9}$	$16.9_{\pm 1.0}$	$5.8_{\pm 1.9}$	$3.3_{\pm 1.4}$
AVP	$59.0_{\pm 3.9}$	$13.7_{\pm 0.4}$	$68.2_{\pm 3.7}$	$39.7_{\pm 3.9}$
TRACE	$63.5_{\pm 3.9}$	$29.1_{\pm 1.5}$	$78.5_{\pm 3.3}$	$50.7_{\pm 4.0}$

Appendix

Table 6:

95\%

bootstrap confidence intervals at

k\!=\!2

(Gemini-3-Flash). Paired bootstrap shows TRACE vs. AVP differences in EP, Cov@

2

, and ECA@

2

are significant at

p<0.01

; AR rescales EP by the constant

\mathrm{EP}_{\mathrm{ref}}

and carries the same test.

Variant	Acc $\uparrow$	Cov@ $k$ $\uparrow$	ECA@ $k$ $\uparrow$
		$1\,/\,2\,/\,3$	$1\,/\,2\,/\,3$
TRACE	63.5	98.3 / 78.5 / 60.3	63.0 / 50.7 / 39.0
w/o final answer replay	59.7	97.8 / 77.3 / 59.5	58.0 / 42.0 / 31.0
w/o growth trajectory	55.0	90.3 / 56.3 / 30.5	49.5 / 27.5 / 16.3
w/o EXPAND	53.0	84.2 / 49.0 / 24.8	44.0 / 25.5 / 13.7
fixed-round stopping	52.3	82.3 / 47.2 / 23.5	43.5 / 24.0 / 13.0

Appendix

Table 7: Ablation of TRACE components on VES-Bench across density levels. The row for TRACE matches Table 4 ; the columns expose how each ablated component shifts Cov@

k

and ECA@

k

k\!=\!1,2,3

Source	License
Ego4D ( Grauman et al. 2022 )	Ego4D License Agreement
Public YouTube	Terms of the original uploader

Appendix

Table 8: Licenses of the VES-Bench source datasets.

Video QA

Video question answering studies how AI systems use visual and audio evidence over time to answer questions about video.

See all 180 papers

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification
Apr 2, 2026Jiahao Meng, Yue Tan, Qi Xu +9Video-Language Model Evaluation Long-Video QA
PKU · WHU · CASIA +2
Video Evidence Indexing: Learning Where to Look from Video Previews for Token-Budgeted Long-Video Question Answering
Sep 30, 2026Haowen Guan, Shengzhi Li, Shichao PeiVideo QA Long-Video QA
Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering
Aug 3, 2026Fan Wei, Siru Zhong, Runmin Dong +3Long-Video Understanding Video Frame Selection
Department of Earth System Science, Tsinghua University · 5The Hong Kong University of Science and Technology (Guangzhou) · School of Artificial Intelligence, Sun Yat-Sen University +2

Apr 2, 2026J/K move · Enter open · S save

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

Jiahao Meng, Yue Tan, Qi Xu, Haochen Wang+8

PKU · WHU · CASIA+2

Video-Language Model Evaluation Long-Video QA Temporal Video Grounding

Open paper AskPDF arXiv

Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can locate the evidence needed to answer a question. We introduce VideoZeroBench, a challenging long-video benchmark with manually annotated question-answer pairs spanning 13 video domains. Questions target fine-grained cues, fleeting events, and evidence distributed across multiple segments. Temporal intervals and key-frame boxes are annotated where applicable. All questions undergo two rounds of cross-verification for answer validity and evidence quality. Our five-level diagnostic protocol compares answering with and without evidence hints, then combines answer correctness with independently evaluated temporal and spatial grounding. Across 19 evaluated models, the best standard QA accuracy is 24.8% (Level-3), achieved by Gemini-3.7-Flash. No model exceeds 1.8% when correct answers and accurate spatio-temporal localization are jointly required (Level-5). Analyses of atomic abilities, evidence spans, input modalities, and thinking-with-videos inference further characterize where the evaluated systems struggle. These findings motivate more precise evidence search and localization for long-video question answering. Our code and data are publicly released.

Abstract

Video QA

Explore similar work

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

Video Evidence Indexing: Learning Where to Look from Video Previews for Token-Budgeted Long-Video Question Answering

Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering