OVET-Consensus: Rank-Preserving Counterfactual Event Verification for Video Moment Grounding
Organizations: Queen Mary University of London London, UK
Abstract
Modern video moment retrieval systems often propose windows that overlap the target event, yet still select the wrong one. We study this scorer bottleneck on QVHighlights with a fixed QD-DETR-derived temporal expert: the parent-aware temporal expert reaches 64.71 [email protected], while diagnostic oracles over the same candidate family reach 75.68 [email protected] within the parent-reranked pool and 94.97 [email protected] over the full 100-candidate evidence pool. We introduce OVET-Consensus, a rank-preserving counterfactual event verification layer that keeps the temporal expert's ranked list as the default and permits only high-confidence rank-1 overrides from a policy calibrated on held-out training videos. The verifier uses parent-aware interval evidence, structured Qwen3-VL event judgments, VideoLLaMA3 pairwise temporal signals, and dense temporal evidence, but never lets those signals freely reorder the top-K list. On QVHighlights validation, OVET-Consensus improves the parent-aware temporal expert from 64.71 to 65.35 [email protected] with a positive paired-bootstrap interval, while mAP is largely unchanged, from 42.67 to 42.74; it makes 20 overrides, with 15 helpful corrections and 3 harmful corrections. The results point to a practical constraint on verifier design: semantic compatibility is useful evidence, but temporal grounding still requires deciding where the event starts, changes, and ends inside a visually compatible scene.
Figures & tables
| OVET-Consensus calibration and inference | |
|---|---|
| Input | Video , query , temporal expert , candidate-pool size , evidence functions, frozen gate thresholds. |
| 1 | Generate candidate intervals for and group them by parent event hypothesis. |
| 2 | For each parent, score child boundary variants and keep the best child as the parent representative. |
| 3 | Rank the parent representatives with the parent-aware temporal expert; let be the default top interval. |
| 4 | For eligible alternatives, precompute or load Qwen3-VL CEV fields, VideoLLaMA3 pairwise fields, dense temporal scores, metric-calibrated ranks, and interval-contrast features. |
| 5 | On held-out training videos only, select consensus thresholds under an override-rate constraint; freeze the policy before validation. |
| Policy constraint | Selected value |
| Minimum consensus score | 8.747896 |
| Minimum votes | 6 |
| Maximum penalties | 0 |
| Maximum duration ratio | 1.15 |
| Maximum source rank | 2 |
| Minimum temporal-expert margin | -0.20 |
| Evidence group | Role in OVET-Consensus |
|---|---|
| Parent-aware interval evidence | Default ranking, parent identity, boundary and contrast features. |
| Qwen3-VL CEV | Structured semantic/event plausibility and outside-context evidence. |
| VideoLLaMA3 pairwise | Conservative temporal comparison between default and alternative. |
| DOTS dense selector | Weak dense evidence from visual, audio, and text/subtitle features. |
| Metric-calibrated selector | Auxiliary metric-quality rank; OVET-Consensus uses only whether this selector ranks the candidate first. |
| Consensus gate | Frozen training-calibrated rule to override rank 1 or preserve the default. |
| Method | Reported setting | R@[email protected] | R@[email protected] | mAP Avg. |
|---|---|---|---|---|
| Moment-DETR [ Lei et al.(2021)Lei, Berg, and Bansal ] | test | 52.89 | 33.02 | 30.73 |
| QD-DETR [ Moon et al.(2023)Moon, Hyun, Park, Park, and Heo ] | test | 62.40 | 44.98 | 39.86 |
| CG-DETR [ Moon et al.(2026b)Moon, Hyun, Lee, and Heo ] | test | 65.43 | 48.38 | 42.86 |
| Sim-DETR [ Tang et al.(2025)Tang, Wei, Zhu, Shi, Li, Lin, and Yang ] | test | 67.64 | 50.91 | 46.93 |
| FlashVTG [ Cao et al.(2025)Cao, Zhang, Du, Yu, Li, and Wang ] | test, InternVideo2 | 70.69 | 53.96 | 52.00 |
| CVA [ Moon et al.(2026a)Moon, Lee, Seo, and Im ] | test | 70.05 | 55.32 | 47.49 |
| Method | [email protected] | [email protected] | mAP | [email protected] | [email protected] | short | middle | long |
|---|---|---|---|---|---|---|---|---|
| Feature-pack reranker | 63.87 | 47.87 | 41.86 | 63.57 | 42.06 | 8.30 | 43.92 | 49.39 |
| Geometry-only reranker | 63.94 | 47.94 | 42.19 | 63.69 | 42.66 | 6.49 | 45.19 | 49.73 |
| Parent-aware boundary selector | 64.00 | 47.94 | 42.57 | 64.10 | 43.06 | 6.96 | 45.61 | 49.86 |
| Parent-aware temporal expert | 64.71 | 47.74 | 42.67 | 64.34 | 43.09 | 7.17 | 45.82 | 49.50 |
| Metric-calibrated selector | 64.45 | 48.26 | 42.67 | 64.24 | 43.16 | 7.03 | 45.74 | 49.84 |
| OVET-Consensus | 65.35 | 48.13 | 42.74 | 64.51 | 43.18 | 7.27 | 46.02 | 49.48 |
| Metric | OVET-Consensus parent-aware temporal expert | 95% bootstrap CI |
|---|---|---|
| [email protected] | +0.645 pp | [0.258, 1.097] |
| [email protected] | +0.387 pp | [0.065, 0.774] |
| mAP | +0.0736 pp | [-0.0324, 0.1854] |
| mean tIoU@1 | +0.004345 | [0.001163, 0.007950] |
| Method | mean tIoU@1 | R@[email protected] | R@[email protected] | R@[email protected] |
|---|---|---|---|---|
| Parent-aware boundary selector | 0.5867 | 0.6400 | 0.4794 | 0.8510 |
| Parent-aware temporal expert | 0.5906 | 0.6471 | 0.4774 | 0.8516 |
| Within-parent oracle, parent order | 0.6724 | 0.7200 | 0.5865 | 0.9213 |
| Within-parent oracle, parent reranker | 0.7023 | 0.7568 | 0.6116 | 0.9355 |
| Candidate subset | mean tIoU@1 | R@[email protected] | R@[email protected] |
|---|---|---|---|
| source rank | 0.6613 | 0.7077 | 0.5768 |
| source rank | 0.7551 | 0.8206 | 0.6755 |
| source rank | 0.7875 | 0.8639 | 0.7181 |
| source rank | 0.8293 | 0.9142 | 0.7761 |
| source rank / full pool | 0.8583 | 0.9497 | 0.8200 |
| Slice | Method | Queries | mean tIoU@1 | R@[email protected] | R@[email protected] |
|---|---|---|---|---|---|
| top-1 fail, top-10 success | Feature-pack reranker | 365 | 0.2360 | 0.0904 | 0.0712 |
| top-1 fail, top-10 success | Parent-aware temporal expert | 365 | 0.2549 | 0.1260 | 0.0740 |
| top-1 fail, top-10 success | Diagnostic oracle | 365 | 0.4207 | 0.4082 | 0.2082 |
| boundary near miss | Parent-aware temporal expert | 1108 | 0.5734 | 0.5957 | 0.4359 |
| boundary near miss | Diagnostic oracle | 1108 | 0.6860 | 0.7301 | 0.5623 |
| short interval | Parent-aware temporal expert | 1439 | 0.5924 | 0.6546 | 0.4830 |
| Variant | Decision rule | mean tIoU@1 | R@[email protected] | R@[email protected] | R@[email protected] |
|---|---|---|---|---|---|
| Parent-aware temporal expert | temporal default | 0.5906 | 0.6471 | 0.4774 | 0.8516 |
| Qwen CEV ranker | absolute VLM score | 0.4603 | 0.4542 | 0.2690 | 0.8510 |
| Qwen confidence gate | absolute VLM + confidence | 0.4898 | 0.4942 | 0.3097 | 0.8510 |
| Manual CEV delta gate | pairwise scalar gate | 0.5911 | 0.6445 | 0.4742 | 0.8510 |
| VideoLLaMA3 pairwise | conservative comparator | 0.5867 | 0.6400 | 0.4794 | 0.8510 |
| Direct DOTS selector | dense full rerank | 0.5740 | 0.6142 | 0.4645 | 0.7258 |
| Variant | [email protected] | [email protected] | mAP | Overrides | Helpful | Harmful |
|---|---|---|---|---|---|---|
| OVET-Consensus, all sources | 65.35 | 48.13 | 42.74 | 20 | 15 | 3 |
| without Qwen3-VL CEV | 64.71 | 47.74 | 42.67 | 0 | 0 | 0 |
| without VideoLLaMA3 | 65.35 | 48.13 | 42.74 | 20 | 15 | 3 |
| without DOTS | 64.84 | 47.81 | 42.68 | 3 | 3 | 0 |
| without metric-calibrated vote | 65.29 | 48.13 | 42.74 | 16 | 12 | 2 |
| temporal and interval evidence only | 64.71 | 47.74 | 42.67 | 0 | 0 | 0 |
| Verifier | Unit/query | Train | Val | Total | Parse fail. | Evidence-generation cost |
|---|---|---|---|---|---|---|
| Qwen3-VL CEV | 5 candidates | 36,090 | 7,750 | 43,840 | 0% | 4.8GB cache; val 4.9h, train 22.4h |
| VideoLLaMA3 pairwise | 4 pairs | 28,872 | 6,200 | 35,072 | 0% | 30GB cache; pair-level deduplication |
| Variant | Overrides | Helpful | Harmful | Neutral | Precision | Mean tIoU |
|---|---|---|---|---|---|---|
| Raw CEV override policy | 927 | 241 | 608 | 78 | 0.2600 | – |
| Manual CEV delta gate | 186 | 82 | 77 | 27 | 0.4409 | – |
| VideoLLaMA3 guarded selector | 0 | 0 | 0 | 0 | – | 0.0000 |
| OVET-Consensus | 20 | 15 | 3 | 2 | 0.7500 | +0.3367 |
| Method | Unique parents@5 | Mean pairwise tIoU@5 | R@[email protected] |
|---|---|---|---|
| Parent-aware temporal expert | 5.0000 | 0.1540 | 0.8516 |
| OVET-Consensus | 5.0000 | 0.1540 | 0.8516 |
| Gate | [email protected] | [email protected] | mAP | Overrides | Helpful | Harmful | Precision |
|---|---|---|---|---|---|---|---|
| reported gate | 65.35 | 48.13 | 42.74 | 20 | 15 | 3 | 0.750 |
| votes | 65.55 | 48.26 | 42.82 | 46 | 25 | 12 | 0.543 |
| votes | 65.03 | 48.00 | 42.70 | 11 | 7 | 3 | 0.636 |
| source rank | 65.42 | 48.13 | 42.74 | 23 | 16 | 4 | 0.696 |
| duration ratio | 65.23 | 48.06 | 42.72 | 18 | 13 | 3 | 0.722 |
| duration ratio | 65.42 | 48.13 | 42.74 | 21 | 16 | 3 | 0.762 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Vote condition | Votes |
|---|---|
| CEV margin | 1 |
| Event margin or contrast margin | 1 |
| Temporal-expert margin | 1 |
| Source rank | 1 |
| Duration ratio in | 1 |
| DOTS score exists and DOTS margin | 1 |
| Penalty condition | Penalties |
|---|---|
| Outside-context match increases relative to the default | 1 |
| Duration ratio | 1 |
| Source rank | 1 |
| Temporal-expert margin | 1 |
| VideoLLaMA3 broader/shifted flag | 1 |
Temporal Video Grounding
Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.
See all 179 papers