Selecting or Solving? What Best-of-N Verification Delivers in Video Reasoning
Organizations: Meta · Carnegie Mellon University
Abstract
Answering a complex question about a long video is a grounding problem: the answer turns on a few seconds of evidence, and vision-language models remain well below human accuracy at finding them. The standard way to spend extra test-time compute on this kind of video reasoning is best-of-N selection: sample several chain-of-thought answers from a fixed model, then let a second model re-watch the video and pick one. Such pipelines are reported by a single headline number, their margin over a majority vote. That number conflates two effects with opposite deployment consequences: the benefit the second model adds by selecting among candidates, and the benefit it would have supplied by solving the question itself. We separate them, measuring the Selection Increment (SI) -- selection accuracy minus the same verifier's accuracy with the candidates withheld -- alongside the margin over self-consistency, across five long-video benchmarks. The Selection Increment is largest for the weakest verifiers: a verifier that solves poorly on its own leaves the candidates the most room to help. Beating the majority vote follows the opposite profile: where the vote already sits near its ceiling no verifier we test improves on it; below that ceiling only the strongest do. On Video-Holmes, the two come apart on a single pool of candidates -- a weak verifier gains +19.3 points from the candidates yet lands 7.1 points below the vote, while the frontier verifier gains the least and beats the vote. The weakest verifier also shows a failure mode: on EgoSchema it picks by where an option sits in the list rather than by what the video shows. Read as a merit score, SI puts a verifier that fails to beat a majority vote at the top of its ranking; read as an audit, it says which of the two effects a headline number delivers. Code is available at https://github.com/facebookresearch/best-n-selection-video-reasoning.
Figures & tables
| Role | Model(s) | Frames |
|---|---|---|
| Base generator ( CoT, ) | Qwen3.5-27B | 64 (32 on EgoSchema) |
| Verifier, cross-model (headline) | Muse Spark 1.1 | 48 |
| Verifier, cross-model (strong) | GPT-5.6 Sol | 32 |
| Verifier, same-family | Qwen3.5-27B / -397B | 48 / 48, 96 |
| Verifier, weak cross-family | Phi-3.5-V | 16, 24, 48, 64 |
| Verifier, mid cross-family | InternVL3-8B / -38B | 48 / 24, 48 |
| Rule | acc. (%) |
|---|---|
| pass@1 | |
| self-consistency | |
| best verifier | |
| oracle@8 |
| Benchmark | Majority (%) | Selection (%) | (pt) | SI (pt) | h (pt) | |
|---|---|---|---|---|---|---|
| TempCompass (Muse) | 1,574 | 79.2 | 87.2 | |||
| TempCompass (GPT-5.6 Sol) | 1,570 | 79.1 | 88.5 | |||
| MLVU-Test (Muse) | 496 | 73.2 | 74.0 | n.s. | ||
| MLVU-Test (GPT-5.6 Sol) | 502 | 72.9 | 70.9 | n.s. | ||
| VRBench (Muse) | 983 | 91.0 | 87.8 | |||
| VRBench (GPT-5.6 Sol) | 989 | 90.9 | 89.3 |
| Qwen-27B verifier | Muse verifier | |
| Base pool | (weak, std. ) | (strong, std. ) |
| Panel A: Selection Increment (SI selection standalone) | ||
| Qwen-base (split ) | ||
| Muse-base (split ) | ||
| Panel B: gain over self-consistency ( selection maj@8) | ||
| Qwen-base | ( vs. ) | ( vs. ) |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Marker mode ( Answer: X ) | Terse mode (bare letter) | ||||||
|---|---|---|---|---|---|---|---|
| Verifier–benchmark arm | acc. | acc. | A-lift | ||||
| MLVU-Test, | |||||||
| MLVU-Test, | |||||||
| MLVU-Test, | |||||||
| Video-Holmes, | |||||||
| Video-Holmes, | |||||||
| Base pool | Verifier | SI | |
|---|---|---|---|
| Qwen-base ( , maj ) | Qwen-27B | ||
| Qwen-base | Muse | ||
| Muse-base ( , maj ) | Qwen-27B | ||
| Muse-base | Muse |
| Verifier | Class | A-lift (pp) |
|---|---|---|
| Phi-3.5-Vision @16f | weak, cross-family | |
| Phi-3.5-Vision @48f | weak, cross-family | |
| Muse (holistic-48) | strong, cross-model | |
| InternVL3-8B @48f | mid, cross-family | |
| InternVL3-38B @48f | mid, cross-family | |
| Qwen3.5-27B (self-verifier) @48f | same-family (self) |
| Benchmark | Phi-3.5 (weak) | Qwen3.5-27B (self) | InternVL3-8B | InternVL3-38B |
|---|---|---|---|---|
| TempCompass | ||||
| MLVU-Test | ||||
| VRBench | — | — | ||
| EgoSchema | ||||
| Video-Holmes |
Video Reasoning
Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.
See all 220 papers