Where Do Reported Long-Video QA Gains Come From? A Controlled Audit of Frame Selection, Evidence Injection, and Expert Routing
Organizations: Independent Researcher
Abstract
Long-video question answering is usually studied as frame selection: which few frames should a vision-language model see? We introduce VideoRouteAudit, a controlled audit that separates this choice from two alternatives, injecting non-visual evidence and routing to another answering model. It holds the backbone, frame budget, decoding, prompt and grader fixed while varying one decision at a time. On the full Video-MME long bucket (n=900, K=8), no tested selector significantly improves over uniform sampling. The largest selection-only estimate is +1.9 pt (95% CI [-1.2,+5.0]) and our cascade is +1.2 pt [-1.7,+4.1]; all are equivalent to uniform within +/-5 pt. The same contrasts are unstable at the n=200 subset scale commonly used: three of four are up to 3.8x larger, one reverses sign, and resampled 200-item subsets reverse the full-bucket effect's sign in 35.8% of draws. The audit also corrects our own first results. The anomaly-domain "+ASR" arms injected no text, so the pooled +4.7 pt gain was selection plus a development prefix. With transcripts, ASR lowers XD-Violence accuracy by 10.0 pt when appended after the question and not at all when placed before it, and lowers UCF-Crime by 13.0 pt either way; on Video-MME, placing it first adds about 4 pt (7 on Qwen2-VL) to uniform sampling and to the cascade alike (post hoc, unadjusted). Swapping the answering model gives the largest contrast (+15.1 pt on MUSIC-AVQA, +33.3 pt on its worst type), but the same audio-visual expert with its soundtrack silenced keeps 92% of the gain (69% with a mismatched soundtrack), so the contrast is mostly a model swap. Code, per-item predictions and scripts that recompute every statistic are available at https://github.com/mikeslin-ir/long-video-qa-audit.
Figures & tables
| Method ( , ) | Acc. (%) | 95% CI | Wall s/item | ||
|---|---|---|---|---|---|
| Uniform- | 46.2 | — | — | — | 10.0 |
| Midpoint uniform- | 46.2 | 1.000 | — | ||
| AKS [ 38 ] (port) | 48.1 | 0.273 | 22.3 | ||
| KeyScore [ 22 ] (port) | 47.1 | 0.624 | 22.4 | ||
| VideoTree [ 42 ] (port) | 47.8 | 0.360 | 22.3 | ||
| Cascade (selection only) | 47.4 | 0.452 | 13.3 |
| Method ( , ) | Video-MME long | XD F/N | UCF-Crime |
|---|---|---|---|
| Uniform- (no inject) | 50.0 | 87.0 | 81.5 |
| Cascade (no inject) | 53.5 | 91.0 | 85.0 |
| Cascade ASR, as published | 56.5 | 91.0 † | 85.0 † |
| Cascade ASR, after the question | 56.5 | 81.0 | 72.0 |
| Cascade ASR, before the question | 57.0 | 91.0 | 72.0 |
| Cascade always-inject OCR | 55.5 | 89.5 | 83.0 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm | Pool | Acc. (%) | ||
|---|---|---|---|---|
| Uniform- | — | 45.4 | — | — |
| Cascade | 132 | 44.8 | 0.685 | |
| Cascade ASR ∗ (original) | 198 | 48.0 | 0.107 | |
| Cascade ASR (pool-matched) | 132 | 44.6 | 0.614 |
| Stage | ms/item | Notes |
| Plan (rule-based) | 0.1 | keyword match |
| Propose (EARS, uniform) | 9 | cached proposals |
| Decode candidate frames | 3,058 | frames, CPU |
| Filter (motion diff.) | 15 | |
| Align (SigLIP) | 18 | cached embeddings ∗ |
| Select (facility loc.) | 91 |
| audit (published runs) | stacfp (released default) | |
| Planned proposers | shot, EARS, uniform | STACFP, EARS, uniform |
| Executed | EARS, uniform | STACFP, EARS, uniform |
| Pool (long video) | ( ) | ( ) |
| Filter | motion-difference | redundancy |
| Alignment | SigLIP (+ keyword aligners) | same |
| Select | facility-location + uniform | same |
| Tool | Stage | Behaviour |
|---|---|---|
| uniform_stride | Propose | evenly spaced frames, endpoints included |
| ears_event | Propose | audio-energy optical-flow event attention; top peaks |
| shot_boundary | Propose | PySceneDetect ContentDetector (threshold ); inactive in audited runs |
| stacfp | Propose | colour-histogram clustering, cut-density adaptive |
| motion_diff | Filter | keep the frames with the largest grayscale change from their predecessor |
| redundancy | Filter | drop near-duplicate frames by histogram similarity |
| Selector ( ) | Acc. (%) | Wall s/item | vs. ours | |
|---|---|---|---|---|
| Random- | 200 | 48.5 | 10.0 | 0.154 |
| Uniform- | 200 | 50.0 | 10.0 | 0.324 |
| Motion (motion-diff only) | 200 | 46.5 | 20.1 | 0.049 |
| CLIP (SigLIP only) | 200 | 49.5 | 20.9 | 0.312 |
| Forced multi-cue (Obj ASR SigLIP) | 200 | 50.5 | 76.9 | 0.327 |
| Cascade (anchored relevance) | 200 | 53.5 | 14.3 | — |
| Domain | Arm | Acc. | vs. cascade | , | |
|---|---|---|---|---|---|
| XD F/N | Cascade, no text | 200 | 91.0 | — | — |
| ASR, as published † | 200 | 91.0 | , | ||
| ASR, after the question | 200 | 81.0 | , | ||
| ASR, before the question | 200 | 91.0 | , | ||
| UCF-Crime | Cascade, no text | 200 | 85.0 | — | — |
| ASR, as published † | 200 | 85.0 | , |
| Position of the ASR block | Acc. | ||
|---|---|---|---|
| No text | 46.2 | — | — |
| After the answer instruction (audited) | 47.8 | 0.15 | |
| Between options and instruction | 48.1 | 0.082 | |
| Between question and options | 47.2 | 0.57 | |
| Before the question | 50.7 | 0.002 |
| 1 | 2 | 4 | 6 | 8 | |
| Uniform- | 38.0 | 43.0 | 49.0 | 52.5 | 50.0 |
| Cascade | 48.0 | 47.0 | 50.0 | 51.0 | 53.5 |
| 0.009 | 0.302 | 0.875 | 0.728 | 0.324 |
| Source | Contrast | Gain | ||
|---|---|---|---|---|
| AKS [ 38 ] T1 | vs. uniform, Qwen2-VL-7B | 2700 | 32 | |
| AKS [ 38 ] T1 | vs. uniform, LLaVA-OV-7B | 2700 | 32 | |
| AKS [ 38 ] T2 | ADA vs. UNI, LLaVA-Video-7B | 2700 | 64 | |
| VideoTree [ 42 ] T2 | vs. LLoVi, Video-MME long | 900 | adapt. | |
| VideoTree [ 42 ] T11 | vs. uniform, EgoSchema sub. | 500 | adapt. | |
| VideoTree [ 42 ] T5 | depth-expansion component | 500 | adapt. |
| Contrast ( ) | 95% CI | Pow. | Equiv. | |||
| F1 — selection-only is null (equivalence wanted) | ||||||
| Motion vs uniform | 200 | 0.349 | 0.19 | – | ||
| Clip vs uniform | 200 | 1.000 | 0.05 | – | ||
| Forced-multicue vs unif. | 200 | 1.000 | 0.05 | – | ||
| Cascade vs uniform | 200 | 0.324 | 0.21 | n/a | ||
| F3(a) — injection effect (effect wanted) | ||||||
| Contrast ( ) | MDE | ||||
| Selection axis | |||||
| Motion vs uniform | 200 | 17 | 24 | 20.5% | 9.0 |
| CLIP vs uniform | 200 | 27 | 28 | 27.5% | 10.4 |
| Forced-multicue vs unif. | 200 | 18 | 17 | 17.5% | 8.3 |
| Cascade vs uniform | 200 | 22 | 15 | 18.5% | 8.5 |
| Cascade vs uniform | 900 | 94 | 83 | 19.7% | 4.1 |
| Cascade | Cascade ASR | |
| vs unif. | vs unif. | |
| Full-bucket ( ) | ||
| Paper’s prefix ( ) | ||
| Median over -subsets | ||
| range | ||
| Standard deviation |
| Method | Train-free | Reasoner | Frames | Scoring |
|---|---|---|---|---|
| TSPM [ 16 ] (AVQA) | specialist | 60 | closed/exact | |
| VideoTree [ 42 ] | ✓ | GPT-4 | dense | open |
| AKS [ 38 ] | ✓ | 7B | 8 | open |
| VideoRouteAudit (ours) | ✓ | 7B | 8 | open judge |
| Method | Acc. (%) | CI | ||
|---|---|---|---|---|
| Uniform- | 67.8 | — | — | — |
| Cascade ‡ | 66.2 | 0.350 | ||
| AKS (port) | 65.2 | 0.182 | ||
| KeyScore (port) | 63.8 | 0.049 | ||
| VideoTree (port) ‡ | 67.6 | 1.000 |
| Question type | Audio-blind | AV expert | |||
| Visual-Counting | 1197 | 83.0 | 89.1 | ||
| Visual-Location | 1225 | 65.7 | 93.1 | ||
| Audio-Counting | 1017 | 85.6 | 88.1 | ||
| Audio-Comparative | 594 | 52.2 | 69.0 | ||
| AV-Existential | 988 | 81.1 | 83.8 | ||
| AV-Counting | 1265 | 66.7 | 81.0 |
| Question type | Blind | Silenced | AV | AV blind | AV silenced | |
|---|---|---|---|---|---|---|
| Visual-Counting | 200 | 83.5 | 90.0 | 89.5 | ||
| Visual-Location | 200 | 61.0 | 95.0 | 95.0 | ||
| Audio-Counting | 200 | 83.5 | 85.5 | 86.0 | ||
| Audio-Comparative | 200 | 56.0 | 62.5 | 69.0 | ||
| AV-Existential | 200 | 83.0 | 85.5 | 87.0 | ||
| AV-Counting | 200 | 67.0 | 83.0 | 83.0 |
Video Question Answering
Video question answering studies how AI systems use visual and audio evidence over time to answer questions about video.
See all 197 papers