FrameRouter: Frame Budget Routing for Long-Video Understanding on Video-MME-v2 and Beyond
Authors: Bingjun Luo, Jialin Guo, Siqi Li
Organizations: Tsinghua University, Beijing, China · Harbin Engineering University, Harbin, China · BNRist, THUIBCS, School of Software, Tsinghua University, Beijing, China
Long-video question answering benchmarks like Video-MME-v2 typically evaluate systems under a uniform frame budget for every question, despite stark variation in the evidence each query actually demands. We analyze this mismatch on Video-MME-v2 and find that different question subsets prefer different frozen frame-budget policies, while uniformly increasing the frame count does not reliably improve all metrics. These findings motivate fixed-mean frame-budget routing, a controlled inference setting in which each question may receive a different frame budget while the average budget matches a uniform reference. We introduce FrameRouter, a training-free test-time framework that combines an evidence-demand router, a budget-constrained allocation rule, and a frozen bank of frame-sampling policies. By controlling how much visual evidence each question receives while leaving the underlying samplers unchanged, FrameRouter is orthogonal to query-aware frame selection. Across Video-MME-v2 and several long-video understanding benchmarks, FrameRouter improves long-video question answering by using the same average visual budget more selectively across questions. Code is available at https://github.com/bingjunluo/FrameRouter.
Figures & tables
Figure 1: Overview of fixed-mean frame-budget routing. Uniform inference gives every question the same visual budget despite heterogeneous evidence demands. FrameRouter routes each question to an appropriate frozen budget policy using question-only signals, while preserving the same dataset-level average budget.
Figure 2: Question-level budget heterogeneity in the frozen Think with-subtitles policy bank. Left: among errors made by the mid policy, 29.8% are answered correctly by at least one neighboring budget. Right: pairwise correction matrices split by group_type and level ; rows denote the policy that is correct and columns denote the policy that is wrong.
Model
MVBench
VideoMMMU
L.V.Bench (w. sub)
L.V.Bench (wo sub)
Frames
Acc.
Frames
Acc.
Frames
Acc.
Frames
Acc.
Commercialization Models
Gemini-3-Pro
-
74.1
-
87.6
-
77.7
-
-
GPT-5
50
74.1
50
84.6
50
72.6
-
-
Kimi-K2.5
-
73.5
-
86.6
-
79.8
-
-
Open-Source Models (Thinking)
Table 2: Comparison across long-video and video understanding benchmarks in %. Missing or unreported entries are denoted by “-”.
Method
plow
phigh
Non-Lin
Consistency
Coherence
Avg. Acc.
Baseline
×
×
30.60
29.50
32.70
48.90
High-only
×
✓
33.94
33.10
35.47
53.03
Low-only
✓
×
34.64
32.86
37.91
52.66
Ours
✓
✓
34.86
33.20
37.93
53.34
Table 3: Score-channel ablation on Video-MME-v2 with the Think backbone and subtitles. All routed variants use the same route quota and mean frame budget; in single-channel variants, the enabled score supplies the ranking signal for both quota-constrained selections.
Figure 3: Allocator diagnostics on Video-MME-v2 Think with subtitles. Left: quota sensitivity under the matched mean budget. Right: realized route shares within each official group × level split.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Pairwise correction matrices split by group_type and level in the Instruct setting. We show the labeled L1–L3 splits for logic and relevance questions; Video-MME-v2 logic questions with empty level fields are excluded from this diagnostic plot. Each mini-matrix is normalized within its split; rows denote the policy that is correct and columns denote the policy that is wrong. The off-diagonal mass varies across question groups and difficulty levels, supporting question-conditioned budget routing rather than a single monotone frame-scaling rule.
Figure 5: Parser evidence-anchor profiles within each route on Video-MME-v2 Think with subtitles. Routes are decided from the current question only, prior to answer generation. Evidence-anchor labels are parser outputs and are used only to interpret routing behavior.
Multimodal Large Language Models (MLLMs) have made strong progress in video understanding, yet long videos remain difficult: the visual token budget grows with video length, so temporally sparse evidence is easily lost. Existing methods compress the input through uniform sampling or frame selection, but these strategies optimize different objectives, either broad temporal coverage or local question relevance, and neither preserves both global storyline context and fine-grained evidence. We propose VideoRouter (VR), which rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. VideoRouter first organizes each video into a question-agnostic temporal hierarchy that partitions it into coarse-to-fine temporally coherent segments. Upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router that judges which view is better supported by its own selected evidence and decides the final answer. Across six backbones, routing improves over both views in all settings, and the choice of view is shown to be dataset-dependent, confirming that no single evidence granularity is universally preferable. On VideoMME, our method outperforms state-of-the-art frame selection methods by 2.5 points, under the LLaVA-Video-7B backbone. We will release the code.
Long-video question answering is usually studied as frame selection: which few frames should a vision-language model see? We introduce VideoRouteAudit, a controlled audit that separates this choice from two alternatives, injecting non-visual evidence and routing to another answering model. It holds the backbone, frame budget, decoding, prompt and grader fixed while varying one decision at a time. On the full Video-MME long bucket (n=900, K=8), no tested selector significantly improves over uniform sampling. The largest selection-only estimate is +1.9 pt (95% CI [-1.2,+5.0]) and our cascade is +1.2 pt [-1.7,+4.1]; all are equivalent to uniform within +/-5 pt. The same contrasts are unstable at the n=200 subset scale commonly used: three of four are up to 3.8x larger, one reverses sign, and resampled 200-item subsets reverse the full-bucket effect's sign in 35.8% of draws. The audit also corrects our own first results. The anomaly-domain "+ASR" arms injected no text, so the pooled +4.7 pt gain was selection plus a development prefix. With transcripts, ASR lowers XD-Violence accuracy by 10.0 pt when appended after the question and not at all when placed before it, and lowers UCF-Crime by 13.0 pt either way; on Video-MME, placing it first adds about 4 pt (7 on Qwen2-VL) to uniform sampling and to the cascade alike (post hoc, unadjusted). Swapping the answering model gives the largest contrast (+15.1 pt on MUSIC-AVQA, +33.3 pt on its worst type), but the same audio-visual expert with its soundtrack silenced keeps 92% of the gain (69% with a mismatched soundtrack), so the contrast is mostly a model swap. Code, per-item predictions and scripts that recompute every statistic are available at https://github.com/mikeslin-ir/long-video-qa-audit.
Recent multimodal large language models (MLLMs) have substantially advanced video understanding, yet long-form video QA remains challenging under fixed input token budgets, where uniform sampling can be inefficient for evidence localization. We propose ReQuest , an uncertainty-driven, question-adaptive keyframe selection pipeline that aligns question intent with relevant video content through selective computation. ReQuest integrates (i) a lightweight question-aware selector distilled from MLLM-generated supervision, (ii) Re-thinking Routing that triggers additional inference only when the model is uncertain with a length-adaptive criterion, and (iii) uncertainty-guided adaptive non-maximum suppression that selects temporally diverse frames while adjusting spacing based on question difficulty. As a plug-andplay method, ReQuest improves long-video QA without modifying or fine-tuning the underlying MLLM. Experiments on Video-MME, MLVU, and LongVideoBench demonstrate consistent accuracy gains with competitive computational cost, with particularly strong improvements in medium and long video regimes.
Minkuk Kim, Suyong Yun, Young Tae Kim +3
Kyung Hee University, Republic of Korea · Electronics and Telecommunications Research Institute (ETRI), Republic of Korea