Long-video question answering forces multimodal large language models to reason under strict visual-token budgets, creating a tradeoff between broad temporal coverage and fine-grained spatial detail. Uniform sampling preserves coverage but often obscures decisive local cues, while caption-based or frame-similarity retrieval operates over lossy proxies that can miss motion, state changes, and before-after relations. We recast long-video reasoning as a visual-token allocation problem: given a fixed budget, decide how much to spend on global context versus high-resolution local evidence, and where in the video to draw that local evidence from. We introduce AdaAlloc, a training-free inference framework that adaptively allocates visual tokens between a global video overview and localized visual detail. AdaAlloc first plans a global/local allocation policy, then uses temporal grounding to locate candidate evidence segments and refines them into a compact temporal memory. The selected policy controls both frame sampling density and resolution, combining sparse, lower-resolution global frames with denser, higher-resolution local frames drawn from the refined memory. The final answerer reasons over the allocated global context and refined local evidence. Across LVBench, MLVU, VideoMME, and LongVideoBench, AdaAlloc improves over budget-matched uniform sampling by up to 10.5 percentage points and outperforms the compared frame-selection and agentic methods on their reported benchmarks.
Figures & tables
Figure 1: Visual token allocation for long-video reasoning. Uniform sampling preserves global coverage but loses fine details, while retrieval-based selection focuses on a few clips but can lose context. AdaAlloc allocates a fixed token budget between low-resolution global coverage and high-resolution local evidence for both context and fine-grained verification.
Figure 2: AdaAlloc pipeline. AdaAlloc splits a fixed visual-token budget B between low-res global context (blue) and high-res local evidence (orange). (1) From a sparse overview O , an MLLM planner picks a policy π and generates queries Q . (2) A temporal grounder (UniversalVTG) localizes Q ; the MLLM refines the candidates into M∗ , discarding an off-topic clip and the first return to the stove. (3) The MLLM answers from a global stream Eg and a local stream El drawn from M∗ .
Benchmark
#Video
#Question
Avg. (min.)
Range (min.)
Total Hours
Main Stressor
LVBench [ 54 ]
103
1,549
67.3
30.1–139.9
115.5
Hour-scale context
MLVU-Test [ 78 ]
349
502
17.8
3.1–136.8
103.4
Hard six-way QA
VideoMME [ 12 ]
900
2,700
17.0
0.18–59.7
255.3
Duration robustness
LongVideoBench-Val [ 60 ]
753
1,337
7.9
0.13–59.6
99.8
Long-context referring
Table 1: Summary of the long-form video QA benchmarks used in our evaluation.
Budget
Method
LVBench
MLVU-Test
VideoMME w/o sub
LongVideoBench-Val
16K tokens
Uniform sampling
42.7
55.4
67.1
62.8
AdaAlloc
51.7 (+9.0)
61.2 (+5.8)
68.4 (+1.3)
66.0 (+3.2)
32K tokens
Uniform sampling
46.1
61.4
69.5
63.9
AdaAlloc
56.6 (+10.5)
67.1 (+5.7)
70.7 (+1.2)
67.5 (+3.6)
Table 2: Controlled main results under matched visual-token budgets. Following the protocol in Section 5 , all methods use the same backbone, decoding setup, and visual-token budget within each block. We report accuracy (%) on four long-form video QA benchmarks in the video-only setting. Numbers in parentheses denote absolute gains over uniform sampling under the same budget.
Method
Visual Tokens
LongVideo Bench-Val
VideoMME w/o sub
Medium
Long
AKS [ 51 ]
≈ 19K †
53.0
54.4
51.1
DIG [ 24 ]
≈ 19K †
58.3
58.4
51.1
AdaAlloc
16K
66.0
68.7
57.2
AKS [ 51 ]
≈ 38K †
55.6
59.2
53.2
DIG [ 24 ]
≈ 38K †
61.2
61.6
53.8
Table 3: Comparison with recent frame-selection methods under the same backbone. All methods use Qwen3-VL-8B-Instruct. † Approximate budgets converted from the reported 128- and 256-frame settings at about 150 visual tokens per frame [ 24 ] ; these exceed AdaAlloc’s 16K and 32K budgets, making the budget favorable to the baselines. We report LongVideoBench and the medium/long subsets of VideoMME w/o sub to match DIG [ 24 ] .
Method
Size
Visual input
LVBench
MLVU-Test
VideoMME w/o sub
LongVideoBench-Val
Proprietary models
GPT-4o [ 20 ]
–
384 frames
48.9
54.9
71.9
66.7
Gemini-1.5-Pro [ 15 ]
–
256 frames
33.1
53.8
75.0
64.0
Open-source long-video MLLMs / trained or adapted video models
mPLUG-Owl3 [ 64 ]
7B
128 frames
43.5
–
59.3
52.1
NVILA [ 32 ]
8B
256 frames
–
–
64.2
57.7
Table 4: Contextual comparison with representative long-video understanding systems. Prior numbers are taken from the respective papers when available. We include methods that report results on comparable long-video benchmarks and specify a frame budget or sampling setting for similar-scale systems. Visual inputs are reported in their original units, since frame-to-token conversion depends on the backbone, resolution, and compression scheme. Because methods differ in backbone MLLM, training data, inference pipeline, visual preprocessing, and evaluation protocol, this table is intended as a cross-paper comparison rather than a strictly controlled head-to-head evaluation. Bold marks the best result excluding proprietary models.
Method
LVBench
MLVU-Test
VideoMME w/o sub
LongVideoBench-Val
Uniform sampling
46.1
61.4
69.5
63.9
Retrieval-only
48.2
59.4
67.6
63.5
Fixed policy
55.0
60.6
69.0
64.7
w/o temporal memory refinement
55.9
66.5
70.2
67.2
AdaAlloc
56.6
67.1
70.7
67.5
Table 5: Ablation study under the 32K visual-token budget. All variants use Qwen3-VL-8B-Instruct with the same final-answering visual-token budget. Uniform sampling answers from uniformly sampled frames. Retrieval-only answers from segments localized by UniversalVTG using the original question. Fixed policy uses a Balanced global/local split. W/o temporal memory refinement removes the refinement stage (Section 4.2 ).
Figure 3: Qualitative example on LVBench. The overview supplies global context, while the question’s request to read a handwritten number motivates a Local-heavy policy for detailed inspection. AdaAlloc retrieves multiple temporal candidates and refines them into a compact memory that preserves the decisive interval. AdaAlloc correctly answers 25 , while Uniform predicts 5 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Budget
Method
Holistic
Single-detail
Multi-detail
M-Avg
TR
AR
NQA
ER
PQA
AO
AC
16K tokens
Uniform sampling
91.3
76.0
76.9
63.4
75.0
62.9
40.3
69.4
AdaAlloc
92.4
78.0
83.1
67.0
80.3
68.3
52.9
74.6 (+5.2)
32K tokens
Uniform sampling
92.0
76.5
81.4
66.5
78.3
68.3
49.0
73.2
AdaAlloc
93.2
79.5
83.9
67.3
81.4
76.4
58.3
77.2 (+4.0)
Appendix
Table 6: MLVU-Dev results. Accuracy (%) on the seven multiple-choice tasks of MLVU-Dev, grouped as in MLVU [ 78 ] : Topic Reasoning (TR), Anomaly Recognition (AR), Needle QA (NQA), Ego Reasoning (ER), Plot QA (PQA), Action Order (AO), and Action Count (AC). M-Avg is the mean over the seven tasks. All methods use Qwen3-VL-8B-Instruct. Numbers in parentheses denote absolute gains over uniform sampling under the same budget.
Budget
VideoMME w/o sub
Short
Medium
Long
Overall
16K
79.4
68.7
57.2
68.4
32K
79.4
70.6
62.2
70.7
Appendix
Table 7: VideoMME duration-split breakdown. We report AdaAlloc performance on VideoMME without subtitles under 16K and 32K visual-token budgets. VideoMME groups videos into short, medium, and long duration buckets.
Sub-task
#Q
Uniform
AdaAlloc
Δ
Information Synopsis
323
79.6
82.4
+2.8
Spatial Reasoning
56
78.6
80.4
+1.8
Attribute Perception
222
82.0
81.1
− 0.9
Object Recognition
354
73.4
76.8
+3.4
Temporal Perception
55
85.5
80.0
− 5.5
OCR Problems
139
77.7
79.9
+2.2
Appendix
Table 8: VideoMME sub-task breakdown. Accuracy (%) of uniform sampling and AdaAlloc on VideoMME without subtitles under the same 32K visual-token budget, broken down by the official sub-task categories. #Q is the number of questions in each sub-task.
Method
Planning & Refinement
Visual Input
Adaptive Allocation
LVBench
MLVU-Test
VideoMME w/o sub
LongVideoBench-Val
Uniform sampling
×
Global only
×
42.7
55.4
67.1
62.8
Retrieval-only
×
Local only
×
46.5
58.8
65.9
62.5
Local-only
✓
Local only
×
48.9
58.0
66.1
64.8
Fixed policy
✓
Global + local
×
49.6
58.0
66.9
63.4
AdaAlloc
✓
Global + local
✓
51.7
61.2
68.4
66.0
Appendix
Table 9: Component ablation under the 16K visual-token budget. All variants use Qwen3-VL-8B-Instruct with the same final-answering visual-token budget and answerer. Planning & refinement denotes overview-conditioned planning with temporal-memory refinement; variants that use it share the same overview planner, grounding model, and refinement procedure.
Planner Overview
LVBench
MLVU-Test
VideoMME w/o sub
LongVideoBench-Val
Default
51.7
61.2
68.4
66.0
2× denser
51.6
61.4
68.9
65.8
Difference
− 0.1
+0.2
+0.5
− 0.2
Appendix
Table 10: Sensitivity to overview density. The final answering budget is fixed at 16K and only the planner’s visual input changes. The 2× denser overview samples twice as many frames at the same resolution and spends additional planning compute, so it is a diagnostic rather than a budget-matched alternative.
Budget
Allocation
LVBench
MLVU-Test
VideoMME w/o sub
LongVideoBench-Val
Avg.
16K
Direct ratio prediction
51.1
61.4
68.8
65.5
61.68
Policy selection (ours)
51.7
61.2
68.4
66.0
61.82
32K
Direct ratio prediction
54.9
67.1
70.7
67.0
64.94
Policy selection (ours)
56.6
67.1
70.7
67.5
65.51
Appendix
Table 11: Discrete policy selection versus direct ratio prediction. Both variants use the same planner, grounding model, token budget, and answering pipeline, and differ only in whether the planner selects one of the five predefined policies or directly predicts the global/local ratio.
Budget
Retriever
LVBench
MLVU-Test
VideoMME w/o sub
LongVideoBench-Val
Avg.
16K
Uniform sampling
42.7
55.4
67.1
62.8
56.99
Query–frame similarity
53.3
61.8
69.9
63.5
62.13
UniversalVTG
51.7
61.2
68.4
66.0
61.82
32K
Uniform sampling
46.1
61.4
69.5
63.9
60.21
Query–frame similarity
51.9
67.3
70.5
65.7
63.85
UniversalVTG
56.6
67.1
70.7
67.5
65.51
Appendix
Table 12: Sensitivity to the temporal retrieval module. AdaAlloc with UniversalVTG [ 2 ] (default) or query–frame similarity retrieval, with the planner, allocation policy, token budget, memory refinement, and final answerer unchanged.
Method
LVBench
MLVU-Test
VideoMME w/o sub
LongVideoBench-Val
Avg.
Uniform sampling
43.4
56.2
64.8
61.0
56.35
AdaAlloc
47.2 (+3.8)
58.4 (+2.2)
65.9 (+1.1)
62.9 (+1.9)
58.60 (+2.25)
Appendix
Table 13: Generalization to another MLLM backbone. Accuracy (%) with VideoLLaMA3-7B [ 70 ] under a matched 32K visual-token budget. Numbers in parentheses denote absolute gains over uniform sampling.
Figure 4: Distribution of selected allocation policies across benchmarks. We report the fraction of planner-routed examples assigned to each AdaAlloc policy on LVBench, MLVU, VideoMME, and LongVideoBench. Examples that bypass planning and use direct answering are excluded.
Question Family
Global-only
Global-heavy
Balanced
Local-heavy
Descriptive
4.4
13.5
33.1
49.0
Causal
0.2
7.5
43.8
48.6
Temporal
0.1
8.7
38.6
52.5
Appendix
Table 14: Selected policies by NExT-QA question family (%). NExT-QA validation split (4,996 questions) under the 16K budget. No question selected Local-only , so the column is omitted. Question families are benchmark metadata and are not available to the planner. Rows may not sum to 100 due to rounding.
Budget
Method
Compute (TFLOPs)
Runtime (s)
16K
Uniform sampling
468
14
DIG [ 24 ]
1,871
109
AdaAlloc
1,146
35
32K
Uniform sampling
1,261
25
DIG [ 24 ]
2,664
119
AdaAlloc
2,873
58
Appendix
Table 15: Inference cost under matched visual-token budgets. Total compute and end-to-end wall-clock runtime of uniform sampling, DIG [ 24 ] , and AdaAlloc, all with the Qwen3-VL-8B-Instruct backbone.