Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.
Figures & tables
Figure 1: Overview of LongVU-TTT. Video frames are encoded independently by a pretrained vision encoder to produce raw visual tokens. The TTT-Conv layer then processes all frames recurrently via continuously updated grouped convolutional fast weights, propagating temporal context across the sequence in chunks. The temporally enriched representations are passed through spatial pooling and importance-based frame sampling to produce a compact set of visual tokens. The existing MLP projector maps the selected features into the LLM embedding space, where they are concatenated with text embeddings and used to generate the response.
Model
Frame policy
MLVU
LongVideoBench
Video-MME
NExT-QA
LVBench
LongVA ( Zhang et al., 2024a )
128
56.3
-
52.6
-
-
InternVL2 ( InternVL Team, 2024 )
16
48.1
51.8
54.0
-
-
LongVILA ( Chen et al., 2024 )
256
-
57.1
60.1
80.7
-
VideoChat2 ( Li et al., 2024c )
-
54.6
36.0
-
-
-
LLaVA-OV ( Li et al., 2024a )
32
64.7
56.3
58.2
79.4
-
Qwen2-VL ( Wang et al., 2024a )
256
64.2
55.6
63.3
-
-
Table 1: Performance comparison of LongVU-TTT with existing video MLLMs on long and medium video understanding benchmarks. Frame policy reports either the fixed frame count or the temporal sampling policy used by each model. LongVU-TTT processes up to 512 frames through TTT-Conv before selecting 128 frames for the LLM. Results from prior work are shown at their reported precision; our results use two decimals. Bold denotes the best result and underline denotes the second best, including ties.
Method
MLVU
LongVideoBench
Video-MME
LLaVA-Video baseline
60.35
52.54
58.20
Bidirectional Mamba2
62.46
53.55
60.80
Gated DeltaNet
62.92
53.96
61.31
TTT-MLP
63.38
54.23
61.14
TTT-MLP (causal)
64.38
55.16
61.73
Temporal Transformer
63.94
54.84
61.67
Table 2: Controlled temporal-operator study using the same fixed 30% subset of the Stage 2 training mixture. All variants use the same base model, insertion point, feature width, frame input, LLM token budget, and training setup. (a) compares projector-stage temporal operators; TTT variants use chunks of 8 frames. (b) isolates fast-weight adaptation, cross-chunk state carry, and 2D locality.
Model
#Frames
Selection
MLVU
LongVideoBench
LLaVA-Video (64 frames)
64
Uniform
70.80
58.20
LLaVA-Video (512 frames)
128
Uniform
65.40
55.14
LongVU-TTT (512 frames)
64
Uniform
68.42
58.46
LongVU-TTT (512 frames)
128
Hybrid
71.80
60.40
LongVU-TTT (512 frames)
256
Uniform
70.80
59.42
LongVU-TTT (128 frames)
128
All
71.50
61.02
Table 3: Temporal contextualization and frame selection. (a) separates the number of frames processed by the temporal module from the visual budget delivered to the LLM. (b) compares selection policies with 512 input frames and a fixed 128-frame LLM budget; all variants use the same TTT-Conv checkpoint.
Method
0–25%
25–50%
50–75%
75–100%
No temporal layer
53.18
50.81
48.95
48.31
TTT-Conv, reset every 8 frames
54.37
51.22
49.10
48.42
TTT-Conv, state carried
56.42
53.07
49.84
48.66
TTT-Conv + explicit evidence (upper bound)
61.35
58.68
55.29
53.11
Table 4: Evidence-withholding diagnostic on the LongVideoBench temporal subset. Source evidence is removed from the explicit LLM input for the first three rows; the final row retains it as an explicit-evidence upper bound. Columns denote normalized temporal-gap bins between the evidence and question timestamp. Bold denotes the best result among the evidence-withheld methods.
Figure 2: System optimization results measured on A100 40GB GPUs with NVLink intra-node and InfiniBand inter-node connections. (a) Maximum trainable frame count for standard gradient checkpointing versus asynchronous CPU activation offloading on a fixed 8-GPU setup. (b) Training step latency for batch-tiled ViT processing versus the standard per-shard loop from 8 to 64 GPUs at 256 frames with 8 tiles. (c) Throughput scaling efficiency under the same 256-frame, 8-tile setting relative to ideal linear scaling.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
#Frames
MLVU
LongVideoBench
128
71.50
61.02
256
71.72
60.86
512
71.80
60.40
Appendix
Table 5: Input-length sweep at a fixed 128-frame LLM budget.
#Frames
TTT-Conv
TTT-MLP
Gated DeltaNet
Temporal Transformer
64
123.97
26.98
13.83
217.85
128
247.99
45.82
27.37
862.42
256
496.60
91.76
54.23
3448.04
512
993.15
184.60
108.89
13704.77
1024
1987.31
369.38
219.25
54921.84
Appendix
Table 6: Temporal-module inference latency (ms) on one A100 40GB GPU.
Figure 3: Importance score components across 128 frames of a representative video. From top to bottom: gradient-direction shift Gt , alignment-loss delta Rt , and combined importance It from Equation 9 . Star markers indicate chunk boundaries, and shaded regions indicate selected frames.
Figure 4: Importance score components Gt , Rt , and It for a lecture video from LongVideoBench ( Wu et al., 2024 ) . Star markers indicate chunk boundaries and shaded regions indicate selected frames.
Long-video understanding remains challenging for Multimodal Large Language Models (MLLMs) due to limited context length. Uniform sampling may miss crucial moments, while agent-based frame video understanding methods often evaluate frames independently, overlooking the temporal organization of videos. Ideally, evidence selection should mimic how humans answer questions about long videos: first locating the relevant segment from the global context, then zooming into local objects and details. We propose Temporal Tree of Thought T^3, a training-free framework for adaptive coarse-to-fine long-video understanding. T^3 constructs a question-agnostic hierarchical temporal tree via recursive temporally constrained clustering, where each node represents a contiguous segment with an informative key frame. During inference, T^3 performs an answer-retrieve-explore loop: it reasons over coarse representative frames, generates a search statement when evidence is insufficient, and expands relevant branches for finer-grained evidence. This process adaptively shifts the search target from temporal regions to specific objects and visual details to help video understanding. Experiments on VideoMME, LongVideoBench, and LVBench show that T^3 improves Qwen2.5-VL-7B by 0.5%, 4.6%, and 4.4%, respectively, under the same frame budget, demonstrating the effectiveness of structured temporal reasoning.
Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: https://haozheqi.github.io/adapt-token
Long-form video understanding remains challenging for Video Large Language Models (VideoLLMs), as the dense frame sampling introduces massive visual tokens while sparse sampling risks missing critical temporal evidence and leading to LLM hallucination. Existing training-free token reduction methods either treat videos equally as static images or rely on segment-level merging heuristics, which weaken fine-grained spatiotemporal modeling and introduce additional overhead. In this paper, we propose EchoPrune, a lightweight and training-free token pruning method that improves temporal resolution under a fixed LLM-side visual token budget. Our core idea is to interpret redundant video tokens as temporal echoes: if a token is well reconstructed from the previous frame, it is merely a temporally redundant echo; otherwise, it may capture new events, motion, or query-relevant visual evidence. Based on this insight, EchoPrune scores visual tokens by (i) query-guided crossmodal relevance and (ii) temporal reconstruction error, measured by correspondence matching and echo matching across consecutive frames. The selected tokens preserve task-relevant cues and temporal novelty while suppressing predictable redundancy, allowing VideoLLMs to observe more frames without increasing the decoding budget. Extensive experiments on LLaVA-OV, Qwen2.5VL, and Qwen3VL across six video understanding benchmarks show that EchoPrune enables VideoLLMs to process up to 20x frames under the same token budget, yielding improved performance (+8.6%) and inference speedup (5.6x for prefilling) on Qwen2.5VL-7B.
Jiameng Li, Minye Wu, Jiezhang Cao +2
KU Leuven · Shanghai Jiaotong University · Weill Cornell Medicine