Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.
Figures & tables
Figure 1: Overview of LongVU-TTT. Video frames are encoded independently by a pretrained vision encoder to produce raw visual tokens. The TTT-Conv layer then processes all frames recurrently via continuously updated grouped convolutional fast weights, propagating temporal context across the sequence in chunks. The temporally enriched representations are passed through spatial pooling and importance-based frame sampling to produce a compact set of visual tokens. The existing MLP projector maps the selected features into the LLM embedding space, where they are concatenated with text embeddings and used to generate the response.
Model
Frame policy
MLVU
LongVideoBench
Video-MME
NExT-QA
LVBench
LongVA ( Zhang et al., 2024a )
128
56.3
-
52.6
-
-
InternVL2 ( InternVL Team, 2024 )
16
48.1
51.8
54.0
-
-
LongVILA ( Chen et al., 2024 )
256
-
57.1
60.1
80.7
-
VideoChat2 ( Li et al., 2024c )
-
54.6
36.0
-
-
-
LLaVA-OV ( Li et al., 2024a )
32
64.7
56.3
58.2
79.4
-
Qwen2-VL ( Wang et al., 2024a )
256
64.2
55.6
63.3
-
-
Table 1: Performance comparison of LongVU-TTT with existing video MLLMs on long and medium video understanding benchmarks. Frame policy reports either the fixed frame count or the temporal sampling policy used by each model. LongVU-TTT processes up to 512 frames through TTT-Conv before selecting 128 frames for the LLM. Results from prior work are shown at their reported precision; our results use two decimals. Bold denotes the best result and underline denotes the second best, including ties.
Method
MLVU
LongVideoBench
Video-MME
LLaVA-Video baseline
60.35
52.54
58.20
Bidirectional Mamba2
62.46
53.55
60.80
Gated DeltaNet
62.92
53.96
61.31
TTT-MLP
63.38
54.23
61.14
TTT-MLP (causal)
64.38
55.16
61.73
Temporal Transformer
63.94
54.84
61.67
Table 2: Controlled temporal-operator study using the same fixed 30% subset of the Stage 2 training mixture. All variants use the same base model, insertion point, feature width, frame input, LLM token budget, and training setup. (a) compares projector-stage temporal operators; TTT variants use chunks of 8 frames. (b) isolates fast-weight adaptation, cross-chunk state carry, and 2D locality.
Model
#Frames
Selection
MLVU
LongVideoBench
LLaVA-Video (64 frames)
64
Uniform
70.80
58.20
LLaVA-Video (512 frames)
128
Uniform
65.40
55.14
LongVU-TTT (512 frames)
64
Uniform
68.42
58.46
LongVU-TTT (512 frames)
128
Hybrid
71.80
60.40
LongVU-TTT (512 frames)
256
Uniform
70.80
59.42
LongVU-TTT (128 frames)
128
All
71.50
61.02
Table 3: Temporal contextualization and frame selection. (a) separates the number of frames processed by the temporal module from the visual budget delivered to the LLM. (b) compares selection policies with 512 input frames and a fixed 128-frame LLM budget; all variants use the same TTT-Conv checkpoint.
Method
0–25%
25–50%
50–75%
75–100%
No temporal layer
53.18
50.81
48.95
48.31
TTT-Conv, reset every 8 frames
54.37
51.22
49.10
48.42
TTT-Conv, state carried
56.42
53.07
49.84
48.66
TTT-Conv + explicit evidence (upper bound)
61.35
58.68
55.29
53.11
Table 4: Evidence-withholding diagnostic on the LongVideoBench temporal subset. Source evidence is removed from the explicit LLM input for the first three rows; the final row retains it as an explicit-evidence upper bound. Columns denote normalized temporal-gap bins between the evidence and question timestamp. Bold denotes the best result among the evidence-withheld methods.
Figure 2: System optimization results measured on A100 40GB GPUs with NVLink intra-node and InfiniBand inter-node connections. (a) Maximum trainable frame count for standard gradient checkpointing versus asynchronous CPU activation offloading on a fixed 8-GPU setup. (b) Training step latency for batch-tiled ViT processing versus the standard per-shard loop from 8 to 64 GPUs at 256 frames with 8 tiles. (c) Throughput scaling efficiency under the same 256-frame, 8-tile setting relative to ideal linear scaling.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
#Frames
MLVU
LongVideoBench
128
71.50
61.02
256
71.72
60.86
512
71.80
60.40
Appendix
Table 5: Input-length sweep at a fixed 128-frame LLM budget.
#Frames
TTT-Conv
TTT-MLP
Gated DeltaNet
Temporal Transformer
64
123.97
26.98
13.83
217.85
128
247.99
45.82
27.37
862.42
256
496.60
91.76
54.23
3448.04
512
993.15
184.60
108.89
13704.77
1024
1987.31
369.38
219.25
54921.84
Appendix
Table 6: Temporal-module inference latency (ms) on one A100 40GB GPU.
Figure 3: Importance score components across 128 frames of a representative video. From top to bottom: gradient-direction shift Gt , alignment-loss delta Rt , and combined importance It from Equation 9 . Star markers indicate chunk boundaries, and shaded regions indicate selected frames.
Figure 4: Importance score components Gt , Rt , and It for a lecture video from LongVideoBench ( Wu et al., 2024 ) . Star markers indicate chunk boundaries and shaded regions indicate selected frames.