Hour-long videos, streaming feeds, and movie-length content are pushing vision-language models (VLMs) past their breaking point -- the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length, causing prohibitive memory consumption and inference latency. Existing approaches, such as token reduction and keyframe selection, shorten the token sequence but leave the O(n) scaling intact, failing to resolve the fundamental bottleneck. We propose FramePack-U, a strikingly simple, training-free frame packing framework that compresses arbitrarily long videos into a fixed size visual representation at constant memory cost. FramePack-U requires no fine-tuning or architectural modification -- it plugs directly into off-the-shelf VLMs at inference time. Inspired by FramePack from the generation literature, we propose a content-aware compression strategy for visual understanding under a fixed token budget. Specifically, FramePack-U (1) scores each frame by its semantic similarity to the text prompt and ranks frames in a diversity-driven manner, and (2) packs the selected frames at varying resolutions into a fixed-length context window. This concentrates the memory budget on frames that are both instruction-relevant and visually diverse. While FramePack-U can in theory pack arbitrarily many frames into the fixed budget, we identify an empirical instantiation that generalizes robustly across settings. Extensive experiments on LongVideoBench, MLVU, and Video-MME show that FramePack-U consistently outperforms full-resolution baselines and prior token reduction and keyframe selection methods across diverse model families, parameter scales, and frame budgets. Notably, on Qwen3.5-35B-A3B with a 128-frame budget, FramePack-U yields gains of +12.9%, +10.6%, and +9.7% over the official model on the three benchmarks, respectively.
Figures & tables
Figure 1: Overview of FramePack-U . Given a long video and a text query, FramePack-U selects query-relevant and visually diverse frames, then packs them with heterogeneous resolutions under a fixed memory budget. Higher-priority frames are kept at higher resolution, while lower-priority frames are progressively compressed. The packed frames are fed into the VLM for answer generation.
Method
LVB
MLVU
V-MME
Proprietary models
GPT-5 [ 17 ]
72.6
77.7
83.3
GPT-5 mini [ 17 ]
69.7
69.1
77.3
Gemini 3 Pro [ 10 ]
75.9
75.7
88.6
Gemini 2.5 Flash [ 9 ]
73.1
75.1
84.2
Claude Sonnet 4.5 [ 1 ]
65.1
64.0
74.2
Table 1: Main results on Qwen3.6-27B ( left ) and Qwen3.5-35B-A3B ( right ) across different memory budgets. Subscripts indicate the gap to Vanilla at the same budget; bold and underline denote the best and second-best results among open-weight models, respectively. We find that: (1) FramePack-U outperforms all baselines substantially, (2) our top results are competitive with, and often surpasses, those of much larger proprietary VLMs.
Table 3
Memory
Scheme
Strategy
LongVideoBench
MLVU
Video-MME
512 frames
–
vanilla
71.8
78.0
74.4
d1f128d2f1536
FramePack-U
74.9
82.1
76.3
− Relevance
74.0 (-0.9)
82.3 (+0.2)
76.3 (+0.0)
− Diversity
73.9 (-1.0)
80.4 (-1.7)
69.8 (-6.5)
d1f256d2f1024
FramePack-U
74.7
82.2
75.9
− Relevance
74.6 (-0.1)
82.3 (+0.1)
76.0 (+0.1)
Table 4: Ablation on prompt relevance and frame diversity using Qwen3.6-27B . − Relevance removes prompt relevance and uses a prompt-agnostic diversity-based ordering, while − Diversity ranks frames only by query-frame similarity. FramePack-U combines prompt relevance and visual diversity. Values in parentheses denote the difference between each ablated variant and FramePack-U . See Appendix B for ablations on 128- and 256-frame budgets.
Figure 3: Qualitative comparison. Relevance-only sampling selects frames based only on query-frame similarity and often produces redundant frames, while uniform sampling preserves temporal coverage but may miss question-relevant evidence. Our strategy combines prompt relevance and visual diversity, allowing it to select frames that better cover the video while retaining key evidence for answering the question. Red boxes highlight frames that contain important visual evidence.
Figure 4: t-SNE visualization. We compare the feature distributions of frames selected by FramePack-U ( n=848 , scheme d1f16d2f320d4f512 ) against Uniform Sampling ( n=128 , full resolution) across three distinct queries. Circles mark representative frames identified by each strategy, whose corresponding visual frames are displayed in the callout thumbnails. Points are colored according to their downsampling tiers ( d=1,2,4 ).
Figure 5: Memory overhead of FramePack-U on Qwen3.5-9B. We evaluate the preprocess, VLM inference and overall memory overhead of both vanilla models and FramePack-U on LongVideoBench. Results of different video duration tiers are reported. More results can be found at Table 5 .
Figure 6: Latency evaluation of FramePack-U on Qwen3.5-9B. We evaluate the preprocess, VLM inference and overall latency of both vanilla models and FramePack-U on LongVideoBench. Results of different video duration groups are reported. More results can be found at Table 6 .
Stage
Model
15s
60s
600s
3600s
Avg.
Preprocess
Qwen3.5-4B
–
–
–
–
–
+ FramePack-U
0.779
1.117
1.576
1.602
1.288
Qwen3.5-9B
–
–
–
–
–
+ FramePack-U
0.777
1.115
1.574
1.600
1.286
Qwen3.6-27B
–
–
–
–
–
+ FramePack-U
0.775
1.115
1.575
1.600
1.307
Table 5: Memory overhead (GiB) across different model sizes. We evaluate the memory overhead of three inference stages on different video duration tiers of LongVideoBench. Preprocess stage represents the introduced CLIP-based frame ranking process. VLM Inference stage denotes the inference process of the off-the-shelf VLM. Overall results reflect the peak memory used during the end-to-end process.
Stage
Model
15s
60s
600s
3600s
Avg.
Preprocess
Qwen3.5-4B
–
–
–
–
–
+ FramePack-U
0.183
0.518
3.460
4.729
2.354
Qwen3.5-9B
–
–
–
–
–
+ FramePack-U
0.181
0.502
3.383
4.655
2.309
Qwen3.6-27B
–
–
–
–
–
+ FramePack-U
0.169
0.480
3.210
4.366
2.178
Table 6: Inference latency (s) across different model sizes. We evaluate the memory overhead of three inference stages on different video duration tiers of LongVideoBench. Preprocess stage represents the introduced CLIP-based frame ranking process. VLM Inference stage denotes the inference process of the off-the-shelf VLM. Overall results reflect the total latency during the end-to-end process. Noticeably, we restrict the vanilla Qwen3.6-27B model to sample a maximum of 224 frames for the memory constraint, leading to a significantly lower I/O latency.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
R←argsorti∈{1,…,T}(ri)[:N]
Appendix
Algorithm 3 CLIP-Guided Sampling and Ranking
Memory
Scheme
#Frame
LongVideoBench
MLVU
Video-MME
128 frames
Vanilla
128
68.0
75.9
73.6
d1f16d2f448
464
74.4
83.9
76.3
d1f64d2f256
320
73.5
83.1
77.2
d1f96d2f128
224
73.6
83.1
77.4
d1f16d2f320d4f512
848
75.2
84.9
76.7
256 frames
Vanilla
256
70.9
76.7
75.9
Appendix
Table 7: Additional ablation study on resolution allocation schemes using Qwen3.6-27B. We report results under 128-, 256-, and 512-frame memory budgets, where each budget denotes the cost equivalent to processing the corresponding number of full-resolution frames. All mixed-resolution schemes use the diversity-aware sampling strategy. Bold numbers indicate the best result within each memory budget.
Memory
Scheme
Strategy
LongVideoBench
MLVU
Video-MME
128 frames
–
Vanilla
68.0
75.9
73.6
d1f16d2f448
FramePack-U
74.4
83.9
76.3
− Relevance
73.2 (-1.2)
81.5 (-2.4)
76.1 (-0.2)
− Diversity
72.8 (-1.6)
81.7 (-2.2)
68.9 (-7.4)
d1f64d2f256
FramePack-U
73.5
83.1
77.2
− Relevance
72.2 (-1.3)
80.1 (-3.0)
76.2 (-1.0)
Appendix
Table 8: Additional ablation study on prompt relevance and frame diversity using Qwen3.6-27B. − Relevance removes prompt relevance and uses a prompt-agnostic diversity-based ordering, while − Diversity ranks frames only by query-frame similarity. FramePack-U combines prompt relevance and visual diversity. Values in parentheses denote the difference between each ablated variant and FramePack-U .
Memory
Model
LongVideoBench
MLVU
Video-MME
128 frames ( d1f96d2f128 )
Qwen3.5-4B
+1.7
+4.0
+1.1
Qwen3.5-9B
+2.8
+3.0
+0.6
Qwen3.6-27B
+5.6
+7.2
+3.8
256 frames ( d1f64d2f768 )
Qwen3.5-4B
+0.2
+1.6
-0.9
Qwen3.5-9B
+1.1
+1.4
-0.2
Qwen3.6-27B
+3.9
+7.0
+0.4
Appendix
Table 9: Additional ablation study on model size. We report the performance gain of FramePack-U over the vanilla baseline across different Qwen-series base models and memory budgets. Larger models generally obtain larger gains, suggesting that stronger VLMs can better exploit the packed multi-resolution visual evidence.
Model
Setting
Method
LVB Overall
LVB 15 / 60 / 600 / 3600
V-MME Overall
V-MME S / M / L
Qwen3.6-27B
128-frame memory d1f16d2f448
Vanilla
68.0
82.0 / 83.1 / 68.9 / 58.0
73.6
85.2 / 74.1 / 61.4
FramePack-U
74.4
82.5 / 84.9 / 73.5 / 69.1
76.3
85.4 / 77.7 / 65.7
Δ
+6.4
+0.5 / +1.8 / +4.6 / +11.1
+2.7
+0.2 / +3.6 / +4.3
256-frame memory d1f250d4f96
Vanilla
70.9
83.1 / 83.7 / 72.8 / 61.5
75.9
85.7 / 76.6 / 65.6
FramePack-U
74.3
83.6 / 83.7 / 73.8 / 68.6
77.9
86.1 / 79.2 / 68.3
Δ
+3.4
+0.5 / +0.0 / +1.0 / +7.1
+2.0
+0.4 / +2.6 / +2.7
Appendix
Table 10: Fine-grained results on LongVideoBench and Video-MME. For LongVideoBench, we report the overall accuracy and the results on the 15s / 60s / 600s / 3600s subsets. For Video-MME, we report the overall accuracy and the short / medium / long subsets. Only the default diversity-aware packing strategy of FramePack-U is reported, with the vanilla baseline included as reference. Δ denotes the difference between FramePack-U and Vanilla under the same setting. Bold indicates the best result in each column or sub-column.
Method
LVB
MLVU
VideoMME
Overall
Vanilla Model
64.5
70.3
70.5
68.4
+ Q-Frame
62.6
76.4
66.1
68.4
Δ
-1.9
+6.1
-4.4
+0.0
+ LongVU-Style Ranking
58.5
71.5
63.9
64.6
Δ
-6.0
+1.2
-6.6
-3.8
+ FramePack-U
66.2
74.3
71.6
70.7
Appendix
Table 11: Performance comparison with similar token compression or key frame selection methods. Δ denotes the performance gap with the vanilla model. Bold indicates the best result in each column.
Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards one token per frame at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into learnable and progressive modules for token-level compression (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate frame-level compression, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named question-conditioned compression (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, i.e., the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined token-level and frame-level leads to an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling. Our XComp is finetuned from VideoChat-Flash with a data-efficient supervised compression tuning stage that only requires 2.5% of the supervised fine-tuning data, yet boosts the accuracy from 42.9% to 46.2% on LVBench and enhances multiple other long video benchmarks.
Zheyu Zhang, Ziqi Pang, Shixing Chen +3
University of Illinois Urbana-Champaign · Amazon Prime Video
Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent Memory (PREM), a memory-token-free framework for frozen vision-language models (VLMs). PREM separates video ingestion from query answering: a recurrent writer distills visual streams into a compact 256 KiB multi-slot associative state, while a question-conditioned readout adds memory-derived key/value (K/V) steering modulations to existing non-visual prompt prefixes during prefill. This enables write-once, query-many inference without extra prompt tokens or decoding recurrence. Across six long-video benchmarks in offline and streaming end-of-stream settings, PREM consistently outperforms frozen baselines at every evaluated visual budget. Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle retrieval. These gains require tuning 0.24% of backbone parameters at 0.03 GiB of peak GPU memory overhead.
Siru Zhong, Qiongyan Wang, Xiaohui Lv +5
The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University
The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on "post-hoc" token reduction -- reducing visual tokens after feature extraction to alleviate the LLM's computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame, a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier -- compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8× more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets.