Organizations: InfRec, Cardinal AI Lab · Shanghai Jiao Tong University · National University of Singapore · The Hong Kong University of Science and Technology
Omni-modal large language models (Omni-LLMs) typically process large numbers of multimodal tokens from audio-visual inputs, incurring substantial computational costs and a large memory footprint for the key-value (KV) cache. Aggressive token pruning can alleviate this problem while maintaining competitive performance on multiple-choice tasks, which require only a single option as output. However, we observe substantial quality degradation when generating long-form, open-ended responses. We attribute this gap to query-guided selection, which can retain sufficient evidence for answer selection while discarding details needed for comprehensive generation. This motivates reducing overlapping information across tokens, termed structural redundancy, while preserving diverse content. However, the retained representations may still contain precision redundancy due to unnecessarily high numerical precision in their cached values. To address these two levels of redundancy in token representations, we introduce a training-free framework named R2-Omni. First, Structure-Aware Token Consolidation (STC) consolidates redundant representations using spatiotemporal similarity in video and local continuity in audio. Then, Semantic-Sensitivity Guided Precision Allocation (SSPA) allocates value precision to the retained representations according to the estimated reduction in attention-output error under quantized keys, with query attention and audio-visual alignment providing a semantic prior. Extensive experiments on long-form generation benchmarks demonstrate an average 5.0× reduction in the memory footprint while preserving 97.98% of baseline generation quality and achieving a 1.48× end-to-end speedup, highlighting the benefits of jointly reducing structural and precision redundancy.
Figures & tables
Figure 1: Motivation of R 2 -Omni. (a) Aggressive token pruning degrades long-form generation on the Video-SALMONN 2 testset [ 6 ] more severely than multiple-choice performance on Daily-Omni [ 16 ] . Details are provided in Appendix A.1 (b) After structural redundancy is reduced ( gray masks ), we further reduce the precision of cached values ( red masks ), and the model holds all details. This suggests that precision redundancy remains, enabling further reduction in the memory footprint.
Figure 2: Overview of R 2 -Omni. Structure-Aware Token Consolidation(STC) reduces overlapping information across audio-visual tokens before prefill. Semantic-Sensitivity Guided Precision Allocation(SSPA) reduces excess precision in the resulting cache by evaluating value upgrades through attention-output distortion under quantized keys, guided by a multimodal semantic prior.
Method
Video-SALMONN 2
LongVALE
OmniDCBench
Avg. Rel. Quality
Correct ↑
EER ↓
Ratio ↑
SODA-C ↑
Ratio ↑
SODA-M ↑
Ratio ↑
Qwen2.5-Omni-3B
24.45
85.66
1.00 ×
2.06
1.00 ×
5.01
1.00 ×
100.0
+ OmniSelect
15.18
95.65
5.12 ×
2.20
5.08 ×
3.39
5.14 ×
81.53
+ OmniZip
17.49
92.79
5.12 ×
2.12
5.18 ×
4.65
5.29 ×
89.89
+ SEATS
16.76
93.04
4.88 ×
2.21
5.28 ×
4.67
5.27 ×
90.28
+ KIVI
19.42
89.64
5.29 ×
2.15
5.22 ×
3.46
5.17 ×
87.10
Table 1: Comparison with token-pruning and KV-cache quantization methods on three long-form generation benchmarks. Ratio refers to the compression ratio of the final memory footprint relative to the baseline. Original model , Token Pruning , KV Quantization , and R 2 -Omni indicate the corresponding method groups. The best and second-best results among compressed methods are highlighted in bold and underline .
Method
Video-SALMONN 2
LongVALE
OmniDCBench
Avg. Rel. Quality
Correct ↑
EER ↓
Ratio ↑
SODA-C ↑
Ratio ↑
SODA-M ↑
Ratio ↑
Qwen2.5-Omni-3B
24.45
85.66
1.00 ×
2.06
1.00 ×
5.01
1.00 ×
100.0
+ OmniZip + KIVI
18.99
89.41
5.11 ×
1.97
5.23 ×
3.79
5.07 ×
86.19
+ OmniSelect + OTT
22.48
88.14
5.12 ×
1.92
5.37 ×
5.02
5.19 ×
95.63
+ SEATS + OScaR
21.44
87.49
5.32 ×
1.98
5.34 ×
2.71
5.51 ×
83.95
+ R 2 -Omni
23.54
85.63
5.39 ×
2.23
5.34 ×
4.78
5.06 ×
99.99
Table 2: Comparison with naively composed token-pruning and KV-cache quantization baselines. comp. Ratio refers to the compression ratio of the final memory footprint relative to the baseline. The best and second-best quality results among completed compressed methods are highlighted in bold and underline .
Figure 5
Figure 5: End-to-end efficiency of R 2 -Omni and the BF16 baseline. R 2 -Omni consistently improves inference latency, memory efficiency, decoding throughput, and maximum batch capacity. Detials are provided in Appendix A.2
Method
Video-SALMONN 2
LongVALE
OmniDCBench
Efficiency (256K)
Correct ↑
EER ↓
Ratio
SODA-C ↑
Ratio
SODA-M ↑
Ratio
TTFT(s)
TPOT(ms/tok)
E2E(s)
R 2 -Omni
21.02
88.57
7.07 ×
2.12
6.79 ×
5.11
6.93 ×
27.03
25.90
40.34
w/o SSPA
17.66
91.80
7.11 ×
2.03
5.13 ×
3.47
7.01 ×
25.50
21.86
36.67
w/o STC
20.39
90.16
3.97 ×
1.77
3.95 ×
5.35
3.95 ×
58.54
39.61
78.86
Table 3: Component ablation of R 2 -Omni. Ratio refers to the compression ratio of the final memory footprint relative to the baseline. Quality is evaluated with 256 frames on Video-SALMONN 2; LongVALE and OmniDCBench follow the main input settings.
Method
Video-SALMONN 2
LongVALE
OmniDCBench
Correct ↑
EER ↓
Ratio
SODA-C ↑
Ratio
SODA-M ↑
Ratio
R 2 -Omni
23.54
85.63
5.39 ×
2.23
5.34 ×
4.78
5.06 ×
w/o A–V Alignment
23.57
86.81
5.36 ×
2.12
5.34 ×
4.55
5.00 ×
w/o Semantic Prior
23.48
87.99
5.34 ×
2.01
5.33 ×
4.61
5.04 ×
Table 4: Ablation of SSPA allocation signals on Qwen2.5-Omni-3B. Ratio refers to the compression ratio of the final memory footprint relative to the baseline. All variants use the same STC output and configured precision budget.
Threshold
Video-SALMONN 2
LongVALE
OmniDCBench
τs
τa
Token Ret. (%)
Correct ↑
EER ↓
Token Ret. (%)
SODA-C ↑
Token Ret. (%)
SODA-M ↑
0.7
0.8
56.14
23.54
85.63
59.73
2.23
61.84
4.78
0.5
0.6
18.49
18.32
90.45
23.50
2.28
22.77
4.26
0.3
0.4
6.97
16.00
92.56
12.24
2.00
10.32
3.18
Table 5: Token retention and generation quality across STC similarity thresholds on Qwen2.5-Omni-3B. Token Ret. refers to Token retention ratio after STC.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Video-SALMONN 2
DailyOmni
Method
Pruning
Correct ↑
Rel. Quality ↑
Accuracy ↑
Rel. Quality ↑
BF16
0%
24.45
100.00
55.56
100.00
OmniZip
40%
25.09
102.60
53.72
96.69
50%
24.43
99.90
53.38
96.09
60%
23.09
94.41
51.63
92.93
70%
21.07
86.18
51.38
92.48
Appendix
Table 6: Token pruning quality on Video-SALMONN 2 and DailyOmni. Video-SALMONN 2 reports Correct rate, while DailyOmni reports multiple-choice accuracy. Relative quality is normalized to the corresponding BF16 baseline.
Method
Context
TTFT (s)
TPOT (ms/tok)
E2E (s)
Peak (GiB)
Final (GiB)
BF16
16K
0.964
13.871
29.357
19.920
0.978
R 2 -Omni
16K
1.223
17.563
37.184
18.790
0.209
OScaR
16K
0.950
16.124
33.963
19.654
0.191
SEATS
16K
0.645
14.482
30.288
20.387
0.288
BF16
32K
2.364
14.037
31.090
23.134
1.824
R 2 -Omni
32K
2.305
17.757
38.653
20.402
0.334
Appendix
Table 7: End-to-end efficiency across context lengths. TTFT denotes time-to-first-token, TPOT denotes time-per-output-token, and E2E denotes the total latency for generating 2,048 tokens. Peak is the peak allocated GPU memory, and Final is the physical memory footprint after generation. Context lengths refer to raw multimodal sequences before token consolidation. Lower is better for all metrics.
Method
Compression
Video-SALMONN 2
LongVALE
OmniDCBench
Correct ↑
EER ↓
SODA-C ↑
SODA-M ↑
R 2 -Omni
3 ×
25.75
84.28
2.09
4.56
4 ×
24.02
86.05
2.13
4.77
6 ×
22.58
87.33
2.25
4.81
7 ×
22.86
86.30
2.12
5.11
8 ×
22.18
87.02
2.11
4.94
Appendix
Table 8: Quality across compression ratios on Qwen2.5-Omni-3B. For R 2 -Omni, each operating point adjusts only the STC thresholds and SSPA value-precision budget.
Method
Video-SALMONN 2
LongVALE
OmniDCBench
Correct ↑
EER ↓
KV Comp.
SODA-C ↑
KV Comp.
SODA-M ↑
KV Comp.
Qwen2.5-Omni-7B
OmniZip + KIVI
29.97
78.57
5.07 ×
1.44
5.34 ×
6.33
4.89 ×
OmniSelect + OTT
30.99
79.07
5.00 ×
1.40
5.35 ×
6.84
4.81 ×
SEATS + OScaR
15.12
93.92
5.27 ×
1.15
5.30 ×
3.57
5.22 ×
MiniCPM-o-4.5
Appendix
Table 9: Direct pruning–quantization compositions on Qwen2.5-Omni-7B and MiniCPM-o-4.5.
Component
Parameter
Default setting
Video STC
Spatial / temporal similarity threshold
0.70 / 0.70
Video STC
Region size / prototypes / isolated retention
[2,128] / one per region / 100%
Audio STC
Chunk size / merge threshold / maximum merge ratio
50 / 0.80 / 0.50
Semantic prior
Query anchors / recency decay
8 / 0.8
Semantic prior
Temporal chunks / alignment top- k / page top- k
16 / 4 / 4
Semantic prior
Attention–alignment weight ω
0.6
Appendix
Table 10: Default R 2 -Omni implementation parameters.
Figure 6: Qualitative comparison of long-range temporal coverage. R 2 -Omni preserves more complete event coverage across the full clip, while OmniZip and OScaR terminate their semantic coverage earlier. The key evidence is highlighted in red.
Figure 7: Qualitative comparison of fine-grained detail preservation. Red text highlights details preserved by both R 2 -Omni and OScaR but omitted by OmniZip, while blue text highlights additional details preserved by R 2 -Omni beyond OScaR.
Figure 8: Qualitative comparison of factuality under compression. R 2 -Omni matches BF16 with zero judge-flagged hallucinated events, while OmniZip and OScaR introduce additional unsupported content. Unsupported events are highlighted in red.
Figure 9: Comparison between STC and token pruning with the same final memory footprint. STC preserves substantially more original visual information after compression.
Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.
Wanshun Su, Yang Shi, Feihu Liu +10
Northwestern Polytechnical University · Peking University · Alibaba Group +1
Omnimodal large language models (OmniLLMs) are rapidly extending multimodal reasoning to cover synchronized audio and video. However, the resulting audio-video token sequences are long, leading to high prefill latency and GPU memory usage at inference time. Existing token pruning methods, designed mainly for vision-only inputs, miss both the cross-modal links between audio and video and the user query that decides which content matters. To bridge this gap, we present Omni-Prune, a training-free, query-aware audio-visual token pruning framework that jointly removes redundancy from both modalities while keeping task-relevant cross-modal evidence. Specifically, Omni-Prune first splits the token sequence into adaptive time windows placed at audio saliency peaks, then scores audio and video tokens on a single scale that combines encoder attention with text-query relevance, and pairs related audio-video tokens so that they are kept together. Within each window, a final K-medoids step then selects a few representative tokens, adding diverse cues that score-based selection alone would miss. Extensive experiments demonstrate that Omni-Prune outperforms established baseline methods, delivering up to 3.25x prefill speedup and 1.3x memory reduction while retaining over 99% of full-model performance.
Existing token compression methods for omnimodal large language models typically rely on one modality to determine what to retain in the other. We show that this assumption often breaks down: for the same query, audio and video relevance often peaks at different moments. This cross-modal salience mismatch makes unidirectional guidance prone to discarding answer-critical cues under aggressive compression. We propose OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video. OmniScope allocates modality-specific token budgets, prunes visual tokens with an anchor-delta strategy that preserves both global context and temporal changes, and merges audio tokens within each second to reduce redundancy while maintaining temporal continuity. Across four audio-video benchmarks and two Qwen2.5-Omni model scales, OmniScope achieves the best average accuracy across all compression settings. At 25% overall token retention, it delivers up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy. These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates. The code is available at https://github.com/MAC-AutoML/OmniScope.
Jinsen Su, Yongdong Luo, Yuexiao Ma +3
Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China · School of Informatics, Xiamen University, Xiamen, China +1