OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models

Yuchen Deng; Zidang Cai; Feidiao Yang; Yufei Wang; Jie Wang; Hai-Tao Zheng; Yuxing Han

cs.CVSep 29, 2026

OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models

Authors: Yuchen Deng, Zidang Cai, Feidiao Yang, Yufei Wang, Jie Wang, Hai-Tao Zheng, Yuxing Han

Organizations: Shenzhen International Graduate School, Tsinghua University, China · Pengcheng Laboratory, China

Abstract

Omnimodal large language models (Omni-LLMs) encode audio and visual streams into temporally interleaved token sequences for multimodal reasoning. However, processing long audio-visual token sequences incurs substantial prefill costs. Existing compression methods have made progress, but often overlook temporal changes in audio-visual semantic relevance. Motivated by temporal variation and local continuity, we propose OmniRoute, a training-free, two-stage compression framework. First, Temporal Evidence-Guided Budgeting (TEGB) derives chunk-wise modality preferences and initial leading-modality budgets from semantic relevance and local content variation. Second, Budget-Constrained Semantic Compression (BCSC) compresses the leading modality and then calibrates the follower's retention target using the actual retained fraction. For video, it combines spatiotemporal grouping with query-guided selection; for audio, it selects tokens based on encoder attention and query relevance, then merges residual tokens into context anchors under visual guidance. Experiments on four representative benchmarks demonstrate a better trade-off between inference efficiency and performance than competitive baselines. The code and interface will be released to facilitate further research.

Figures & tables

Figure 1: WorldSense accuracy using Qwen2.5-Omni-3B/7B.

Figure 2: Temporal audio-visual evidence. (a) The

D_{i}

trajectory and its centered five-chunk moving average for a WorldSense video. (b)-(c) Distributions of

V_{s}

and

C_{s}

over 500 videos.

Figure 3: OmniRoute overview. TEGB derives chunk-wise modality preferences and leading-modality budgets from semantic relevance and local variation. BCSC calibrates follower targets using actual leading-modality retention. Video compression combines spatial and temporal grouping with query-guided selection; audio compression combines attention- and query-based selection with visually guided merging.

Model	Method	Retained Ratio	FLOPs Ratio	WorldSense	AVUTBench	VideoMME	DailyOmni	Avg.
Qwen2.5- Omni-7B	Full Tokens	100%	100%	46.8	64.5	66.0	62.9	100%
	Random	55%	48%	43.6	61.0	65.4	58.6	95.0%
	DyCoke (V&A)	50%	44%	44.6	62.0	65.5	55.8	94.8%
	FlashVID	45%	39%	46.0	63.0	65.9	57.0	96.6%
	OmniZip	45%	39%	45.9	63.0	66.1	59.5	97.6%
	OmniRoute (Ours)	45%	39%	46.9	63.4	66.0	60.6	98.7%

Table 1: Comparison with token compression methods across different omni-models.

Method	Mem. $\downarrow$	Prefill (ms) $\downarrow$	Acc. $\uparrow$	Latency (s) $\downarrow$
Full Tokens (100%)	44 G	2371 (1.00 $\times$ )	46.8	10.99 (1.00 $\times$ )
DyCoke (50%)	36 G	1386 (1.71 $\times$ )	44.6	8.59 (1.28 $\times$ )
FlashVID (45%)	35 G	1073 (2.21 $\times$ )	46.0	9.63 (1.14 $\times$ )
OmniZip (45%)	32 G	894 (2.65 $\times$ )	45.9	7.99 (1.38 $\times$ )
Ours (45%)	27 G	936 (2.53 $\times$ )	46.9	8.12 (1.35 $\times$ )

Table 2: Efficiency on WorldSense with Qwen2.5-Omni-7B. Mem. denotes peak GPU memory.

Method	Mem. $\downarrow$	Prefill (ms) $\downarrow$	Acc. $\uparrow$	Latency (s) $\downarrow$
Full Tokens (100%)	44 G	2371 (1.00 $\times$ )	46.8	10.99 (1.00 $\times$ )
DyCoke (50%)	36 G	1386 (1.71 $\times$ )	44.6	8.59 (1.28 $\times$ )
FlashVID (45%)	35 G	1073 (2.21 $\times$ )	46.0	9.63 (1.14 $\times$ )
OmniZip (45%)	32 G	894 (2.65 $\times$ )	45.9	7.99 (1.38 $\times$ )
Ours (45%)	27 G	936 (2.53 $\times$ )	46.9	8.12 (1.35 $\times$ )

Table 2: Efficiency on WorldSense with Qwen2.5-Omni-7B. Mem. denotes peak GPU memory.

Setting	$m_{i}$ , $n_{i}$	Routing	Retained Ratio	Acc.
Full OmniRoute	✓	Dynamic	45%	46.9
w/o $m_{i}$ / $n_{i}$	✗	Dynamic	45%	46.4
Audio-led only	✓	Audio-led	45%	45.9
Video-led only	✓	Video-led	45%	45.2

Table 3: Ablation of OmniRoute’s core components on WorldSense with Qwen2.5-Omni-7B.

Figure 4: Hyperparameter analysis on WorldSense: modality-specific weights

\gamma_{v}

and

\gamma_{a}

, and cross-modal coupling

\beta

Retained Ratio	FLOPs Ratio	Mem. $\downarrow$	Prefill (ms) $\downarrow$	Acc. $\uparrow$	Latency (s) $\downarrow$
45%	39%	27 G	936	46.9	8.12
40%	31%	27 G	788	46.3	8.00
35%	26%	27 G	640	46.1	7.73
30%	18%	27 G	558	45.6	7.59

Table 4: Performance of OmniRoute under different retained ratios on WorldSense with Qwen2.5-Omni-7B.

Explore similar work

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
Jul 25, 2026Jinsen Su, Yongdong Luo, Yuexiao Ma +3Token Compression Omni-Modal
Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China · School of Informatics, Xiamen University, Xiamen, China +1
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
Jul 3, 2026Shijie Cao, Qingyu Zhang, Boxi Yu +6Token Compression Audio-Visual Reasoning
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Limerick +1
OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models
May 12, 2026Yuchen Deng, Zidang Cai, Hai-Tao Zheng +3Token Compression Audio-Visual Reasoning
Tsinghua Shenzhen International Graduate School, Tsinghua University · Pengcheng Laboratory

Jul 25, 2026, cs.CVJ/K move · Enter open · S save

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

Jinsen Su, Yongdong Luo, Yuexiao Ma, Yibo Hu+2

Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China · School of Informatics, Xiamen University, Xiamen, China+1

Token Compression Omni-Modal Modality-Specific Encoders Multimodal Large Language Models Cross-Modal Conflicts Relevance

Open paper AskPDF arXiv

Existing token compression methods for omnimodal large language models typically rely on one modality to determine what to retain in the other. We show that this assumption often breaks down: for the same query, audio and video relevance often peaks at different moments. This cross-modal salience mismatch makes unidirectional guidance prone to discarding answer-critical cues under aggressive compression. We propose OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video. OmniScope allocates modality-specific token budgets, prunes visual tokens with an anchor-delta strategy that preserves both global context and temporal changes, and merges audio tokens within each second to reduce redundancy while maintaining temporal continuity. Across four audio-video benchmarks and two Qwen2.5-Omni model scales, OmniScope achieves the best average accuracy across all compression settings. At 25% overall token retention, it delivers up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy. These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates. The code is available at https://github.com/MAC-AutoML/OmniScope.

Abstract

Explore similar work

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models