Omni-modal large language models (Omni-LLMs) can jointly process a broad range of modalities, such as text, audio, and video, enabling stronger multimodal understanding and reasoning capabilities, but often at substantial inference cost. We observe that the benefit of additional modalities varies across queries, indicating that a fixed modality configuration can be inefficient. Motivated by this, we study query-wise modality selection, which keeps the underlying Omni-LLM fixed while adaptively selecting which modalities to provide for each query. We formulate the problem as an accuracy-cost optimization problem and develop a cost-sensitive learning approach for modality selection. We evaluate our approach on WorldSense using Qwen2.5-Omni and Qwen3-Omni under three inference-cost measures: input tokens, latency, and monetary cost. Across both models, query-wise modality selection consistently improves the accuracy-cost trade-off and can even outperform the standard setting in which all available modalities are provided to the Omni-LLM. For Qwen2.5-Omni, our cost-sensitive approach matches or exceeds this full-modality accuracy while requiring only approximately 41% of the input tokens, 39% of the latency, and 61% of the monetary cost.
Figures & tables
Figure 1: Examples from WorldSense illustrating query-dependent modality requirements. Checkmarks ✓ and crosses × indicate prediction correctness using Qwen2.5-Omni.
Figure 2: Comparison between fixed modality configurations and per-query modality selection. (a) In fixed-modality inference, a single input configuration is used for every query, either Text+Audio (TA) or Text+Audio+Video (TAV). (b) In our framework, a modality selector determines for each query whether to use TA or TAV before invoking the Omni-LLM for prediction.
Figure 3: Accuracy–cost trade-off for query-wise modality selection on WorldSense using Qwen2.5-Omni (top) and Qwen3-Omni (bottom). The three columns measure cost by average input tokens, monetary cost, and latency per query, respectively. Semi-transparent markers denote operating points obtained by sweeping each method’s decision threshold or hyperparameter, while solid lines show the corresponding Pareto frontiers. Filled circles denote the Always-TA and Always-TAV endpoints, and random routing serves as a non-adaptive reference.
Model
Qwen2.5-Omni
Qwen3-Omni
Method
TokenIn
Latency
Money
TokenIn
Latency
Money
Random Router
0.5046
0.4987
0.4920
0.4986
0.4985
0.4985
VerbConf Cascade
0.4355
0.4327
0.4965
0.4110
0.4193
0.4195
LogitConf Cascade
0.6963
0.6967
0.6043
0.7143
0.7108
0.7133
Estimate Then Predict
0.8178
0.8047
0.8992
0.6941
0.6860
0.7895
Cost-Sensitive Learning
0.9265
0.9125
0.9878
0.8670
0.8587
0.9422
Table 1: Normalized accuracy–cost AUC of different modality-selection methods across three cost measures on WorldSense.
Model
Method
TokenIn
Δ %
Latency
Δ %
Money
Δ %
Qwen2.5
Always-TAV
41050
0.00
47.13
0.00
0.0963
0.00
VerbConf Cascade
44224
-7.73
49.17
-4.33
0.1338
-38.94
LogitConf Cascade
29289
28.65
31.81
32.51
0.0999
-3.74
Estimate Then Predict
28099
31.55
31.74
32.65
0.0762
20.87
Cost-Sensitive Learning
16877
58.89
18.35
61.07
0.0588
38.94
Qwen3
Always-TAV
29874
0.00
36.89
0.00
0.0652
0.00
Table 2: Cost reduction at Always-TAV accuracy. For each method, we report the input-token, monetary, and latency costs at which its accuracy–cost Pareto frontiers reach the Always-TAV accuracy (48.00% for Qwen2.5-Omni; 51.79% for Qwen3-Omni). Costs are averaged over test queries, and Δ % ( 12 ) denotes the relative cost reduction compared with Always-TAV.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Paradigm
h(q)
Training
Additional Requirement
Oracle
Routing
Omni-LLM correctness
No
Ground-truth
Random Router
Routing
Random score
No
None
Task-Type Router
Routing
Task metadata
No
Metadata
Estimate then Predict
Routing
Text embedding
Yes
Text Encoder
Cost-Sensitive Router
Routing
Text embedding
Yes
Text Encoder
LLM-as-a-Judge
Routing
External LLM judgment
No
External LLM
Appendix
Table 3: Summary of the modality-selection methods evaluated in our experiments.
Figure 4: Potential of query-wise modality selection on WorldSense using Qwen2.5-Omni. (a) Accuracy–cost frontier of an oracle selector that chooses between T+A and TAV using ground-truth query outcomes. (b) A simple task-type router that selects the modality configuration based only on the query’s task type. The Always-TA and Always-TAV points are shown as reference configurations, and cost is measured by average input tokens per query.
The unified audio-visual understanding capability of omni-modal large language models (om-LLMs) comes at the high cost of processing tens of thousands of visual and audio tokens throughout the LLM. To reduce the cost, training-free methods for om-LLM token selection are being actively developed, operating either pre-LLM or inner-LLM. Recognizing that the non-textual token redundancy varies across stages and LLM layers, we propose in this paper a novel stage-adaptive token selection (SEATS) method that operates in both pre-LLM and inner-LLM stages. Given an overall token budget measured by the token retention rate (TRR), SEATS performs top-down token budget allocation in an input-dependent manner. Specifically, a larger TRR is used for pre-LLM token selection, whilst within the LLM, all tokens are retained in shallow layers, increasingly decreased TRRs are applied in middle layers, and the non-textual tokens are fully removed in late layers. Extensive experiments on three representative om-LLMs and six benchmarks verify the viability of the proposed method. By retaining only 10% of visual and audio tokens, SEATS preserves 94.9% of the full-token performance on Qwen2.5-Omni while achieving a 9.3x FLOPs reduction and a 4.9x prefill speedup.
Zijie Xin, Jie Yang, Ruixiang Zhao +4
Renmin University of China · WeChat Vision, Tencent Inc.
Omni-modal language models are intended to jointly understand audio, visual inputs, and language, but benchmark gains can be inflated when visual evidence alone is enough to answer a query. We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration, and how post-training behaves under a visually debiased evaluation setting. We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets when filtering is undefined or would make comparisons unstable. This yields OmniClean, a cleaned evaluation view with 8,551 retained queries from 16,968 audited queries. On OmniClean, we evaluate OmniBoost, a three-stage post-training recipe based on Qwen2.5-Omni-3B: mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data. Balanced bi-modal SFT gives limited and uneven gains, RLVR provides the first broad improvement, and self-distillation reshapes the benchmark profile. After SFT on self-distilled data, the 3B model reaches performance comparable to, and in aggregate slightly above, Qwen3-Omni-30B-A3B-Instruct without using a stronger omni-modal teacher. These results show that omni-modal progress is easier to interpret when evaluation controls visual leakage, and that small omni-modal models can benefit from staged post-training with self-distilled omni-query supervision. Project page: https://cheliu-computation.github.io/omni/
Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (e.g., vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks. Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly. We term this disconnect knows-but-cannot-say. To investigate this phenomenon, we introduce Token-level Modality Alignment (TMA), which measures the alignment between individual modality representations and response representations. Our analysis shows that non-textual representations are less aligned with response tokens and receive less attention than textual representations, suggesting a potential explanation for uneven modality preferences. Motivated by these findings, we propose OmniCatalyst, a simple perceive → infer strategy that complements the original modality inputs with self-generated text, making non-textual knowledge more accessible. Across seven Omni-LLMs, OmniCatalyst consistently improves performance (+7.29%) and yields more balanced modality preferences, with further gains from diverse training strategies. Further analysis demonstrates that the knowledge extraction phases increase TMA, suggesting that improved alignment makes non-textual information more accessible during response generation. In summary, our findings uncover how Omni-LLMs can better express modality knowledge through language. Our code is available at https://github.com/naver-ai/OmniLLM.