Efficient Omni-modal Large Language Model Inference via Per-Query Modality Selection

Shuqi Liu; Yuzhou Cao; Luke Ong

cs.LGOct 22, 2026

Efficient Omni-modal Large Language Model Inference via Per-Query Modality Selection

Authors: Shuqi Liu, Yuzhou Cao, Luke Ong

Organizations: College of Computing and Data Science, Nanyang Technological University

Abstract

Omni-modal large language models (Omni-LLMs) can jointly process a broad range of modalities, such as text, audio, and video, enabling stronger multimodal understanding and reasoning capabilities, but often at substantial inference cost. We observe that the benefit of additional modalities varies across queries, indicating that a fixed modality configuration can be inefficient. Motivated by this, we study query-wise modality selection, which keeps the underlying Omni-LLM fixed while adaptively selecting which modalities to provide for each query. We formulate the problem as an accuracy-cost optimization problem and develop a cost-sensitive learning approach for modality selection. We evaluate our approach on WorldSense using Qwen2.5-Omni and Qwen3-Omni under three inference-cost measures: input tokens, latency, and monetary cost. Across both models, query-wise modality selection consistently improves the accuracy-cost trade-off and can even outperform the standard setting in which all available modalities are provided to the Omni-LLM. For Qwen2.5-Omni, our cost-sensitive approach matches or exceeds this full-modality accuracy while requiring only approximately 41% of the input tokens, 39% of the latency, and 61% of the monetary cost.

Figures & tables

Panel of Figure 1 — Figure 1: Examples from WorldSense illustrating query-dependent modality requirements. Checkmarks $\checkmark$ and crosses $\times$ indicate prediction correctness using Qwen2.5-Omni.

Panel of Figure 2 — Figure 2: Comparison between fixed modality configurations and per-query modality selection. (a) In fixed-modality inference, a single input configuration is used for every query, either Text+Audio (TA) or Text+Audio+Video (TAV). (b) In our framework, a modality selector determines for each query whether to use TA or TAV before invoking the Omni-LLM for prediction.

Panel of Figure 3 — Figure 3: Accuracy–cost trade-off for query-wise modality selection on WorldSense using Qwen2.5-Omni (top) and Qwen3-Omni (bottom). The three columns measure cost by average input tokens, monetary cost, and latency per query, respectively. Semi-transparent markers denote operating points obtained by sweeping each method’s decision threshold or hyperparameter, while solid lines show the corresponding Pareto frontiers. Filled circles denote the Always-TA and Always-TAV endpoints, and random routing serves as a non-adaptive reference.

Model	Qwen2.5-Omni			Qwen3-Omni
Method	TokenIn	Latency	Money	TokenIn	Latency	Money
Random Router	0.5046	0.4987	0.4920	0.4986	0.4985	0.4985
VerbConf Cascade	0.4355	0.4327	0.4965	0.4110	0.4193	0.4195
LogitConf Cascade	0.6963	0.6967	0.6043	0.7143	0.7108	0.7133
Estimate Then Predict	0.8178	0.8047	0.8992	0.6941	0.6860	0.7895
Cost-Sensitive Learning	0.9265	0.9125	0.9878	0.8670	0.8587	0.9422

Table 1: Normalized accuracy–cost AUC of different modality-selection methods across three cost measures on WorldSense.

Model	Method	TokenIn	$\Delta$ %	Latency	$\Delta$ %	Money	$\Delta$ %
Qwen2.5	Always-TAV	41050	0.00	47.13	0.00	0.0963	0.00
	VerbConf Cascade	44224	-7.73	49.17	-4.33	0.1338	-38.94
	LogitConf Cascade	29289	28.65	31.81	32.51	0.0999	-3.74
	Estimate Then Predict	28099	31.55	31.74	32.65	0.0762	20.87
	Cost-Sensitive Learning	16877	58.89	18.35	61.07	0.0588	38.94
Qwen3	Always-TAV	29874	0.00	36.89	0.00	0.0652	0.00

Table 2: Cost reduction at Always-TAV accuracy. For each method, we report the input-token, monetary, and latency costs at which its accuracy–cost Pareto frontiers reach the Always-TAV accuracy (48.00% for Qwen2.5-Omni; 51.79% for Qwen3-Omni). Costs are averaged over test queries, and

\Delta

% ( 12 ) denotes the relative cost reduction compared with Always-TAV.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Paradigm	$\bm{h}(\bm{q})$	Training	Additional Requirement
Oracle	Routing	Omni-LLM correctness	No	Ground-truth
Random Router	Routing	Random score	No	None
Task-Type Router	Routing	Task metadata	No	Metadata
Estimate then Predict	Routing	Text embedding	Yes	Text Encoder
Cost-Sensitive Router	Routing	Text embedding	Yes	Text Encoder
LLM-as-a-Judge	Routing	External LLM judgment	No	External LLM

Appendix

Table 3: Summary of the modality-selection methods evaluated in our experiments.

Panel of Figure 4 — Figure 4: Potential of query-wise modality selection on WorldSense using Qwen2.5-Omni. (a) Accuracy–cost frontier of an oracle selector that chooses between T+A and TAV using ground-truth query outcomes. (b) A simple task-type router that selects the modality configuration based only on the query’s task type. The Always-TA and Always-TAV points are shown as reference configurations, and cost is measured by average input tokens per query.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 226 papers

Stage-adaptive Token Selection for Efficient Omni-modal LLMs
May 19, 2026Zijie Xin, Jie Yang, Ruixiang Zhao +4Efficient Multimodal Inference Multimodal Large Language Models
Renmin University of China · WeChat Vision, Tencent Inc.
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
May 12, 2026Che Liu, Lichao Ma, Xiangyu Tony Zhang +4Cross-Modal Learning Multimodal Large Language Models
StepFun-Audio Team
Omni-Modal LLMs Know More Than They Say
Oct 12, 2026Na Min An, Taekyung Kim, Jaemyung Yu +2Multimodal Large Language Models Cross-Modal Alignment
KAIST AI · NAVER AI Lab

May 19, 2026J/K move · Enter open · S save

Stage-adaptive Token Selection for Efficient Omni-modal LLMs

Zijie Xin, Jie Yang, Ruixiang Zhao, Tianyi Wang+3

Renmin University of China · WeChat Vision, Tencent Inc.

Efficient Multimodal Inference Multimodal Large Language Models Token Pruning Multimodal Token Compression

Open paper AskPDF arXiv

The unified audio-visual understanding capability of omni-modal large language models (om-LLMs) comes at the high cost of processing tens of thousands of visual and audio tokens throughout the LLM. To reduce the cost, training-free methods for om-LLM token selection are being actively developed, operating either pre-LLM or inner-LLM. Recognizing that the non-textual token redundancy varies across stages and LLM layers, we propose in this paper a novel stage-adaptive token selection (SEATS) method that operates in both pre-LLM and inner-LLM stages. Given an overall token budget measured by the token retention rate (TRR), SEATS performs top-down token budget allocation in an input-dependent manner. Specifically, a larger TRR is used for pre-LLM token selection, whilst within the LLM, all tokens are retained in shallow layers, increasingly decreased TRRs are applied in middle layers, and the non-textual tokens are fully removed in late layers. Extensive experiments on three representative om-LLMs and six benchmarks verify the viability of the proposed method. By retaining only 10% of visual and audio tokens, SEATS preserves 94.9% of the full-token performance on Qwen2.5-Omni while achieving a 9.3x FLOPs reduction and a 4.9x prefill speedup.

Abstract

Efficient Multimodal Inference

Explore similar work

Stage-adaptive Token Selection for Efficient Omni-modal LLMs

Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation

Omni-Modal LLMs Know More Than They Say