Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference

Bo-Wei Chen; Chung-Chi Chen; An-Zi Yen

cs.CLFeb 25, 2026

Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference

Authors: Bo-Wei Chen, Chung-Chi Chen, An-Zi Yen

Organizations: Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Artificial Intelligence Research Center, AIST, Japan

Abstract

Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks, with larger models performing better but at higher computational costs. We propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates. By assessing a model's confidence in handling the task and response accuracy, tasks that are likely to be solved correctly are retained, while more uncertain or complex cases are delegated to a larger model, ensuring reliability while minimizing computation. Specifically, we evaluate a model's likelihood of knowing the correct answer and the probability that its response is accurate. Experiments on the Massive Multitask Language Understanding (MMLU) benchmark show that our approach achieves accuracy comparable to the largest model while reducing computational costs by 20% to 40%. When applied to GPT-4o API calls, it reduces token usage by approximately 60%, further improving cost efficiency. These findings indicate the potential of confidence-based model selection to enhance real-world LLM deployment, particularly in resource-constrained settings such as edge devices and commercial API applications.

Figures & tables

Figure 1: An illustration of our strategy.

LLM(s)	Acc.	PD ( $\downarrow$ )	Reduced CC ( $\uparrow$ )	Time ( $\downarrow$ )
3B	63.78%	–	–	–
8B	70.21%	–	–	–
70B	83.57%	–	–	–
3B $\to$ 8B	69.44%	-0.77%	20.34%	-18.90%
3B $\to$ 70B	81.89%	-1.68%	32.47%	-26.51%
8B $\to$ 70B	83.22%	-0.35%	36.46%	-33.03%

Table 1: Experimental results with LLaMA models on MMLU. Acc., Reduced CC, Time, and PD denote accuracy, reduced computational cost (measured in GFLOPs), end-to-end time, and performance drop. Each metric is calculated relative to those of

M_{k}

LLM(s)	P	R	F1
3B	0.6444	0.6372	0.6368
8B	0.7055	0.7021	0.7017
70B	0.8386	0.8354	0.8357
3B $\to$ 8B	0.6974	0.6941	0.6940
3B $\to$ 70B	0.8198	0.8160	0.8166
8B $\to$ 70B	0.8350	0.8317	0.8322

Table 2: Macro-Precision (P), Macro-Recall (R), and Macro-F1 (F1) scores of LLaMA models on MMLU.

LLM(s)	Acc.	PD ( $\downarrow$ )	Reduced CC ( $\uparrow$ )	Time ( $\downarrow$ )
4B	66.73%	–	–	–
8B	72.43%	–	–	–
32B	79.51%	–	–	–
4B $\to$ 8B	70.21%	-3.07%	17.80%	2.55%
4B $\to$ 32B	75.03%	-5.63%	52.70%	-40.21%
8B $\to$ 32B	80.00%	0.62%	33.18%	-23.76%

Table 3: Experimental results with Qwen 3 models on MMLU.

LLM(s)	Acc.	Tokens	PD	Reduced Tokens
GPT-4o	86.43%	36,225	–	–
3B $\to$ GPT-4o	84.48%	30,671	-1.95%	15.33%
8B $\to$ GPT-4o	85.66%	28,941	-0.77%	20.11%
70B $\to$ GPT-4o	86.85%	14,505	0.42%	59.96%

Table 4: Experimental results with GPT-4o.

Setting	LLM(s)	PD	Reduced CC
w/ $P(IK)$	8B $\to$ 70B	-0.35%	36.46%
w/ $P(IK)$	3B $\to$ 8B $\to$ 70B	-2.03%	44.48%
w/o $P(IK)$	8B $\to$ 70B	-1.26%	36.83%
w/o $P(IK)$	3B $\to$ 8B $\to$ 70B	-4.27%	47.17%

Table 5: Ablation studies of

P(IK)

LLM(s)	Acc.	PD	Reduced CC
3B	31.47%	–	–
8B	31.92%	–	–
70B	52.23%	–	–
8B $\to$ 70B	51.79%	-0.44%	3.93%
3B $\to$ 8B $\to$ 70B	51.56%	-0.67%	5.11%

Table 6: Experimental results of LLaMA models in OOD scenario.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt used for multiple choice questions.

The following are multiple choice questions about {subject}. Please determine the correct answer in the format “Answer: <answer>”, where <answer>is A, B, C or D.

Question: {question}

Choices: {choices}

Answer:

Prompt used for open-ended questions.

(system) The following are questions about entity relationship. Please determine the correct answer in the format “Answer: <answer>”.

Appendix

Table 7: Prompt templates used in this work.

LLM(s)	3B	8B	70B
3B $\to$ 8B	487	943	–
3B $\to$ 70B	487	–	943
8B $\to$ 70B	–	594	836
3B $\to$ 8B $\to$ 70B	487	198	745

Appendix

Table 8: Number of queries processed by each model stage under different configurations.

Comparison	Chi-squared	p-value
3B $\to$ 8B vs 8B	4.00	0.0455
8B $\to$ 70B vs. 70B ∗	N/A	0.4048
3B $\to$ 8B $\to$ 70B vs. 70B	19.22	0.00001
3B $\to$ 70B vs. 70B	20.48	0.00001
3B $\to$ GPT-4o vs. GPT-4o	15.84	0.00007
8B $\to$ GPT-4o vs. GPT-4o	4.00	0.0455

Appendix

Table 9: Significance test results. The exact p-value for 8B

\to

70B vs 70B is calculated using a binomial test instead of the chi-squared approximation.

	Accuracy	$\Delta$
3B (w/o grounding)	0.1916	-
3B (w/ grounding)	0.2669	+0.0753
8B (w/o grounding)	0.2520	-
8B (w/ grounding)	0.3514	+0.0994
70B (w/o grounding)	0.4767	-
70B (w/ grounding)	0.6608	+0.1841

Appendix

Table 10: Accuracy and delta for different model sizes with and without grounding.

LLM(s)	Acc.	Hallucination Rate	Cost (USD)	PD	Reduced Cost
3B	0.2689	0.4169	0.0030	-	-
8B	0.3387	0.4679	0.0028	-	-
70B	0.6585	0.3208	0.0257	-	-
3B $\to$ 8B	0.3317	0.4923	0.0027	-2.07%	2.64%
3B $\to$ 70B	0.6131	0.3547	0.0226	-6.89%	12.29%
8B $\to$ 70B	0.6459	0.3422	0.0239	-1.91%	7.11%

Appendix

Table 11: Experimental results with LLaMA models on PopQA.

Model	Input (per 1M tokens)	Output (per 1M tokens)
3B	$0.08	$0.16
8B	$0.10	$0.20
70B	$0.60	$1.20
GPT-4o	$2.50	$10.00

Appendix

Table 12: Token pricing for different models.

LLM(s)	Total Cost	Reduced Cost
3B	$0.017	-
8B	$0.021	-
70B	$0.129	-
3B $\to$ 8B	$0.020	4.76%
3B $\to$ 70B	$0.096	25.58%
8B $\to$ 70B	$0.089	31.01%

Appendix

Table 13: Estimated cost for each setting.

Model	Accuracy	F1	AUROC
3B	69.16%	62.76%	75.17%
8B	76.50%	69.21%	81.78%

Appendix

Table 14: Performance of

P(IK)

classifier of LLaMA 3B and 8B models.

8B $\to$ 70B
Setting	Accuracy	PD	Reduced CC
$P(T)\geq 95\%$	0.8350	-0.08%	33.25%
$P(T)\geq 90\%$	0.8322	-0.42%	36.46%
$P(T)\geq 70\%$	0.8287	-0.84%	38.56%
$P(T)\geq 50\%$	0.8266	-1.09%	39.33%
8B $\to$ 70B (w/o $P(IK)$ )

Appendix

Table 15: Model Performance Comparison for LLaMA 8B

\to

70B.

Cost-Aware Inference

Cost-aware inference allocates models, computation, and serving resources to meet quality goals at lower latency, energy use, and monetary cost.

See all 280 papers

Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages
Oct 16, 2026Md Sabbir Hossen, Anichur Rahman, Pabon Shaha +3LLM Routing LLM Inference Acceleration
Bangladesh University, Bangladesh · Georgia Southern University, USA · University of Southern Mississippi, USA
Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware Inference Small Language Models
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago
Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
Jun 25, 2026Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana +11LLM Serving Cost-Aware Inference
ADAPT Centre, Trinity College Dublin · Huawei Research

Oct 16, 2026J/K move · Enter open · S save

Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages

Md Sabbir Hossen, Anichur Rahman, Pabon Shaha, Airin Afroj Aishi+2

Bangladesh University, Bangladesh · Georgia Southern University, USA · University of Southern Mississippi, USA

LLM Routing LLM Inference Acceleration Multilingual Language Model Evaluation Confidence Estimation in Language Models

Open paper AskPDF arXiv

Small-to-large language model routing can reduce inference cost by escalating difficult inputs while retaining easier ones on a smaller model. Its success, however, depends on identifying examples that benefit from escalation, and the reliability of confidence across multilingual tasks remains unclear. We present a systematic evaluation of confidence-based routing across three model families, four languages, and three task types, covering 72 model, dataset, language, and scale configurations and 42,288 predictions. We compare global confidence routing with random routing, equal-quota confidence routing, and an oracle upper bound. Confidence identifies beneficial escalations on the Belebele reading comprehension and SIB-200 topic classification datasets, with mean correction-detection AUCs of 0.612 and 0.705. At 50% escalation, it improves over random routing by 1.40--4.67 percentage points on Belebele and 2.54--5.63 points on SIB-200. In contrast, the mean AUC on the MMLU-ProX-Lite knowledge and reasoning dataset is only 0.510, and routing provides no consistent improvement over random selection because small models often remain highly confident while incorrect. Global confidence routing also creates language escalation gaps up to 30.33 percentage points, whereas equal-quota routing removes this disparity with little aggregate accuracy cost. These results show that confidence routing is task-dependent and can allocate computation unevenly across languages. Code is available at https://github.com/sabbir5622r/multilingual-llm-routing

Abstract

Cost-Aware Inference

Explore similar work

Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages

Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving