Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning

Chuang Zhang; Zizhen Zhu; Yihao Wei; Bing Tian; Junyi Liu; Henan Wang; Xavier Wang; Yaxiao Liu

cs.CLMar 4, 2026

Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning

Authors: Chuang Zhang, Zizhen Zhu, Yihao Wei, Bing Tian, Junyi Liu, Henan Wang, Xavier Wang, Yaxiao Liu

Organizations: Amazon Web Services · Tsinghua University

Abstract

Large language models (LLMs) demonstrate superior reasoning capabilities compared to small language models (SLMs), but incur substantially higher costs. We propose COllaborative REAsoner (COREA), a system that cascades an SLM with an LLM to achieve a balance between accuracy and cost in complex reasoning tasks. COREA first attempts to answer questions using the SLM, which outputs both an answer and a verbalized confidence score. Questions with confidence below a predefined threshold are deferred to the LLM for more accurate resolution. We introduce a reinforcement learning-based training algorithm that aligns the SLM's confidence through an additional confidence calibration reward. Extensive experiments demonstrate that our method jointly improves the SLM's reasoning ability and confidence calibration across diverse datasets and model backbones. Compared to using the LLM alone, COREA reduces cost by 21.5% and 16.8% on out-of-domain math and non-math datasets, respectively, with only an absolute pass@1 drop within 2%.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Architecture of COREA : COREA cascades an SLM with an LLM. For each query, the SLM is prompted to generate reasoning steps, an answer, and a confidence score. LLM is invoked only when the SLM’s confidence is less than the predefined threshold. (b) Confidence calibration of the SLM: The SLM is trained using GRPO with a multi-objective reward function to improve its reasoning ability and calibrate its confidence.

System	DeepMath500			OOD Math			OOD Non-Math
System	Pass@1% ↑	Avg Cost ↓	LLM%	Pass@1% ↑	Avg Cost ↓	LLM%	Pass@1% ↑	Avg Cost ↓	LLM%
Standalone
SLM	42.7	4423	0.0	60.4	2622	0.0	45.2	2004	0.0
RLVR-SLM	57.6	2511	0.0	71.8	1928	0.0	62.4	1438	0.0
Brier-SLM	55.7	3490	0.0	72.7	2322	0.0	63.5	1708	0.0
L1-SLM*	56.6	3766	0.0	72.1	2589	0.0	64.7	1902	0.0

Table 1: Performance comparison across different systems and datasets. Pass@1, average cost per question (Avg Cost), and LLM usage percentage (LLM%) are reported. OOD Math includes Math500, GSM8K, and OlympiadBench. OOD Non-Math includes GPQA and CommonsenseQA. In all collaborative systems, the Baseline LLM’s Pass@1 serves as the confidence threshold. Values in parentheses indicate the difference from the Baseline LLM. COREA (L1-SLM-Verb) utilizes L1-SLM and verbalized confidence. * indicates our method.

Figure 2: Results on DeepMath-500: (a) Pass@1, (b) average cost, and (c) LLM usage percentage versus confidence threshold; (d) Pass@1 versus average cost for different systems. The vertical dashed lines in (a)-(c) mark the Baseline LLM’s Pass@1 (0.69).

Panel of Figure 3 — Figure 3: Training process of Qwen2.5-7BI with different reward configurations: RLVR, Brier (RLCC with Brier score), and L1 (RLCC with L1 confidence reward) on DeepMath500, with 8 sampled answers per question.

Method	Pass@1% ↑	ECE ↓	AUROC ↑
SLM-Verb	$52.8\pm 0.9$	$0.45\pm 0.01$	$0.50\pm 0.01$
RLVR-SLM-Verb	$66.9\pm 0.7$	$0.30\pm 0.01$	$0.50\pm 0.01$
RLVR-SLM-AvgProb	$66.9\pm 0.7$	$0.28\pm 0.01$	$0.64\pm 0.01$
RLVR-SLM-Probe	$66.9\pm 0.7$	$0.17\pm 0.01$	$\mathbf{0.72\pm 0.01}$
Router+RLVR-SLM	$66.9\pm 0.7$	$0.20\pm 0.01$	$0.56\pm 0.01$
Brier-SLM-Verb	$67.5\pm 0.7$	$0.17\pm 0.01$	$\mathbf{0.72\pm 0.01}$

Table 2: Comparison of different methods on accuracy and calibration metrics, averaged across all datasets. Here, the metrics are only evaluated on the SLM.

Figure 4: Comparison of Pass@1 and ECE across several models trained with different reward configurations, averaged across all datasets. Detailed results on each dataset are provided in Appendix E .

Conf. Reward	Pass@1% $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$
L1	$67.8\pm 0.7$	$0.12\pm 0.01$	$0.72\pm 0.01$
Sample_L1	$\mathbf{68.3\pm 0.7}$	$0.16\pm 0.01$	$0.68\pm 0.01$
L2	$67.5\pm 0.7$	$0.31\pm 0.01$	$0.62\pm 0.01$
Brier	$67.5\pm 0.7$	$0.17\pm 0.01$	$0.72\pm 0.01$
KL	$66.5\pm 0.7$	$0.15\pm 0.01$	$0.69\pm 0.01$
Sample_KL	$67.1\pm 0.7$	$\mathbf{0.11\pm 0.01}$	$\mathbf{0.74\pm 0.01}$

Table 3: Comparison of different confidence reward functions on accuracy and calibration metrics, averaged across all datasets.

Rollout Size	Pass@1% $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$
4	$67.2\pm 0.8$	$\mathbf{0.10\pm 0.01}$	$0.68\pm 0.01$
8	$67.8\pm 0.7$	$0.12\pm 0.01$	$\mathbf{0.72\pm 0.01}$
16	$66.7\pm 0.7$	$\mathbf{0.10\pm 0.01}$	$0.69\pm 0.01$
32	$\mathbf{68.3\pm 0.5}$	$0.17\pm 0.01$	$0.65\pm 0.01$

Table 4: Comparison of different rollout sizes with L1 confidence reward on accuracy and calibration metrics, averaged across all datasets.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Question and Response of Qwen2.5-7BI.

Panel of Figure 6 — Figure 6: Response of Qwen2.5-7BI continued.

Panel of Figure 7 — Figure 7: Response of RLVR-Qwen2.5-7BI.

Panel of Figure 8 — Figure 8: Response of L1-Qwen2.5-7BI (our approach).

System	Pass@1%	SLM Cost	LLM Cost	Total Cost	LLM%
Standalone
SLM	42.7	4423	0	4423	0
RLVR-SLM	57.6	2511	0	2511	0
Brier-SLM	55.7	3490	0	3490	0
L1-SLM*	56.6	3766	0	3766	0
Baseline LLM	69.0	0	14882	14882	100

Appendix

Table 5: Performance comparison across different systems on DeepMath500. Pass@1, SLM Cost per question, LLM Cost per question, Total Cost per question, and LLM usage percentage (LLM%) are reported. In all collaborative systems, the Baseline LLM’s Pass@1 serves as the confidence threshold. Values in parentheses indicate the difference from the Baseline LLM. COREA (L1-SLM-Verb) utilizes L1-SLM and verbalized confidence. * indicates our method.

System	Pass@1%	SLM Cost	LLM Cost	Total Cost	LLM%
Standalone
SLM	54.1	3113	0	3113	0
RLVR-SLM	70.5	2125	0	2125	0
Brier-SLM	74.4	2627	0	2627	0
L1-SLM*	72.4	3020	0	3020	0
Baseline LLM	81.2	0	11499	11499	100

Appendix

Table 6: Performance comparison across different systems on Math500. Pass@1, SLM Cost per question, LLM Cost per question, Total Cost per question, and LLM usage percentage (LLM%) are reported. In all collaborative systems, the Baseline LLM’s Pass@1 serves as the confidence threshold. Values in parentheses indicate the difference from the Baseline LLM. COREA (L1-SLM-Verb) utilizes L1-SLM and verbalized confidence. * indicates our method.

System	Pass@1%	SLM Cost	LLM Cost	Total Cost	LLM%
Standalone
SLM	81.7	1388	0	1388	0
RLVR-SLM	90.9	1348	0	1348	0
Brier-SLM	90.9	1369	0	1369	0
L1-SLM*	90.3	1487	0	1487	0
Baseline LLM	95.6	0	6073	6073	100

Appendix

Table 7: Performance comparison across different systems on GSM8K. Pass@1, SLM Cost per question, LLM Cost per question, Total Cost per question, and LLM usage percentage (LLM%) are reported. In all collaborative systems, the Baseline LLM’s Pass@1 serves as the confidence threshold. Values in parentheses indicate the difference from the Baseline LLM. COREA (L1-SLM-Verb) utilizes L1-SLM and verbalized confidence. * indicates our method.

System	Pass@1%	SLM Cost	LLM Cost	Total Cost	LLM%
Standalone
SLM	23.4	4671	0	4671	0
RLVR-SLM	35.2	2916	0	2916	0
Brier-SLM	36.0	3960	0	3960	0
L1-SLM*	36.5	4424	0	4424	0
Baseline LLM	47.0	0	17167	17167	100

Appendix

Table 8: Performance comparison across different systems on OlympiadBench. Pass@1, SLM Cost per question, LLM Cost per question, Total Cost per question, and LLM usage percentage (LLM%) are reported. In all collaborative systems, the Baseline LLM’s Pass@1 serves as the confidence threshold. Values in parentheses indicate the difference from the Baseline LLM. COREA (L1-SLM-Verb) utilizes L1-SLM and verbalized confidence. * indicates our method.

System	Pass@1%	SLM Cost	LLM Cost	Total Cost	LLM%
Standalone
SLM	21.6	3927	0	3927	0
RLVR-SLM	29.5	2451	0	2451	0
Brier-SLM	32.8	3282	0	3282	0
L1-SLM*	34.2	3775	0	3775	0
Baseline LLM	43.4	0	13420	13420	100

Appendix

Table 9: Performance comparison across different systems on GPQA. Pass@1, SLM Cost per question, LLM Cost per question, Total Cost per question, and LLM usage percentage (LLM%) are reported. In all collaborative systems, the Baseline LLM’s Pass@1 serves as the confidence threshold. Values in parentheses indicate the difference from the Baseline LLM. COREA (L1-SLM-Verb) utilizes L1-SLM and verbalized confidence. * indicates our method.

System	Pass@1%	SLM Cost	LLM Cost	Total Cost	LLM%
Standalone
SLM	53.9	1298	0	1298	0
RLVR-SLM	74.5	1067	0	1067	0
Brier-SLM	74.8	1130	0	1130	0
L1-SLM*	75.9	1215	0	1215	0
Baseline LLM	81.7	0	5229	5229	100

Appendix

Table 10: Performance comparison across different systems on CommonsenseQA. Pass@1, SLM Cost per question, LLM Cost per question, Total Cost per question, and LLM usage percentage (LLM%) are reported. In all collaborative systems, the Baseline LLM’s Pass@1 serves as the confidence threshold. Values in parentheses indicate the difference from the Baseline LLM. COREA (L1-SLM-Verb) utilizes L1-SLM and verbalized confidence. * indicates our method.

System	DeepMath500			OOD Math			OOD Non-Math
System	Pass@1% ↑	Avg Cost ↓	LLM%	Pass@1% ↑	Avg Cost ↓	LLM%	Pass@1% ↑	Avg Cost ↓	LLM%
Standalone
SLM	25.8	3422	0.0	41.5	2435	0.0	42.8	1558	0.0
L1-SLM	26.5	1311	0.0	35.3	1273	0.0	35.7	449	0.0
Baseline LLM	69.0	69450	100.0	79.6	47430	100.0	71.4	34663	100.0
Collaborative

Appendix

Table 11: Performance comparison of 1.5B SLM + 32B LLM pairing across different systems and datasets. Pass@1, average cost per question (Avg Cost), and LLM usage percentage (LLM%) are reported. OOD Math includes Math500, GSM8K, and OlympiadBench. OOD Non-Math includes GPQA and CommonsenseQA. In all collaborative systems, the Baseline LLM’s Pass@1 serves as the confidence threshold. Values in parentheses indicate the difference from the Baseline LLM. * indicates our method.

Panel of Figure 9 — Figure 9: Comparison of Pass@1 and ECE across several models trained with different reward configurations on DeepMath500, Math500, GSM8K, OlympiadBench, GPQA, OlympiadBench, respectively.

Method	DeepMath500			Math500			GSM8K
Method	Pass@1%	ECE	AUROC	Pass@1%	ECE	AUROC	Pass@1%	ECE	AUROC
SLM	41.7±1.5	0.57±0.02	0.47±0.02	52.8±1.2	0.47±0.01	0.48±0.02	81.3±0.5	0.19±0.01	0.52±0.00
RLVR	58.2±0.8	0.40±0.01	0.49±0.01	70.6±0.8	0.28±0.01	0.50±0.01	91.0±0.3	0.07±0.00	0.49±0.01
L1	56.3±0.9	0.11±0.01	0.77±0.01	71.5±0.6	0.11±0.01	0.81±0.01	90.5±0.3	0.07±0.00	0.78±0.01
Sample_L1	58.4±0.8	0.17±0.01	0.75±0.01	73.6±0.9	0.07±0.01	0.81±0.01	91.1±0.3	0.05±0.00	0.70±0.01
L2	54.8±1.1	0.22±0.02	0.63±0.01	72.6±0.9	0.39±0.01	0.67±0.01	89.7±0.2	0.53±0.00	0.67±0.01

Appendix

Table 12: Comparison of different confidence reward functions on accuracy and calibration metrics for DeepMath500, Math500, and GSM8K. Results are reported as mean ± standard deviation.

Method	OlympiadBench			GPQA			CommonsenseQA
Method	Pass@1%	ECE	AUROC	Pass@1%	ECE	AUROC	Pass@1%	ECE	AUROC
SLM	23.2±1.0	0.76±0.01	0.48±0.01	21.2±1.0	0.71±0.01	0.49±0.03	54.7±0.9	0.39±0.01	0.52±0.01
RLVR	34.6±0.9	0.63±0.01	0.49±0.01	29.0±1.9	0.66±0.02	0.50±0.03	74.6±0.6	0.19±0.01	0.52±0.01
L1	36.9±1.0	0.11±0.01	0.77±0.01	34.5±1.2	0.15±0.02	0.54±0.02	75.9±0.6	0.19±0.01	0.63±0.01
Sample_L1	37.5±1.4	0.22±0.01	0.74±0.01	32.3±0.8	0.40±0.01	0.56±0.02	75.8±0.6	0.18±0.01	0.60±0.00
L2	36.5±0.5	0.10±0.00	0.60±0.01	32.2±1.6	0.04±0.01	0.54±0.02	76.9±0.9	0.28±0.01	0.60±0.01

Appendix

Table 13: Comparison of different confidence reward functions on accuracy and calibration metrics for OlympiadBench, GPQA, and CommonsenseQA. Results are reported as mean ± standard deviation.

Rollout Size	DeepMath500			Math500			GSM8K
Rollout Size	Pass@1%	ECE	AUROC	Pass@1%	ECE	AUROC	Pass@1%	ECE	AUROC
4	55.8±1.0	0.12±0.01	0.67±0.01	71.2±1.0	0.13±0.01	0.74±0.02	91.7±0.4	0.07±0.01	0.73±0.02
8	56.3±0.9	0.11±0.01	0.77±0.01	71.5±0.6	0.11±0.01	0.81±0.01	90.5±0.3	0.07±0.00	0.78±0.01
16	54.7±1.1	0.09±0.01	0.69±0.01	68.9±1.3	0.08±0.01	0.74±0.01	91.3±0.3	0.07±0.00	0.74±0.00
32	57.2±0.9	0.19±0.01	0.73±0.01	72.1±0.5	0.11±0.01	0.78±0.02	91.7±0.2	0.06±0.00	0.62±0.01

Appendix

Table 14: Comparison of different rollout sizes with L1 confidence reward on accuracy and calibration metrics for DeepMath500, Math500, and GSM8K. Results are reported as mean ± standard deviation.

Rollout Size	OlympiadBench			GPQA			CommonsenseQA
Rollout Size	Pass@1%	ECE	AUROC	Pass@1%	ECE	AUROC	Pass@1%	ECE	AUROC
4	35.2±0.9	0.07±0.01	0.63±0.02	29.0±1.1	0.20±0.01	0.56±0.02	75.4±0.9	0.11±0.01	0.69±0.01
8	36.9±1.0	0.11±0.01	0.77±0.01	34.5±1.2	0.15±0.02	0.54±0.02	75.9±0.6	0.19±0.01	0.63±0.01
16	33.3±1.0	0.08±0.01	0.69±0.01	31.7±1.2	0.24±0.01	0.55±0.02	75.6±0.5	0.09±0.01	0.66±0.01
32	34.2±0.7	0.20±0.01	0.80±0.01	34.0±1.4	0.42±0.02	0.53±0.02	77.5±0.3	0.21±0.00	0.55±0.01

Appendix

Table 15: Comparison of different rollout sizes with L1 confidence reward on accuracy and calibration metrics for OlympiadBench, GPQA, and CommonsenseQA. Results are reported as mean ± standard deviation.

Language Model Cascading

See all 21 papers

Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning
Apr 26, 2026Zichuan Fu, Xian Wu, Guojing Li +7Efficient Language Model Reasoning LLM Inference Acceleration
City University of Hong Kong · Tencent Jarvis Lab · Renmin University of China +1
Offline Guidance, Online Reasoning: Reusing LLM Feedback for Small Language Models
Sep 30, 2026Bohan Zhang, Linan Yue, Weibo Gao +3Efficient Language Model Reasoning Small Language Models
Southeast University · Hong Kong Polytechnic University · ZTE Corporation
PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
Jul 22, 2026Niqi Lyu, Pengtao Shi, Wei Qiu +4RL for Language Model Reasoning Cost-Aware Inference
Pyromind Dynamics Inc.

Apr 26, 2026J/K move · Enter open · S save

Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning

Zichuan Fu, Xian Wu, Guojing Li, Yejing Wang+6

City University of Hong Kong · Tencent Jarvis Lab · Renmin University of China+1

Efficient Language Model Reasoning LLM Inference Acceleration Multi-LLM Collaboration LLM Code Generation LLM Reasoning

Open paper AskPDF arXiv

Recent advancements in large language models (LLMs) have catalyzed the rise of reasoning-intensive inference paradigms, where models perform explicit step-by-step reasoning before generating final answers. While such approaches improve answer quality and interpretability, they incur substantial computational overhead due to the prolonged generation sequences. In this paper, we propose Tandem, a novel collaborative framework that synergizes large and small language models (LLMs and SLMs) to achieve high-quality reasoning with significantly reduced computational cost. Specifically, the LLM serves as a strategic coordinator, efficiently generating a compact set of critical reasoning insights. These insights are then used to guide a smaller, more efficient SLM in executing the full reasoning process and delivering the final response. To balance efficiency and reliability, Tandem introduces a cost-aware termination mechanism that adaptively determines when sufficient reasoning guidance has been accumulated, enabling early stopping of the LLM's generation. Experiments on mathematical reasoning and code generation benchmarks demonstrate that Tandem reduces computational costs by approximately 40% compared to standalone LLM reasoning, while achieving superior or competitive performance. Furthermore, the sufficiency classifier trained on one domain transfers effectively to others without retraining. The code is available at: https://github.com/Applied-Machine-Learning-Lab/ACL2026_Tandem.

Abstract

Language Model Cascading

Explore similar work

Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning

Offline Guidance, Online Reasoning: Reusing LLM Feedback for Small Language Models

PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference