Reliable large language models (LLMs) must not only produce accurate answers but also express confidence that faithfully reflects their probability of being correct. Such calibration is essential for identifying uncertain predictions and supporting reliable decision-making in real-world deployment. Recent studies incorporate calibration learning into reinforcement learning (RL), jointly optimizing answer correctness and verbalized confidence using ground-truth correctness supervision. However, their reliance on labeled data limits their applicability in practical test-time settings, where ground-truth labels are unavailable and calibration may need to adapt to newly encountered target tasks. To address this challenge, we propose Test-Time Calibration Learning (TTCL), a label-free framework that jointly adapts reasoning accuracy and verbalized confidence directly on unlabeled target-task data. Specifically, TTCL derives self-supervision signals for both correctness and calibration from multiple model-generated responses, enabling calibration learning at test time without ground-truth labels. Theoretical analysis further establishes TTCL as a bounded surrogate for the ideal calibration objective. Extensive experiments on mathematical reasoning and factual question answering demonstrate that TTCL consistently improves both accuracy and calibration across diverse models and tasks. On base models, TTCL achieves an average relative accuracy improvement of +40.13% and an ECE reduction of +70.80% across eight benchmarks. Moreover, TTCL can further improve both accuracy and calibration for already calibrated models under domain shift, particularly when source-domain calibration transfers poorly to target tasks. In the math-to-factQA setting, TTCL achieves an average relative accuracy gain of +20.35% and reduces ECE by +53.83%. The source code is released at https://github.com/tmlr-group/TTCL.
Figures & tables
Figure 1: Overview of our work : TTCL enables label-free calibration learning directly on unlabeled target-task data, consistently improving reasoning accuracy while reducing calibration error across models and tasks, with confidence progressively aligning with accuracy during test-time learning.
Figure 2: Calibration performance of Qwen3-8B and its RLCR-calibrated model on DAPO-Math-14K. Left: Base Qwen3-8B exhibits over-confidence on both AMC and NQ, with predicted confidence exceeding empirical accuracy. Right: RLCR on DAPO-Math-14K transfers well to AMC but remains limited on NQ, where further TTCL adaptation markedly improves calibration.
Figure 3: Illustration of TTCL method : Given an unlabeled question, TTCL samples multiple rollouts to construct label-free supervision: the most supported answer serves as the pseudo label, while the support of each answer provides its confidence target. A target cache stabilizes these signals and is updated via EMA across epochs. Correctness and calibration rewards are optimized separately, with the former updating reasoning and answer tokens and the latter updating only confidence tokens.
Methods
MATH500
AMC
AIME24
AIME25
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
Qwen3-4B
Base Model
80.6
0.155
0.670
59.38
0.383
0.703
20.42
0.739
0.652
16.25
0.807
0.692
w/ TTCL
86.6
0.029
0.866
77.81
0.091
0.910
28.33
0.065
0.856
20.42
0.261
0.878
# Improv
+7.44%
+81.52%
+29.28%
+31.05%
+76.25%
+29.52%
+38.78%
+91.26%
+31.15%
+25.64%
+67.68%
+27.00%
Qwen3-8B
Table 1: Performance of TTCL for base models on mathematical reasoning benchmarks. Improved results after TTCL are highlighted in bold , with positive relative improvements shown in red color. AUROC is not reported for Llama-3.1-8B-Instruct on AIME25 because its accuracy is 0.
Methods
SimpleQA
NQ
HotpotQA
TriviaQA
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
Qwen3-4B
Base Model
1.40
0.912
0.481
8.63
0.859
0.705
9.56
0.831
0.703
15.63
0.789
0.660
w/ TTCL
2.40
0.081
0.731
10.63
0.155
0.848
10.06
0.367
0.816
17.00
0.257
0.766
# Improv
+71.43%
+91.16%
+51.96%
+23.25%
+81.95%
+20.32%
+5.20%
+55.82%
+16.13%
+8.80%
+67.41%
+16.08%
Qwen3-8B
Table 2: Performance of TTCL for base models on factual QA benchmarks. Improved results after TTCL are highlighted in bold , with positive relative improvements shown in red .
Math-to-Math
MATH500
AMC
AIME24
AIME25
AIME26
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
RLCR-calibrated Qwen3-4B on DAPO-Math-14K
Base Model
80.6
0.155
0.670
59.38
0.383
0.703
20.42
0.739
0.652
16.25
0.807
0.692
15.42
0.813
0.606
RLCR Model
89.0
0.158
0.867
79.38
0.131
0.865
29.17
0.291
0.881
24.17
0.311
0.888
23.33
0.348
0.823
w/ TTCL
90.4
0.026
0.924
88.13
0.101
0.902
42.08
0.096
0.933
25.00
0.122
0.910
24.58
0.189
0.839
# Improv
+1.57%
+83.37%
+6.67%
+11.02%
+22.76%
+4.31%
+44.29%
+67.18%
+5.93%
+3.45%
+60.57%
+2.52%
+5.36%
+45.52%
+2.02%
Table 3: In-domain performance of TTCL on RLCR-calibrated models. Improved results are highlighted in bold , with positive and negative relative gains shown in red and green respectively.
Math-to-FactQA
SimpleQA
ChineseSimpleQA
NQ
HotpotQA
TriviaQA
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
RLCR-calibrated Qwen3-4B on DAPO-Math-14K
Base Model
1.40
0.912
0.481
12.13
0.819
0.658
8.63
0.859
0.705
9.56
0.831
0.703
15.63
0.789
0.660
RLCR Model
2.00
0.623
0.662
13.00
0.600
0.730
11.00
0.651
0.732
11.13
0.560
0.753
19.38
0.551
0.713
w/ TTCL
2.70
0.277
0.747
14.03
0.307
0.798
12.81
0.228
0.867
13.50
0.293
0.850
20.38
0.290
0.812
# Improv
+35.00%
+55.61%
+12.84%
+7.95%
+48.83%
+9.18%
+16.48%
+65.01%
+18.47%
+21.35%
+47.62%
+12.87%
+5.16%
+47.34%
+13.91%
Table 4: Out-of-domain adaptation of TTCL on RLCR-calibrated models. Improved results after TTCL are highlighted in bold , with positive relative improvements shown in red .
Table 5: Performance comparison with existing label-free calibration baselines. The best results among label-free methods are highlighted in bold . RLCR is included as a label-leakage reference. AUROC is not reported for Llama-3.1-8B-Instruct on AIME25 because its accuracy is 0.
Figure 9
Model
Method
AMC23
AIME24
NQ
HotpotQA
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
ACC ↑
ECE ↓
AUROC ↑
Qwen3-4B
Pseudo-RLCR
74.38
0.158
0.698
26.67
0.595
0.574
10.94
0.836
0.688
11.00
0.853
0.554
TTCL
77.81
0.091
0.910
28.33
0.065
0.856
10.63
0.155
0.848
10.50
0.244
0.744
Qwen3-8B
Pseudo-RLCR
72.50
0.249
0.735
31.25
0.450
0.744
15.50
0.751
0.553
14.00
0.847
0.553
TTCL
80.63
0.168
0.844
30.83
0.120
0.882
14.81
0.190
0.752
15.31
0.330
0.798
Table 7: Performance of binary pseudo-correctness as confidence supervision vs. TTCL confidence supervision. Pseudo-RLCR uses the majority-vote pseudo-label for both correctness and confidence supervision signals, whereas TTCL uses answer frequency as the confidence supervision.
Figure 11
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
MATH500
AMC
AIME24/AIME25/AIME26
SimpleQA
ChineseSimpleQA
NQ/HotpotQA/TriviaQA
Training epochs
10
40
40
10
10
10
Training batch size
32
8
8
64
128
32
Training Temperature
0.7
Rollouts G
32
Max prompt length
1,024
Max response length
4,096
Appendix
Table 8: Detailed implementation configurations.
Task
Dataset Name
# Data Size
Mathematics
MATH500 ( Lightman et al., 2024 )
500
AMC ( Yang et al., 2024 )
40
AIME24 ( Zhang and Math-AI, 2024 )
30
AIME25 ( Zhang and Math-AI, 2025 )
30
AIME26 ( Zhang and Math-AI, 2026 )
30
DAPO-Math-14K ( Yu et al., 2025 )
14,109
Appendix
Table 9: Statistics of benchmarks used in our experiments.
Methods
MATH500
AMC
AIME24
AIME25
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
Qwen3-4B
Base Model
80.6
0.184
0.984
59.38
0.387
0.984
20.42
0.731
0.951
16.25
0.790
0.979
w/ TTCL
86.6
0.071
0.881
77.81
0.131
0.709
28.33
0.144
0.325
20.42
0.185
0.470
# Improv
+7.44%
+61.48%
-
+31.05%
+66.15%
-
+38.78%
+80.29%
-
+25.64%
+76.58%
-
Qwen3-8B
Appendix
Table 10: Brier score and mean confidence of TTCL for base models on mathematical reasoning benchmarks. Improved results after TTCL are highlighted in bold , with positive relative improvements shown in red . As mean confidence is not a metric for which higher or lower values are better, we do not report its relative improvement, and it only serves as a reference alongside other metrics.
Methods
SimpleQA
NQ
HotpotQA
TriviaQA
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
Qwen3-4B
Base Model
1.40
0.875
0.927
8.63
0.855
0.961
9.56
0.795
0.927
15.63
0.772
0.954
w/ TTCL
2.40
0.041
0.095
10.63
0.176
0.420
10.06
0.198
0.349
17.00
0.099
0.191
# Improv
+71.43%
+95.36%
-
+23.25%
+79.41%
-
+5.20%
+75.06%
-
+8.80%
+87.22%
-
Qwen3-8B
Appendix
Table 11: Brier score and mean confidence of TTCL for base models on factual QA benchmarks. Improved results after TTCL are highlighted in bold , with positive relative improvements shown in red color. As mean confidence is not a metric for which higher or lower values are better, we do not report its relative improvement and include it only as a reference alongside the other metrics.
Math-to-Math
MATH500
AMC
AIME24
AIME25
AIME26
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
RLCR-calibrated Qwen3-4B on DAPO-Math-14K
Base Model
80.6
0.184
0.984
59.38
0.387
0.984
20.42
0.731
0.951
16.25
0.790
0.979
15.42
0.803
0.974
RLCR Model
89.0
0.090
0.772
79.38
0.134
0.701
29.17
0.228
0.557
24.17
0.238
0.566
23.33
0.257
0.568
w/ TTCL
90.4
0.069
0.913
88.13
0.078
0.804
42.08
0.160
0.453
25.00
0.112
0.360
24.58
0.152
0.397
# Improv
+1.57%
+23.76%
-
+11.02%
+41.66%
-
+44.29%
+29.60%
-
+3.45%
+52.98%
-
+5.36%
+40.71%
-
Appendix
Table 12: In-domain performance of Brier score and mean confidence of TTCL on RLCR-calibrated models. Improved results are highlighted in bold , with positive relative gains shown in red . Mean confidence serves as a reference for analysis alongside other metrics. Since neither higher nor lower mean confidence is inherently better, we do not report its relative improvement.
Math-to-FactQA
SimpleQA
ChineseSimpleQA
NQ
HotpotQA
TriviaQA
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
ACC ↑
Brier ↓
Conf
RLCR-calibrated Qwen3-4B on DAPO-Math-14K
Base Model
1.40
0.875
0.927
12.13
0.801
0.950
8.63
0.855
0.961
9.56
0.795
0.927
15.63
0.772
0.954
RLCR Model
2.00
0.431
0.643
13.00
0.472
0.730
11.00
0.518
0.762
11.13
0.413
0.671
19.38
0.456
0.745
w/ TTCL
2.70
0.134
0.304
14.03
0.204
0.447
12.81
0.159
0.356
13.50
0.176
0.428
20.38
0.207
0.494
# Improv
+35.00%
+68.92%
-
+7.95%
+56.77%
-
+16.48%
+69.35%
-
+21.35%
+57.56%
-
+5.16%
+54.55%
-
Appendix
Table 13: Out-of-domain performance of Brier score and mean confidence of TTCL on RLCR-calibrated models. Improved results are highlighted in bold , with positive relative gains shown in red . Mean confidence serves as a reference for analysis alongside other metrics. Since neither higher nor lower mean confidence is inherently better, we do not report its relative improvement.
Figure 7: Visualization of calibration effects before and after TTCL on base models for mathematical benchmarks. TTCL effectively calibrates models to the target benchmarks, substantially narrowing the gap between confidence and empirical accuracy. Blue bars denote empirical accuracy within each confidence bin, while light blue bars indicate the deviation from perfect calibration. For Llama-3.1-8B-Instruct on AIME25, no accuracy bars is shown because its empirical accuracy is 0.
Figure 8: Visualization of calibration effects before and after TTCL on base models for factual QA benchmarks. TTCL effectively calibrates models to the target benchmarks, substantially narrowing the gap between confidence and empirical accuracy. Blue bars denote empirical accuracy within each confidence bin, while light blue bars indicate the deviation from perfect calibration.
Figure 9: Visualization of in-domain calibration effects before and after TTCL on math-calibrated models for mathematical benchmarks. TTCL calibrates models to the target benchmarks, further narrowing the gap between confidence and empirical accuracy. Blue bars denote empirical accuracy within each confidence bin, while light blue bars indicate the deviation from perfect calibration.
Figure 10: Visualization of out-of-domain calibration effects before and after TTCL on math-calibrated models for factual benchmarks. The calibration learned on the source domain transfers poorly to the target domain in both Math-to-FactQA and FactQA-to-Math settings, whereas TTCL effectively adapts the models to the target benchmarks and further narrows the gap between confidence and empirical accuracy. Blue bars denote empirical accuracy within each confidence bin, while light blue bars indicate the deviation from perfect calibration.
Figure 11: More training dynamics of TTCL on base models. The red curves show mean confidence and the cyan-blue curves show empirical accuracy during TTCL training. The shaded area indicates the confidence-accuracy gap, and the inset zooms in on the accuracy trajectory. TTCL progressively aligns confidence with empirical accuracy while improving task accuracy.
Figure 12: More training dynamics of TTCL on calibrated models. The red curves show mean confidence and the cyan-blue curves show empirical accuracy during TTCL training. The shaded area indicates the confidence-accuracy gap, and the inset zooms in on the accuracy trajectory. TTCL progressively aligns confidence with empirical accuracy while improving task accuracy.
TTCL Target
ARC
GPQA-D
IFEval
MMLU-Pro
TruthfulQA
TTCL Target
ARC
GPQA-D
IFEval
MMLU-Pro
TruthfulQA
Challenge
Easy
MC1
MC2
Challenge
Easy
MC1
MC2
Qwen3-8B
Base Model
55.20
75.34
50.00
84.62
63.70
32.80
51.60
Base Model
55.20
75.34
50.00
84.62
63.70
32.80
51.60
MATH500
55.29
74.83
51.52
85.47
64.82
33.29
51.65
SimpleQA
54.52
74.83
45.96
84.65
63.86
33.17
51.20
AMC
55.29
74.83
45.96
85.83
63.87
33.05
51.35
NQ
55.12
74.62
51.52
84.77
63.12
33.05
51.54
AIME24
54.52
74.07
48.99
86.20
62.66
32.68
50.95
HotpotQA
55.46
74.79
45.45
86.11
63.87
33.29
51.03
Appendix
Table 14: More results of TTCL-trained models on multiple other domains. These results examine whether TTCL improves calibration at the cost of model general capabilities on other domains.
TTCL Target
Avg.
Prompt Strict
Inst. Strict
Prompt Loose
Inst. Loose
TTCL Target
Avg.
Prompt Strict
Inst. Strict
Prompt Loose
Inst. Loose
Qwen3-8B
Base Model
84.62
80.04
86.45
83.36
88.61
Base Model
84.62
80.04
86.45
83.36
88.61
MATH500
85.47
81.15
87.05
84.47
89.21
SimpleQA
84.65
79.85
86.33
83.55
88.85
AMC
85.83
81.33
87.29
85.03
89.69
NQ
84.77
80.41
83.36
86.69
88.61
AIME24
86.20
82.07
87.89
85.03
89.81
HotpotQA
86.11
82.07
85.03
87.65
89.69
AIME25
84.98
80.59
86.57
83.92
88.85
TriviaQA
84.56
80.22
83.18
86.33
88.49
Appendix
Table 15: Detailed IFEval results of TTCL-trained models. IFEval reports four metrics: prompt-level strict accuracy, prompt-level loose accuracy, instruction-level strict accuracy, and instruction-level loose accuracy. In previous tables, we report the average of these four metrics. Here, we provide the detailed results as a supplement to Table 6 and Table 14 .
Figure 13: Reward hacking without the target cache. Four AIME24 examples in which the model’s rollouts increasingly concentrate on an incorrect answer while mean confidence rises toward 1.0. The blue lines show the frequency-based probability of the most frequent answer, and the red lines show mean confidence. When supervision signals are derived from the the rollout group of current policy, increasing agreement on a wrong answer raises its confidence target, leading to reward hacking.
Label-free RL Methods
MATH500
AMC
AIME24
AIME25
AIME26
ACC ↑
ECE ↓
Conf
ACC ↑
ECE ↓
Conf
ACC ↑
ECE ↓
Conf
ACC ↑
ECE ↓
Conf
ACC ↑
ECE ↓
Conf
Qwen3-4B-Base
Base Model
58.8
0.347
0.935
38.44
0.541
0.926
10.00
0.826
0.926
4.17
0.876
0.918
6.25
0.861
0.924
Self-Certainty ( Zhao et al., 2026 )
61.6
0.342
0.958
41.25
0.540
0.953
7.50
0.877
0.952
5.00
0.894
0.944
5.00
0.900
0.950
Entropy ( Prabhudesai et al., 2025 )
77.2
0.206
0.978
51.88
0.462
0.981
14.58
0.833
0.978
8.75
0.890
0.978
9.58
0.885
0.980
Majority-voting ( Zuo et al., 2025 )
72.6
0.230
0.956
43.13
0.509
0.940
9.58
0.831
0.927
4.58
0.876
0.921
5.83
0.864
0.923
Appendix
Table 16: Calibration results of existing label-free RL methods. We evaluate several label-free RL methods using the released checkpoints from Zhang et al. (2026) , and report accuracy, ECE, and mean confidence values. Mean confidence values that are substantially higher than the corresponding empirical accuracy, indicating over-confidence of calibration, are highlighted in red .
While test-time scaling has enabled large language models to solve highly difficult tasks, state-of-the-art results come at exorbitant compute costs. These inefficiencies can be attributed to the miscalibration of post-trained language models, and the lack of calibration in popular sampling techniques. Here, we present Online Reasoning Calibration (ORCA), a framework for calibrating the sampling process that draws upon conformal prediction and test-time training. Specifically, we introduce a meta-learning procedure that updates the calibration module for each input. This allows us to provide valid confidence estimates under distributional shift, e.g. in thought patterns that occur across different stages of reasoning, or in prompt distributions between model development and deployment. ORCA not only provides theoretical guarantees on conformal risks, but also empirically shows higher efficiency and generalization across different reasoning tasks. At risk level δ=0.1, ORCA improves Qwen2.5-32B efficiency on in-distribution tasks with savings up to 47.5% with supervised labels and 40.7% with self-consistency labels. Under zero-shot out-of-domain settings, it improves MATH-500 savings from 24.8% of the static calibration baseline to 67.0% while maintaining a low empirical error rate, and the same trend holds across model families and downstream benchmarks. Our code is publicly available at https://github.com/wzekai99/ORCA.
Cai Zhou, Zekai Wang, Menghua Wu +6
Department of Electrical Engineering and Computer Science (MIT EECS) · Computer Science and Artificial Intelligence Laboratory (MIT CSAIL) · Laboratory for Information and Decision Systems (MIT LIDS) +1
Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin (RLCM), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation.
Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks. However, prevailing RL reward designs typically prioritize response correctness, neglecting to incentivize models to express their confidence accurately. This leads to a critical problem: performance gains are often accompanied by poor calibration between confidence and accuracy, misleading models to overconfidently hallucinate when uncertain. To address this limitation, we propose Correctness and Confidence Calibration Reinforcement Learning (C3RL), a novel RL algorithm integrating correctness, calibration and dataset-informed reference accuracy rewards together. Comprehensive evaluation across 8 text and multimodal datasets demonstrates that C3RL enhances calibration without sacrificing accuracy, outperforming the current state-of-the-art method in both performance and calibration metrics. Utilizing the well-calibrated verbalized confidence from C3RL, we further introduce Confidence-based Adaptive Test Time Scaling (CAS), an adjustable inference-time strategy that allocates computational resources based on response confidence. Experiments show that CAS surpasses majority voting on both in-domain and out-of-domain datasets while reducing the inference budget by up to 12.33 times. We believe the synergy of C3RL and CAS paves the way for deploying more reliable and resource-efficient LLMs. The code, data and models will be released.
Xuqing Yang, Yi Yuan, Shanzhe Lei +1
Shanghai Jiao Tong University · Southeast University · Shanghai AI Laboratory