Test-Time Confidence Adaptation under RL-Induced Trajectory Shift

Zhenzhen Ren; Xiaoqing Zhang; GuoBiao Li; Bowen Fang; Yanxi Wang; Xinpeng Zhang; Ke Han

cs.AIOct 26, 2026

Test-Time Confidence Adaptation under RL-Induced Trajectory Shift

Authors: Zhenzhen Ren, Xiaoqing Zhang, GuoBiao Li, Bowen Fang, Yanxi Wang, Xinpeng Zhang, Ke Han

Organizations: Fudan University · Zhongguancun Academy · Renmin University of China · Institute of Automation, Chinese Academy of Sciences ·

Abstract

Trajectory-based confidence estimation has shown promising performance for large language models (LLMs) by leveraging rich signals from their reasoning processes. However, we find that such confidence estimators can degrade substantially after agentic reinforcement learning (Agentic RL) as it induces substantial shifts in the reasoning trajectories. More importantly, despite these shifts, we uncover a reliability structure that is largely preserved across Base and RL models: $\textbf{more}$ $\textbf{stable}$ $\textbf{trajectories}$ $\textbf{remain}$ $\textbf{more}$ $\textbf{likely}$ $\textbf{to}$ $\textbf{produce}$ $\textbf{correct}$ $\textbf{answers}$ . This preserved relationship provides a transferable, label-free reliability signal for post-RL confidence adaptation. Building on this finding, we propose $\textbf{StaR-TTA}$ , a stability-guided framework for label-free test-time confidence adaptation. StaR-TTA jointly learns from correctness supervision and stability-derived reliability supervision on labeled Base trajectories, establishing the connection between trajectory stability and answer reliability. At test time, it reuses stability-induced reliability rankings from unlabeled RL trajectories to adapt confidence scores to the shifted trajectory distribution, without requiring target correctness labels. Across four QA benchmarks, StaR-TTA achieves an average AUROC of $0.721$ , with consistent improvements across unseen datasets, model families and scales, and different Agentic RL algorithms.

Figures & tables

Figure 1: (a) Agentic RL shifts reasoning trajectories and degrades uncertainty quantification (UQ). (b) Stability-correctness ordering remains highly positive after RL. (c) Stability-guided test-time adaptation restores reliable confidence estimation. Using unlabeled RL trajectories, we exploit trajectory stability to construct pairwise ordering and adapt a confidence estimator at test time with a ranking loss, leading to more reliable confidence estimates.

Method	Base AUROC $\uparrow$	RL AUROC $\uparrow$	Relative drop (%)
MICE-LR	0.741	0.567	23.5
MICE-RF	0.773	0.559	27.7
HTC	0.765	0.675	11.8
CCPS	0.745	0.491	34.1
DACA	0.659	0.656	0.5
$P(\mathrm{Sufficient})$	0.627	0.689	$-9.9$

Table 1: Base-to-RL confidence transfer on NQ.

Method	Tool control	Answer control	Trajectory replay	Recovery (%)
MICE-LR	$.567\,{\scriptscriptstyle(-.174)}$	$.514\,{\scriptscriptstyle(-.110)}$	$.563\,{\scriptscriptstyle(+.049)}$	44.5
MICE-RF	$.559\,{\scriptscriptstyle(-.215)}$	$.577\,{\scriptscriptstyle(-.087)}$	$.623\,{\scriptscriptstyle(+.045)}$	51.9
CCPS	$.491\,{\scriptscriptstyle(-.255)}$	$.498\,{\scriptscriptstyle(-.133)}$	$.513\,{\scriptscriptstyle(+.015)}$	11.1

Table 2: Controlled AUROC analysis on NQ.

Figure 2: Overview of our method. (a) Firstly, an SRM model is trained to capture the relationship between trajectory stability and correctness. Then a scorer (TCS) is optimized, followed by a calibrator (CC) to convert scores into calibrated probabilities. (b) At test time, we adapt the results of TCS to Agentic-RL trajectories. We extract stability signals from both Agentic-RL and Base trajectories and use this information to adjust the score while keeping TCS and CC fixed .

Figure 3: AUROC of sampling-stability signals for answer correctness.

	TriviaQA				HotpotQA				MuSiQue				PopQA
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
DACA	.668	.027	.223	.638	.613	.155	.243	.679	.625	.310	.229	.650	.680	.110	.244	.681
P(Sufficient)	.661	.331	.337	1.131	.636	.526	.498	1.551	.648	.643	.551	1.584	.641	.459	.446	1.382
HTC	.653	.289	.316	.989	.507	.175	.268	.738	.721	.081	.132	.423	.678	.328	.334	.937
MICE-LR	.675	.125	.232	.670	.640	.112	.234	.662	.623	.054	.135	.443	.595	.282	.321	.882
MICE-RF	.629	.046	.229	.651	.615	.048	.219	.629	.610	.064	.136	.442	.590	.119	.254	.702

Table 3: Main results of confidence estimation on the RL model’s test trajectories.

Figure 4: t-SNE visualization of the H64 hidden representations for Base and RL.

	TriviaQA				2WikiMultiHopQA				Bamboogle				NQ
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HTC	.653	.289	.316	.989	.482	.289	.337	1.333	.531	.212	.270	.909	.672	.131	.239	.737
MICE-LR	.675	.125	.232	.670	.502	.299	.346	.948	.585	.399	.379	1.027	.653	.271	.302	.837
MICE-RF	.629	.046	.229	.651	.503	.132	.268	.732	.520	.243	.289	.787	.607	.136	.250	.693
CCPS	.420	.405	.429	1.335	.453	.261	.324	1.368	.524	.220	.277	1.793	.491	.256	.320	1.791
DACA	.668	.027	.223	.638	.509	.072	.247	.686	.556	.218	.249	.692	.623	.165	.258	.710

Table 4: Cross-dataset generalization of confidence estimation.

	TriviaQA				HotpotQA				MuSiQue				PopQA
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HTC	.683	.114	.233	.669	.631	.071	.202	.596	.688	.103	.090	.313	.611	.211	.274	.896
MICE-LR	.730	.047	.202	.593	.663	.108	.208	.597	.651	.071	.085	.300	.700	.098	.218	.622
MICE-RF	.715	.029	.202	.587	.681	.097	.197	.569	.661	.069	.080	.286	.713	.103	.212	.601
CCPS	.707	.105	.210	.607	.686	.126	.201	.580	.652	.103	.087	.308	.691	.083	.213	.619
DACA	.772	.204	.237	.943	.583	.152	.234	.661	.682	.394	.265	.753	.752	.297	.319	1.076

Table 5: Cross-model generalization: Llama-3.2-3B-Instruct with PPO .

	TriviaQA				HotpotQA				MuSiQue				PopQA
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HTC	.717	.355	.353	2.684	.647	.152	.246	.710	.794	.149	.161	.477	.697	.075	.220	.630
MICE-LR	.641	.064	.220	.639	.614	.096	.236	.657	.673	.041	.140	.440	.581	.068	.242	.674
MICE-RF	.669	.036	.209	.607	.629	.087	.228	.638	.645	.061	.146	.456	.652	.059	.227	.639
CCPS	.655	.059	.202	.587	.659	.078	.224	.631	.672	.087	.147	.458	.625	.146	.253	.697
DACA	.725	.265	.268	2.005	.657	.372	.379	1.823	.697	.315	.314	1.155	.676	.389	.387	2.679

Table 6: Cross-model generalization: Qwen2.5-7B-Instruct with PPO.

	TriviaQA				HotpotQA				MuSiQue				PopQA
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HTC	.663	.321	.331	1.171	.422	.245	.274	1.039	.551	.844	.767	2.479	.636	.579	.570	2.613
MICE-LR	.640	.150	.251	.716	.624	.149	.230	.657	.567	.051	.052	.217	.524	.267	.327	.890
MICE-RF	.564	.080	.244	.681	.605	.068	.204	.593	.587	.079	.057	.234	.521	.112	.254	.701
CCPS	.425	.536	.546	4.192	.437	.259	.275	1.661	.483	.210	.104	.434	.467	.318	.354	2.919
DACA	.692	.093	.232	.655	.611	.244	.261	.716	.642	.465	.263	.718	.697	.134	.252	.696

Table 7: Generalization across RL algorithms: Qwen2.5-3B with GRPO.

Dataset	Setting	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
TriviaQA	w/o TTA	0.8018	0.0455	0.1794	0.5386
	w/ TTA	0.8000	0.0535	0.1808	0.5421
HotpotQA	w/o TTA	0.7112	0.0349	0.1997	0.5828
	w/ TTA	0.7154	0.0409	0.1996	0.5824
MuSiQue	w/o TTA	0.6296	0.1212	0.1478	0.5503
	w/ TTA	0.6721	0.1198	0.1450	0.5316

Table 8: Effect of removing TTA.

$N$	Average	Trivia	Hotpot	MuSiQue	PopQA
0	0.7035	0.8018	0.7112	0.6296	0.6714
1	0.7035	0.8018	0.7112	0.6296	0.6714
2	0.6306	0.7537	0.5125	0.6383	0.6181
4	0.6680	0.7828	0.5744	0.6671	0.6477
8	0.7089	0.7911	0.6839	0.6683	0.6924
12	0.7072	0.7914	0.6893	0.6703	0.6779

Table 9: AUROC under different TTA sample sizes.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Setting	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
TriviaQA	w/o Stability input $c^{m}$	0.7413	0.1529	0.2250	0.6408
	w/o Statistics $d^{m}$	0.7871	0.0473	0.1850	0.5526
	Ours	0.8000	0.0535	0.1808	0.5421
HotpotQA	w/o Stability input $c^{m}$	0.6170	0.1222	0.2337	0.6679
	w/o Statistics $d^{m}$	0.6992	0.0724	0.2079	0.6030
	Ours	0.7154	0.0409	0.1996	0.5824

Appendix

Table 10: Independent ablations of Scorer input features. Each ablation removes only one feature group.

Dataset	Setting	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
TriviaQA	w/o JS agreement	0.8000	0.0521	0.1816	0.5441
	Ours	0.8000	0.0535	0.1808	0.5421
HotpotQA	w/o JS agreement	0.7067	0.0402	0.2022	0.5885
	Ours	0.7154	0.0409	0.1996	0.5824
MuSiQue	w/o JS agreement	0.6743	0.1198	0.1449	0.5310
	Ours	0.6721	0.1198	0.1450	0.5316

Appendix

Table 11: Effect of removing Base–RL JS agreement from TTA.

Setting	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	Average degradation (%) $\downarrow$
w/o Label BCE	0.6952	0.1151	0.2372	0.6659	25.03
w/o Label Ranking	0.6886	0.0647	0.2248	0.6408	2.23
w/o Stability Ranking	0.7180	0.1131	0.2250	0.6457	21.22
w/o SRM BCE	0.6972	0.0634	0.2223	0.6345	0.86
Full method	0.6956	0.0610	0.2226	0.6354	0.00

Appendix

Table 12: Ablation of individual training losses on PopQA. Average degradation denotes the mean relative degradation across the four metrics compared with the full method.

Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
Logistic regression	0.6803	0.0783	0.1950	0.5811
Probit regression	0.6807	0.0781	0.1958	0.5855
Complementary log-log	0.6802	0.0791	0.1953	0.5825
Sigmoid (Brier loss)	0.6821	0.0736	0.1942	0.5775
Clipped ridge regression	0.6803	0.0699	0.1944	0.5782
Ours	0.7208	0.0688	0.1870	0.5729

Appendix

Table 13: Confidence estimation averaged equally across four datasets.

Figure 5: Sensitivity to the anchoring weight

\lambda_{a}

. Curves show the equal-weight average over four datasets.

$\lambda_{\mathrm{h}}$	$\lambda_{r}$	$\lambda_{s}$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	Change
0.5	1.0	0.5	0.7208	0.0688	0.1870	0.5729	-
1.0 $\uparrow$	1.0	0.5	0.7227	0.0656	0.1860	0.5662	$-0.0023$
0.5	0.2 $\downarrow$	0.5	0.7229	0.0711	0.1870	0.5785	$+0.0025$
0.5	1.0	1.0 $\uparrow$	0.7193	0.0710	0.1879	0.5786	$+0.0018$

Appendix

Table 14: Robustness to training loss weightss.

TriviaQA
	Base trajectories				RL trajectories
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
DACA †	0.6979	0.2257	0.2625	0.7223	0.6675	0.0265	0.2232	0.6376
P(Sufficient) †	0.6815	0.4786	0.4497	1.3340	0.6605	0.3306	0.3372	1.1313
HTC	0.7976	0.0162	0.1765	0.5391	0.6529	0.2887	0.3161	0.9893
MICE-LR	0.7723	0.0223	0.1871	0.5540	0.6754	0.1249	0.2324	0.6703

Appendix

Table 15: Confidence estimation on Base and RL test trajectories.

Dataset	Total original tokens	Five answers tokens	Five answers / Total original (%)
TriviaQA	5,034.5	45.0	0.9
HotpotQA	5,289.3	45.8	0.9
MuSiQue	6,330.2	47.0	0.7
PopQA	6,642.5	46.8	0.7
Overall	5,824.1	46.1	0.8

Appendix

Table 16: Token usage of original trajectories and five sampled answers.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 249 papers

Sep 10, 2026

Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning

Test-time adaptation has emerged as a lightweight alternative to costly post-training for improving the reasoning capabilities of Large Language Models (LLMs) on downstream tasks. Predictive entropy provides a model-derived signal for such adaptation, guiding models toward higher-confidence reasoning states without external verifiers or reward models. However, higher confidence does not necessarily imply correctness, as LLMs may remain highly confident along incorrect reasoning trajectories. We observe that high-confidence reasoning is more likely to be correct when confidence remains stable under local perturbations. Based on this observation, we propose Test-Time Adaptation via Stability-Aware Confidence Optimization (TASCO), a framework that incorporates local stability into confidence-based test-time adaptation while keeping the LLM frozen. TASCO operationalizes local stability by optimizing a lightweight task-level prefix under two alternative perturbation strategies: Random Perturbation promotes distributional stability across trajectories induced by nearby perturbed prefixes, whereas Sharpness-Aware Perturbation targets worst-case local sensitivity. Experiments demonstrate that TASCO improves reasoning accuracy and token efficiency across diverse LLMs and reasoning benchmarks, while behavioral analyses show that it maintains stable confidence under local perturbations without prematurely concentrating the model's predictive distribution.

Bincheng Gu, Min Gao, Zongwei Wang +3

Key Laboratory of Dependable Service Computing in Cyber Physical Society, Chongqing University, China · King’s College London, London, UK · Griffith University, Brisbane, Australia

Test-Time Adaptation LLM Reliability Inference-Time Adaptation of Language Models LLM Reasoning

Ask PDF

Jul 2, 2026

Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks. However, prevailing RL reward designs typically prioritize response correctness, neglecting to incentivize models to express their confidence accurately. This leads to a critical problem: performance gains are often accompanied by poor calibration between confidence and accuracy, misleading models to overconfidently hallucinate when uncertain. To address this limitation, we propose

\textbf{C}

orrectness and

\textbf{C}

onfidence

\textbf{C}

alibration

\textbf{R}

einforcement

\textbf{L}

earning (

\textbf{C3RL}

), a novel RL algorithm integrating correctness, calibration and dataset-informed reference accuracy rewards together. Comprehensive evaluation across 8 text and multimodal datasets demonstrates that C3RL enhances calibration without sacrificing accuracy, outperforming the current state-of-the-art method in both performance and calibration metrics. Utilizing the well-calibrated verbalized confidence from C3RL, we further introduce

\textbf{C}

onfidence-based

\textbf{A}

daptive Test Time

\textbf{S}

caling (

\textbf{CAS}

), an adjustable inference-time strategy that allocates computational resources based on response confidence. Experiments show that CAS surpasses majority voting on both in-domain and out-of-domain datasets while reducing the inference budget by up to 12.33 times. We believe the synergy of C3RL and CAS paves the way for deploying more reliable and resource-efficient LLMs. The code, data and models will be released.

Xuqing Yang, Yi Yuan, Shanzhe Lei +1

Shanghai Jiao Tong University · Southeast University · Shanghai AI Laboratory

LLM Inference Efficiency RL for Language Models Test-Time Scaling Language Model Calibration

Ask PDF

Apr 25, 2026

Process Supervision of Confidence Margin for Calibrated LLM Reasoning

Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin (RLCM), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation.

Liaoyaqi Wang, Chunsheng Zuo, William Jurayj +2

Johns Hopkins University

Confidence Estimation in Language Models RL for Language Model Reasoning Language Model Calibration LLM Reliability Process Supervision LLM Uncertainty Estimation

Ask PDF

Abstract

Confidence Estimation in Language Models

Explore similar work

Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning

Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

Process Supervision of Confidence Margin for Calibrated LLM Reasoning