Test-Time Confidence Adaptation under RL-Induced Trajectory Shift

Zhenzhen Ren; Xiaoqing Zhang; GuoBiao Li; Bowen Fang; Yanxi Wang; Xinpeng Zhang; Ke Han

cs.AIOct 26, 2026

Test-Time Confidence Adaptation under RL-Induced Trajectory Shift

Authors: Zhenzhen Ren, Xiaoqing Zhang, GuoBiao Li, Bowen Fang, Yanxi Wang, Xinpeng Zhang, Ke Han

Organizations: Fudan University · Zhongguancun Academy · Renmin University of China · Institute of Automation, Chinese Academy of Sciences ·

Abstract

Trajectory-based confidence estimation has shown promising performance for large language models (LLMs) by leveraging rich signals from their reasoning processes. However, we find that such confidence estimators can degrade substantially after agentic reinforcement learning (Agentic RL) as it induces substantial shifts in the reasoning trajectories. More importantly, despite these shifts, we uncover a reliability structure that is largely preserved across Base and RL models: $\textbf{more}$ $\textbf{stable}$ $\textbf{trajectories}$ $\textbf{remain}$ $\textbf{more}$ $\textbf{likely}$ $\textbf{to}$ $\textbf{produce}$ $\textbf{correct}$ $\textbf{answers}$ . This preserved relationship provides a transferable, label-free reliability signal for post-RL confidence adaptation. Building on this finding, we propose $\textbf{StaR-TTA}$ , a stability-guided framework for label-free test-time confidence adaptation. StaR-TTA jointly learns from correctness supervision and stability-derived reliability supervision on labeled Base trajectories, establishing the connection between trajectory stability and answer reliability. At test time, it reuses stability-induced reliability rankings from unlabeled RL trajectories to adapt confidence scores to the shifted trajectory distribution, without requiring target correctness labels. Across four QA benchmarks, StaR-TTA achieves an average AUROC of $0.721$ , with consistent improvements across unseen datasets, model families and scales, and different Agentic RL algorithms.

Figures & tables

Figure 1: (a) Agentic RL shifts reasoning trajectories and degrades uncertainty quantification (UQ). (b) Stability-correctness ordering remains highly positive after RL. (c) Stability-guided test-time adaptation restores reliable confidence estimation. Using unlabeled RL trajectories, we exploit trajectory stability to construct pairwise ordering and adapt a confidence estimator at test time with a ranking loss, leading to more reliable confidence estimates.

Method	Base AUROC $\uparrow$	RL AUROC $\uparrow$	Relative drop (%)
MICE-LR	0.741	0.567	23.5
MICE-RF	0.773	0.559	27.7
HTC	0.765	0.675	11.8
CCPS	0.745	0.491	34.1
DACA	0.659	0.656	0.5
$P(\mathrm{Sufficient})$	0.627	0.689	$-9.9$

Table 1: Base-to-RL confidence transfer on NQ.

Method	Tool control	Answer control	Trajectory replay	Recovery (%)
MICE-LR	$.567\,{\scriptscriptstyle(-.174)}$	$.514\,{\scriptscriptstyle(-.110)}$	$.563\,{\scriptscriptstyle(+.049)}$	44.5
MICE-RF	$.559\,{\scriptscriptstyle(-.215)}$	$.577\,{\scriptscriptstyle(-.087)}$	$.623\,{\scriptscriptstyle(+.045)}$	51.9
CCPS	$.491\,{\scriptscriptstyle(-.255)}$	$.498\,{\scriptscriptstyle(-.133)}$	$.513\,{\scriptscriptstyle(+.015)}$	11.1

Table 2: Controlled AUROC analysis on NQ.

Figure 2: Overview of our method. (a) Firstly, an SRM model is trained to capture the relationship between trajectory stability and correctness. Then a scorer (TCS) is optimized, followed by a calibrator (CC) to convert scores into calibrated probabilities. (b) At test time, we adapt the results of TCS to Agentic-RL trajectories. We extract stability signals from both Agentic-RL and Base trajectories and use this information to adjust the score while keeping TCS and CC fixed .

Figure 3: AUROC of sampling-stability signals for answer correctness.

	TriviaQA				HotpotQA				MuSiQue				PopQA
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
DACA	.668	.027	.223	.638	.613	.155	.243	.679	.625	.310	.229	.650	.680	.110	.244	.681
P(Sufficient)	.661	.331	.337	1.131	.636	.526	.498	1.551	.648	.643	.551	1.584	.641	.459	.446	1.382
HTC	.653	.289	.316	.989	.507	.175	.268	.738	.721	.081	.132	.423	.678	.328	.334	.937
MICE-LR	.675	.125	.232	.670	.640	.112	.234	.662	.623	.054	.135	.443	.595	.282	.321	.882
MICE-RF	.629	.046	.229	.651	.615	.048	.219	.629	.610	.064	.136	.442	.590	.119	.254	.702

Table 3: Main results of confidence estimation on the RL model’s test trajectories.

Figure 4: t-SNE visualization of the H64 hidden representations for Base and RL.

	TriviaQA				2WikiMultiHopQA				Bamboogle				NQ
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HTC	.653	.289	.316	.989	.482	.289	.337	1.333	.531	.212	.270	.909	.672	.131	.239	.737
MICE-LR	.675	.125	.232	.670	.502	.299	.346	.948	.585	.399	.379	1.027	.653	.271	.302	.837
MICE-RF	.629	.046	.229	.651	.503	.132	.268	.732	.520	.243	.289	.787	.607	.136	.250	.693
CCPS	.420	.405	.429	1.335	.453	.261	.324	1.368	.524	.220	.277	1.793	.491	.256	.320	1.791
DACA	.668	.027	.223	.638	.509	.072	.247	.686	.556	.218	.249	.692	.623	.165	.258	.710

Table 4: Cross-dataset generalization of confidence estimation.

	TriviaQA				HotpotQA				MuSiQue				PopQA
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HTC	.683	.114	.233	.669	.631	.071	.202	.596	.688	.103	.090	.313	.611	.211	.274	.896
MICE-LR	.730	.047	.202	.593	.663	.108	.208	.597	.651	.071	.085	.300	.700	.098	.218	.622
MICE-RF	.715	.029	.202	.587	.681	.097	.197	.569	.661	.069	.080	.286	.713	.103	.212	.601
CCPS	.707	.105	.210	.607	.686	.126	.201	.580	.652	.103	.087	.308	.691	.083	.213	.619
DACA	.772	.204	.237	.943	.583	.152	.234	.661	.682	.394	.265	.753	.752	.297	.319	1.076

Table 5: Cross-model generalization: Llama-3.2-3B-Instruct with PPO .

	TriviaQA				HotpotQA				MuSiQue				PopQA
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HTC	.717	.355	.353	2.684	.647	.152	.246	.710	.794	.149	.161	.477	.697	.075	.220	.630
MICE-LR	.641	.064	.220	.639	.614	.096	.236	.657	.673	.041	.140	.440	.581	.068	.242	.674
MICE-RF	.669	.036	.209	.607	.629	.087	.228	.638	.645	.061	.146	.456	.652	.059	.227	.639
CCPS	.655	.059	.202	.587	.659	.078	.224	.631	.672	.087	.147	.458	.625	.146	.253	.697
DACA	.725	.265	.268	2.005	.657	.372	.379	1.823	.697	.315	.314	1.155	.676	.389	.387	2.679

Table 6: Cross-model generalization: Qwen2.5-7B-Instruct with PPO.

	TriviaQA				HotpotQA				MuSiQue				PopQA
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
HTC	.663	.321	.331	1.171	.422	.245	.274	1.039	.551	.844	.767	2.479	.636	.579	.570	2.613
MICE-LR	.640	.150	.251	.716	.624	.149	.230	.657	.567	.051	.052	.217	.524	.267	.327	.890
MICE-RF	.564	.080	.244	.681	.605	.068	.204	.593	.587	.079	.057	.234	.521	.112	.254	.701
CCPS	.425	.536	.546	4.192	.437	.259	.275	1.661	.483	.210	.104	.434	.467	.318	.354	2.919
DACA	.692	.093	.232	.655	.611	.244	.261	.716	.642	.465	.263	.718	.697	.134	.252	.696

Table 7: Generalization across RL algorithms: Qwen2.5-3B with GRPO.

Dataset	Setting	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
TriviaQA	w/o TTA	0.8018	0.0455	0.1794	0.5386
	w/ TTA	0.8000	0.0535	0.1808	0.5421
HotpotQA	w/o TTA	0.7112	0.0349	0.1997	0.5828
	w/ TTA	0.7154	0.0409	0.1996	0.5824
MuSiQue	w/o TTA	0.6296	0.1212	0.1478	0.5503
	w/ TTA	0.6721	0.1198	0.1450	0.5316

Table 8: Effect of removing TTA.

$N$	Average	Trivia	Hotpot	MuSiQue	PopQA
0	0.7035	0.8018	0.7112	0.6296	0.6714
1	0.7035	0.8018	0.7112	0.6296	0.6714
2	0.6306	0.7537	0.5125	0.6383	0.6181
4	0.6680	0.7828	0.5744	0.6671	0.6477
8	0.7089	0.7911	0.6839	0.6683	0.6924
12	0.7072	0.7914	0.6893	0.6703	0.6779

Table 9: AUROC under different TTA sample sizes.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Setting	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
TriviaQA	w/o Stability input $c^{m}$	0.7413	0.1529	0.2250	0.6408
	w/o Statistics $d^{m}$	0.7871	0.0473	0.1850	0.5526
	Ours	0.8000	0.0535	0.1808	0.5421
HotpotQA	w/o Stability input $c^{m}$	0.6170	0.1222	0.2337	0.6679
	w/o Statistics $d^{m}$	0.6992	0.0724	0.2079	0.6030
	Ours	0.7154	0.0409	0.1996	0.5824

Appendix

Table 10: Independent ablations of Scorer input features. Each ablation removes only one feature group.

Dataset	Setting	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
TriviaQA	w/o JS agreement	0.8000	0.0521	0.1816	0.5441
	Ours	0.8000	0.0535	0.1808	0.5421
HotpotQA	w/o JS agreement	0.7067	0.0402	0.2022	0.5885
	Ours	0.7154	0.0409	0.1996	0.5824
MuSiQue	w/o JS agreement	0.6743	0.1198	0.1449	0.5310
	Ours	0.6721	0.1198	0.1450	0.5316

Appendix

Table 11: Effect of removing Base–RL JS agreement from TTA.

Setting	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	Average degradation (%) $\downarrow$
w/o Label BCE	0.6952	0.1151	0.2372	0.6659	25.03
w/o Label Ranking	0.6886	0.0647	0.2248	0.6408	2.23
w/o Stability Ranking	0.7180	0.1131	0.2250	0.6457	21.22
w/o SRM BCE	0.6972	0.0634	0.2223	0.6345	0.86
Full method	0.6956	0.0610	0.2226	0.6354	0.00

Appendix

Table 12: Ablation of individual training losses on PopQA. Average degradation denotes the mean relative degradation across the four metrics compared with the full method.

Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
Logistic regression	0.6803	0.0783	0.1950	0.5811
Probit regression	0.6807	0.0781	0.1958	0.5855
Complementary log-log	0.6802	0.0791	0.1953	0.5825
Sigmoid (Brier loss)	0.6821	0.0736	0.1942	0.5775
Clipped ridge regression	0.6803	0.0699	0.1944	0.5782
Ours	0.7208	0.0688	0.1870	0.5729

Appendix

Table 13: Confidence estimation averaged equally across four datasets.

Figure 5: Sensitivity to the anchoring weight

\lambda_{a}

. Curves show the equal-weight average over four datasets.

$\lambda_{\mathrm{h}}$	$\lambda_{r}$	$\lambda_{s}$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	Change
0.5	1.0	0.5	0.7208	0.0688	0.1870	0.5729	-
1.0 $\uparrow$	1.0	0.5	0.7227	0.0656	0.1860	0.5662	$-0.0023$
0.5	0.2 $\downarrow$	0.5	0.7229	0.0711	0.1870	0.5785	$+0.0025$
0.5	1.0	1.0 $\uparrow$	0.7193	0.0710	0.1879	0.5786	$+0.0018$

Appendix

Table 14: Robustness to training loss weightss.

TriviaQA
	Base trajectories				RL trajectories
Method	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
DACA †	0.6979	0.2257	0.2625	0.7223	0.6675	0.0265	0.2232	0.6376
P(Sufficient) †	0.6815	0.4786	0.4497	1.3340	0.6605	0.3306	0.3372	1.1313
HTC	0.7976	0.0162	0.1765	0.5391	0.6529	0.2887	0.3161	0.9893
MICE-LR	0.7723	0.0223	0.1871	0.5540	0.6754	0.1249	0.2324	0.6703

Appendix

Table 15: Confidence estimation on Base and RL test trajectories.

Dataset	Total original tokens	Five answers tokens	Five answers / Total original (%)
TriviaQA	5,034.5	45.0	0.9
HotpotQA	5,289.3	45.8	0.9
MuSiQue	6,330.2	47.0	0.7
PopQA	6,642.5	46.8	0.7
Overall	5,824.1	46.1	0.8

Appendix

Table 16: Token usage of original trajectories and five sampled answers.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 249 papers

Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning
Sep 10, 2026Bincheng Gu, Min Gao, Zongwei Wang +3Test-Time Adaptation LLM Reliability
Key Laboratory of Dependable Service Computing in Cyber Physical Society, Chongqing University, China · King’s College London, London, UK · Griffith University, Brisbane, Australia
Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
Jul 2, 2026Xuqing Yang, Yi Yuan, Shanzhe Lei +1LLM Inference Efficiency RL for Language Models
Shanghai Jiao Tong University · Southeast University · Shanghai AI Laboratory
Process Supervision of Confidence Margin for Calibrated LLM Reasoning
Apr 25, 2026Liaoyaqi Wang, Chunsheng Zuo, William Jurayj +2Confidence Estimation in Language Models RL for Language Model Reasoning
Johns Hopkins University

Sep 10, 2026J/K move · Enter open · S save

Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning

Bincheng Gu, Min Gao, Zongwei Wang, Yibing Bai+2

Key Laboratory of Dependable Service Computing in Cyber Physical Society, Chongqing University, China · King’s College London, London, UK · Griffith University, Brisbane, Australia

Test-Time Adaptation LLM Reliability Inference-Time Adaptation of Language Models LLM Reasoning

Open paper AskPDF arXiv

Test-time adaptation has emerged as a lightweight alternative to costly post-training for improving the reasoning capabilities of Large Language Models (LLMs) on downstream tasks. Predictive entropy provides a model-derived signal for such adaptation, guiding models toward higher-confidence reasoning states without external verifiers or reward models. However, higher confidence does not necessarily imply correctness, as LLMs may remain highly confident along incorrect reasoning trajectories. We observe that high-confidence reasoning is more likely to be correct when confidence remains stable under local perturbations. Based on this observation, we propose Test-Time Adaptation via Stability-Aware Confidence Optimization (TASCO), a framework that incorporates local stability into confidence-based test-time adaptation while keeping the LLM frozen. TASCO operationalizes local stability by optimizing a lightweight task-level prefix under two alternative perturbation strategies: Random Perturbation promotes distributional stability across trajectories induced by nearby perturbed prefixes, whereas Sharpness-Aware Perturbation targets worst-case local sensitivity. Experiments demonstrate that TASCO improves reasoning accuracy and token efficiency across diverse LLMs and reasoning benchmarks, while behavioral analyses show that it maintains stable confidence under local perturbations without prematurely concentrating the model's predictive distribution.

Abstract

Confidence Estimation in Language Models

Explore similar work

Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning

Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

Process Supervision of Confidence Margin for Calibrated LLM Reasoning