Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Models

Lucas Biechy; Cédric Eichler; Adrien Boiret; Nicolas Anciaux

cs.AISep 28, 2026

Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Models

Authors: Lucas Biechy, Cédric Eichler, Adrien Boiret, Nicolas Anciaux

Organizations: Petscraft, Inria · Université Paris-Saclay · INSA CVL · Université d’Orléans · LIFO

Abstract

While Large Reasoning Models (LRMs) excel at complex reasoning, alignment through reinforcement learning often induces systemic overconfidence. In production environments, where logits may be unavailable, robust black-box uncertainty quantification (UQ) is essential for trustworthiness and safety. Focusing on question-answering for LRMs, we show that existing black-box methods, such as paraphrase-based self-consistency and confidence verbalization, offer little to no improvement over simple repeated sampling, suggesting that alignment suppresses useful output variability. We introduce prompt-level relaxation operators that broaden the model's effective output distribution by approximating the effect of an optimal policy obtained with a stronger KL-regularization parameter, hence closer to the reference model. Theoretically, we demonstrate that relaxation improves calibration. We propose Jailbreak for Uncertainty (J4U), a jailbreak-derived technique for UQ that empirically reproduces the behavioral signatures predicted by our relaxation theory. Across 3 datasets and 4 LRMs, including a closed-source production model, J4U's improvement over repeated sampling achieves statistical significance in up to 6 times more LRM-dataset-metric settings than the strongest black-box UQ state-of-the-art baseline we evaluate, with average ECE reductions up to 5 times larger. These results provide a practical tool for UQ in black-box LRM deployment.

Figures & tables

	VC	Rephrase	J4U-Suffix	J4U-Prog	J4U-Art
	( Xiong et al., 2024 )	( Yang et al., 2024a )	(ours)	(ours)	(ours)
Acc. $\uparrow$	-1.2% -0.005 $\Delta$	+1.4% -0.025 $\Delta$	+2.8% -0.001 $\Delta$	-1.3% -0.017 $\Delta$	+7.0% -0.008 $\Delta$
ECE $\downarrow$	+8.6% +0.035 $\Delta$	-3.1% -0.018 $\Delta$	-2.9% -0.008 $\Delta$	-15.8% -0.068 $\Delta$	-13.8% -0.066 $\Delta$
Brier $\downarrow$	+3.5% +0.008 $\Delta$	+2.0% +0.002 $\Delta$	-1.7% -0.006 $\Delta$	-4.4% -0.015 $\Delta$	-4.8% -0.018 $\Delta$
NLL $\downarrow$	+5.0% +0.55 $\Delta$	-2.0% -0.24 $\Delta$	-6.4% -0.30 $\Delta$	-14.4% -1.73 $\Delta$	-14.5% -1.80 $\Delta$

Table 1: Comparison versus RS (across 12 LRM-dataset pairs). Left: relative (%) and absolute (

\Delta

) average differences; bold values denote the best-performing approach per metric. Variation per LRMs and dataset is encoded as a 4×3 heatmap (rows = LRMs, columns = datasets): green = improvement, red = degradation; darker shades indicate statistical significance.

LRM	J4U-Suffix	J4U-Prog	J4U-Art
Qwen3-4B	+7.6% +0.015 $\Delta$	+102.9% +0.308 $\Delta$	+72.5% +0.210 $\Delta$
gpt-oss-20b	+9.2% +0.030 $\Delta$	+39.9% +0.155 $\Delta$	+17.4% +0.069 $\Delta$
DeepSeek-R1-32B	+13.2% +0.035 $\Delta$	+57.7% +0.158 $\Delta$	+38.6% +0.106 $\Delta$
GPT-5.6 Luna	+17.0% +0.018 $\Delta$	+57.9% +0.109 $\Delta$	+62.4% +0.114 $\Delta$
Global Mean	+11.8% +0.024 $\Delta$	+64.6% +0.182 $\Delta$	+47.7% +0.125 $\Delta$

Table 2: Increase in entropy vs

\pi_{\text{RL}}

, in relative change (in %) and absolute difference (

\Delta

LRM	J4U-Suffix	J4U-Prog	J4U-Art
Qwen3-4B	-5.7% -0.037 $\Delta$	-16.8% -0.113 $\Delta$	-28.8% -0.187 $\Delta$
gpt-oss-20b	-1.1% -0.013 $\Delta$	-22.3% -0.183 $\Delta$	-31.5% -0.250 $\Delta$
DeepSeek-R1-32B	-3.7% -0.017 $\Delta$	-4.8% -0.037 $\Delta$	-23.7% -0.147 $\Delta$
Global Mean	-3.5% -0.022 $\Delta$	-14.6% -0.111 $\Delta$	-28.0% -0.195 $\Delta$

Table 3: Reduction in reasoning traces cosine similarity vs

\pi_{\text{RL}}

, in relative change (in %) and absolute difference (

\Delta

), with the exclusion of GPT-5.6 Luna (reasoning traces are not accessible).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	Accuracy $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
SynthPAI	RS	0.56 $\pm$ 0.03	0.34 $\pm$ 0.03	0.37 $\pm$ 0.03	8.27 $\pm$ 0.88
	VC	0.57 $\pm$ 0.03	0.33 $\pm$ 0.03	0.36 $\pm$ 0.03	8.30 $\pm$ 0.85
	Rephrase	0.56 $\pm$ 0.03	0.32 $\pm$ 0.03	0.36 $\pm$ 0.02	7.30 $\pm$ 0.88
	J4U-Suffix	0.56 $\pm$ 0.03	0.30 $\pm$ 0.03	0.35 $\pm$ 0.02	6.19 $\pm$ 0.83
	J4U-Prog	0.58 $\pm$ 0.03	0.23 $\pm$ 0.03	0.31 $\pm$ 0.02	3.91 $\pm$ 0.64
	J4U-Art	0.57 $\pm$ 0.03	0.25 $\pm$ 0.03	0.32 $\pm$ 0.02	4.22 $\pm$ 0.64

Appendix

Table 4: Comparison of UQ techniques on Qwen3-4B . Bold indicates statistically significant improvements w.r.t. RS by bootstrap test with

\alpha=0.05

Dataset	Method	Accuracy $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
SynthPAI	RS	0.61 $\pm$ 0.03	0.26 $\pm$ 0.03	0.31 $\pm$ 0.02	5.41 $\pm$ 0.74
	VC	0.59 $\pm$ 0.03	0.28 $\pm$ 0.03	0.32 $\pm$ 0.03	5.85 $\pm$ 0.78
	Rephrase	0.57 $\pm$ 0.03	0.26 $\pm$ 0.03	0.32 $\pm$ 0.02	4.72 $\pm$ 0.71
	J4U-Suffix	0.60 $\pm$ 0.03	0.22 $\pm$ 0.03	0.30 $\pm$ 0.02	3.63 $\pm$ 0.64
	J4U-Prog	0.60 $\pm$ 0.03	0.24 $\pm$ 0.03	0.31 $\pm$ 0.02	4.75 $\pm$ 0.72
	J4U-Art	0.60 $\pm$ 0.03	0.24 $\pm$ 0.03	0.30 $\pm$ 0.02	4.21 $\pm$ 0.65

Appendix

Table 5: Comparison of UQ techniques on gpt-oss-20b . Bold indicates statistically significant improvements w.r.t. RS by bootstrap test with

\alpha=0.05

Dataset	Method	Accuracy $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
SynthPAI	RS	0.57 $\pm$ 0.03	0.30 $\pm$ 0.03	0.35 $\pm$ 0.02	7.25 $\pm$ 0.83
	VC	0.57 $\pm$ 0.03	0.31 $\pm$ 0.03	0.35 $\pm$ 0.02	7.30 $\pm$ 0.82
	Rephrase	0.53 $\pm$ 0.03	0.32 $\pm$ 0.03	0.35 $\pm$ 0.02	5.63 $\pm$ 0.75
	J4U-Suffix	0.57 $\pm$ 0.03	0.28 $\pm$ 0.03	0.33 $\pm$ 0.02	6.01 $\pm$ 0.78
	J4U-Prog	0.58 $\pm$ 0.03	0.25 $\pm$ 0.03	0.32 $\pm$ 0.02	4.71 $\pm$ 0.65
	J4U-Art	0.58 $\pm$ 0.03	0.25 $\pm$ 0.03	0.31 $\pm$ 0.02	4.41 $\pm$ 0.62

Appendix

Table 6: Comparison of UQ techniques on DeepSeek-R1-32B . Bold indicates statistically significant improvements w.r.t. RS by bootstrap test with

\alpha=0.05

. Red indicates significant deterioration.

Dataset	Method	Accuracy $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$
SynthPAI	RS	0.61 $\pm$ 0.03	0.32 $\pm$ 0.03	0.35 $\pm$ 0.03	9.86 $\pm$ 0.99
	VC	0.59 $\pm$ 0.03	0.37 $\pm$ 0.04	0.39 $\pm$ 0.03	11.67 $\pm$ 1.2
	Rephrase	0.57 $\pm$ 0.03	0.36 $\pm$ 0.03	0.38 $\pm$ 0.03	10.61 $\pm$ 0.96
	J4U-Suffix	0.60 $\pm$ 0.03	0.30 $\pm$ 0.03	0.33 $\pm$ 0.03	7.94 $\pm$ 0.98
	J4U-Prog	0.59 $\pm$ 0.03	0.30 $\pm$ 0.03	0.34 $\pm$ 0.02	8.63 $\pm$ 0.94
	J4U-Art	0.61 $\pm$ 0.03	0.26 $\pm$ 0.03	0.32 $\pm$ 0.03	7.45 $\pm$ 0.84

Appendix

Table 7: Comparison of UQ techniques on GPT-5.6 Luna . Bold indicates statistically significant improvements w.r.t. RS by bootstrap test with

\alpha=0.05

. Red indicates significant deterioration.

LRM	Method	Accuracy $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
Qwen3-4B	RS	0.02 $\pm$ 0.01	0.72 $\pm$ 0.03	8.04 $\pm$ 1.16
	VC	0.05 $\pm$ 0.02	0.73 $\pm$ 0.03	8.17 $\pm$ 1.22
	Rephrase	0.02 $\pm$ 0.01	0.61 $\pm$ 0.02	8.17 $\pm$ 0.97
	J4U-Suffix	0.03 $\pm$ 0.01	0.68 $\pm$ 0.03	8.32 $\pm$ 1.22
	J4U-Prog	0.02 $\pm$ 0.01	0.62 $\pm$ 0.02	8.60 $\pm$ 0.98
	J4U-Art	0.01 $\pm$ 0.01	0.53 $\pm$ 0.02	8.36 $\pm$ 0.99

Appendix

Table 8: Accuracy and confidence calibration on HLE (exact match). Bold indicates statistically significant improvements w.r.t. RS by bootstrap test with

\alpha=0.05

. Red indicates significant deterioration.

		Qwen3-4B			gpt-oss-20b			DeepSeek-R1-32B			GPT-5.6 Luna
Dataset	Method	Mean	Var.	p-value	Mean	Var.	p-value	Mean	Var.	p-value	Mean	Var.	p-value
SynthPAI	RS	0.22	0.07		0.30	0.08		0.26	0.08		0.14	0.06
	Rephrase	0.26	0.08	7 $\times 10^{-7}$	0.35	0.07	7 $\times 10^{-11}$	0.31	0.08	5 $\times 10^{-12}$	0.15	0.07	6 $\times 10^{-2}$
	J4U-Suffix	0.28	0.07	3 $\times 10^{-19}$	0.37	0.07	1 $\times 10^{-17}$	0.30	0.08	2 $\times 10^{-11}$	0.22	0.08	7 $\times 10^{-17}$
	J4U-Prog	0.39	0.07	2 $\times 10^{-35}$	0.34	0.08	1 $\times 10^{-5}$	0.33	0.07	1 $\times 10^{-10}$	0.21	0.08	1 $\times 10^{-11}$
	J4U-Art	0.38	0.07	3 $\times 10^{-32}$	0.35	0.07	4 $\times 10^{-8}$	0.35	0.07	2 $\times 10^{-17}$	0.22	0.08	8 $\times 10^{-16}$

Appendix

Table 9: Mean and variance of Shannon entropy across policies, together with p-value of one-side Mann–Whitney U test against RS.

		Qwen3-4B		gpt-oss-20b		DeepSeek-R1-32B
Dataset	Method	Mean	Var.	Mean	Var.	Mean	Var.
SynthPAI	RS	0.67	0.15	0.85	0.09	0.73	0.14
	Rephrase	0.63	0.16	0.87	0.09	0.72	0.15
	J4U-Suffix	0.53	0.15	0.77	0.13	0.65	0.15
	J4U-Prog	0.48	0.14	0.65	0.15	0.61	0.16
	J4U-Art	0.34	0.07	0.45	0.12	0.35	0.14

Appendix

Table 10: Mean and variance of average cosine similarity between CoT embeddings across policies.

Methods	Complexity	Tokens			Wall-clock
		In (%)	Out (%)	Total (%)	Time (%)
VC	$O(2K)$	+530	+64	+135	+66.29
Rephrase	$O(2K)$	+141	+56	+63	+25.55
J4U-SUFFIX	$O(K)$	+6	+0	+0	-7.23
J4U-PROG	$O(K)$	+10	+17	+11	-8.14
J4U-ART	$O(K)$	+81	+21	+27	+5.37

Appendix

Table 11: Relative overhead compared to Repeated Sampling in

O(K)

LRM	Method	Accuracy $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
Qwen3-4B	RS	0.96 $\pm$ 0.01	0.04 $\pm$ 0.01	0.71 $\pm$ 0.31
	VC	0.96 $\pm$ 0.01	0.04 $\pm$ 0.01	0.65 $\pm$ 0.29
	Rephrase	0.95 $\pm$ 0.01	0.04 $\pm$ 0.01	0.61 $\pm$ 0.31
	NOISE 10%	0.94 $\pm$ 0.02	0.18 $\pm$ 0.02	0.66 $\pm$ 0.29
	NOISE 20%	0.69 $\pm$ 0.04	0.19 $\pm$ 0.03	0.91 $\pm$ 0.27
	J4U-Suffix	0.96 $\pm$ 0.01	0.04 $\pm$ 0.01	0.64 $\pm$ 0.28

Appendix

Table 12: Accuracy and confidence calibration on gsm8k where RS exhibit high accuracy and confidence calibration. Red indicates statistically significant deterioration.

Explore similar work

Stop Before You Fail: Operational Capability Boundaries for Mitigating Unproductive Reasoning in Large Reasoning Models
Sep 29, 2025Qingjie Zhang, Yujia Fu, Yang Wang +5Large Reasoning Models Early Failure Prediction
Tsinghua University · 2Ant Group
Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
Sep 29, 2026Feiyang Li, Shengjing Liu, Qi Zhan +7Large Language Model Uncertainty Confidence Calibration
College of Computer Science and Software Engineering, Shenzhen University · Tsinghua University · Behavioral and Spatial AI Lab, Peking University & Tongji University
Quantifying Faithful Confidence Expression in Large Reasoning Models
Jun 2, 2026Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu +1Confidence Calibration Large Reasoning Models
1Yale University

Sep 29, 2025, cs.AIJ/K move · Enter open · S save

Stop Before You Fail: Operational Capability Boundaries for Mitigating Unproductive Reasoning in Large Reasoning Models

Qingjie Zhang, Yujia Fu, Yang Wang, Liu Yan+4

Tsinghua University · 2Ant Group

Large Reasoning Models Early Failure Prediction

Open paper AskPDF arXiv

Current answering paradigms for Large Reasoning Models (LRMs) often fail to account for the fact that some questions may lie beyond the model's operational capability boundary, leading to long but unproductive reasoning. In this paper, we study whether LRMs expose early signals predictive of such cases, and whether these signals can be used to mitigate unproductive reasoning. In black-box settings, we find that reasoning expressions contain failure-predictive signals. In white-box settings, we show that the hidden states of the last input token contain information that is predictive of whether a question will not be solved correctly under our evaluation setup. Building on these observations, we propose two test-time monitoring strategies: reasoning expression monitoring and hidden states monitoring, that reduce token usage by 62.7-93.6%, substantially improving efficiency and reliability while largely preserving accuracy.

Abstract

Explore similar work

Stop Before You Fail: Operational Capability Boundaries for Mitigating Unproductive Reasoning in Large Reasoning Models

Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

Quantifying Faithful Confidence Expression in Large Reasoning Models