Conformity Breaks Conformal Prediction

Yibo Hu; Hanyu Su

cs.LGSep 3, 2026

Conformity Breaks Conformal Prediction

Authors: Yibo Hu, Hanyu Su

Organizations: Illinois Institute of Technology

Abstract

A conformal certificate can be valid when an LLM answers alone and invalid when the same LLM sees peers that unanimously assert a wrong answer. The question is unchanged; the model's score for the correct answer changes. We call this a score-mechanism shift: clean calibration certifies how the model scores answers alone, but not how it scores them under peer pressure. We show that this shift silently breaks conformal prediction in multi-agent LLM systems. Across open-weight models and multiple-choice QA tasks, coverage falls from a calibrated 90% to 74% under unanimous-wrong peers at the standard alpha = 0.10 operating point. The average hides a sharper failure: by targeting the low-confidence items the certificate still covers, an attacker nearly halves coverage on that subgroup, from 87% to 47%, while the monitored average remains much higher. The failure also reaches the decision layer: a system that should escalate when uncertain can instead become confident enough to act on the attacker's wrong answer. Standard conformal fixes do not solve the problem, because the question distribution has not changed; the model's scoring behavior has.

Figures & tables

Condition	$\alpha=0.10$ (target 90%)			$\alpha=0.05$ (target 95%)
Condition	seed0	seed1	seed2	seed0	seed1	seed2
Round 1 (no pressure)	90.1%	90.0%	90.0%	95.0%	95.0%	95.0%
Unan. wrong (no auth.)	73.5%	74.1%	74.2%	85.3%	86.6%	87.1%
Unan. wrong (+ auth.)	75.1%	74.0%	74.6%	87.1%	87.4%	86.7%
Mixed (no auth.)	94.2%	94.0%	94.4%	98.4%	99.0%	99.2%
Mixed (+ auth.)	93.4%	93.5%	94.1%	98.6%	98.9%	98.9%

Table 1: Coverage collapses only under unanimous-wrong peers. Three seeds (common subset: 4 models

\times

ARC

+

TruthfulQA), target

90\%

(

\alpha=0.10

) or

95\%

(

\alpha=0.05

Figure 1: Coverage collapses only under unanimous-wrong peers and improves under unanimous-correct ones. Per model; $\alpha=0.10$ (left), $\alpha=0.05$ (right). Llama’s immunity is a global-calibration artifact (Section 4.3 ).

Figure 2: Conditional coverage collapses while marginal coverage holds. The targeted band is the decile of solo confidence immediately above the calibrated threshold, where the certificate covers $87\%$ of items without pressure. Under unanimous-wrong pressure the pooled predictor keeps marginal coverage at $76\%$ while coverage on the band falls to $47\%$ ; every model drops. Gemma-9B is the sharpest illustration: its monitored marginal coverage is exactly at target, and its band is at $71\%$ . Error bars are standard deviations over $2{,}000$ calibration splits.

Unit	Singleton-ACT on wrong answer (%)
POOLED	12.1
Qwen-7B	28.7
Mistral-7B	26.9
Llama-8B	17.5
Gemma-9B	0.2

Table 2: The act-vs-escalate defense fails. Among items where conformity flips the pooled top-1 from correct to wrong, the fraction yielding a singleton set equal to the wrong answer, so the clean-calibrated layer commits the attacker’s answer with no escalation (the complement either escalates or acts on the correct answer). LAC,

\alpha=0.10

, seed-0, with the authority cue.

Condition	Down-cross	Up-cross	Net	Coverage gap
Unan. wrong	19.8%	3.2%	$+$ 16.6 pp	$+$ 16.6 pp
Unan. wrong ( $+$ auth.)	19.4%	4.5%	$+$ 15.0 pp	$+$ 15.0 pp
Mixed	2.0%	6.1%	$-$ 4.1 pp	$-$ 4.1 pp
Mixed ( $+$ auth.)	2.7%	6.0%	$-$ 3.4 pp	$-$ 3.4 pp
Unan. correct	3.9%	7.3%	$-$ 3.4 pp	$-$ 3.4 pp
Unan. correct ( $+$ auth.)	3.6%	7.1%	$-$ 3.5 pp	$-$ 3.5 pp

Table 3: Threshold-crossing flow, scored per (model, question) pair against the shared threshold of Section 3.1 (

\alpha=0.10

, seed-0, mean over

2{,}000

splits). Net flow equals the coverage gap in every condition, as Proposition 1 (i) requires.

Unit	Threshold	Mean $p_{\mathrm{gt}}^{(r1)}$	Items with $p_{\mathrm{gt}}^{(r1)}<0.001$
Shared (four models)	0.000388	—	—
Llama-8B	0.012596	0.557	1.7%
Gemma-9B	0.000499	0.743	12.3%
Qwen-7B	0.000124	0.753	13.7%
Mistral-7B	0.000025	0.634	20.7%

Table 4: Calibrated thresholds on the probability scale (an answer is kept when

p\geq

the entry;

\alpha=0.10

, seed-0, mean over

2{,}000

splits). Low thresholds come from items the model scores near zero, not from low average confidence.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Robustness decomposition across four models. Panel A: round-1 confidence (Gemma and Qwen are similar). Panel B: answer-flip rate (Gemma $16\%$ , Qwen $76\%$ ). Panel C: $p_{\mathrm{gt}}$ drop under unanimous-wrong pressure (Gemma $-0.15$ , Qwen $-0.75$ ). Panel D: answer-flip rate vs. coverage under pressure, each model on its own threshold so that the shared-calibration artifact of Section 4.3 does not enter. The robustness gap is driven by $\Delta p_{\mathrm{gt}}$ , not static confidence or buffer.

Figure 4: Per-model (Mondrian) calibration does not rescue coverage at $\alpha=0.10$ . It removes the global-pooling artifact (Llama’s apparent immunity) but three of four models still fail under unanimous-wrong pressure (Qwen $61\%$ , Llama $\approx 72\%$ , Mistral $\approx 74\%$ ; only Gemma stays near $90\%$ ). At $\alpha=0.05$ Mondrian largely rescues the vulnerable models ( $88$ – $92\%$ ), so the persistent failure is the $\alpha=0.10$ finding shown here.

Figure 5: Targeting without the answer key. Attacking $10\%$ of items, three adversaries with no access to the correct answers cost $-36$ to $-40$ pp on the items they choose, against $-14.3$ pp for random targeting. Ranking by $p_{\mathrm{gt}}$ instead selects the decile the threshold already excludes, where coverage is $13.4\%$ to begin with and the attack raises it. Overall coverage moves by at most $4$ pp throughout (right).

Model	Flip rate	$\bar{p}_{\mathrm{gt}}^{(r1)}$	Median $p_{\mathrm{gt}}^{(r1)}$	$\Delta p_{\mathrm{gt}}$	Buffer	Coverage (UW)
Gemma-9B	16.2%	0.753	0.999	$-$ 0.151	0.818	90.6%
Mistral-7B	31.2%	0.612	0.981	$-$ 0.226	0.779	68.4%
Llama-8B	49.3%	0.583	0.707	$-$ 0.384	0.555	99.3% (artifact)
Qwen-7B	75.8%	0.765	1.000	$-$ 0.747	0.862	37.5%

Appendix

Table 5: Robustness decomposition (three-seed pooled). “Buffer”

=\mathrm{mean}(p_{\mathrm{gt}}^{(r1)}-(1-\hat{\tau}))

for covered calibration items;

\Delta p_{\mathrm{gt}}

is the mean change in ground-truth probability under unanimous-wrong pressure. The driver of collapse is

\Delta p_{\mathrm{gt}}

, not static confidence or buffer. The coverage column uses the shared threshold, which is why Llama reads as immune; on per-model thresholds the same four models read

61

72

74

and

90\%

(Section 5.1 ).

Dataset	Round-1 coverage	Unanimous-wrong	Gap
ARC-Challenge	96.6%	84.7%	$-$ 5.3pp
BBH (multi-task)	90.3%	65.7%	$-$ 24.3pp
MMLU-Pro	87.7%	62.3%	$-$ 27.7pp
TruthfulQA	85.5%	70.7%	$-$ 19.3pp

Appendix

Table 6: Coverage under unanimous-wrong pressure per task type (seeds 1–2, 6 models pooled, global calibration,

\alpha=0.10

). The four BIG-Bench Hard subtasks (geometric shapes, logical deduction, temporal sequences, tracking) are bundled into one “BBH” row. Every task type shows substantial collapse. The gap column is measured against the

90\%

target. TruthfulQA’s round-1 coverage (

85.5\%

) is already below nominal with no pressure (a known truth-direction confound), so part of its

-19.3

pp is not the conformity effect; against its own clean baseline the pressure costs

-14.8

pp. The headline numbers elsewhere, computed against the calibrated threshold, are unaffected.

Stratum (round-1 $p_{\mathrm{gt}}$ percentile)	No pressure	Unanimous wrong	Change
$0$ – $10$ (below threshold)	13.4%	23.6%	$+$ 10.2 pp
$10$ – $20$ (the targeted band)	87.3%	47.4%	$-$ 39.9 pp
$20$ – $40$	100.0%	70.7%	$-$ 29.3 pp
$40$ – $70$	100.0%	84.3%	$-$ 15.7 pp
$70$ – $100$	100.0%	97.3%	$-$ 2.7 pp

Appendix

Table 7: Coverage by stratum of round-1 confidence (pooled predictor, seed-0,

\alpha=0.10

, mean over

2{,}000

splits). The calibrated threshold falls at the tenth percentile.

	Marginal		Band		Conditional
Unit	Clean	Unan.-wrong	Clean	Unan.-wrong	Bottom-20%	Bottom-10%
POOLED	90.1% $\pm$ 3.3	75.7% $\pm$ 7.0	87.3% $\pm$ 20.2	47.4% $\pm$ 21.0	35.5% $\pm$ 15.3	23.6% $\pm$ 13.2
Qwen-7B	90.0% $\pm$ 3.2	60.1% $\pm$ 16.5	87.0% $\pm$ 20.6	36.3% $\pm$ 14.3	26.2% $\pm$ 14.0	16.1% $\pm$ 16.5
Llama-8B	90.1% $\pm$ 3.3	73.0% $\pm$ 11.9	87.7% $\pm$ 20.1	69.0% $\pm$ 15.6	58.2% $\pm$ 15.2	47.3% $\pm$ 17.8
Mistral-7B	90.1% $\pm$ 3.5	81.2% $\pm$ 2.8	86.7% $\pm$ 21.0	76.3% $\pm$ 8.8	78.9% $\pm$ 5.4	81.4% $\pm$ 8.5
Gemma-9B	90.0% $\pm$ 3.4	90.1% $\pm$ 3.6	86.6% $\pm$ 21.0	70.6% $\pm$ 16.3	53.8% $\pm$ 14.2	37.0% $\pm$ 15.1

Appendix

Table 8: Marginal vs. conditional coverage under unanimous-wrong pressure, LAC split-conformal calibrated on clean round-1 data (

\alpha=0.10

, target

90\%

; mean

\pm

std over

2{,}000

splits). Band is the headline subgroup: the decile of round-1

p_{\mathrm{gt}}

immediately above the calibrated threshold. The bottom-

k\%

columns are given for reference; they straddle the threshold and are discussed below.

Figure 6: Coverage vs. peer-pressure intensity (logit interpolation). Qwen-7B falls below $80\%$ at $t\approx 0.65$ , Mistral-7B at $t\approx 0.80$ ; Gemma-9B and Llama-8B (global-calibration artifact) never break $80\%$ .

Predictor	ARC $+$ TruthfulQA	Seven task types
Answer-flip rate	$-0.89$	$-0.71$
$\|\Delta p_{\mathrm{gt}}\|$	$-0.89$	$-0.49$
Mean solo $p_{\mathrm{gt}}$	$-0.20$	$+0.37$
Clean accuracy	$-0.20$	$+0.37$

Appendix

Table 9: Spearman rank correlation between each predictor and pressured coverage, over the six models of each regime. Negative is the correct direction for the dynamic quantities, positive for the static ones.

Measured on	ARC	TQA	MMLU-Pro	LogDed	Track	Temporal	Geom
ARC	$-0.94$	$-0.49$	$-0.54$	$-0.71$	$-0.31$	$-0.94$	$+0.09$
TQA	$-0.83$	$-0.20$	$-0.43$	$-0.77$	$-0.43$	$-0.83$	$+0.26$
MMLU-Pro	$-1.00$	$-0.66$	$-0.60$	$-0.77$	$-0.37$	$-1.00$	$+0.03$
LogDed	$-0.83$	$-0.60$	$-0.31$	$-0.77$	$-0.49$	$-0.83$	$+0.31$
Track	$-0.67$	$-0.49$	$-0.14$	$-0.23$	$+0.14$	$-0.67$	$+0.26$
Temporal	$-0.71$	$-0.94$	$-0.09$	$-0.26$	$-0.03$	$-0.71$	$+0.20$

Appendix

Table 10: Cross-task transfer. Row: the task the flip rate is measured on. Column: the task whose pressured coverage is ranked. Off-diagonal entries are out-of-sample. Seeds 1–2, six models with the complete task grid.

Model	Flip rate	$\bar{\Delta p}_{\mathrm{gt}}$	Coverage (UW)	Gap
Qwen-1.5B	32.8%	$-$ 0.301	96.0%	$+$ 6.0pp
Qwen-3B	31.3%	$-$ 0.290	45.2%	$-$ 44.8pp
Qwen-7B	53.7%	$-$ 0.528	48.1%	$-$ 41.9pp

Appendix

Table 11: Qwen size scaling (seeds 1–2, all datasets,

\alpha=0.10

). The coverage column uses the threshold shared by the three sizes, which is what makes the spread below as wide as it is; on per-size thresholds it narrows to

8

pp.

Conformal Prediction

Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.

See all 352 papers

Conformalized Large Language Models under Configuration Shift
Aug 2, 2026Yuqicheng Zhu, Jialin Yu, Lin Li +7Conformal Prediction Conformal Prediction for Language Models
University of Stuttgart · Robert Bosch GmbH · University of Oxford +4
Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations
Apr 17, 2026Yanli Wang, Peng Kuang, Xiaoyu Han +2Language Model Robustness Conformal Prediction
Imperial College London · Zhejiang University · University of Illinois Urbana-Champaign +1
Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity
Jun 1, 2026Jiaming Qu, Lucheng Fu, Yibo HuMulti-Agent LLM Systems Normative Conformity in Language Models
Amazon · Georgia Institute of Technology · Illinois Institute of Technology

Aug 2, 2026J/K move · Enter open · S save

Conformalized Large Language Models under Configuration Shift

Yuqicheng Zhu, Jialin Yu, Lin Li, Gengyuan Zhang+6

University of Stuttgart · Robert Bosch GmbH · University of Oxford+4

Conformal Prediction Conformal Prediction for Language Models Language Model Calibration LLM Uncertainty Estimation

Open paper AskPDF arXiv

Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets with finite-sample coverage guarantees under exchangeability. Yet for LLMs, nonconformity scores are often induced by an inference pipeline, not just a fixed model, making them depend not only on the data distribution but also on configurable factors such as the prompt template, decoding parameters, and deployment setting. Since such configurations are routinely modified in practice but rarely treated as a source of shift, their impact on CP validity remains poorly understood. We call this configuration shift and study it systematically along three axes: prompt template, decoding temperature, and weight quantization. In a broad empirical study spanning

9

LLMs,

4

datasets, and

4

nonconformity scores, we find that configuration shift consistently erodes CP validity, often driving empirical coverage below the target. By contrast, efficiency is largely preserved: valid prediction sets remain close in size to the i.i.d. baseline. We derive coverage lower bounds that attribute this loss to a discrepancy between calibration and test score distributions, and use their finite-sample plug-in versions as empirical diagnostics of shift severity. We further show that these findings lead to practical mitigations: bound-inspired recalibration is effective with limited test examples, while fragility-aware calibration ensembling recovers much of the lost coverage without test data.

Abstract

Conformal Prediction

Explore similar work

Conformalized Large Language Models under Configuration Shift

Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations

Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity