CS-WCP: Robust Conformal Sets for LLM-Judge Traffic Shifts with Uncertain Group Proportions

Ibne Farabi Shihab; Fariya Afrin

cs.LGAug 23, 2026

CS-WCP: Robust Conformal Sets for LLM-Judge Traffic Shifts with Uncertain Group Proportions

Authors: Ibne Farabi Shihab, Fariya Afrin

Organizations: Department of Computer Science, Iowa State University · Department of Computer Science, Kalinga Institute of Industrial Technology

Abstract

Prediction sets built from an LLM judge can undercover when deployment traffic changes the prevalence of task or policy groups. Weighted conformal prediction is exact under covariate shift when the density ratio is known, but group proportions must usually be estimated from finite unlabeled samples. We introduce confidence-set weighted conformal prediction (CS-WCP), which constructs simultaneous exact intervals for source and target group masses and returns the union of weighted conformal sets over every compatible ratio vector. For a fixed or independently learned finite partition, CS-WCP attains coverage at least 1-alpha-delta_w-tau_A-kappa, where tau_A measures within-cell covariate mismatch and kappa measures conditional shift. A linear endpoint rule computes the robust union in O(G|Y|) time. Across 336 constructed shared-support traffic shifts, CS-WCP reaches 0.973 mean coverage with 13 point failures, compared with 0.954 and 44 failures for source conformal prediction, at mean binary set sizes 1.74 and 1.65. On 336 natural cross-task transfers, coverage rises from 0.882 to 0.962, but mean set size reaches 1.87 and a size-matched group plug-in baseline is competitive. The method therefore supplies an auditable coverage safeguard under uncertain mixture weights; its value is conservative tail protection, not scalar probability calibration or uniformly smaller sets.

Figures & tables

Method	Shift structure	Unknown quantity	Guarantee target	Difference from CS-WCP
WCP	covariate shift	full ratio	marginal coverage	assumes known ratio for exact validity
GWCP	observed finite groups	group proportions	near/exact corrected coverage	pools group score CDFs rather than robustly unioning WCP sets
Unknown-subpopulation CP	latent mixture	group assignment or classifier	coverage or risk control	learns latent structure under classifier assumptions
Clipped WCP	general covariate shift	learned, possibly unbounded ratio	expected and dataset-conditional coverage	controls clipping bias rather than finite mass-count uncertainty
PAC label-shift sets	label shift	class-ratio weights	PAC set validity	propagates confusion-matrix and target-label-mass uncertainty
CS-WCP	fixed finite traffic partition	source and target group masses	$1-\alpha-\delta_{w}-\tau_{A}-\kappa$	robust union with an exact endpoint test

Table 1: Scope comparison with the closest conformal-shift families. The table compares assumptions and guarantee targets, not empirical superiority.

Method	Coverage $\uparrow$	Size $\downarrow$	Singleton $\uparrow$	Point failures $\downarrow$	FWER certified $\uparrow$
Source CP	$0.954$	$1.652$	$0.342$	$44$	$202$
Oracle WCP	$0.964$	$1.702$	$0.294$	$27$	$227$
Classifier-odds WCP	$0.957$	$1.662$	$0.333$	$38$	$208$
KMM-WCP	$0.957$	$1.670$	$0.324$	$42$	$214$
Group plug-in WCP	$0.964$	$1.701$	$0.295$	$28$	$225$
Clipped uLSIF WCP	$0.956$	$1.662$	$0.332$	$41$	$209$

Table 2: Constructed shared-support traffic-mixture results at

1-\alpha=0.90

over 336 transfers. CS-WCP has the fewest point failures and the most familywise certifications among executed rows, but it also returns the largest sets.

Method	Coverage $\uparrow$	Size $\downarrow$	Point failures $\downarrow$	FWER certified $\uparrow$
Source CP ( $\alpha=.10$ )	$0.882$	$1.61$	$128$	$181$
Classifier-odds WCP ( $\alpha=.10$ )	$0.900$	$1.65$	$112$	not recorded
Group plug-in ( $\alpha=.10$ )	$0.951$	$1.84$	$59$	$256$
Group plug-in ( $\alpha=.075$ )	$0.970$	$1.89$	$38$	$280$
CS-WCP ( $\alpha=.10$ )	$0.962$	$1.87$	$48$	$276$
Target-tuned group plug-in	$0.984$	$1.92$	$11$	$301$

Table 3: Natural cross-task results over 336 transfers with an independently frozen partition and disjoint mass counts. The target-tuned row consumes 60 target labels per transfer and is not label-free. The fixed conservative group plug-in row is competitive with CS-WCP, so no universal dominance claim is made.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Tier	Transfers	Target labels used	Construction
Constructed primary	$336$	none for set construction	$288$ QA generator-mixture plus $48$ SAMSum policy-mixture transfers; fixed traffic partition
Natural cross-task	$336$	none for set construction	$8$ judges $\times 42$ ordered pairs over seven non-code domains; partition frozen on an independent design role
HumanEval sensitivity	$48$	official tests at evaluation	$8$ judges $\times 6$ ordered generator-mixture pairs; reported separately because the eligible pool is smaller
Penalty audit	$336+336$	independent source and target audits	audit cells and multiplicity budget frozen before labels are opened

Appendix

Table 4: Disjoint experimental tiers. Counts and results are not transferred across protocols.

	Constructed mixture tier				Natural cross-task tier
Method	Cov.	Size	Fail	Cert.	Cov.	Size	Fail	Cert.
Source CP ( $.10$ )	$.954$	$1.65$	$44$	$202$	$.882$	$1.61$	$128$	$181$
Source CP ( $.075$ )	$.971$	$1.74$	$16$	$236$	$.898$	$1.65$	$110$	$196$
Source CP ( $.05$ )	$.982$	$1.81$	$3$	$270$	$.927$	$1.74$	$80$	$225$
Source CP ( $.02$ )	$.991$	$1.86$	$1$	$287$	$.971$	$1.88$	$31$	$278$
Group plug-in ( $.10$ )	$.964$	$1.70$	$28$	$225$	$.951$	$1.84$	$59$	$256$

Appendix

Table 5: Coverage–size frontier. CS-WCP appears once at its predeclared level and ratio budget. Other methods are swept over fixed nominal levels to reveal the cost of conservatism. “Cert.” counts familywise certifications of the predeclared coverage floor on the sealed evaluation labels.

Explore similar work

Conformalized Large Language Models under Configuration Shift
Aug 2, 2026Yuqicheng Zhu, Jialin Yu, Lin Li +7Conformal Prediction Language Model Calibration
University of Stuttgart · Robert Bosch GmbH · University of Oxford +4
Sharp training-conditional coverage for conformal prediction under covariate shift
Sep 27, 2026Mehrdad PournaderiWeighted Conformal Prediction Set Prediction
Mofid Securities
Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift
Feb 16, 2026Farbod Siahkali, Ashwin Verma, Vijay GuptaConformal Prediction Distribution Shift Robustness
Elmore Family School of Electrical and Computer Engineering, at Purdue University, West Lafayette, IN 47907 USA

Aug 2, 2026J/K move · Enter open · S save

Conformalized Large Language Models under Configuration Shift

Yuqicheng Zhu, Jialin Yu, Lin Li, Gengyuan Zhang+6

University of Stuttgart · Robert Bosch GmbH · University of Oxford+4

Conformal Prediction Language Model Calibration LLM Uncertainty Estimation

Open paper AskPDF arXiv

Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets with finite-sample coverage guarantees under exchangeability. Yet for LLMs, nonconformity scores are often induced by an inference pipeline, not just a fixed model, making them depend not only on the data distribution but also on configurable factors such as the prompt template, decoding parameters, and deployment setting. Since such configurations are routinely modified in practice but rarely treated as a source of shift, their impact on CP validity remains poorly understood. We call this configuration shift and study it systematically along three axes: prompt template, decoding temperature, and weight quantization. In a broad empirical study spanning

9

LLMs,

4

datasets, and

4

nonconformity scores, we find that configuration shift consistently erodes CP validity, often driving empirical coverage below the target. By contrast, efficiency is largely preserved: valid prediction sets remain close in size to the i.i.d. baseline. We derive coverage lower bounds that attribute this loss to a discrepancy between calibration and test score distributions, and use their finite-sample plug-in versions as empirical diagnostics of shift severity. We further show that these findings lead to practical mitigations: bound-inspired recalibration is effective with limited test examples, while fragility-aware calibration ensembling recovers much of the lost coverage without test data.

Abstract

Explore similar work

Conformalized Large Language Models under Configuration Shift

Sharp training-conditional coverage for conformal prediction under covariate shift

Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift