SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

Liang You; Hengyu Shi; Dongwen Ou

cs.LGSep 30, 2026

SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

Authors: Liang You, Hengyu Shi, Dongwen Ou

Organizations: University of Pittsburgh · Duke University

Abstract

Conformal prediction guarantees marginal coverage, but a single calibration threshold can still spread that coverage unevenly, over-covering easy regions and under-covering hard ones. SimplexUQ is, to our knowledge, the first benchmark and reproducible protocol for measuring this allocation problem on simplex-valued predictions; it compares existing conformal wrappers rather than proposing a new one. Its task suite, SimplexTasks-12, combines six controlled synthetic regimes with six frozen-predictor real tasks spanning class probabilities, topic mixtures, spectral abundances, cell-type fractions, age distributions, and emotion mixtures. Each comparison fixes the predictor, score, and response-free stratification map, varies only the wrapper, and reports marginal coverage, worst-stratum coverage, max disparity, and within-task radius and compute. Global calibration can look valid while failing badly: on CIFAR-10 it attains 0.900 marginal coverage but only 0.542 in the worst entropy stratum, and Mondrian calibration raises that stratum to 0.886 while reducing max disparity from 0.358 to 0.022. No wrapper dominates, however. Under smooth synthetic heterogeneity, several repairs are competitive; fixed-map analyses show that rankings depend on the evaluation groups and protocol; and in a 12-task comparison, Mondrian has lower disparity on its single target partition for all 12 tasks, whereas BatchMVP has lower disparity over overlapping groups on five. These are empirical comparisons, not new coverage guarantees. A controlled predictor-bias sweep shows that removing predictor bias only partly reduces global-threshold disparity. We release task cards, result provenance, permitted derived arrays, and rebuild instructions, and treat wrapper selection as a diagnostic comparison rather than a universal ranking.

Figures & tables

Wrapper	Threshold rule	Guarantee or role
Global	One split-conformal threshold from all calibration scores	Marginal coverage (Theorem 6 )
Mondrian	One threshold per stratum of the fixed map $g$	Coverage within each stratum (Theorem 9 )
TwoStage	Local scale $\hat{\sigma}$ fitted on separate data; calibrate $R/\hat{\sigma}(U)$	Marginal coverage (Theorem 7 ); primary normalization wrapper
OneShot	kNN scale fitted on the same calibration residuals	Not guaranteed (Proposition 2 ); diagnostic
TrainRes	Scale fitted on training residuals, then held fixed	Marginal, but can misallocate (Proposition 3 ); diagnostic
FullCP	Full-conformal local-scale variant	Exact reference; highest compute

Table 1: Wrappers compared in SimplexUQ. All operate on the same frozen predictor output

U

and score

R=d(Y,U)

; they differ only in how the threshold is calibrated. Formal statements are in Appendix A.

Figure 1: Allocation failure on the D2 smooth-scale synthetic task. (A) The true local residual scale is highest near simplex vertices and lowest toward the interior. (B) Under the pre-declared boundary stratification, a single global split-conformal threshold under-covers boundary-adjacent strata despite near-nominal pooled coverage. (C) Both Mondrian and TwoStage improve allocation; this profile does not establish unique superiority of normalization.

Figure 2: Synthetic benchmark summary. Each cell reports max disparity. D2/D5 support several competitive repairs, aligned D3 favors group-wise calibration, and D6 shows large global disparity. For D6 † , FullCP and Jackknife+ use a smaller exact-reference run. Paired comparisons are provided in Appendix C.3 .

Task	Domain	$K$	$n$	Design hypothesis
CIFAR-10 softmax	Vision	10	10000	Classification-style grouped stress test
20 Newsgroups topics	NLP	10	7539	Mild / smooth heterogeneity
SemEval AffectiveText	NLP/Affect	6	1000	Small-sample weak structure
Samson NMF unmixing	Remote sensing	3	9025	Aligned grouped heterogeneity
PBMC deconvolution	Genomics	8	5000	Semi-synthetic sensitivity case
UTKFace age LDL	Demo- graphics	10	23687	Strong grouped heterogeneity

Table 2: Real tasks in SimplexUQ.

Figure 3: Archived task-specific disparity results. CIFAR-10, Samson, and UTKFace favor grouped repair in these runs; Topics and AffectiveText differ under their archived protocols. Reference-only cells: UTKFace FullCP uses 2,000 samples; PBMC Jackknife+, OneShot, and TrainRes use five-stratum auxiliary runs, while its other displayed methods use two-stratum main runs. Those PBMC cells do not support cross-protocol rankings. Fixed-map robustness is reported separately in Appendix C.5 .

Figure 4: Within-task disparity versus mean score radius from the same sources as the summary tables; bars show

\pm 1

standard deviation. Radius measures radial extent, not volume. Smaller values are preferable only alongside acceptable coverage and comparable protocols. UTKFace FullCP and PBMC auxiliary-method points retain the reference-run limitations in Figure 3 ; axes are not comparable across tasks.

Observed pattern	Diagnostic evidence	Prefer	Main caveat
Near-homogeneous regime	Low max disparity and acceptable worst-stratum coverage	Global split CP	Extra adaptation can add variance or compute without improving allocation
Coarse aligned heterogeneity	Hard/easy regions line up with entropy bins, dominant class, or other meaningful groups	Group-wise / Mondrian CP	Fragmented or weakly aligned groups can become unstable in small samples
Smooth heterogeneity	Coverage changes gradually across entropy or boundary proximity	Compare two-stage normalization with Mondrian; exact variants if affordable	Naive OneShot normalization is not guaranteed, and exactness can be expensive
Bias-type or predictor-driven failure	High disparity remains after both grouping and normalization	No wrapper recommendation; investigate the predictor or score	Wrapper choice alone may not repair structural misspecification
Small-sample weak-structure setting	Few calibration points per group and similar floors across valid methods	Conservative global or smooth normalization strategies	Mondrian can overfit the calibration split and hurt worst-stratum behavior

Table 3: Benchmark-derived candidate-selection heuristic, to be applied after a task’s diagnostics are computed. Recommendations are starting points for held-out comparison, not guarantees or a validated universal selection rule, and must be re-derived after a material predictor change.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Source asset	Citation	Usage terms in this benchmark
CIFAR-10	Official CIFAR-10 image benchmark, accessed through cached torchvision downloads	[ 21 ]	Public benchmark dataset used only to build frozen softmax predictions. The paper cites the official technical report requested by the source page and does not redistribute the image archive as part of the benchmark tables or figures.
Topics	20 Newsgroups text collection, accessed through scikit-learn’s fetcher	[ 22 ]	Public educational/research benchmark used only to derive cached topic-mixture arrays. We treat the corpus as a source-cited benchmark asset rather than republishing the raw text collection.
Affective- Text	SemEval-2007 Task 14 headlines and gold emotion labels	[ 23 ]	Shared-task benchmark data used locally for evaluation. We do not repackage the raw headline collection in the paper outputs, and the conformal benchmark operates on derived cached predictions and normalized gold labels.
Samson	Public 95 $\times$ 95 Samson ROI plus endmember/abundance benchmark files used in hyperspectral unmixing	[ 24 ]	Academic benchmark files used unmodified with attribution. The public bundle we used does not ship an explicit license file, so we document it as a source-cited research asset and avoid presenting it as a newly redistributable paper asset.
PBMC	10x Genomics PBMC3K single-cell reference, accessed through scanpy.datasets.pbmc3k	[ 25 ]	Public 10x tutorial data. Our benchmark uses derived pseudobulk mixtures and frozen deconvolution outputs rather than redistributing the raw count matrix. Source terms and any permitted redistribution are documented separately in the task card.
UTKFace	Official UTKFace aligned-and-cropped face images	[ 26 ]	Non-commercial research only according to the dataset homepage. Copyright remains with the original image owners, so the benchmark uses derived features and predictions and does not repackage the face-image archive in the paper outputs.

Appendix

Table 4: Real-benchmark asset provenance and usage terms. Because the paper contributes an evaluation artifact, these source and usage conditions are part of the benchmark specification rather than ancillary metadata.

Item	Benchmark setting
CPU-only conformal evaluation on a personally owned Apple MacBook Air.
Predictor regeneration	Not part of repeated benchmarking. Each task first constructs or caches a fixed simplex predictor $U$ , after which all conformal runs operate on frozen arrays.
Synthetic repetitions	200 random calibration/test splits per regime.
Real repetitions	50 random calibration/test splits for most tasks; AffectiveText and PBMC use 200.
Cheap methods	Global, Mondrian/partition, and most normalization-style methods are typically sub-second per repetition once $(Y,U)$ is cached.
Expensive methods	FullCP: 2.30 sec/rep on D2, 3.35 on Topics, 3.71 on Samson, and 3.25 on full-scale CIFAR-10.

Appendix

Table 5: Compute environment and repetition counts for the benchmark.

Figure 5: Representative synthetic stratified coverage profiles. Normalization is competitive in D2, aligned grouped calibration helps in D3, and global coverage is uneven in D6. These profiles do not establish a unique winner among wrapper families; paired synthetic comparisons are reported separately.

Figure 6: Stratified coverage from the archived task runs used in the summary tables. Grouped repair helps on CIFAR-10 and UTKFace; Topics and AffectiveText have different rankings under their archived protocols. PBMC displays only methods from its two-stratum main protocol, not the incompatible five-stratum auxiliary runs.

Figure 7: Runtime-versus-disparity tradeoff on the smooth-scale synthetic benchmark. Exact and leave-one-out variants can improve allocation reliability, but they do so at a nontrivial computational cost relative to split, partition, and normalization-based methods.

Task	Boundary	Entropy	Dominant	KMeans	Winner stability
CIFAR-10	Mondrian (0.021, 0.901)	Mondrian (0.022, 0.902)	Mondrian (0.020, 0.902)	Mondrian (0.024, 0.902)	Stable: Mondrian
Topics	Mondrian (0.024, 0.902)	Mondrian (0.025, 0.902)	Mondrian (0.038, 0.901)	Mondrian (0.028, 0.902)	Stable: Mondrian
AffectiveText	Mondrian (0.070, 0.919)	Mondrian (0.071, 0.919)	Global (0.106, 0.906)	Mondrian (0.073, 0.918)	Mixed: Global, Mondrian
Samson	Mondrian (0.020, 0.902)	Mondrian (0.025, 0.903)	Mondrian (0.011, 0.901)	Mondrian (0.079, 0.903)	Stable: Mondrian
UTKFace	Mondrian (0.014, 0.900)	Mondrian (0.013, 0.901)	Mondrian (0.027, 0.901)	Mondrian (0.019, 0.902)	Stable: Mondrian
PBMC	Mondrian (0.016, 0.902)	Mondrian (0.029, 0.903)	Mondrian (0.031, 0.903)	Mondrian (0.031, 0.904)	Stable: Mondrian

Appendix

Table 6: Real-task stratification sensitivity across fixed alternative strata. Each cell reports the best valid method under that stratification, shown as method name with

(\text{max disparity},\text{marginal coverage})

. A method is treated as valid when mean marginal coverage is at least nominal minus the configured tolerance.

Figure 8: PBMC sensitivity checks. Left: alternative fixed stratifications of frozen outputs. Right: archived pseudobulk configurations with Dirichlet concentrations 0.5, 1.0, and 2.0; the 1.0 point uses the retained two-stratum main run. The right panel changes generated data and realized strata as well as concentration, so it is descriptive rather than a concentration-only causal comparison.

Task	Method	Coverage	Worst	Disparity	Radius	Runtime (s)
CIFAR-10	Global	0.900	0.542	0.358	0.153	0.000
CIFAR-10	Mondrian	0.902	0.886	0.022	0.188	0.001
CIFAR-10	TwoStage	0.901	0.641	0.259	0.119	0.079
CIFAR-10	FullCP	0.900	0.656	0.244	0.107	3.247
CIFAR-10	Jackknife+	0.900	0.657	0.243	0.108	0.151
CIFAR-10	OneShot	0.882	0.609	0.291	0.091	0.150

Appendix

Table 7: Real-task metrics from the same saved sources as the original figure.

\dagger

denotes reference-only cells: UTKFace FullCP uses 2,000 samples; PBMC auxiliary methods use a different five-stratum evaluation from the two-stratum main run. These cells do not establish head-to-head rankings. Weighted is an inverse-scale diagnostic. Runtime is taken from each method’s own run, not another configuration.

Task	Method	Coverage	Worst	Disparity	Radius	Runtime (s)
AffectiveText	Global	0.904	0.892	0.100	29.073	0.000
AffectiveText	Mondrian	0.902	0.582	0.356	28.893	0.000
AffectiveText	TwoStage	0.905	0.893	0.100	86.693	0.003
AffectiveText	FullCP	0.903	0.889	0.101	38.979	0.308
AffectiveText	Jackknife+	0.894	0.880	0.101	37.746	0.004
AffectiveText	OneShot	0.890	0.873	0.102	34.973	0.004

Appendix

Table 8: Real-task metrics from the same saved sources as the original figure.

\dagger

Stratum	Global	Mondrian	TwoStage	FullCP	Jackknife+	OneShot	TrainRes	Weighted
S0	0.999	0.902	0.999	0.998	0.998	0.998	0.999	0.650
S1	0.998	0.901	0.998	0.996	0.996	0.995	0.996	0.000
S2	0.992	0.905	0.983	0.978	0.978	0.970	0.977	0.000
S3	0.972	0.902	0.882	0.872	0.872	0.838	0.871	0.000
S4	0.542	0.902	0.641	0.656	0.657	0.609	0.663	0.000

Appendix

Table 9: CIFAR-10: mean stratum coverage from the corresponding summary sources. Reference cells (

\dagger