Between Resolution Collapse and Variance Inflation: Weighted Conformal Anomaly Detection in Low-Data Regimes

Oliver Hennhöfer; Christine Preisach

stat.MLMar 24, 2026

Between Resolution Collapse and Variance Inflation: Weighted Conformal Anomaly Detection in Low-Data Regimes

Authors: Oliver Hennhöfer, Christine Preisach

Organizations: Intelligent Systems Research Group (ISRG), Karlsruhe University of Applied Sciences, Germany

Abstract

Standard conformal anomaly detection provides marginal finite-sample guarantees under the assumption of exchangeability . However, real-world data often exhibit distribution shifts, necessitating a weighted conformal approach to adapt to local non-stationarity. We show that this adaptation induces a critical trade-off between the minimum attainable p-value and its stability. As importance weights localize to relevant calibration instances, the effective sample size decreases. This can render standard conformal p-values overly conservative for effective error control, while the smoothing technique used to mitigate this issue introduces conditional variance, potentially masking anomalies. We propose a continuous inference relaxation that resolves this dilemma by decoupling local adaptation from tail resolution via continuous weighted kernel density estimation. While relaxing finite-sample exactness to asymptotic validity, our method eliminates Monte Carlo variability and recovers the statistical power lost to discretization. Empirical evaluations confirm that our approach not only restores detection capabilities where discrete baselines yield zero discoveries, but outperforms standard methods in statistical power while maintaining valid marginal error control in practice.

Figures & tables

Panel of Figure 1 — Figure 1: The Resolution–Variance Dilemma. Under distribution shift, high importance weights create large steps in the conservative estimator, imposing a resolution floor of minimum attainable $p$ -values that prevents rejection even for extreme scores. Standard randomization resolves the floor but introduces variance inflation via noise that can mask the signal. The proposed Weighted KDE decouples resolution from sample size, enabling rejection below the floor without stochastic noise.

Dataset	Method	Deterministic		Randomized		$n_{\text{train}}$	$n_{\text{test}}$
	Homogeneous	FDR	Power	FDR	Power
WBC	EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.062\pm 0.129$	$0.200\pm 0.332$	106	56
	Weighted EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.075\pm 0.245$	$0.100\pm 0.244$
	KDE	$0.095\pm 0.152$	$0.500\pm 0.351$	—
	Weighted KDE	$0.078\pm 0.142$	$0.417\pm 0.373$	—
Ionosphere	EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.076\pm 0.199$	$0.150\pm 0.221$	112	88

Table 1: Performance of conformal inference strategies across anomaly detection benchmarks. All weighted methods employ WCS with homogeneous pruning to guarantee finite-sample FDR control. Values represent the mean

\pm

standard deviation of the empirical marginal FDR and statistical power aggregated over 20 independent trials with randomized splits. We compare deterministic and randomized baselines against the proposed continuous (KDE-based) approach. The calibration and test set sizes are denoted by

n_{\text{train}}

and

n_{\text{test}}

. Underlined values indicate validity violations (see Section 7.3 ).

Panel of Figure 2 — Figure 2: Statistical power of weighted and randomized conformal methods with WCS pruning strategies, including the weighted KDE-based approach. Labels refer to the WCS pruning method. Results for Musk are omitted due to ceiling performance across all strategies. Error bars denote mean $\pm$ standard error over 20 randomized trials.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	PR-AUC (mean $\pm$ std)	ROC-AUC (mean $\pm$ std)	Brier (mean $\pm$ std)	Wins
WBC	IForest	0.987 $\pm$ 0.009	0.993 $\pm$ 0.004	0.051 $\pm$ 0.005	12/20
	COPOD	0.984 $\pm$ 0.008	0.992 $\pm$ 0.004	0.048 $\pm$ 0.011	3/20
	ECOD	0.984 $\pm$ 0.008	0.992 $\pm$ 0.004	0.049 $\pm$ 0.012	2/20
	HBOS	0.972 $\pm$ 0.014	0.988 $\pm$ 0.006	0.050 $\pm$ 0.006	3/20
	LODA	0.913 $\pm$ 0.029	0.962 $\pm$ 0.011	0.071 $\pm$ 0.009	0/20
	INNE	0.908 $\pm$ 0.042	0.954 $\pm$ 0.019	0.219 $\pm$ 0.011	0/20

Appendix

Table 4: Per-dataset detector comparison (mean

\pm

std over 20 seeds). Results are aggregated across independent trials with per-seed model selection. Wins counts how often (out of 20) a detector is selected for the dataset according to the lexicographic rule PR-AUC

\uparrow

, ROC-AUC

\uparrow

, Brier

\downarrow

Dataset	Model	PR-AUC (mean $\pm$ std)	ROC-AUC (mean $\pm$ std)	Brier (mean $\pm$ std)	Wins
Vowels	INNE	0.594 $\pm$ 0.092	0.913 $\pm$ 0.028	0.176 $\pm$ 0.012	20/20
	IForest	0.312 $\pm$ 0.087	0.783 $\pm$ 0.049	0.188 $\pm$ 0.013	0/20
	LODA	0.243 $\pm$ 0.065	0.696 $\pm$ 0.057	0.204 $\pm$ 0.012	0/20
	HBOS	0.240 $\pm$ 0.076	0.701 $\pm$ 0.053	0.236 $\pm$ 0.015	0/20
	ECOD	0.227 $\pm$ 0.070	0.628 $\pm$ 0.066	0.228 $\pm$ 0.017	0/20
	COPOD	0.114 $\pm$ 0.028	0.526 $\pm$ 0.061	0.221 $\pm$ 0.010	0/20

Appendix

Table 5: Per-dataset detector comparison (mean

\pm

\uparrow

, ROC-AUC

\uparrow

, Brier

\downarrow

Dataset	Method	Deterministic		Randomized		$n_{\text{train}}$	$n_{\text{test}}$
	Deterministic	FDR	Power	FDR	Power
WBC	EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.062\pm 0.129$	$0.200\pm 0.332$	106	56
	Weighted EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.050\pm 0.224$	$0.017\pm 0.074$
	KDE	$0.095\pm 0.152$	$0.500\pm 0.351$	—
	Weighted KDE	$0.078\pm 0.142$	$0.417\pm 0.373$	—
Ionosphere	EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.076\pm 0.199$	$0.150\pm 0.221$	112	88

Appendix

Table 6: Performance of conformal inference strategies across anomaly detection benchmarks. All weighted methods employ WCS with deterministic pruning to guarantee finite-sample FDR control. Values represent the mean

\pm

n_{\text{train}}

and

n_{\text{test}}

. Underlined values indicate validity violations (see Section 7.3 ).

Dataset	Method	Deterministic		Randomized		$n_{\text{train}}$	$n_{\text{test}}$
	Heterogeneous	FDR	Power	FDR	Power
WBC	EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.062\pm 0.129$	$0.200\pm 0.332$	106	56
	Weighted EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.075\pm 0.245$	$0.083\pm 0.183$
	KDE	$0.095\pm 0.152$	$0.500\pm 0.351$	—
	Weighted KDE	$0.078\pm 0.142$	$0.417\pm 0.373$	—
Ionosphere	EDF	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.076\pm 0.199$	$0.150\pm 0.221$	112	88

Appendix

Table 7: Performance of conformal inference strategies across anomaly detection benchmarks. All weighted methods employ WCS with heterogeneous pruning to guarantee finite-sample FDR control. Values represent the mean

\pm

n_{\text{train}}

and

n_{\text{test}}

. Underlined values indicate validity violations (see Section 7.3 ).

Anomaly Detection

Anomaly detection finds rare or unexpected patterns in data, from defective products and system faults to fraud and security threats.

See all 652 papers

Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with 'nonconform'
May 13, 2026Oliver Hennhöfer, Maximilian Kirsch, Christine PreisachFDR Control Conformal Prediction
Intelligent Systems Research Group, Karlsruhe University of Applied Sciences
Weight Clipping for Robust Conformal Inference under Unbounded Covariate Shifts
May 3, 2026James Wang, Surbhi GoelDistribution Shift Robustness Weighted Conformal Prediction
Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA.
Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference
May 20, 2026Ziang Song, Ying Jin, Emmanuel J. CandèsFDR Control Conformal Prediction
Department of Statistics, Stanford University · Department of Statistics and Data Science, University of Pennsylvania · Department of Mathematics, Stanford University

May 13, 2026J/K move · Enter open · S save

Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with 'nonconform'

Oliver Hennhöfer, Maximilian Kirsch, Christine Preisach

Intelligent Systems Research Group, Karlsruhe University of Applied Sciences

FDR Control Conformal Prediction Anomaly Detection

Open paper AskPDF arXiv

Most anomaly detection systems output scores rather than calibrated decisions, leaving practitioners to choose thresholds heuristically and without clear statistical interpretation. Conformal anomaly detection addresses this limitation by converting anomaly scores into calibrated p-values that are valid under the statistical assumption of data exchangeability, with a growing literature extending this idea beyond that setting. We present 'nonconform', a Python package for applying conformal anomaly detection within existing machine-learning workflows, and use it as the basis for an implementation-grounded introduction to the field. The package integrates with 'scikit-learn', 'pyod', and custom anomaly detectors, and provides a unified interface for calibration, p-value generation, and false discovery rate control. It supports several conformalization strategies, ranging from simple split-conformal calibration to more data-efficient and shift-aware extensions. Through a progression from foundational concepts to advanced conformalization strategies, complemented by code examples, the paper connects the statistical ideas behind conformal anomaly detection to their practical use in 'nonconform'. Empirical results demonstrate that the implemented methods enable statistically principled anomaly detection. Together, the package and exposition aim to make core conformal anomaly detection workflows more accessible and reproducible in experimental and production-oriented settings.

Abstract

Anomaly Detection

Explore similar work

Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with 'nonconform'

Weight Clipping for Robust Conformal Inference under Unbounded Covariate Shifts

Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference