Conformal test martingales (CTMs) are e-processes for sequentially testing exchangeability via p-values produced by a conformity function A. However, any distribution shift that preserves the push-forward P∘A−1 leaves the p-value distribution invariant, rendering it invisible to every CTM based on A. We formalise this blind spot through the \emph{A-equivalence class} [P]A and show that, for any target distribution T, there is a unique closest element Q∗∈[P]A in KL divergence---the \emph{A-cryptic I-projection}---with explicit density q∗(z)=t(z)pS(A(z))/tS(A(z)) and a Pythagorean decomposition into detectable and cryptic components. When A is the Neyman-Pearson likelihood ratio, the cryptic component vanishes, connecting our result to optimal e-value design. We demonstrate the theory on high-dimensional image data by empirically constructing this I-projection. By conditionally resampling EMNIST letters to identically match the conformity scores of MNIST digits, we demonstrate that a distribution can be shifted to a disjoint support while leaving the test martingale mathematically blinded.
Figures & tables
Figure 1: The A -cryptic I-projection Q∗ satisfies a Pythagorean identity.
Figure 2: Empirical A -cryptic I-projection. Left: Conformity score densities. By construction, the cryptic projection Q∗ empirically matches the null P . Middle: Sleeper/Stayer martingales. The naive shift to T is rapidly detected, whereas Q∗ perfectly tracks the null hypothesis. Right: Image triplets. The Q∗ samples are pristine letters, demonstrating that a distribution can be shifted to a disjoint support within the equivalence class [P]A without altering the test statistic.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Exact A -cryptic I-projection in the Gaussian setting. Left: T with μT=[5,5] (orange) is distant from P (blue) in feature space, but Q∗ (purple) reweights T to recover P ’s score distribution (middle). The CTM for Q∗ remains non-increasing (right).
Figure 4: Exact A -cryptic I-projection in the Gaussian setting. Left: T with μT=[10,10] (orange) is distant from P (blue) in feature space, but Q∗ (purple) reweights T to recover P ’s score distribution (middle). The CTM for Q∗ remains non-increasing (right).
Figure 5: Exact A -cryptic I-projection in the Gaussian setting. Left: T with μT=[9,−2] (orange) is distant from P (blue) in feature space, but Q∗ (purple) reweights T to recover P ’s score distribution (middle). The CTM for Q∗ remains non-increasing (right).
Conditional conformal test martingales provide sequential tests for detecting distribution shifts, using a fixed reference dataset to avoid contamination from test-time samples. Prior work used a penalized betting function via online optimization with smoothing, clipping, and other hyperparameters, which can adapt slowly when a shift occurs after a long stationary period. In this work, we introduce an approach that requires no hyperparameters and provably adapts quickly to shifts occurring at unknown times. Our key idea is to frame the penalized betting problem as portfolio selection in a three-asset market. Then, we use a restarting universal portfolio, with a heavy-tailed wealth allocation over time. We establish anytime false-alarm control and derive optimal finite-sample bounds on detection delay under detectable alternatives. We also develop an FFT-based implementation that computes the portfolio sequence over T rounds in O(T2lnT) time, instead of O(T3) of the naive implementation. Synthetic and real-world experiments suggest that the proposed approach adapts rapidly after long periods of null samples, with detection delays depending substantially less on shift time than prior baselines.
Tuo Liu, Drew Prinster, Francesco Orabona
King Abdullah University of Science and Technology, Saudi Arabia · Johns Hopkins University, USA
Split conformal prediction provides finite-sample marginal coverage under exchangeability, but this guarantee averages over the random calibration sample. We study instead the law of the calibration-conditional coverage induced by a realized conformal threshold. In the continuous i.i.d. setting this law is exactly Beta(k,n+1−k), so the usual marginal guarantee corresponds to its mean. We take this beta law as a finite-sample reference object and quantify departures from it using Wasserstein distances on [0,1]. The framework yields direct bounds on marginal coverage gaps and on bad-calibration probabilities, and separates different sources of non-i.i.d. behavior according to how they deform the beta reference: test-side shift acts through a transport map on the coverage scale, while calibration dependence changes the order-statistic law itself. We instantiate the framework in scale-shift, clustered, and stationary mixing settings, where the induced deformations can be characterized explicitly or through Berry-Esseen approximations. Simulations on dependent processes confirm that the first-order approximation tracks the empirical Wasserstein distance even at moderate sample sizes.
Thiago R. Ramos, Helton Graziadei, Luben M. C. Cabezas
Federal University of S˜ao Carlos · Federal University of S˜ao Carlos, University of S˜ao Paulo, Inria and Universit´e Grenoble Alpes
A fundamental assumption in statistics and machine learning is that ``the future looks like the past,'' formalized as exchangeability: the joint data distribution is order-invariant. In practice, this assumption is often violated due to distribution shifts over time. Early detection of exchangeability violations is crucial to prevent performance degradation and enable timely interventions like model retraining. Conformal test martingales offer a flexible, distribution-free framework for sequential exchangeability testing with guaranteed false-alarm rate control by betting against the uniformity of conformal p-values. While alternatives such as plug-in martingales and mixture-based strategies exist, computationally efficient baselines like the Simple Jumper are limited to detecting mean location shifts. We propose a family of conformal test martingales based on shifted Legendre polynomials that extend the Simple Jumper to higher-order moments. The Simple Legendre Jumper replaces linear betting functions with polynomials of arbitrary degree, enabling rapid detection of variance, skewness, and other higher-order deviations. The Product Legendre Jumper combines multiple polynomial degrees into a single betting function but suffers from exponential state-space growth, termed the jumping tax. To resolve this, we introduce the Variational Legendre Jumper, which employs a mean-field approximation to reduce complexity to constant time per step with minimal power loss, providing an expressive, scalable framework for real-time distribution shift monitoring.