Distribution Shift Detection

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 36

Oct 6, 2026cs.DC

Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations

Concept drift threatens production machine learning, yet the empirical behavior of multivariate two-sample drift detectors at scale remains under-characterized. Existing benchmarks rarely address the hundreds of millions of rows and high-cardinality features typical of industrial-operational datasets. We evaluate five multi-column two-sample tests (marginal, projection-based, and kernel embedding methods) across three complementary environments: the Harvard Dataverse, a validated Failing Loudly reproduction (mean absolute error between 0.030 and 0.053), and a novel synthetic-injection benchmark on the 137.5-million-row Trendyol collection-ranking feature table. Testing four drift types across two severity-scope regimes, we demonstrate that distributed Maximum Mean Discrepancy with Random Fourier Features on Apache Spark scales robustly. Averaged over the four drift types in the strong regime and under a calibrated threshold, it achieves a Pearson correlation of r = 0.940 with expected drift magnitude, an 80.4% true positive rate, and a 3.2% false positive rate. Conversely, the per-dimension Kolmogorov-Smirnov test failed due to statistic saturation from ID-like columns under asymmetric sampling, establishing a critical constraint for large-scale sampling design. At weak configurations (realized-flip fractions of at most 0.57%), detectors struggled to reliably discriminate, highlighting the need for future intensity-grid power analyses to distinguish fundamental sensitivity bounds from scalable threshold shifts.

Cagdas Pullu, Mahmut Emir Arslan, Bugra Balkac +4

Data & Analytics, Trendyol Group, Istanbul, 34485, Istanbul, Türkiye. · Department of Data Science and Analytics, Istanbul Technical University, Maslak, Istanbul, 34469, Istanbul, Türkiye.

Two-Sample Testing Concept Drift Maximum Mean Discrepancy Distribution Shift Detection

Oct 5, 2026cs.LG

The Blind Spot Paradox: When Adaptive Classifiers Defeat Drift Detectors

Monitoring concept drift from an adaptive classifier's error stream creates an operational conflict with the model's own update loop. When internal adaptation outpaces evidence accumulation, accuracy recovers before cumulative detectors (CUSUM, Page-Hinkley) can reach threshold. Instrumenting an Adaptive Random Forest (ARF) shows that surviving trees absorb 98.6% of the post-drift error transient through incremental leaf updates alone. The first background tree swap accounts for just 0.71% of this erased error volume, but drops external detection rates by 31 percentage points. We derive the finite-horizon boundary where cumulative evidence fails to cross threshold and measure a critical magnitude floor (

Δe_c = 0.120

) below which false-alarm budgets preclude detection. This failure manifests as missed shifts on stationary streams and false-alarm flooding triggered by internal tree swaps on noisy baselines. We validate on synthetic shifts, ARMA-GARCH series (ProteuS), and tabular benchmarks (BAF, INSECTS); on the synthetic sweep at a standard threshold, the blind spot appears at

Δe \approx 0.25

, showing why classical benchmarks like SEA (

Δe \le 0.21

) failed to reach it.

Raphaël Minato, Fabrice Popineau, Arpad Rimmel +1

Université Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique, 91190, Gif-sur-Yvette, France

Concept Drift Distribution Shift Detection Change-Point Detection

Oct 1, 2026stat.ML

Zero Flux: Flow-Based Comparison of High-Dimensional Discrete Distributions

Comparing two high-dimensional discrete distributions has always been a challenging task due to the exponentially growing state space and complex changes in interactions. A recent work suggests comparing distributions through a vector field trained using flow matching between two continuous distributions. The resulting vector field at mid-point vanishes if and only if two distributions identical. However, such a flow-based criterion does not naturally apply to discrete distributions. We extend this principle to the discrete domain and introduce the \emph{Zero Flux} criterion, a discrepancy based on local probability fluxes. Under independent coupling, we show that all local probability fluxes vanish at the midpoint if and only if two distributions are the same. This discrepancy decomposes the joint distributional difference into smaller, local contributions and can be efficiently estimated from samples. We establish finite sample error bounds for our estimator. Experiments on synthetic and real categorical data demonstrate reliable recovery of sparse dependence signals and stable tracking of distribution shifts in high dimensions.

Leyang Wang, Yakun Wang, Song Liu +1

Department of Mathematical Informatics, The University of Tokyo, Japan · Center for Advanced Intelligence Project, RIKEN, Japan · School of Mathematics, University of Bristol, United Kingdom

Distribution Shift Distribution Shift Detection Distribution Matching Discrete Flow Matching

Oct 1, 2026cs.LG

Dataset Identity, Not Novelty: The Source of an Inflated OOD Detection Gain

A post-hoc out-of-distribution (OOD) detector reads the activations of a trained classifier and returns a score. It fits that score on in-distribution data, and the benchmarks that evaluate it supply a second piece of OOD data for the fitting itself. Some detectors tune a constant on it. Others fit a direction in feature space or train a flexible combiner and report the number that it reaches as the gain that is still available. Every such fit is validated on held-out samples of the same OOD dataset. That check rules out memorizing individual images. It says nothing about a fit that has instead learned which dataset it is looking at, and a direction that recognizes one OOD dataset rather than novelty passes it perfectly. The detector that a practitioner installs meets OOD data from a source that nobody fitted it on, so the difference decides what the reported number is worth. We measure it by holding out the whole OOD dataset rather than a sample of it, and we call that gap the inflation. We read it across a range of combiners on ImageNet and CIFAR-100 backbones. Most of the gain that the usual protocol reports turns out to be dataset identity rather than novelty. The size of the fit does not move what survives, so the effect is not ordinary overfitting. The share depends instead on whether the input exposes class identity, and two controls that vary that property alone separate the inflation on every backbone of both benchmarks. A closed form accounts for the effect and computes it from the fitting rows, so a practitioner can tell which fits will inflate without running the hold-out protocol. One of these fits survives, namely the single constant that the field already picks on a designated validation dataset. Anything above it reports a gain that the hold-out protocol does not return, and on one benchmark what survives falls while what is reported climbs.

Donghoon Lee, Shinjin Kang

Hongik University

Distribution Shift Detection OOD Detection Cross-Dataset Generalization

Sep 29, 2026cs.RO

When to Adapt: Multi-Signal Domain Shift Detection for Efficient Training-Free Adaptation in Open-Vocabulary Segmentation

Robust and reliable perception is essential for autonomous robots operating in real-world environments, particularly in long-term missions where environmental conditions may change significantly over time. Although recent advances in Visual Foundation Models (VFMs) have improved open-vocabulary semantic segmentation, these models can still suffer from domain shift, which can significantly degrade performance if they are not adapted to the current environment. Training-free domain adaptation is a relevant paradigm for adaptation, consisting of adjusting the model online using lightweight adapters. Recent approaches apply this on a per-frame basis, which is impractical for deployments on resource-constrained robotic hardware. To tackle this, we propose a multi-signal domain shift detection method for training-free continual test-time adaptation (TF-CTTA) in open-vocabulary segmentation. Our method leverages temporal coherence across consecutive frames by monitoring and combining complementary aspects of domain shift (visual change, adapter mismatch, and semantic drift) to trigger adaptation only when needed. We validate our approach on a benchmark including indoor and outdoor environments and using real robotic data. We demonstrate that our approach maintains segmentation accuracy while substantially reducing adaptations, making training-free adaptation practical and feasible for long-term, real-world robotic deployments.

Michele Antonazzi, Alejandra C. Hernandez, José Araujo +2

EECS and Digital Futures, KTH Royal Institute of Technology, Stockholm, Sweden · Ericsson Research, Ericsson AB, Stockholm, Sweden

Distribution Shift Continual Test-Time Adaptation Vision Foundation Model Adaptation Distribution Shift Detection Open-Vocabulary Semantic Segmentation Robotic Perception

Sep 27, 2026cs.LG

Behavioral Monitoring of JEPA World Models with Jacobian Centroids

Detecting failures in World Model (WM)-based planning requires monitoring whether the model is behaviorally aligned with the current task, which in turn requires studying its internal representations. Here, we show that centroids---sub-component Jacobian row-sums---effectively identify the behavioral properties of WMs, complementing traditional activation-based knowledge signals. The centroids of a model are easily computed through Jacobian vector products and characterize how the model organizes the geometry of its input space, yielding an efficient perspective on internal representations, including the generation of task-relevant saliency maps. Evaluated on continuous control tasks using JEPA WMs, this behavioral view reveals a structural dissociation, where the encoder correctly represents the goal while the predictor remains behaviorally unresponsive. This failure mode directly predicts planning failure before any action is taken, allowing for goal resampling to recapture out-of-distribution success. Moreover, centroid-based methods outperform baseline methods as distribution-shift detectors. Together, these tools yield a behavioral monitoring stack that is operational and consequential under distribution shifts.

Thomas Walker, Randall Balestriero, Richard Baraniuk

Rice University · Brown University

Distribution Shift World Model-Based Planning Joint-Embedding Predictive Architecture Distribution Shift Detection World Model Evaluation

Sep 14, 2026cs.LG

A Decision-Support Audit Protocol for Supervision Drift in Proxy-Labeled Credit-Risk Prediction

Credit-risk models are trained on proxy labels and deployed under temporal and segment change, yet no single transfer metric separates base-rate shift, probability-scale shift, and feature-label relationship change. We contribute a design-science artifact: a locked, multi-signal audit protocol for supervision drift in proxy-labeled credit-risk prediction. Five layers (transfer performance, an oracle-gap probe, a calibration diagnostic, feature-label stability, and a synthetic positive control), thresholds, and decision rules were locked before interpretation; a bounded reading is a designed outcome. On a public LendingClub dataset (temporal 2013 to 2016 and cross-segment transfer), ranking is stable and oracle gaps are small; the clearest temporal signal is a prevalence and probability-scale mismatch that intercept-only diagnostic recalibration largely reduces, though its cause is not identifiable from the available release. The positive control responds only to larger injected shifts; subtler drift cannot be excluded. Mapping diagnostic patterns to governance actions is conceptual guidance, not validated here.

Mehrdad Shoeibi, Muhammad Shabanpour, Waldemar Karwowski +1

Department of Industrial Engineering and Management Systems University of Central Florida, Orlando, FL, USA · Department of Economics, Northeastern University, Boston, MA, USA

AI Risk Management Algorithmic Auditing Distribution Shift Detection Credit Scoring

Aug 14, 2026cs.AI

Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers

Safety classifiers deployed with large language models often fail for two reasons: their decisions reflect the policy learned during training rather than the deployer's desired policy, and their performance degrades as deployment traffic evolves. We present Regime-Conditional Verification (RCV), a lightweight wrapper that adapts an off-the-shelf safety classifier without retraining it. RCV estimates, from the classifier's internal representations, the probability that each prediction disagrees with the deployer's policy, and selectively corrects predictions likely to be wrong. The same correctness estimates also provide a label-free signal for detecting distribution shift, enabling a maintenance loop that updates the correctness estimation layer and resorts to classifier fine-tuning only when repair fails within a label budget. Across three off-the-shelf safety classifiers and two benchmark datasets, RCV improves adherence to the deployer's policy in every classifier-dataset combination, catching up to 0.81 of previously missed unsafe content without modifying the underlying classifier. In a deployment study with ten attack campaigns, each a harm category held out of RCV's training, RCV detects every campaign in a dedicated injection panel; in the maintenance census most drift episodes are repaired without updating the classifier, and the fine-tune is reserved for the residual episodes.

Thiago Sandoval, Ufuk Topcu

The University of Texas at Austin

Distribution Shift Selective Prediction Language Model Safety Evaluation Distribution Shift Detection Safety Filtering LLM Safety

Aug 2, 2026stat.ML

How fine a change can moments see? A scale law for detecting distribution shift, with a kernel calibration rule

Detecting that a stream of high-dimensional embeddings has changed is usually framed as a choice of statistic. We give a scale law that constrains any moment-based choice and test it against topological alternatives. The law: certifying a feature of spatial scale eps carrying mass fraction f requires polynomial tests of degree N* >= log(1/f)/(2 eps), proved via the Chebyshev extremal problem; a Gauss-quadrature construction gives N* >= 4b-1 for a b-scale topology, so cost is set by feature fineness, not feature count. The law is one-sided: we exhibit an annulus whose mean, covariance and all fourth-order moments equal those of a filled disk, yet H_1 is nonzero. Its practical content is a calibration rule. The upper bound is attained by Gaussian test functions, the RKHS witness of an RBF kernel, so the law predicts which bandwidth an MMD test should use: the feature scale. On real embedding streams we measure sigma*/eps with median 1.12 (IQR 1.01-1.52, n=26) over three settings and three scales, and a data-driven bandwidth reaches AUC >= 0.95. Against an adversary optimised against the defender's statistics (mean, covariance, k-NN, kurtosis), only a bandwidth-matched kernel test still detects. For persistent homology the verdict is mixed and depends on choices usually left implicit. The summary matters more than the filtration: total persistence attains recall 0.75 at FPR 1% where the first persistence landscape attains 0.00. What survives is a cost gap, not a power gap: where persistence works it costs 116x kurtosis, which works at least as well. We conclude not that topological summaries are useless, but that on this task a kernel test whose bandwidth the law sets dominates them.

Adel Kaleche

Independent researcher, France

Two-Sample Testing Distribution Shift Maximum Mean Discrepancy Distribution Shift Detection

Jul 29, 2026cs.CV

Level, Sharpness, and Corpus: Why Zero-Shot OOD Detector Rankings Do Not Transfer

Selecting a zero-shot out-of-distribution (OOD) detector for a new deployment is typically based on benchmark rankings, implicitly assuming that the highest-ranked detector will transfer across domains. We show that this assumption does not hold. Through a controlled portability audit across seventeen in-distribution datasets, three vision-language models, and seven representative zero-shot OOD detectors, we find that detector rankings reverse across deployments, every detector exceeds

80\%

FPR95 on at least one domain, and the preferred detector depends on both the in-distribution data and the underlying VLM. We trace these reversals to complementary evidence channels in vision-language logits. Corpus-free detectors rely on different combinations of absolute match level and relative or spatial sharpness, while WordNet-based methods additionally depend on external semantic coverage. A simple proposition shows that level and sharpness cannot generally be recovered from one another, explaining why no single detector transfers reliably across deployments. Motivated by this diagnosis, we introduce the Complementary Evidence Guard (CEG), a detector-agnostic wrapper that preserves complementary evidence through a non-compensatory fusion of the base detector, level, and sharpness using only empirical in-distribution percentiles. Controls replacing these channels with entropy, logit variance, or random noise do not reproduce the gains. Without OOD samples, auxiliary corpora, or learned fusion, CEG reduces detector sensitivity and improves GL-MCM from

38.1

to

28.8

and MCM from

42.6

to

30.5

family-balanced FPR95.

Ignacio M. De la Jara, Cristian Rodriguez-Opazo, Stephen Gould +1

University of Adelaide · 3Naval Group Pacific · 2Australian National University

VLM Evaluation Distribution Shift Detection OOD Detection

Jul 29, 2026cs.CV

Representation Trajectories Matters: Complementary Evidence for OOD Detection and Image Classification

Vision models do not form a representation at once; each block revises it. We ask whether the resulting computation path contains evidence that the final representation discards, and whether that evidence improves OOD detection and image classification on clean and shifted data. Unlike approaches that treat intermediate layers as separate snapshots, we retain sample identity across depth and study the transformations connecting successive states. We separate class-coherent transport from input-specific innovation, and coordinate movement from relational reorganization. Across supervised, self-supervised, vision--language, hierarchical, and convolutional encoders, these paths show strong sample-specific continuity and architecture-specific depth profiles that recur across datasets. They are also practically useful. An ID-only transition-surprise score complements strong final-state detectors, reducing FPR95 in 131/152 non-saturated comparisons on a balanced OpenOOD grid; gains are largest for visually disruptive and semantically far shifts, and remain positive on near-OOD for most detectors. Frozen update probes improve 71/72 clean model--dataset cases, while shifted-data gains vary with architecture and corruption type. Computation paths therefore provide a broadly useful reliability signal whose value is determined jointly by model organization and the shift encountered.

Ignacio M. De la Jara, Cristian Rodriguez-Opazo, Hamed Damirchi +2

University of Adelaide · Naval Group Pacific · Australian National University

Distribution Shift Detection OOD Detection Image Classification

Jul 21, 2026cs.AI

Black-Mamba: Biologically-Inspired Leaky Accumulation for Conceptual Knowledge under Distribution Drift

Forecasting under real-world conditions is inherently non-stationary, as the conditional distribution of future observations evolves over time. Recent test-time adaptive sequence models address this challenge by updating internal states during inference, but tie adaptation to instantaneous prediction errors or surprise. This coupling can conflate persistent distribution shift with stochastic innovations, leading to unnecessary updates and inefficient adaptation. We introduce Black-Mamba, a test-time adaptive forecasting architecture that formulates online adaptation as evidence-gated state tracking under distribution drift. The model augments a base predictor with a dynamic memory updated when temporally accumulated surprisal provides sufficient evidence of a regime change. This turns adaptation into a selective, event-driven process rather than a continuous one. Across multiple forecasting benchmarks with non-stationary dynamics, Black-Mamba achieves competitive or improved predictive performance compared to existing test-time adaptation methods while significantly reducing the number of memory updates during inference. Together with mathematical analysis and biological evidence, these results suggest that accumulated surprisal provides a principled signal for distinguishing persistent drift from transient noise, yielding more efficient and robust adaptation.

Giuseppe Soriano, Nicola Tonellotto, Alberto Gotta

University of Pisa National Research Council (CNR) Pisa, Italy · University of Pisa Pisa, Italy · National Research Council (CNR) Pisa, Italy

Test-Time Adaptation Concept Drift Continual Test-Time Adaptation Distribution Shift Detection Non-Stationary Time Series Forecasting

Jul 19, 2026cs.LG

When Drift Detectors cry Wolf: False Alarm Rates in continuous ML Monitoring

Drift detection is a core component of production machine learning monitoring systems, where detectors are used to compare incoming data with a reference distribution and trigger alerts when changes occur. However, these detectors are often evaluated in research settings that emphasize detection accuracy under synthetic shifts, while overlooking false alarms under continuous monitoring. In production environments, models are monitored repeatedly over time and across many features, and even small false positive rates can accumulate into frequent alerts, leading to alarm fatigue. We empirically analyze false positive behavior across five commonly used drift detectors: PSI, KS, MMD, LSDD, and adversarial validation. Consistent with existing literature, PSI exhibits strong sensitivity to batch size, producing frequent false alarms at small sample sizes; however, we further observe that its behavior stabilizes and improves substantially once batch sizes exceed approximately 200 samples. In contrast, KS, MMD, and LSDD display persistent fluctuations across batch sizes, while remaining comparatively more reliable than PSI in low-data regimes. Applying a Bonferroni correction reduces false positive rates, but often at the cost of reduced true positive sensitivity, reinforcing the well-known stability - sensitivity trade-off in drift detection. This work provides a systematic comparison of false positive behavior across multiple drift detectors under continuous monitoring conditions. We identify tradeoffs across detector families and provide practical guidelines for selecting and calibrating drift detectors in production ML systems.

Raj Shekhar Singh

Indian Institute of Technology, Roorkee Roorkee, Uttarakhand, India

Distribution Shift Detection

Jul 18, 2026cs.LG

Dimension-Calibrated Unexplained Mass: An Interpretable Drift Statistic for Contamination Monitoring in Data Streams

Drift detectors that work tend not to explain themselves, and drift detectors that explain themselves tend to fail in high dimension. We close that gap for Gaussian mixture models (GMMs): each fitted component is a named "regime," and the fraction of a stream window matching no regime -- its unexplained mass -- is a drift signal that is simultaneously its own explanation. We identify why this statistic collapses in high dimension and repair it. Under a correct component a normal point in d dimensions lies about sqrt(d) sigma from the mean, so once d exceeds 9 essentially every point exceeds a fixed 3-sigma radius: window-level ROC-AUC is exactly 0.50 on Satellite (d=36) and Optdigits (d=64). Calibrating the radius to sqrt(chi-squared_d(0.99)) removes the collapse -- AUC 1.00 and 0.89 -- while leaving low dimensions unchanged. Across seven public benchmarks, five seeds, and eight model-free detectors spanning the kernel, classifier, projection, density-difference, transport, likelihood and partition families, the repaired statistic is best or tied-best on five of seven datasets at 10% window contamination (its two losses are Pendigits, where the whole field beats it, and Optdigits), and as contamination becomes sparse the sample-level detectors fade toward chance while it degrades most gracefully: at 2% its mean AUC across the benchmarks is 0.86 against at most 0.73 for any model-free detector (1.00 vs. MMD's 0.72 on KDD-http) -- while alone among them reporting which regime the data left and how far outside it the window lies. We delimit its scope honestly: unexplained mass detects and explains novel-regime drift but is blind by construction to in-support re-weighting of known regimes, where distribution-level tests are required and explain nothing; and the underlying density model's EVT-calibrated false-alarm rates degrade above d of about 36. All code and experiments are released.

Behnam Asadi

Concept Drift Gaussian Mixture Models Distribution Shift Detection OOD Detection

Jul 17, 2026cs.LG

ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language

Hallucinations and artificial text in LLM-generated outputs often appear as distributional deviations between prompt and response hidden-state distributions. Since prompts or retrieved contexts typically serve as reference samples and responses as query samples, with major differences in length, these asymmetries motivate the use of change test statistics that treat the two samples differently. We consider an asymmetric two-sample test ASK-NN based on the directed k-nearest-neighbor graph. Our statistic counts reference points whose nearest neighbor in the pooled sample is also a reference point. Under the permutation null, it admits an exact finite-sample conditional mean and variance; we further establish asymptotic normality and consistency under fixed alternatives. ASK-NN is computationally effective and easy to implement. Empirically, it is competitive with kernel and graph-based baselines on synthetic benchmarks, artificial-text detection, and LLM hallucination detection from token-level hidden states.

Sergey Zakharov, Rodion Oblovatny, Alexey Zaytsev

Saint-Peterburg university, Saint-Peterburg, Russia · Risk AI Research Lab, Moscow, Russia · Applied AI Institute, Moscow, Russia

Two-Sample Testing AI-Generated Text Detection Distribution Shift Distribution Shift Detection LLM Hallucination Detection

Jul 4, 2026cs.LG

Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains

Recent AI-generated text detection work often introduces a new benchmark together with a specialized detector tailored to it. We revisit this practice from a baseline-first perspective. Across several benchmarks, we show that a plain, fully fine-tuned RoBERTa matches or exceeds the specialized detectors those benchmarks are built around. This suggests that much of the recent architectural complexity is not what drives strong in-distribution detection. The remaining challenge is the distribution shift. The same strong baseline degrades sharply when the topic domain or generating model changes at test time, and simply adding more source data does not close the gap. We identify a key failure mode: under distribution shift, the detector can assign high-confidence machine labels to human-written text from unseen domains. We then study two lightweight domain adaptation methods to address this problem:

K

-shot adaptation with first-order MAML over LoRA adapters, and a per-sample confidence-weighted ensemble built on top of the adapted detector. Overall, our results suggest that progress in AI-generated text detection should be measured not only by in-distribution performance, but also by robustness under distribution shift.

Zhuoer Shen, Mingyi Wang, Shaofeng Zou +1

Department of Computer Science, University of California, Santa Barbara · School of ECEE, Arizona State University

Distribution Shift Robustness AI-Generated Text Detection Domain Adaptation Distribution Shift Detection Robustness of Machine-Generated Text Detection

Jun 20, 2026cs.LG

Cluster-Specific Localized Drift Detection for Efficient Batch Model Adaptation under Controlled Distribution Shift

Machine learning systems deployed in dynamic environments frequently operate under nonstationary data distributions, where controlled distribution shift can progressively degrade predictive performance. However, many widely used tabular benchmark datasets lack explicit temporal structure, limiting reproducible evaluation of drift adaptation methods. This work proposes a cluster-induced distribution shift simulation framework that transforms static tabular datasets into controlled evolving data streams through structured perturbations across featurespace partitions. Using this framework, six adaptation strategies are systematically evaluated: static learning, sliding-window retraining, global ADWIN retraining, cluster-local ADWIN retraining, random subspace drift detection, and feature-partitioned drift detection. Experiments are conducted on five benchmark datasets covering both classification and regression tasks using diverse predictive model families, including linear models, k-Nearest Neighbours, tree ensembles, boosting methods, and adaptive online learners.

Ignacio Cabrera Martin, Marcello Trovati, Almas Baimagambetov +1

University of Brighton, School of Architecture, Technology and Engineering, Brighton, BN2 4GJ, United Kingdom · University of Lancashire, School of Business, Preston, PR1 2HE, United Kingdom

Distribution Shift Robustness Concept Drift Distribution Shift Detection Change-Point Detection

Jun 18, 2026stat.ML

Betting on Moments: Legendre Jumper Martingales for Online Exchangeability Testing

A fundamental assumption in statistics and machine learning is that ``the future looks like the past,'' formalized as exchangeability: the joint data distribution is order-invariant. In practice, this assumption is often violated due to distribution shifts over time. Early detection of exchangeability violations is crucial to prevent performance degradation and enable timely interventions like model retraining. Conformal test martingales offer a flexible, distribution-free framework for sequential exchangeability testing with guaranteed false-alarm rate control by betting against the uniformity of conformal p-values. While alternatives such as plug-in martingales and mixture-based strategies exist, computationally efficient baselines like the Simple Jumper are limited to detecting mean location shifts. We propose a family of conformal test martingales based on shifted Legendre polynomials that extend the Simple Jumper to higher-order moments. The Simple Legendre Jumper replaces linear betting functions with polynomials of arbitrary degree, enabling rapid detection of variance, skewness, and other higher-order deviations. The Product Legendre Jumper combines multiple polynomial degrees into a single betting function but suffers from exponential state-space growth, termed the jumping tax. To resolve this, we introduce the Variational Legendre Jumper, which employs a mean-field approximation to reduce complexity to constant time per step with minimal power loss, providing an expressive, scalable framework for real-time distribution shift monitoring.

Johan Hallberg Szabadváry

Department of Mathematics, Stockholm University

Conformal Prediction Distribution Shift Detection Change-Point Detection Sequential Hypothesis Testing Temporal Distribution Shift Statistical Hypothesis Testing

Jun 10, 2026cs.LG

TEDD: Robust Detection of Unstable Temporal Features

When working with real-world temporal data, it is common to encounter features whose distribution is changing over time. The naive employment of Machine Learning models on this unstable data might lead to rapidly degrading performance, especially if the new distribution is much different from what was previously seen during training. In order to cope with this problem, it is critical to automatically identify features that are changing over time. With these features detected, data scientists and other practitioners will be able to mitigate the issue (for instance, by applying data transformations), deploying more robust models that retain high performance for longer periods of time. In this paper, we describe which temporal changes a feature should not suffer from, and propose TEDD, a technique to a) identify when a dataset might lead to an unstable Machine Learning model and b) automatically detect which features cause such lack of robustness. In order to achieve it, we leverage a regression model to highlight which features contribute to a good prediction of an instance's timestamp. We compare our approach to other methods in real and synthetic data, testing their detection capability on all simple change patterns. We show that our method: detects all types of basic changes, both for numerical and categorical features; can detect multivariate drifts; returns a comparable value measuring the amount of change of each feature; requires no parameter tuning; and is scalable both on number of features and instances of the dataset.

Ricardo Ribeiro Pereira, Bruno Casal Laraña, Nádia Soares +1

DCC/FCUP University of Porto Porto, Portugal · Feedzai Porto, Portugal · Mindera Porto, Portugal

Distribution Shift Robustness Distribution Shift Detection Change-Point Detection Temporal Distribution Shift

Jun 10, 2026cs.LG

Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers

Safety classifiers deployed in production operate under a stationarity assumption that fails silently: when input distributions drift, accuracy degrades with no error signal until ground-truth labels arrive. We present an online monitor that detects distributional shift in classifier scores via a sliding-window KS statistic with empirically calibrated alarm thresholds. In a pre-registered factorial evaluation (4 classifiers

\times

5 shift conditions

\times

20 seeds

\times

2 window sizes; 800 cells), the monitor achieves 86.6% valid detection (mean latency 39.5 steps) across synthetic-onset, real-jailbreak, and adversarial regimes; a classifier

\times

shift interaction (

η^2 = 0.185

) shows that monitoring must be tuned per classifier. Attempting to recover post-detection coverage via weighted conformal prediction exposes a failure mode: density-ratio estimation collapses for generative classifiers because logistic regression separates source from target perfectly in 3584-4096-dimensional embedding space, clipping all importance weights to zero; projecting to

\leq 32

dimensions restores coverage. We then extend the framework to gradient-based evasion and give the first threat-model characterisation of score-disagreement monitoring as a canary. We falsify three assumptions: that architectural diversity drives the signal (false,

η^2 = 0.011

), that it is generic out-of-distribution detection (false, GCG-specific,

p < 10^{-12}

), and that an adaptive attacker can suppress it (false while the canary is confident). We derive the exact security boundary, a confidence-gated equilibrium at which a monitor-aware attacker stalls at gap

= 1/(2λ)

, and provide a calibration-free scan martingale achieving false-alarm rate

\leq 1\%

across all classifiers with no per-model tuning.

Jun Wen Leong

Distribution Shift Conformal Prediction Distribution Shift Detection AI Safety Evaluation Change-Point Detection Adversarial Example Detection

Jun 5, 2026stat.ML

Empirical Transfer Operators and Finite-Sample Change Detection for Noisy Expanding Interval Maps

We study finite-sample change detection for one-dimensional noisy dynamical systems using partition-based empirical approximations of stationary behaviour. Given observations from an interval-valued process, we partition the state space, estimate a finite transition matrix from observed transitions between partition elements, and apply a small Doeblin-type regularisation to ensure a unique stationary distribution. From an initial reference segment, we compute a baseline empirical stationary distribution

\widehatπ_{0,ρ}

. For each later sliding window, we compute

\widehatπ_{t,ρ}

and define the score

S_t=\|\widehatπ_{t,ρ}-\widehatπ_{0,ρ}\|_1.

Large values of

S_t

indicate a change in stationary behaviour relative to the baseline. The statistic detects changes in invariant density or stationary law, but not all possible changes in transition dynamics. Under explicit assumptions on empirical transition concentration, finite-state stationary distribution stability, partition approximation, regularisation bias, and noise stability, we derive a finite-sample bound for the empirical stationary density. The bound separates sampling error, regularisation bias, partition approximation error, and noise bias. We then obtain a single-window false-alarm guarantee and a sufficient detection condition when the invariant density changes by more than the estimation error. We illustrate the method on synthetic noisy beta-map change-point experiments.

Aparna Rajput

Dynamical Systems Distribution Shift Detection Change-Point Detection Temporal Distribution Shift

May 31, 2026stat.ML

Distribution-free changepoint localization after sequential change detection

This paper introduces a distribution-free framework for constructing post-detection confidence sets for changepoints after stopping a sequential change detection procedure. It is well known that conformal test martingales can be used to sequentially detect changes in distribution, but by themselves provide no inference for the time at which a proclaimed change occurred. Past work on post-detection inference requires pre- and post-change classes of distributions to be known, but this paper accomplishes localization of the changepoint without any distributional assumptions. We establish finite-sample coverage guarantees (conditional on correct detection). We provide non-asymptotic bounds on the conditional expected size of the confidence sets. Under suitable asymptotic regimes, we prove that the conditional expected size of the confidence set remains uniformly bounded and demonstrate strong empirical performance on simulated and real data. To the best of our knowledge, this is the first general distribution-free framework for sequential changepoint localization with valid post-detection coverage.

Aytijhya Saha, Aaditya Ramdas

Massachusetts Institute of Technology · Carnegie Mellon University

Confidence Region Estimation Distribution Shift Detection Change-Point Detection

May 30, 2026eess.AS

Local Diagnostics of Continuous Normalizing Flow for Out-of-Distribution Detection

We address the problem of out-of-distribution (OOD) detection for target observations embedded in a subspace of the high dimensional data space. Using continuous normalizing flows (CNFs), we propose a Lagrangian sub-flow (LSF) framework designed to isolate and estimate the density for the relevant components in the representation and using the remaining components as context. Through experimentation with models for speech synthesis, we show that CNFs, similarly to other deep generative models (DGMs), are susceptible to the "likelihood paradox", where high likelihood is erroneously assigned to OOD samples. This is attributed to the inductive bias of DGMs that prioritize low-level structural details over high-level semantic coherence. To mitigate this phenomenon, we propose a number of geometric diagnostic signals based on the velocity field over the sub-flow trajectory. Based on these signals, we design metrics for the challenging task of zero-shot phoneme-level mispronunciation detection. Finally, we demonstrate the superiority of these metrics compared to likelihood-based methods on a real-world mispronunciation detection benchmark.

Xinwei Cao, Mengxuan Lu, Torbjørn Svendsen +1

Department of Electronic Systems Norwegian University of Science and Technology Trondheim, Norway · School of Future Technology South China University of Technology Guangzhou, China

Speech Processing Distribution Shift Detection Continuous Normalizing Flows OOD Detection

May 29, 2026cs.CV

KLIP: localized distribution shift detection via KL-divergence with diffusion priors in Inverse Problems

Diffusion models have shown promising performance as data-driven priors for computational imaging, as well as some capacity to detect out-of-distribution (OOD) images. However, existing approaches to OOD detection often require some knowledge of the shifted distribution, fail to detect subtle or localized distribution shifts, and operate on full images, rather than the indirect measurements available in inverse problems. We propose an OOD detection metric based on the Kullback-Leibler divergence between the diffusion prior and the posterior distribution, that (i) does not require any calibration data or knowledge of the shifted distribution, and (ii) can detect whole images as OOD as well as localize OOD patches within an image. Experimentally, we show that this metric can detect subtle yet semantically meaningful distribution shifts, such as the shift from healthy liver CT scans to those with tumors, and generalizes across different types of diffusion models, datasets, and inverse problems. Our code can be found at https://github.com/voilalab/KLIP.

Alireza Kheirandish, Jihoon Hong, Sara Fridovich-Keil

Georgia Institute of Technology

Diffusion-Based OOD Detection Distribution Shift Detection Diffusion-Based Inverse Problems OOD Detection Inverse Problems Diffusion Models

May 29, 2026cs.LG

dashi: A Python library for Dataset Shift Characterization to Support Trustworthy AI Development and Deployment

The Artificial Intelligence (AI) life cycle requires a thorough understanding of the underlying data dynamics for robust, safe and cost-effective AI development and use. Dataset shifts are defined as changes between train and test data distributions. Whether occurring over time (temporal) or across different sites (multi-source), they can severely degrade model performance and compromise data quality. This is particularly important in health AI, where the safety and fundamental rights of patients can be severely affected by uncontrolled shifts both at training and operational stages. While the theoretical foundations of covariate, prior, and concept shifts are well established, there is a lack of accessible and comprehensive software tools to perform their analysis. We introduce dashi, an open-source Python library designed for the exploration, quantification, and characterization of dataset shifts. dashi provides a dual approach: an unsupervised approach that leverages information geometry and non-parametric statistical manifolds to data variability characterization and analysis (e.g., Information Geometric Temporal plots and Multi-Source Variability metrics like Global Probabilistic Deviation and Source Probabilistic Outlyingness), and a supervised approach that quantifies and characterizes model performance degradation. Both unsupervised and supervised approaches work across user-defined temporal and domain/source batches. We demonstrate the utility of dashi on three simulated and real-world health AI case studies on gestational diabetes mellitus, COVID-19 and emergency medical dispatch. By providing interactive visual analytics and variability metrics, dashi supports trustworthiness of AI life cycle stages enabling robust and safe machine learning pipelines through the assessment of data coherence and AI performance.

David Fernández-Narro, Pablo Ferri, Ángel Sánchez-García +2

Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de Valéncia, Camino de Vera s/n, Valencia 46022, España

Distribution Shift Detection Temporal Distribution Shift

May 29, 2026cs.LG

Learning Hyperspherical Time-Frequency Representations for Time-Series Out-of-Distribution Detection

Out-of-distribution (OOD) detection for time-series data remains comparatively underexplored compared to vision and language, with a limited principled understanding of how supervised time-series representations can be leveraged for reliable detection under distributional shifts. This work formulates time-series OOD detection as representation learning with hyperspherical embeddings, where class-conditional structure is induced by a von Mises-Fisher (vMF) likelihood-based objective on the unit sphere. The learned representation combines time- and frequency-domain views of the input signal via domain-specific encoders, integrating them into a joint embedding space for OOD detection. Detection uses distance-based scores over the learned embeddings, including k-nearest neighbors (k-NN) and Mahalanobis scores. We evaluate the approach at scale on the complete UCR and UEA time-series archives under a cross-dataset protocol. Empirical results show consistent improvements under both k-NN and Mahalanobis scoring over strong contrastive learning and post-hoc baselines in the same setting. Code is available at https://github.com/tiiuae/hypertf-time-series-ood.

Willian T. Lunardi, Samridha Shrestha, Martin Andreoni

Technology Innovation Institute, Abu Dhabi, United Arab Emirates · Khalifa University, Abu Dhabi, United Arab Emirates

Hyperspherical Representation Learning Distribution Shift Detection Geometric Representation Learning OOD Detection Time Series Representation Learning

May 21, 2026cs.LG

The Signal in the Noise: OOD Detection Through Goodness-of-Fit Testing in Factorised Latent Spaces

Deep generative models offer a natural foundation for out-of-distribution (OOD) detection, yet prior work has shown that their assigned likelihoods are notoriously unreliable indicators for in- vs out-of-distribution data. In this paper, we address this problem by leveraging the diffeomorphic and mass-preserving properties of continuous normalising flows. Our analysis shows that OOD samples are mapped to noise samples that are highly atypical under the noise prior in ways not captured by the likelihood. Based on this observation, we propose a new method -- Signal in the Noise (SITN) -- for OOD detection on the single-sample level. SITN requires no access to OOD data, incurs minimal computational overhead, and provides strict control of false positive rates. Comprehensive evaluations through standard benchmarks and synthetic perturbations highlight the method's effectiveness and the absence of the complexity bias inherent to likelihood-based methods.

Philipp Bomatter, Jack Geary, Henry Gouk

School of Informatics, University of Edinburgh

Distribution Shift Detection OOD Detection

May 19, 2026cs.LG

Tippett-minimum Fusion of Representation-space Diffusion Models for Multi-Encoder Out-of-Distribution Detection

We address out-of-distribution (OOD) detection across the full spectrum of distribution shifts -- global domain changes, semantic divergence, texture differences, and covariate corruptions -- through a multi-encoder fusion of per-encoder representation-space diffusion models (RDMs). We statistically identify each encoder's sensitivity to specific shift types from ID data alone and introduce EncMin2L -- an encoder-agnostic two-level

\min(\cdot)

-gate that combines and calibrates per-encoder diffusion-based likelihood detectors without OOD labels, outperforming monolithic multi-encoder baselines at

2.3\times

lower parameter cost. Two ID-data diagnostics:

η^2

(class-conditional F-test) and

Δμ

(log-likelihood shift under synthetic corruptions) -- quantify encoder specialization, while a Tippett minimum

p

-value combination aggregates per-encoder scores into a single, calibration-stable OOD signal. EncMin2L achieves

\geq 0.94

AUROC across all four shift types simultaneously, outperforming the state-of-the-art representation-space diffusion OOD detectors across overlapping benchmarks.

Neelkamal Bhuyan

Georgia Institute of Technology

Diffusion-Based OOD Detection Distribution Shift Detection OOD Detection Diffusion Models

May 15, 2026stat.ML

Unsupervised Domain Shift Detection with Interpretable Subspace Attribution

We developed a tool for detecting domain shifts, namely subtle differences in the probability distributions of datasets. We identify these shifts using an algorithm designed to detect localised density anomalies in high-dimensional feature spaces. If an anomaly is present, we then identify the feature subspace in which the anomaly is most pronounced. This allows us to trace the domain shift to a small set of features, making the shift interpretable. Moreover, we provide a protocol for compensating domain shifts by extracting, from two unlabelled datasets, subsets of samples with no detectable residual distributional difference. We validate the framework on controlled 20-dimensional benchmarks with known ground truth, recovering both broad and localized shifts together with their supporting feature subspaces. We then apply it to healthy electrocardiogram (ECG) recordings represented by 782 features. In age- and sex-matched cohort comparisons differing in measurement-device composition, the method detects device-induced shifts, extracts representative subsets enriched in the imbalanced device components, and identifies ECG features associated with the acquisition contrast. These results suggest that density-shift detection and subspace attribution provide a practical framework for uncovering hidden cohort biases before downstream modelling.

Sebastian Springer, Alessandro Laio

SISSA

Feature Attribution Distribution Shift Distribution Shift Detection Unsupervised Anomaly Detection

May 5, 2026cs.AI

Geometry over Density: Few-Shot Cross-Domain OOD Detection

Out-of-distribution (OOD) detection identifies test samples that fall outside a model's training distribution, a capability critical for safe deployment in high-stakes applications. Standard OOD detectors are trained on a specific in-distribution (ID) dataset and detect deviations from that single domain. In contrast, we study few-shot cross-domain OOD detection: given a \emph{single} pre-trained model, can we perform OOD detection on \emph{arbitrary} new ID-OOD task pairs using only a handful of ID samples at inference time, with no additional training? We propose \textbf{UFCOD}, a unified framework that achieves this goal through information-geometric analysis of diffusion trajectories. Our key insight is that diffusion noise predictions are score functions (gradients of log-density), and we extract two energy features: \emph{Path Energy} (integrated score magnitude) and \emph{Dynamics Energy} (score smoothness), that form a discrete Sobolev norm capturing how samples interact with the learned diffusion process. The central contribution is a \textbf{train-once, deploy-anywhere} paradigm: a diffusion model trained on a single dataset (e.g., CelebA) serves as a universal feature extractor for OOD detection across semantically unrelated domains (e.g., CIFAR-10, SVHN, Textures). At deployment, each new task requires only

\sim

100 unlabeled ID samples for inference: no retraining, no fine-tuning, no task-specific adaptation. Using 100 ID samples per task, UFCOD achieves 93.7% average AUROC across 12 cross-domain benchmarks, competitive with methods trained on 50k--163k samples, demonstrating

\sim

500

\times

improvement in sample efficiency. See our code in https://github.com/lili0415/UFCOD.

Shawn Li, You Qin, Jiate Li +4

University of Southern California · National University of Singapore · Amazon

Diffusion-Based OOD Detection Distribution Shift Detection Cross-Domain Generalization OOD Detection