The Blind Spot Paradox: When Adaptive Classifiers Defeat Drift Detectors

Raphaël Minato; Fabrice Popineau; Arpad Rimmel; Bich-Liên Doan

cs.LGOct 5, 2026

The Blind Spot Paradox: When Adaptive Classifiers Defeat Drift Detectors

Authors: Raphaël Minato, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan

Organizations: Université Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique, 91190, Gif-sur-Yvette, France

Abstract

Monitoring concept drift from an adaptive classifier's error stream creates an operational conflict with the model's own update loop. When internal adaptation outpaces evidence accumulation, accuracy recovers before cumulative detectors (CUSUM, Page-Hinkley) can reach threshold. Instrumenting an Adaptive Random Forest (ARF) shows that surviving trees absorb 98.6% of the post-drift error transient through incremental leaf updates alone. The first background tree swap accounts for just 0.71% of this erased error volume, but drops external detection rates by 31 percentage points. We derive the finite-horizon boundary where cumulative evidence fails to cross threshold and measure a critical magnitude floor ( $Δe_c = 0.120$ ) below which false-alarm budgets preclude detection. This failure manifests as missed shifts on stationary streams and false-alarm flooding triggered by internal tree swaps on noisy baselines. We validate on synthetic shifts, ARMA-GARCH series (ProteuS), and tabular benchmarks (BAF, INSECTS); on the synthetic sweep at a standard threshold, the blind spot appears at $Δe \approx 0.25$ , showing why classical benchmarks like SEA ( $Δe \le 0.21$ ) failed to reach it.

Figures & tables

Fig. 1: Race between the internal ( $\tau_{\mathrm{ARF}}$ ) and external ( $\tau_{\mathrm{det}}$ ) stopping times at $\Delta e{=}0.25$ ( $200$ seeds/panel); starved runs ( $\tau_{\mathrm{det}}$ censored at horizon $50{,}000$ ) sit in the broken right-hand zone. The transition $\lambda^{*}{\approx}15$ – $25$ separates the monitor-wins, preemption (late-alarm) and total-starvation regimes.

Panel of Figure 2 — Fig. 2: Instrumented race, ARF ( $M{=}10$ ) vs. external StrictCUSUM. (A) $\lambda{=}50$ : $100\%$ miss. (B) $\lambda{=}25$ : paradoxical miss increase with $\Delta e$ (bars: runs where ARF absorbed drift before alarm). (C) $\lambda{=}8$ : reliable detection at $\Delta e{>}0.15$ .

	IID Baseline		Cal. A (Low GARCH)		Cal. B (High GARCH)
Pipeline Configuration	F1	ADD	F1	ADD	F1	ADD
PHT + HT	0.75 $\pm$ 0.04	23 $\pm$ 3	0.86 $\pm$ 0.04	22 $\pm$ 2	0.98 $\pm$ 0.02	23 $\pm$ 3
PHT + ARF ( $c_{\mathrm{int}}=32$ )	0.68 $\pm$ 0.05	19 $\pm$ 1	0.63 $\pm$ 0.05	20 $\pm$ 1	0.56 $\pm$ 0.05	19 $\pm$ 1
PHT + ARF ( $c_{\mathrm{int}}=1$ )	0.00	$\infty$	0.00	$\infty$	0.00	$\infty$
EDDM + HT	0.68 $\pm$ 0.05	37 $\pm$ 4	0.83 $\pm$ 0.04	39 $\pm$ 3	0.94 $\pm$ 0.03	40 $\pm$ 4
EDDM + ARF ( $c_{\mathrm{int}}=1$ )	0.00	$\infty$	0.00	$\infty$	0.00	$\infty$

TABLE I: Concept drift detection on heteroscedastic ProteuS streams (12 transitions

\times

30 seeds; 360 runs per cell). Mean

\pm

95% bootstrap confidence intervals (1,000 resamples);

\infty

: no detection within the tolerance horizon;

[X]

: delay-corrected ADD (KSWIN intrinsic window delay

W/2=15

). Boldface marks the blind-spot collapse (

\mathrm{F1}=0.00

) and full resolution (

\mathrm{F1}=1.00

). Complete separations at the seed level hold for every starved pipeline against its baseline or windowed resolution (paired sign tests, 30/30 seeds favouring the non-starved arm,

p\approx 2\times 10^{-9}

; see text for event counts).

Fig. 3: Regime crossover and non-adaptive resolution (100 streams/point, bands = $\pm 1$ SEM). Left: pre-drift false alarms; center: missed detections; right: post-drift accuracy. Shaded zones: Weak Signal, Safe Zone (ARF outperforms HT, the literature consensus) and Blind Spot ( $\Delta e{\geq}0.25$ ), where the ARF’s recovery starves the PHT ( $100\%$ missed at $\Delta e{=}0.50$ ). The non-adaptive RF (yellow) inherits the ARF’s variance reduction (left) and keeps the HT’s persistent error signal (right), which removes starvation (center).

Benchmark Variant	$\Delta e$ (adaptive)	$\text{F1}_{\mathrm{PHT+HT}}$	$\text{F1}_{\mathrm{PHT+ARF}_{c=1}}$	$\text{F1}_{\mathrm{ADW+ARF}_{c=1}}$	Ratio HT / ARF
BAF Base	$\approx 0$	0.10	0.10	0.00	1.00
BAF Variant I	$\approx 0$	0.09	0.11	0.00	0.83
BAF Variant II	$\approx 0$	0.00	0.00	0.00	–
INSECTS abrupt_balanced	$0.07^{\ast}$	0.13	0.18	0.48	0.71
INSECTS gradual_balanced	0.45	0.25	0.02	0.06	10.57
INSECTS incremental_reoccurring	0.39	0.31	0.19	0.16	1.61

TABLE II: Empirical evaluation across six real-world benchmark streams (30 seeds per configuration). Drift positions follow the canonical protocol of Souza et al. [ 32 ] ( incremental_reoccurring_balanced retains

K=2

in-stream transitions). The

c_{\mathrm{int}}=1

evaluation mode induces precision collapse via false-alarm flooding rather than starvation (e.g., 86 alarms emitted on gradual_balanced for a single true drift). ∗ abrupt_balanced exhibits heterogeneous, partly negative local error shifts where neither detector exceeds baseline false-alarm rates.

Explore similar work

When Drift Detectors cry Wolf: False Alarm Rates in continuous ML Monitoring
Jul 19, 2026Raj Shekhar SinghFalse-Negative Rate Alarm Fatigue
Indian Institute of Technology, Roorkee Roorkee, Uttarakhand, India
Cluster-Specific Localized Drift Detection for Efficient Batch Model Adaptation under Controlled Distribution Shift
Jun 20, 2026Ignacio Cabrera Martin, Marcello Trovati, Almas Baimagambetov +1Distribution Shifts Parameter-Efficient Adaptation
University of Brighton, School of Architecture, Technology and Engineering, Brighton, BN2 4GJ, United Kingdom · University of Lancashire, School of Business, Preston, PR1 2HE, United Kingdom
Concept Drift Detection and Adaptive Retraining of Malware Classification Models
Aug 13, 2026Christofer Washington Berruz Chungata, Martin Jurecek, Katerina Potika +2Malware Support Vector Machines
Department of Computer Science, San Jose State University · Faculty of Information Technology, Czech Technical University in Prague

Jul 19, 2026, cs.LGJ/K move · Enter open · S save

When Drift Detectors cry Wolf: False Alarm Rates in continuous ML Monitoring

Raj Shekhar Singh

Indian Institute of Technology, Roorkee Roorkee, Uttarakhand, India

False-Negative Rate Alarm Fatigue

Open paper AskPDF arXiv

Drift detection is a core component of production machine learning monitoring systems, where detectors are used to compare incoming data with a reference distribution and trigger alerts when changes occur. However, these detectors are often evaluated in research settings that emphasize detection accuracy under synthetic shifts, while overlooking false alarms under continuous monitoring. In production environments, models are monitored repeatedly over time and across many features, and even small false positive rates can accumulate into frequent alerts, leading to alarm fatigue. We empirically analyze false positive behavior across five commonly used drift detectors: PSI, KS, MMD, LSDD, and adversarial validation. Consistent with existing literature, PSI exhibits strong sensitivity to batch size, producing frequent false alarms at small sample sizes; however, we further observe that its behavior stabilizes and improves substantially once batch sizes exceed approximately 200 samples. In contrast, KS, MMD, and LSDD display persistent fluctuations across batch sizes, while remaining comparatively more reliable than PSI in low-data regimes. Applying a Bonferroni correction reduces false positive rates, but often at the cost of reduced true positive sensitivity, reinforcing the well-known stability - sensitivity trade-off in drift detection. This work provides a systematic comparison of false positive behavior across multiple drift detectors under continuous monitoring conditions. We identify tradeoffs across detector families and provide practical guidelines for selecting and calibrating drift detectors in production ML systems.

Abstract

Explore similar work

When Drift Detectors cry Wolf: False Alarm Rates in continuous ML Monitoring

Cluster-Specific Localized Drift Detection for Efficient Batch Model Adaptation under Controlled Distribution Shift

Concept Drift Detection and Adaptive Retraining of Malware Classification Models