Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations

Cagdas Pullu; Mahmut Emir Arslan; Bugra Balkac; Aylin Ondersev Balta; Cihangir Celal Palaci; Fikri Cem Yilmaz; Altan Cakir

Figure 1: Temporal concept-drift profiles, following the taxonomy of Gama et al. (2014) : (a) sudden drift replaces the old concept at once; (b) gradual drift alternates between the old and new concepts until the new one prevails; (c) incremental drift moves through intermediate states; (d) recurring drift makes previously seen concepts reappear. The dashed line marks the drift onset. Terminology varies across surveys; for example, Hinder et al. (2024) describe incremental drift as sampling from both concepts with varying probabilities.

Test	Family	Vector	Decision Rule / Calibration	Complexity
Univariate marginal $\times 6$ 1 1 1 Chi-square, Jensen–Shannon, Kullback–Leibler, PSI, KS, and univariate Wasserstein-1, applied per feature.	Marginal per-feature	$X_{k}$	Fixed threshold or asymptotic $p$	$O(n)$ per column
Per-dim KS	Marginal aggregation	$(X,Y)$ stack	Bonferroni: $\min_{k}p_{k}<\alpha_{0}/d$	$O(nd\log n)$
Projected KS (SRP+KS)	Projection-based	$(X,Y)S$ , $S\in\mathbb{R}^{d\times 32}$	Bonferroni on projected dims	$O(nd+n\log n)$
MMD-RBF	Multivariate kernel	$(X,Y)$ stack	Permutation ( $B=1{,}000$ )	$O(n^{2}d)$ ; kernel matrix
RFF-MMD	Multivariate kernel	$(X,Y)$ stack	Permutation ( $B=200$ )	$O(nDd)$ ; $D=500$
Sliced Wasserstein Distance	Projection-based	$(X,Y)$ projections	Fixed $\tau_{\mathrm{SW}}=0.05$	$O(nP\log n)$ ; $P=100$

Table 1: Overview of two-sample tests evaluated in this study. Family classification follows Section 3.1 (marginal per-feature; marginal per-feature aggregation on the stacked vector; projection-based; multivariate embedding/kernel). Vector column indicates the input to which the test is applied. Complexity is per-position at reference size

n_{R}

and current-window size

n_{C}

. Parameter values are those of the Failing Loudly reproduction; the Trendyol scan settings are given in Section 6.3.1 .

Test Name	Sub-Family	Core Statistic / Mechanism	Threshold / Calibration	Execution Paradigm
Univariate (6 variants)	Marginal	Marginal $P_{R}(X_{k})$ vs $P_{C}(X_{k})$ per feature	Fixed $\alpha_{0}=0.05$ or lit. $\tau$	Hybrid (Driver aggregated)
Per-dim KS	Marginal Agg.	$\max_{k}D_{k}$ across $d$ raw dimensions	Bonferroni ( $\alpha_{0}/d$ )	Driver-side (Subsampled)
Projected KS (SRP)	Projection	$\max$ KS across $K=32$ sparse random projections	Bonferroni ( $\alpha_{0}/K$ )	Driver-side (Subsampled)
Sliced Wasserstein Distance	Projection	Avg. Wasserstein-1 over 100 random unit directions	Fixed distance ( $\tau=0.05$ )	Spark Distributed
MMD-RBF	Kernel Embed.	$\widehat{\mathrm{MMD}}^{2}$ via Gaussian RBF ( $O(n^{2}d)$ )	Permutation ( $B=1000$ )	Driver-side (Subsampled)
RFF-MMD	Kernel Embed.	$\\|\bar{\mathbf{h}}(X_{R})-\bar{\mathbf{h}}(X_{C})\\|_{2}^{2}$ mapping ( $D=500$ )	Permutation ( $B=200$ )	Spark Distributed

Table 2: Summary of Evaluated Two-Sample Tests and Execution Architecture

Layer	Source / Family	Configs	Dim	Ground Truth
Harvard Dataverse	SEA, SINE, STAGGER, MIXED, Random Tree	20	2–9	3 temporal changepoints
Failing Loudly	MNIST, CIFAR-10 (10 shift types)	60	784 / 3072	Image perturbation labels
Trendyol benchmark	Collection-ranking feature table	8	11	Injected row-level drift (Eqs. 6 – 8 )

Table 3: Dataset inventory and feature dimensionality. Feature dimensionality is reported after one-hot encoding of categorical covariates.

Figure 2: Evaluation protocols of this study within the taxonomy of unsupervised drift detectors of Gemaque et al. (2020) (adapted from their Fig. 2, CC BY 4.0). Batch detectors test an accumulated window as a whole; sliding-window scans advance the detection window along the stream and differ in whether the reference window stays fixed or slides with it. The survey’s online detectors check every arriving instance, whereas our scans evaluate at a stride

\Delta

(Harvard Dataverse) or per position bucket (Trendyol benchmark). Partial-batch detection, which tests only a selected subset of each batch, is not evaluated here.

Protocol	Data Structure & Windows	Evaluated Tests	Primary Metrics
Batch (Harvard)	Ref: 1st half. Cur: 2nd half.	All 11 tests (Marginal + Joint)	Binary Recall (TP / TP+FN)
Sliding-Scan (Harvard)	$W=5000$ , stride $\Delta=1000$ . Moving reference $[t-W,t-1]$ .	All 11 tests	Hit rate ( $\rho=200$ ), Latency, FPR
Sliding-Scan (Trendyol)	$N=50$ position buckets. Fixed reference table (first 100,000 rows; 10,000 for driver-side tests). 8 scenarios.	MMD-RBF, Per-dim KS, SRP+KS, RFF-MMD, SWD	Pearson $r$ vs. $\bar{m}_{b}$ , TPR, FPR, Runtime
Failing Loudly	Static sizes $N\in\{10\dots 10,000\}$ . 5 random splits, 3 fractions.	MMD-RBF, Per-dim KS, SRP+KS, RFF-MMD, SWD	$\mathrm{Acc}(N)$ , MAE vs literature

Table 4: Summary of experimental protocols and evaluation metrics.

Drift Type (Regime)	Severity ( $\sigma$ )	Scope ( $\phi$ )	Mean $\bar{\alpha}_{s}$	$\text{FlipRate}_{\text{scenario}}^{\text{expected}}$
sudden_strong	0.75	0.60	0.400	10.3%
sudden_weak	0.25	0.10	0.400	0.57%
gradual_strong	0.75	0.60	0.325	8.4%
gradual_weak	0.25	0.10	0.325	0.47%
incremental_strong	0.75	0.60	0.325	8.4%
incremental_weak	0.25	0.10	0.325	0.47%

Table 5: Trendyol Benchmark Scenario Configurations. The strong regime measures clean detection capabilities, while the weak regime establishes an empirical low-signal failure point.

Test Category	SEA	SINE	STAGGER	MIXED	Random Tree (RT)
Multivariate (Kernel & Proj.)	100%	100%	100%	100%	100%
Per-dim KS (Bonferroni)	100%	0%	100%	0%	100%
Univariate Marginal	0%	0%	0%	0%	100% 1 1 1 Recall artifact resulting from a generator-induced class-marginal rescaling at changepoints.

Table 6: Aggregated Batch Recall on Harvard Dataverse Streams. The bifurcation in recall demonstrates the structural failure of marginal tests under stationary

P(X)

environments, contrasted against the robustness of joint multivariate embeddings.

Figure 3: Sliding-scan

p

-value trajectories on Harvard Dataverse streams. Each panel aggregates streams within one family and plots the median

p

-value at each scan position for MMD-RBF, projected KS, and per-dim KS. Shaded bands are interquartile ranges over streams. Red dashed lines mark ground-truth changepoints; the horizontal line is

\alpha_{0}=0.05

on a symmetric-log axis. STAGGER is omitted because its whole-vector detection uses distance-based statistics without

p

-values after one-hot encoding of the categorical features.

Figure 4: Detection accuracy by test and shift family at

N=1000

on the Failing Loudly benchmark, disaggregated by dataset. Rows are the three reproduced baselines (per-dimension KS on raw pixels, projected KS with SRP, MMD-RBF). Columns are the five shift families of Failing Loudly. Each cell aggregates all comparisons of one family over the three perturbation fractions and five splits: 45 for the Gaussian and image families (three intensities each), 30 for the combined family (two variants), and 15 for the adversarial and class-knockout families.

Figure 5: Reproduced

p

-value curves under the Gaussian-noise shift class of Rabanser et al. (2019) on MNIST and CIFAR-10. Panels correspond to affected-fraction

\{0.10,0.50,1.00\}

. Lines are median

p

-values across 30 shift instances per sample size (3 intensities

\times

5 splits

\times

2 datasets). Shaded bands are the interquartile range. The horizontal reference line is

\alpha_{0}=0.05

. This figure is the reproduction analog of Failing Loudly Figure 3.

Figure 6: Failing Loudly reproduction: image-transformation shifts (rotation, translation, zoom at three intensities). Panels correspond to

\{10\%,50\%,100\%\}

perturbed test data.

Figure 7: Failing Loudly reproduction: adversarial shifts generated with the fast gradient sign method. Panels correspond to

\{10\%,50\%,100\%\}

perturbed test data. Adversarial shifts are the hardest single-class shift family at low perturbation fractions.

Figure 8: Failing Loudly reproduction: class knock-out (label) shifts. Panels correspond to

\{10\%,50\%,100\%\}

perturbed test data. Because knock-out shifts change

P(Y)

rather than

P(X)

, MMD-based tests detect the shift through its indirect effect on the observed covariate distribution.

Figure 9: Failing Loudly reproduction: combined image transformation + class knock-out shifts. Panels correspond to

\{10\%,50\%,100\%\}

perturbed test data. Combined shifts show the sharpest median-

p

descent because covariate and label distributions shift jointly.

Test	TPR (%)	FPR (%)	Ratio	$r$	Time (s)
Per-dim KS	100.0	100.0	0.91	$-0.085$	0.07
MMD-RBF	100.0	98.4	1.99	$-0.022$	36.20
Projected KS	100.0	100.0	1.25	$+0.109$	0.63
Spark RFF-MMD	40.2	3.2	8.41	$+0.427$	2.68
Spark SWD	48.2	10.0	1.84	$+0.422$	5.50

Table 7: Sliding-scan detection metrics on the Trendyol collection-ranking feature table averaged over all 8 test scenarios (4 drift types

\times

2 severity regimes). Grand means include the weak regime, where both RFF-MMD and SWD fail to discriminate; see Table 8 for the per-regime breakdown. 1 1 footnotetext: TPR = rejection rate over injection-active positions (

\mathcal{P}_{1}=\{b:\bar{m}_{b}>0\}

); FPR = injection-relative rejection rate over injection-null positions (

\mathcal{P}_{0}=\{b:\bar{m}_{b}=0\}

; this partition rather than a chronological pre/post-drift split is used because it correctly handles recurring drift). The injection-relative FPR is not the classical statistical Type-I error, since injection-null positions may still carry natural production-side covariate heterogeneity as documented in Section 6.3.5 ; Ratio = injection-active mean statistic divided by injection-null mean statistic;

r

= Pearson correlation between test statistic and per-bucket expected drift magnitude

\bar{m}_{b}

(Eq. 7 ); Time = median compute time per position on the Spark configuration described in Section 4.2 . Spark RFF-MMD uses the calibrated threshold

3\times 10^{-3}

(Section 6.3.1 ); the other tests use their implementation defaults. The three tests reporting near-100% FPR and TPR simultaneously do not perform meaningful binary discrimination at their default thresholds; their failure mechanisms are discussed in Section 6.3.6 .

		Sudden		Gradual		Incremental		Recurring
Test	Metric	Strong	Weak	Strong	Weak	Strong	Weak	Strong	Weak
RFF-MMD	$r$	$+0.948$	$-0.094$	$+0.944$	$-0.081$	$+0.942$	$-0.083$	$+0.926$	$-0.084$
	TPR	95.2	0.0	71.4	0.0	71.4	0.0	83.3	0.0
	FPR	3.4	3.4	3.4	3.4	3.4	3.4	2.6	2.6
SWD	$r$	$+0.844$	$+0.033$	$+0.844$	$+0.095$	$+0.820$	$+0.041$	$+0.767$	$-0.069$
	TPR	95.2	9.5	76.2	14.3	76.2	14.3	83.3	16.7

Table 8: Strong- vs weak-regime detection performance by drift type on the Trendyol collection-ranking feature table. RFF-MMD and SWD are the two tests capable of producing meaningful discrimination; the remaining three (per-dim KS, MMD-RBF, projected KS) exhibit systematic failures at all regimes (see Section 6.3.6 ). 1 1 footnotetext: Metric definitions (

r

, TPR, FPR and the

\mathcal{P}_{1}

/

\mathcal{P}_{0}

partition) as in Table 7 . Each cell is a single scenario; TPR and FPR in %. RFF-MMD uses the calibrated threshold

3\times 10^{-3}

(Section 6.3.1 ); SWD uses the fixed implementation threshold

\tau_{\mathrm{SW}}=0.05

. With 21 active and 29 null positions per scenario (12 and 38 for recurring drift), Wilson 95% intervals are wide, e.g.,

20/21=95.2\%

[77.3, 99.2],

15/21=71.4\%

[50.0, 86.2],

1/29=3.4\%

[0.6, 17.2], and

3/29=10.3\%

[3.6, 26.4].

Figure 10: Sliding-scan trajectories of Spark RFF-MMD (red) and Sliced Wasserstein Distance (blue) across 8 scenarios. Rows correspond to the four drift types; the left column is the strong regime and the right column the weak regime. Solid lines are the test statistic on a shared log y-axis; dashed horizontal lines are the drift threshold of each test in the matching color (RFF-MMD:

3\times 10^{-3}

, calibrated per Section 6.3.1 ; SWD:

5\times 10^{-2}

, implementation default). The dotted vertical line marks the initial drift onset (for recurring drift, subsequent returns between concepts are additional true transition points not marked). Per-panel Pearson

r

is shown in each subplot title.

Figure 11: Per-position runtime aggregated across all 8 scenarios (

n=400

measurements per test: 50 positions

\times

8 scenarios). Bars are medians; whiskers span the interquartile range. Each test is assigned a distinct color that is reused consistently in Figure 12 . MMD-RBF dominates the range at

\sim

36 seconds per position, roughly 6.5

\times

slower than the next test (Sliced Wasserstein Distance) and roughly 480

\times

slower than per-dim KS.

Figure 12: Per-position runtime distribution disaggregated by scenario. Each box aggregates

n=50

per-position measurements from one scenario. Colors match Figure 11 . The y-axis is broken to accommodate MMD-RBF’s range (

{\sim}31

–53 s, top panel) while preserving resolution on the four smaller-scale tests (0–10.5 s, bottom panel). Distributions are similar across scenarios, confirming that runtime is a function of test complexity and execution mode, independent of drift structure or severity regime.

Symbol	Meaning	Defined in
Two-sample testing
$\mathcal{D}_{R},\mathcal{D}_{C}$	Reference and current samples	Sec. 3.1
$P_{R},P_{C}$	Distributions of the reference and current samples	Eq. ( 1 )
$z$ ; $X,Y$	Tested vector ( $(X,Y)$ or $X_{k}$ ); features and label	Sec. 3.1
$d$	Feature dimensionality	Table 1
$s$ ; $p$	Test statistic; $p$ -value	Sec. 3.1

Name in Text	Symbol	Role	Original Column
Source-table columns
Seed-content identifier	–	Metadata (vector)	seed
Owner identifier	–	Metadata (vector)	ownerid
Channel identifier	–	Metadata (vector)	channelId
Collection size	–	Metadata (vector)	content_cnt
Image cosine similarity	$\mathrm{sim}_{\mathrm{img}}$	Similarity (vector); Concept A	image_cosine_similarity

Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations

Abstract

Figures & tables

Appendix

Explore similar work

A Framework for Evaluating and Benchmarking Concept Drift Detection Methods

Learner-based Concept Drift Detection: Analysis and Evaluation

When Drift Detectors cry Wolf: False Alarm Rates in continuous ML Monitoring