Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition

Kentaro Oda

cs.LGAug 20, 2026

Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition

Authors: Kentaro Oda

Organizations: Center for Management of Information Technologies, Kagoshima University

Abstract

After the inputs X are known, how much additional information does the label Y carry about which dataset a sample came from? That single quantity -- estimable as the difference of two discriminators' held-out cross-entropies, D_CJS = CE(Z|X) - CE(Z|X,Y) -- is exactly the part of a dataset difference that covariate shift cannot explain. We propose the Conditional Jensen-Shannon Discrepancy (CJSD): with a task indicator Z, the chain rule I(Z;X,Y) = I(Z;X) + I(Z;Y|X) splits total task discrepancy exactly into a covariate axis and a functional axis, both estimable from two ordinary classifiers, with no task-specific predictors, generative models, or bootstrap surrogates. We prove a covariate-null property (the functional axis is exactly zero under pure covariate shift, however severe), a drift-mass law (D_CJS/ln2 equals the mass of the disagreement region for deterministic labels), a one-sided misspecification-control inequality (each direction of estimation error is bounded, unconditionally, by the excess risk of a single discriminator), and a fixed-measure metrization via an identifiability lemma. Empirically, on a ten-measure battery over 202 dataset pairs (synthetic, Electricity, Covertype), only the two conditional-information estimators -- CJSD and a kNN plug-in for the same estimand -- separate concept from covariate shift with AUC 1.0; the case for CJSD is the estimator: under controlled dimensionality scaling the kNN plug-in fails from d=64 while the discriminator route holds to d=256 with a swappable classifier, and it alone yields paired confidence intervals and sequential extensions from the same learned object. The same estimator audits the conditional fidelity of synthetic-data generators that marginal and joint QA metrics pass, detects annotation-guideline changes invisible to input-space monitors, and supports null-calibrated fairness audits.

Figures & tables

Panel of Figure 1 — Figure 1: The whole estimator. Two ordinary classifiers predict the dataset indicator $Z$ , one from $x$ , one from $(x,y)$ ; subtracting their held-out cross-entropies yields the functional axis—the information about $Z$ that $Y$ adds after $X$ is known. Nothing is ever evaluated off its training support.

Figure 2: Ten-measure comparison over 202 pairs across three data families. Left: concept-vs-covariate discrimination AUC (the two conditional-information estimators, CJSD and kNN-CMI, reach $1.0$ ). Middle: false signal under pure covariate shift, normalized by the concept signal ( $0$ is ideal). Right: drift-mass sensitivity is preserved.

Figure 3: Identity checks. Left: estimator consistency against the population value. Middle: the drift-mass law $D_{\mathrm{CJS}}/\ln 2=\rho$ . Right: the covariate null—the exchange-based score inflates with support separation while $\widehat{D}_{\mathrm{CJS}}$ stays at zero.

Figure 4: Dimensionality scaling (2 informative dims $+$ noise, $\rho=0.3$ , 8 seeds). Left: concept-vs-covariate AUC; kNN geometry fails from $d{=}64$ , the discriminator route holds to $d{=}128$ (RF) / $d{=}256$ (HGB). Right: mean estimate under concept drift; shrinkage is downward.

Figure 5: $n\times d$ phase diagram of concept-vs-covariate AUC (6 seeds per cell). Left: the kNN plug-in is geometry-limited—no row reaches reliable discrimination past $d{=}64$ . Middle/right: the discriminator route is sample-limited—the boundary moves outward with $n$ , and HGB discriminators clear the whole grid at $n{=}12\mathrm{k}$ .

Figure 6: Left: robustness to sample-size asymmetry ( $n_{A}{=}8000$ fixed). Middle/right: over the $s\times\theta$ grid each estimated axis responds to its own factor only (additive decomposition; note the feasible region is anti-coupled, $D_{\mathrm{CJS}}\leq\ln 2-I_{x}$ ).

Figure 7: Real-data pair batteries in the CJSD plane: the four constructed shift types occupy the four quadrants; pure covariate pairs sit at $\widehat{D}_{\mathrm{CJS}}\approx 0$ (the exchange score places them at $0.04$ – $0.10$ ).

Distribution Shift Detection

Distribution shift detection identifies when data or model behavior changes between reference and deployment settings.

See all 62 papers

Sep 10, 2026

General Quantification of Covariate and Concept Shifts

Generalization under distribution shift remains a core challenge in modern machine learning, yet existing learning bound theory is limited to narrow, idealized settings and is non-estimable from samples. In this paper, we bridge the gap between theory and practical applications. We first show that existing definition of concept shift breaks when the source and target supports mismatch. Leveraging entropic optimal transport, we propose a key notion:

γ^{*}\!

-concept shifts, and derive a general error bound unifying covariate and

γ^{*}\!

-concept shifts, which applies to broad loss functions, label spaces, and stochastic labeling. We further develop estimators for these shifts with concentration guarantees, and the DataShifts algorithm, which can quantify distribution shifts and estimate the error bound in most applications - a rigorous and general tool for analyzing learning error under distribution shift.

Hongbo Chen, Li Charlie Xia

Department of Statistics and Financial Mathematics, School of Mathematics, South China University of Technology, Guangzhou, China.

Distribution Shift Concept Drift Covariate Shift Generalization Bounds

Ask PDF

Jul 20, 2026

The Label Complexity of Useful Class-Conditional Prediction Sets under Distribution Shift

Prediction sets can make deployed classifiers safer by returning several plausible labels when a single prediction is uncertain. Their value depends on classwise reliability: average coverage can meet its target while rare or difficult classes fail repeatedly. This concern is sharper after distribution shift, when calibration labels come from a source environment but reliability is needed on the target. We ask what labeled source data and unlabeled target inputs reveal about class-conditional prediction sets, and when target labels are necessary. Under unrestricted joint shift, two target laws can produce the same observable data while requiring different classwise thresholds; any label-free rule covering both must enlarge its sets on one law. We give a labeled target audit that estimates the missing quantiles with a simultaneous guarantee. Probability-scale error is invariant to increasing score transformations, and threshold recovery follows under local regularity. At fixed confidence, achieving threshold tolerance

\varepsilon

with fixed, nonadaptive class-stratified labeled pairs has total complexity

Θ(K\varepsilon^{-2}\log K)

, or

Θ(\varepsilon^{-2}\log K)

labels per class under equal allocation. Class imbalance creates a separate acquisition cost; for foreground class probabilities of order

1/K

, the mixed-stream label complexity is also

Θ(K\varepsilon^{-2}\log K)

at fixed confidence. Experiments on action-recognition and image shifts show that marginal coverage can conceal severe class failures and that source classwise calibration depends on the shift. The results connect the information available at deployment to the target labels needed for useful class-conditional prediction.

Weijia Han, Lisha Qu, Tianxin Zhou +2

University of Washington

Distribution Shift Robustness Conformal Prediction

Ask PDF

Jan 21, 2026

Factorizable joint shift revisited

Factorizable joint shift (FJS) represents a type of distribution shift (or dataset shift) that comprises both covariate and label shift. Recently, it has been observed that FJS actually arises from consecutive label and covariate (or vice versa) shifts. Research into FJS so far has been confined mostly to the case of categorical labels. We propose a framework for analysing distribution shift in the case of a general label space, thus covering both classification and regression models. Based on the framework, we generalise existing results on FJS to general label spaces and present and analyse a related extension to label distribution estimation of the expectation maximisation (EM) algorithm for class prior probabilities. We also take a fresh look at generalized label shift (GLS) in the case of a general label space.

Dirk Tasche

Centre for Business Mathematics and Informatics, North-West University, South Africa

Distribution Shift Robustness Label Shift Distribution Shift Distribution Matching

Ask PDF

Abstract

Distribution Shift Detection

Explore similar work

General Quantification of Covariate and Concept Shifts

The Label Complexity of Useful Class-Conditional Prediction Sets under Distribution Shift

Factorizable joint shift revisited