An assumption often made in supervised learning is that the training and testing sets have the same label distribution. However, in real-life scenarios, this assumption rarely holds. For example, medical diagnosis result distributions change over time and across locations; fraud detection models must adapt as patterns of fraudulent activity shift; the category distribution of social media posts changes based on trending topics and user demographics. In the task of label shift estimation, the goal is to estimate the changing label distribution pt(y) in the testing set, assuming the likelihood p(x∣y) does not change, implying no concept drift. In this paper, we propose a new approach for post-hoc label shift estimation, unlike previous methods that perform moment matching with confusion matrix estimated from a validation set or maximize the likelihood of the new data with an expectation-maximization algorithm. We aim to incrementally update the prior on each sample, adjusting each posterior for more accurate label shift estimation. The proposed method is based on intuitive assumptions on classifiers that are generally true for modern probabilistic classifiers. The proposed method relies on a weaker notion of calibration compared to other methods. As a post-hoc approach for label shift estimation, the proposed method is versatile and can be applied to any black-box probabilistic classifier. Experiments on CIFAR-10 and MNIST show that the proposed method consistently outperforms the current state-of-the-art maximum likelihood-based methods under different calibrations and varying intensities of label shift.
Figures & tables
Shift Estimator
Calibration Method
α=0.1
α=1
α=10
Validation Size
1000
2500
4000
1000
2500
4000
1000
2500
4000
CC
None
53.2
50.1
49.6
9.78
9.23
9.09
1.41
1.30
1.29
RLLS-Hard
None
6.68
3.85
2.61
5.02
2.03
1.58
2.66
1.15
1.10
RLLS
None
6.07
5.27
2.60
5.42
2.55
1.22
3.38
1.62
0.83
EM
None
19.6
14.06
10.4
5.19
3.37
2.73
3.01
1.45
1.03
LEIP
None
17.7
12.62
9.41
4.75
2.83
2.43
2.65
1.08
0.89
Table 1: CIFAR 10: Comparison of CC, RLLS, EM and LEIP under Dirichlet shift Values reported are MSE between the estimated shift weight and the true weight. All numbers are reported on the scale of 10−3 and are means across 50 runs for each α . Weight for EM and LEIP are calculated by dividing the estimated target label distribution by source label distribution calculated with Nv1∑k=0k=Nvp(yi∣xk) for each class over the validation set to be consistent with previous work [ 1 ] . For CC, the source distribution for weight is simply the class distribution of the validation set. Numbers in bold indicate the best-performing method in the group.
Figure 1: Comparison of CC, RLLS, EM and LEIP under Dirichlet shift for a wide range of α . Numbers reported are on the scale of 10−3 ; for each α , the MSE reported are average across 50 runs. Validation set size 2500 for CIFAR10 and 1500 for MNIST.
Shift Estimator
Calibration Method
α=0.1
α=1
α=10
Validation Size
500
1500
2000
500
1500
2000
500
1500
2000
CC
None
13.3
12.0
11.5
2.74
2.49
2.32
0.60
0.48
0.45
RLLS-Hard
None
13.1
2.58
2.32
7.43
1.06
1.04
4.89
0.57
0.82
RLLS
None
10.8
2.62
2.29
7.39
1.12
0.97
5.35
0.65
0.64
EM
None
13.9
2.99
3.20
7.92
1.03
1.16
5.49
0.66
0.65
LEIP
None
14.2
3.14
3.29
8.00
0.97
1.14
5.46
0.67
0.67
Table 2: MNIST: Comparison of CC, RLLS, EM and LEIP under Dirichlet shift Values reported are MSE between the estimated shift weight and the true weight. All numbers are reported on the scale of 10−3 and are means across 50 runs for each α . The same settings for calculating weight and source distribution are used as introduced in the caption of Table 1 .
α
Class 1
Class 2
Class 3
Class 4
Class 5
Class 6
Class 7
Class 8
Class 9
Class 10
0.1
0.008
0.003
0.239
0.144
0.000
0.000
0.002
0.443
0.161
0.000
0.5
0.010
0.084
0.001
0.693
0.011
0.001
0.043
0.025
0.057
0.074
1
0.025
0.066
0.118
0.002
0.084
0.020
0.015
0.464
0.186
0.020
3
0.087
0.038
0.146
0.060
0.098
0.165
0.063
0.203
0.091
0.049
5
0.122
0.071
0.116
0.188
0.087
0.085
0.106
0.094
0.047
0.084
10
0.115
0.088
0.102
0.058
0.104
0.102
0.130
0.102
0.070
0.130
Table 3: Label distribution generated by Dirichlet Shift for different α values each row sums to 1
Shift Estimator
Calibration Method
α=0.1
α=1
α=10
Validation Size
1000
2500
4000
1000
2500
4000
1000
2500
4000
CC
None
53.2
50.1
49.6
9.78
9.23
9.09
1.41
1.30
1.29
RLLS-Hard
None
6.68
3.85
2.61
5.02
2.03
1.58
2.66
1.15
1.10
RLLS
None
6.07
5.27
2.60
5.42
2.55
1.22
3.38
1.62
0.83
EM
None
10.0
8.43
8.65
2.61
2.37
2.42
0.66
0.49
0.53
LEIP
None
8.26
7.21
7.64
2.37
2.2
2.17
0.57
0.55
0.53
Table 4: CIFAR10: Comparison of CC, RLLS, EM, and LEIP under Dirichlet shift . Values reported are MSE between the estimated shift weight and the true weight. All numbers are reported on the scale of 10−3 and are means across 50 runs for each α . Weights for EM and LEIP are calculated by dividing the estimated target label distribution by the source label distribution, determined by hard label counts in the validation set. This method, which differs from previous work [ 1 ] , yields better results for EM and LEIP when the classifier is uncalibrated or calibrated with TS.
Shift Estimator
Calibration Method
α=0.1
α=1
α=10
Validation Size
500
1500
2000
500
1500
2000
500
1500
2000
CC
None
13.3
12.0
11.5
2.74
2.49
2.32
0.60
0.48
0.45
RLLS-Hard
None
13.1
2.58
2.32
7.43
1.06
1.04
4.89
0.57
0.82
RLLS
None
10.8
2.62
2.29
7.39
1.12
0.97
5.35
0.65
0.64
EM
None
0.58
0.28
0.30
1.03
0.48
0.49
1.11
0.46
0.47
LEIP
None
0.30
0.26
0.25
0.46
0.33
0.32
0.45
0.31
0.30
Table 5: MNIST: Comparison of CC, RLLS, EM, and LEIP under Dirichlet shift . Values reported are MSE between the estimated shift weight and the true weight. All numbers are reported on the scale of 10−3 and are means across 50 runs for each α . Weights for EM and LEIP are calculated by dividing the estimated target label distribution by the source label distribution, determined by hard label counts in the validation set. This method, which differs from previous work [ 1 ] , yields better results for EM and LEIP when the classifier is uncalibrated or calibrated with TS.
In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation even when unlabeled target inputs are available. We address the complementary setting where the practitioner has a pre-specified label-shift correction from domain knowledge and asks whether incoming labeled outcomes support it. We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value, so its running product is a nonnegative martingale and Ville's inequality yields an anytime-valid confirmation rule. The log martingale equals the cumulative negative log-predictive density (NLPD) gap between the source and the corrected predictive, converting routine model monitoring into a formal sequential test. Rejection means the incoming data support the posited correction relative to the source predictive, but it is not a precise estimate of the degree of shift. Closed forms are available for GP sources with Gaussian label-shift ratios. GP regression simulations validate Type I control, finite-sample power, miscalibration sensitivity, and the small-batch advantage of a reliable prior over label-based re-estimation.
Seungjin Choi
CROID Research and aSSIST University, Seoul, Korea.
Conformal Bayes combines Bayesian posterior predictives with conformal calibration to produce prediction sets that are both statistically valid and geometrically efficient. We study conformal Bayes under label shift from a unified perspective, identifying two complementary approaches that restore nominal target-domain coverage through importance-weighted conformal calibration but operate through independent mechanisms. Post-hoc calibration tilts the posterior predictive toward the target domain and corrects the conformal threshold via an importance-weighted quantile, leaving the parameter posterior unchanged. In-training adaptation tilts the parameter posterior itself to the target domain, producing a corrected predictive whose highest predictive density region serves as the highest predictive density (HPD)-based prediction set under the fitted target predictive; efficiency is model-dependent and does not imply finite-sample conditional optimality. Two controlled experiments isolate the regime-dependence of each strategy: in the low-dimensional, well-estimated regime StrategyA produces the narrowest valid intervals, while in the high-dimensional, underdetermined regime StrategyB achieves up to 43% width reduction at unchanged coverage, under the stated source-sampling and label-shift assumptions.
Seungjin Choi
1CROID Research and aSSIST University, Seoul, Korea.
Prediction sets can make deployed classifiers safer by returning several plausible labels when a single prediction is uncertain. Their value depends on classwise reliability: average coverage can meet its target while rare or difficult classes fail repeatedly. This concern is sharper after distribution shift, when calibration labels come from a source environment but reliability is needed on the target. We ask what labeled source data and unlabeled target inputs reveal about class-conditional prediction sets, and when target labels are necessary. Under unrestricted joint shift, two target laws can produce the same observable data while requiring different classwise thresholds; any label-free rule covering both must enlarge its sets on one law. We give a labeled target audit that estimates the missing quantiles with a simultaneous guarantee. Probability-scale error is invariant to increasing score transformations, and threshold recovery follows under local regularity. At fixed confidence, achieving threshold tolerance ε with fixed, nonadaptive class-stratified labeled pairs has total complexity Θ(Kε−2logK), or Θ(ε−2logK) labels per class under equal allocation. Class imbalance creates a separate acquisition cost; for foreground class probabilities of order 1/K, the mixed-stream label complexity is also Θ(Kε−2logK) at fixed confidence. Experiments on action-recognition and image shifts show that marginal coverage can conceal severe class failures and that source classwise calibration depends on the shift. The results connect the information available at deployment to the target labels needed for useful class-conditional prediction.