Label Shift Estimation With Incremental Prior Update

Yunrui Zhang; Gustavo Batista; Salil S. Kanhere

cs.LGApr 2, 2026

Label Shift Estimation With Incremental Prior Update

Authors: Yunrui Zhang, Gustavo Batista, Salil S. Kanhere

Organizations: University of New South Wales, School of Computer Science and Engineering, Australia

Abstract

An assumption often made in supervised learning is that the training and testing sets have the same label distribution. However, in real-life scenarios, this assumption rarely holds. For example, medical diagnosis result distributions change over time and across locations; fraud detection models must adapt as patterns of fraudulent activity shift; the category distribution of social media posts changes based on trending topics and user demographics. In the task of label shift estimation, the goal is to estimate the changing label distribution $p_t(y)$ in the testing set, assuming the likelihood $p(x|y)$ does not change, implying no concept drift. In this paper, we propose a new approach for post-hoc label shift estimation, unlike previous methods that perform moment matching with confusion matrix estimated from a validation set or maximize the likelihood of the new data with an expectation-maximization algorithm. We aim to incrementally update the prior on each sample, adjusting each posterior for more accurate label shift estimation. The proposed method is based on intuitive assumptions on classifiers that are generally true for modern probabilistic classifiers. The proposed method relies on a weaker notion of calibration compared to other methods. As a post-hoc approach for label shift estimation, the proposed method is versatile and can be applied to any black-box probabilistic classifier. Experiments on CIFAR-10 and MNIST show that the proposed method consistently outperforms the current state-of-the-art maximum likelihood-based methods under different calibrations and varying intensities of label shift.

Figures & tables

Shift Estimator	Calibration Method	$\alpha=0.1$			$\alpha=1$			$\alpha=10$
Validation Size		1000	2500	4000	1000	2500	4000	1000	2500	4000
CC	None	53.2	50.1	49.6	9.78	9.23	9.09	1.41	1.30	1.29
RLLS-Hard	None	6.68	3.85	2.61	5.02	2.03	1.58	2.66	1.15	1.10
RLLS	None	6.07	5.27	2.60	5.42	2.55	1.22	3.38	1.62	0.83
EM	None	19.6	14.06	10.4	5.19	3.37	2.73	3.01	1.45	1.03
LEIP	None	17.7	12.62	9.41	4.75	2.83	2.43	2.65	1.08	0.89

Table 1: CIFAR 10: Comparison of CC, RLLS, EM and LEIP under Dirichlet shift Values reported are MSE between the estimated shift weight and the true weight. All numbers are reported on the scale of

10^{-3}

and are means across 50 runs for each

\alpha

. Weight for EM and LEIP are calculated by dividing the estimated target label distribution by source label distribution calculated with

\frac{1}{N_{v}}\sum_{k=0}^{k=N_{v}}p(y_{i}|\mathbf{x}_{k})

for each class over the validation set to be consistent with previous work [ 1 ] . For CC, the source distribution for weight is simply the class distribution of the validation set. Numbers in bold indicate the best-performing method in the group.

Panel of Figure 1 — Figure 1: Comparison of CC, RLLS, EM and LEIP under Dirichlet shift for a wide range of $\alpha$ . Numbers reported are on the scale of $10^{-3}$ ; for each $\alpha$ , the MSE reported are average across 50 runs. Validation set size 2500 for CIFAR10 and 1500 for MNIST.

Shift Estimator	Calibration Method	$\alpha=0.1$			$\alpha=1$			$\alpha=10$
Validation Size		500	1500	2000	500	1500	2000	500	1500	2000
CC	None	13.3	12.0	11.5	2.74	2.49	2.32	0.60	0.48	0.45
RLLS-Hard	None	13.1	2.58	2.32	7.43	1.06	1.04	4.89	0.57	0.82
RLLS	None	10.8	2.62	2.29	7.39	1.12	0.97	5.35	0.65	0.64
EM	None	13.9	2.99	3.20	7.92	1.03	1.16	5.49	0.66	0.65
LEIP	None	14.2	3.14	3.29	8.00	0.97	1.14	5.46	0.67	0.67

Table 2: MNIST: Comparison of CC, RLLS, EM and LEIP under Dirichlet shift Values reported are MSE between the estimated shift weight and the true weight. All numbers are reported on the scale of

10^{-3}

and are means across 50 runs for each

\alpha

. The same settings for calculating weight and source distribution are used as introduced in the caption of Table 1 .

$\alpha$	Class 1	Class 2	Class 3	Class 4	Class 5	Class 6	Class 7	Class 8	Class 9	Class 10
0.1	0.008	0.003	0.239	0.144	0.000	0.000	0.002	0.443	0.161	0.000
0.5	0.010	0.084	0.001	0.693	0.011	0.001	0.043	0.025	0.057	0.074
1	0.025	0.066	0.118	0.002	0.084	0.020	0.015	0.464	0.186	0.020
3	0.087	0.038	0.146	0.060	0.098	0.165	0.063	0.203	0.091	0.049
5	0.122	0.071	0.116	0.188	0.087	0.085	0.106	0.094	0.047	0.084
10	0.115	0.088	0.102	0.058	0.104	0.102	0.130	0.102	0.070	0.130

Table 3: Label distribution generated by Dirichlet Shift for different

\alpha

values each row sums to 1

Shift Estimator	Calibration Method	$\alpha=0.1$			$\alpha=1$			$\alpha=10$
Validation Size		1000	2500	4000	1000	2500	4000	1000	2500	4000
CC	None	53.2	50.1	49.6	9.78	9.23	9.09	1.41	1.30	1.29
RLLS-Hard	None	6.68	3.85	2.61	5.02	2.03	1.58	2.66	1.15	1.10
RLLS	None	6.07	5.27	2.60	5.42	2.55	1.22	3.38	1.62	0.83
EM	None	10.0	8.43	8.65	2.61	2.37	2.42	0.66	0.49	0.53
LEIP	None	8.26	7.21	7.64	2.37	2.2	2.17	0.57	0.55	0.53

Table 4: CIFAR10: Comparison of CC, RLLS, EM, and LEIP under Dirichlet shift . Values reported are MSE between the estimated shift weight and the true weight. All numbers are reported on the scale of

10^{-3}

and are means across 50 runs for each

\alpha

. Weights for EM and LEIP are calculated by dividing the estimated target label distribution by the source label distribution, determined by hard label counts in the validation set. This method, which differs from previous work [ 1 ] , yields better results for EM and LEIP when the classifier is uncalibrated or calibrated with TS.

Shift Estimator	Calibration Method	$\alpha=0.1$			$\alpha=1$			$\alpha=10$
Validation Size		500	1500	2000	500	1500	2000	500	1500	2000
CC	None	13.3	12.0	11.5	2.74	2.49	2.32	0.60	0.48	0.45
RLLS-Hard	None	13.1	2.58	2.32	7.43	1.06	1.04	4.89	0.57	0.82
RLLS	None	10.8	2.62	2.29	7.39	1.12	0.97	5.35	0.65	0.64
EM	None	0.58	0.28	0.30	1.03	0.48	0.49	1.11	0.46	0.47
LEIP	None	0.30	0.26	0.25	0.46	0.33	0.32	0.45	0.31	0.30

Table 5: MNIST: Comparison of CC, RLLS, EM, and LEIP under Dirichlet shift . Values reported are MSE between the estimated shift weight and the true weight. All numbers are reported on the scale of

10^{-3}

and are means across 50 runs for each

\alpha

Label Shift

Label shift studies how to adapt predictions when class proportions change between training and target data.

See all 21 papers

Anytime-Valid Confirmation of Label-Shift Corrections
Jun 12, 2026Seungjin ChoiLabel Shift Anytime-Valid Inference
CROID Research and aSSIST University, Seoul, Korea.
Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation
Jun 10, 2026Seungjin ChoiWeighted Conformal Prediction Label Shift
1CROID Research and aSSIST University, Seoul, Korea.
The Label Complexity of Useful Class-Conditional Prediction Sets under Distribution Shift
Jul 20, 2026Weijia Han, Lisha Qu, Tianxin Zhou +2Distribution Shift Robustness Conformal Prediction
University of Washington

Jun 12, 2026J/K move · Enter open · S save

Anytime-Valid Confirmation of Label-Shift Corrections

Seungjin Choi

CROID Research and aSSIST University, Seoul, Korea.

Label Shift Anytime-Valid Inference E-Values

Open paper AskPDF arXiv

In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation even when unlabeled target inputs are available. We address the complementary setting where the practitioner has a pre-specified label-shift correction from domain knowledge and asks whether incoming labeled outcomes support it. We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value, so its running product is a nonnegative martingale and Ville's inequality yields an anytime-valid confirmation rule. The log martingale equals the cumulative negative log-predictive density (NLPD) gap between the source and the corrected predictive, converting routine model monitoring into a formal sequential test. Rejection means the incoming data support the posited correction relative to the source predictive, but it is not a precise estimate of the degree of shift. Closed forms are available for GP sources with Gaussian label-shift ratios. GP regression simulations validate Type I control, finite-sample power, miscalibration sensitivity, and the small-batch advantage of a reliable prior over label-based re-estimation.

Abstract

Label Shift

Explore similar work

Anytime-Valid Confirmation of Label-Shift Corrections

Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation

The Label Complexity of Useful Class-Conditional Prediction Sets under Distribution Shift