Latent-Space Importance Weighting under Covariate Shift

Yuling Jiao; Lican Kang; Yang Xiang; Jerry Zhijian Yang; Fusheng Zhou

stat.MLOct 11, 2026

Latent-Space Importance Weighting under Covariate Shift

Authors: Yuling Jiao, Lican Kang, Yang Xiang, Jerry Zhijian Yang, Fusheng Zhou

Organizations: School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China · Institute for Math and AI, Wuhan University, Wuhan, 430072, China · Department of Mathematics, The Hong Kong University of Science and Technology, Hong Kong, China, and Algorithms of Machine Learning and Autonomous Driving Research Lab, HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, Shenzhen, China · School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China ·

Abstract

In this work, we propose a transfer learning algorithm for covariate shift, integrating sample reweighting with representation learning. A key feature of our approach is the use of a lower-dimensional latent space for modeling, in contrast to conventional methods that operate directly in the high-dimensional ambient space. Our method estimates the density ratio of learned latent features for downstream weighted prediction. We establish explicit target-domain excess-risk rates with independent data folds for strictly sufficient learned representations with regular learned-space prediction scores and density ratios. We give dimension comparisons under stated smoothness conditions and a variance comparison for true weights and a fixed predictor. General finite-sample inequalities retain representation-quality errors. Experiments on simulated and real-world datasets demonstrate gains from combining learned representations and latent weighting, and reveal sensitivity to initialization and domain choice.

Figures & tables

Method	Representation	Upper bound
Latent weights (ours) ( 10 )	✓	$\widetilde{O}(n^{-\frac{\beta_{\mathcal{H}}}{2\beta_{\mathcal{H}}+d}}+m^{-\frac{\beta_{\ddot{\Phi}}}{2\beta_{\ddot{\Phi}}+d}})$
Original weights ( 11 )	✓	$\widetilde{O}(n^{-\frac{\beta_{\mathcal{H}}}{2\beta_{\mathcal{H}}+d}}+m^{-\frac{\beta_{\Phi}}{2\beta_{\Phi}+p}})$
No representation ( 13 )	✗	$\widetilde{O}(n^{-\frac{\beta_{\ddot{\mathcal{H}}}}{2\beta_{\ddot{\mathcal{H}}}+p}}+m^{-\frac{\beta_{\Phi}}{2\beta_{\Phi}+p}})$

Table 1: Target-domain excess-risk bounds

\hat{R}\in\arg\min_{R\in\mathcal{F}}\widehat{\mathcal{L}}_{n_{R}}(R)\quad\text{using }D_{\rm rep}\text{ in \eqref{eq:rep-training}}.

Algorithm 1 Reweighting in the latent space

\hat{R}\in\arg\min_{R\in\mathcal{F}}\widehat{\mathcal{L}}_{n_{R}}(R)\quad\text{using }D_{\rm rep}\text{ in \eqref{eq:rep-training}}.

Algorithm 2 Reweighting in original space

Figure 1: The Top-1 accuracy of four methods on simulated and real datasets.

Dataset	Top-1 accuracy(%)
Dataset	Method ( 10 )	Method ( 11 )	Method ( 12 )	Method ( 13 )
Circle	97.02	96.40	95.21	95.70
Moon	91.08	81.55	84.82	71.72
Blood	74.42	65.98	62.99	50.62
Breast cancer	92.19	90.63	90.63	89.06
Haberman	70.00	66.67	66.67	50.0

Table 2: The performance of four methods on simulated and real datasets.

Methods	{C,D,W} $\to$ A	{A,D,W} $\to$ C	{A, C, D } $\to$ W	{A, C, W} $\to$ D
AFN ( Xu et al., 2019 )	47.6	37.9	45.8	61.3
BSP ( Chen et al., 2019 )	60.7	35.3	55.9	54.8
CDAN ( Long et al., 2018 )	61.3	40.2	57.6	64.5
DANN ( Ganin and Lempitsky, 2015 )	59.2	42.0	64.4	58.1
DAN ( Long et al., 2015 )	59.7	40.2	40.7	64.5
ERM	59.2	35.3	52.5	71.0

Table 3: The Top-1 accuracy(%) comparison of proposed method with others on Office-Caltech dataset without pretrained model weight.

Methods	A $\to$ W	D $\to$ W	W $\to$ D	A $\to$ D	D $\to$ A	W $\to$ A
AFN ( Xu et al., 2019 )	23.7	47.5	77.4	9.7	9.9	19.9
BSP ( Chen et al., 2019 )	42.4	33.9	64.5	16.1	12.6	12.6
CDAN ( Long et al., 2018 )	13.6	50.8	35.5	25.8	9.4	13.6
DANN ( Ganin and Lempitsky, 2015 )	30.5	32.2	32.3	25.8	16.2	6.8
DAN ( Long et al., 2015 )	22.0	50.8	51.6	9.7	5.2	13.1
ERM	11.9	35.6	38.7	9.7	7.9	8.4

Table 4: The Top-1 accuracy (%) comparison of proposed method with others on Office-Caltech dataset without pretrained model weight.

Methods	{C,D,W} $\to$ A	{A,D,W} $\to$ C	{A, C, D } $\to$ W	{A, C, W} $\to$ D
AFN ( Xu et al., 2019 )	96.3	92.9	96.6	96.8
BSP ( Chen et al., 2019 )	95.3	86.2	96.8	96.6
CDAN ( Long et al., 2018 )	95.8	87.9	86.4	93.5
DANN ( Ganin and Lempitsky, 2015 )	95.8	87.5	88.1	96.8
DAN ( Long et al., 2015 )	95.3	87.1	100.0	96.6
ERM	95.8	87.5	100.0	91.5

Table 5: The Top-1 accuracy(%) comparison of proposed method with others on Office-Caltech dataset using pretrained model weight.

Methods	A $\to$ W	D $\to$ W	W $\to$ D	A $\to$ D	D $\to$ A	W $\to$ A
AFN ( Xu et al., 2019 )	93.2	100.0	100.0	100.0	83.8	86.9
BSP ( Chen et al., 2019 )	91.5	100.0	100.0	96.8	84.8	84.3
CDAN ( Long et al., 2018 )	83.1	96.6	100.0	93.5	88.0	84.3
DANN ( Ganin and Lempitsky, 2015 )	81.4	100.0	100.0	64.5	88.5	83.8
DAN ( Long et al., 2015 )	89.8	96.6	100.0	90.3	86.4	84.8
ERM	79.7	81.4	100.0	83.9	73.3	78.5

Table 6: The Top-1 accuracy (%) comparison of proposed method with others on Office-Caltech dataset using pretrained model weight.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Datasets	Number- Source	Number- Target	Covariates	Labels	Features
Tabular	Circle	10000	10000	2	2	(1, 32, 32)
	Moon	10000	10000	2	2	(1, 32, 32)
	Breast cancer	365	318	9	2	(1, 20, 20)
	Blood	369	379	3	2	(1, 20, 20)
	Haberman	158	148	3	2	(1, 20, 20)
Office-Caltech	Caltech (C)	1123	-	-	10	(Variable)

Appendix

Table 7: Dataset sizes and feature dimensions. Office-Caltech counts are shown for transfer from C, D and W to A.

Dataset	$lr$	$\alpha_{w}$	latent dim
Circle	0.5	0.9	2
Moon	3.0	0.4	2
Blood	1.0	0.5	3
Breast cancer	3.0	0.1	6
Haberman	1.5	0.3	3
Office-Caltech	0.2	0.8	128

Appendix

Table 8: Hyperparameters.

lr

is the learning rate for representation training.

Layers	Details	Output shape
Conv-1	$7\times 7$ Conv	$(64,112,112)$
Maxpool-1	$3\times 3$ Maxpool	$(64,56,56)$
Conv-2	$\left[\begin{matrix}3\times 3~\text{Conv, BN, ReLU}\\ 3\times 3~\text{Conv, BN, ReLU}\end{matrix}\right]\times 2$	$(64,56,56)$
Conv-3	$\left[\begin{matrix}3\times 3~\text{Conv, BN, ReLU}\\ 3\times 3~\text{Conv, BN, ReLU}\end{matrix}\right]\times 2$	$(128,28,28)$
Conv-4	$\left[\begin{matrix}3\times 3~\text{Conv, BN, ReLU}\\ 3\times 3~\text{Conv, BN, ReLU}\end{matrix}\right]\times 2$	$(256,14,14)$
Conv-5	$\left[\begin{matrix}3\times 3~\text{Conv, BN, ReLU}\\ 3\times 3~\text{Conv, BN, ReLU}\end{matrix}\right]\times 2$	$(512,7,7)$

Appendix

Table 9: The network architecture based on ResNet-18.

Layers	Details	Output shape
Conv-1	$3\times 3$ Conv	$(24,12,12)$
Dense Block-1	$\left[\begin{matrix}1\times 1~\text{Conv, BN}\\ 3\times 3~\text{Conv, BN}\end{matrix}\right]\times 2$	$(48,12,12)$
Transition Layer-1	BN, ReLU, $2\times 2$ Average Pool, $1\times 1$ Conv,	$(24,6,6)$
Dense Block-2	$\left[\begin{matrix}1\times 1~\text{Conv, BN}\\ 3\times 3~\text{Conv, BN}\end{matrix}\right]\times 2$	$(48,6,6)$
Transition Layer-2	BN, ReLU, $2\times 2$ Average Pool, $1\times 1$ Conv,	$(24,3,3)$
Dense Block-3	$\left[\begin{matrix}1\times 1~\text{Conv, BN}\\ 3\times 3~\text{Conv, BN}\end{matrix}\right]\times 2$	$(48,3,3)$

Appendix

Table 10: The DenseNet architectures.

Layers	Details	Output shape
Linear-1	Linear	$(64)$
Linear-2	$\left[\begin{matrix}\text{Linear, LeakyReLU(0.2)}\\ \text{Linear, LeakyReLU(0.2)}\end{matrix}\right]\times 2$	$(128)$
Linear-3	LeakyReLU(0.2), Linear	$(1)$

Appendix

Table 11: The network architecture of full connect linear network with Leaky ReLU

Layers	Details	Output shape
Linear-1	Linear, ReLU	$(128)$
Linear-2	Linear, ReLU	$(128)$
Linear-3	Linear	$(\text{category number})$

Appendix

Table 12: The network architecture of prediction head

Figure 2: True and estimated log density ratios for

N(0,0.8)

and

N(3,1.0)

. Blue and orange histograms show

\log r^{*}

and

\log\hat{r}

, respectively.

Covariate Shift

Covariate shift studies how to adapt and evaluate models when input distributions change but the relationship between inputs and labels remains stable.

See all 51 papers

Unbounded Density Ratio Estimation and Its Application to Covariate Shift Adaptation
Mar 31, 2026Ren-Rui Liu, Jun Fan, Lei Shi +1Density Ratio Estimation Statistical Learning Theory
School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, China · Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong · School of Mathematical Sciences and Shanghai, Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China
Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression
Sep 14, 2026Ren-Rui Liu, Zheng-Chu GuoKernel Regression Distribution Shift
School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, China
Backward Conformal Prediction under Covariate Shift
Oct 12, 2026Amin Meghrazi, Yuanyuan LiConformal Prediction Weighted Conformal Prediction
The Ohio State University · Munich RE

Mar 31, 2026J/K move · Enter open · S save

Unbounded Density Ratio Estimation and Its Application to Covariate Shift Adaptation

Ren-Rui Liu, Jun Fan, Lei Shi, Zheng-Chu Guo

School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, China · Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong · School of Mathematical Sciences and Shanghai, Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China

Density Ratio Estimation Statistical Learning Theory Covariate Shift

Open paper AskPDF arXiv

This paper focuses on the problem of unbounded density ratio estimation -- an understudied yet critical challenge in statistical learning -- and its application to covariate shift adaptation. Much of the existing literature assumes that the density ratio is either uniformly bounded or unbounded but known exactly. These conditions are often violated in practice, creating a gap between theoretical guarantees and real-world applicability. In contrast, this work directly addresses unbounded density ratios and integrates them into importance weighting for effective covariate shift adaptation. We propose a three-step estimation method that leverages unlabeled data from both the source and target distributions: (1) estimating a relative density ratio; (2) applying a truncation operation to control its unboundedness; and (3) transforming the truncated estimate back into the standard density ratio. The estimated density ratio is then employed as importance weights for regression under covariate shift. We establish rigorous, non-asymptotic convergence guarantees for both the proposed density ratio estimator and the resulting regression function estimator, demonstrating optimal or near-optimal convergence rates. Our findings offer new theoretical insights into density ratio estimation and learning under covariate shift, extending classical learning theory to more practical and challenging scenarios.

Abstract

Covariate Shift

Explore similar work

Unbounded Density Ratio Estimation and Its Application to Covariate Shift Adaptation

Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression

Backward Conformal Prediction under Covariate Shift