Unbounded Density Ratio Estimation and Its Application to Covariate Shift Adaptation

Ren-Rui Liu; Jun Fan; Lei Shi; Zheng-Chu Guo

stat.MLMar 31, 2026

Unbounded Density Ratio Estimation and Its Application to Covariate Shift Adaptation

Authors: Ren-Rui Liu, Jun Fan, Lei Shi, Zheng-Chu Guo

Organizations: School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, China · Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong · School of Mathematical Sciences and Shanghai, Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China

Abstract

This paper focuses on the problem of unbounded density ratio estimation -- an understudied yet critical challenge in statistical learning -- and its application to covariate shift adaptation. Much of the existing literature assumes that the density ratio is either uniformly bounded or unbounded but known exactly. These conditions are often violated in practice, creating a gap between theoretical guarantees and real-world applicability. In contrast, this work directly addresses unbounded density ratios and integrates them into importance weighting for effective covariate shift adaptation. We propose a three-step estimation method that leverages unlabeled data from both the source and target distributions: (1) estimating a relative density ratio; (2) applying a truncation operation to control its unboundedness; and (3) transforming the truncated estimate back into the standard density ratio. The estimated density ratio is then employed as importance weights for regression under covariate shift. We establish rigorous, non-asymptotic convergence guarantees for both the proposed density ratio estimator and the resulting regression function estimator, demonstrating optimal or near-optimal convergence rates. Our findings offer new theoretical insights into density ratio estimation and learning under covariate shift, extending classical learning theory to more practical and challenging scenarios.

Figures & tables

Panel of Figure 1 — Figure 1: A comparison between the standard density ratio $\theta$ and the relative density ratio $\phi$ . The mixing coefficient is set to $\alpha=1/2$ . Left: the source distribution $\rho^{\textnormal{S}}_{\mathcal{X}}$ is uniform on $[0,1]$ , while the target distribution $\rho^{\textnormal{T}}_{\mathcal{X}}$ has a density proportional to $(\log x)^{2}$ . Right: the source and target distributions are $\mathcal{N}(0,1)$ and $\mathcal{N}(1,1)$ , respectively.


	Definition		Definition

$L_{\textnormal{S}}$	$\displaystyle L_{\textnormal{S}}\,f=\int_{\mathcal{X}}f(x)\,K(\cdot,x)\,\mathrm{d}\rho^{\textnormal{S}}_{\mathcal{X}}(x)$	$\widehat{L}_{\textnormal{S}}$	$\displaystyle\widehat{L}_{\textnormal{S}}\,f=\frac{1}{n_{\theta}}\sum_{j=1}^{n_{\theta}}f(x_{j})\,K(\cdot,x_{j})\rule{0.0pt}{18.49988pt}$
$L_{\textnormal{T}}$	$\displaystyle L_{\textnormal{T}}\,f=\int_{\mathcal{X}}f(x)\,K(\cdot,x)\,\mathrm{d}\rho^{\textnormal{T}}_{\mathcal{X}}(x)$	$\widehat{L}_{\textnormal{T}}$	$\displaystyle\widehat{L}_{\textnormal{T}}\,f=\frac{1}{n_{\theta}}\sum_{j=1}^{n_{\theta}}f(x^{\prime}_{j})\,K(\cdot,x^{\prime}_{j})$
$L_{\textnormal{R}}$	$\displaystyle L_{\textnormal{R}}\,f=(1-\alpha)L_{\textnormal{S}}\,f+\alpha L_{\textnormal{T}}\,f$	$\widehat{L}_{\textnormal{R}}$	$\displaystyle\widehat{L}_{\textnormal{R}}\,f=(1-\alpha)\widehat{L}_{\textnormal{S}}\,f+\alpha\widehat{L}_{\textnormal{T}}\,f$
$L_{\textnormal{W}}$	$\displaystyle L_{\textnormal{W}}\,f=\int_{\mathcal{X}}\widehat{\theta}(x)f(x)\,K(\cdot,x)\,\mathrm{d}\rho^{\textnormal{S}}_{\mathcal{X}}(x)$	$\widehat{L}_{\textnormal{W}}$	$\displaystyle\widehat{L}_{\textnormal{W}}\,f=\frac{1}{n_{f}}\sum_{i=1}^{n_{f}}\widehat{\theta}(x_{i})f(x_{i})\,K(\cdot,x_{i})$

Table 1: Definitions of integral operators and their empirical counterparts. The unlabeled samples

\{x_{j}\}_{j=1}^{n_{\theta}}

and

\{x^{\prime}_{j}\}_{j=1}^{n_{\theta}}

are drawn from

\rho^{\textnormal{S}}_{\mathcal{X}}

and

\rho^{\textnormal{T}}_{\mathcal{X}}

, respectively, while the labeled sample

\{x_{i}\}_{i=1}^{n_{f}}

is drawn from

\rho^{\textnormal{S}}_{\mathcal{X}}

. The density ratio estimator

\widehat{\theta}=\widehat{\theta}_{\mu,D}

is obtained via ( 14 ) with the optimal regularization parameter

\mu

and truncation threshold

D

specified in Theorem 2 .

Density Ratio Estimation

Density ratio estimation compares probability mass across distributions to support inference, distribution-shift correction, and data selection.

See all 26 papers

Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression
Sep 14, 2026Ren-Rui Liu, Zheng-Chu GuoKernel Regression Distribution Shift
School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, China
Anytime PAC-Bayes for Constrained Density-Ratio Networks under Covariate Shift
May 17, 2026Paulo Akira F. Enabe, Rodrigo ProvasiPAC-Bayesian Generalization Bounds Domain Adaptation
Escola Politénica University of São Paulo Department of Structural and Geotechnical Engineering
Nonparametric Goodness-of-fit Testing under Covariate Shift
Aug 5, 2026Zhen Hou, Dong XiaKernel Ridge Regression Covariate Shift
Department of Mathematics Hong Kong University of Science and Technology

Sep 14, 2026J/K move · Enter open · S save

Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression

Ren-Rui Liu, Zheng-Chu Guo

School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, China

Kernel Regression Distribution Shift Density Ratio Estimation

Open paper AskPDF arXiv

We study density ratio estimation and importance-weighted regression under target shift with continuous outputs. Under target shift, the conditional distribution of the inputs given the outputs remains invariant across the training and test distributions, while the output marginal distribution may change. Although this problem has been extensively studied for discrete outputs, the continuous setting is substantially less understood: the importance weights are determined by an unknown density ratio function, for which existing estimation methods lack explicit finite-sample convergence rates. We propose a spectral regularization method in a reproducing kernel Hilbert space (RKHS) for estimating the continuous density ratio from labeled training samples and unlabeled test inputs. Under a source condition with regularity parameter

ι>0

, we establish high-probability finite-sample guarantees and show that the estimator achieves the capacity-independent minimax-optimal RKHS-norm rate

O(n_η^{-ι/(2ι+2)})

. We then incorporate the estimated density ratio into importance-weighted regression and characterize the propagation of density-ratio estimation error to the final predictor. When sufficiently many samples are available for density ratio estimation, the resulting regression estimator attains the minimax-optimal rates of standard kernel regression. These results establish a finite-sample theory for continuous density ratio estimation and importance-weighted learning under target shift.

Abstract

Density Ratio Estimation

Explore similar work

Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression

Anytime PAC-Bayes for Constrained Density-Ratio Networks under Covariate Shift

Nonparametric Goodness-of-fit Testing under Covariate Shift