Pseudo-Labeling for Unsupervised Domain Adaptation with Kernel GLMs

Nathan Weill; Kaizheng Wang

stat.MLMar 19, 2026

Pseudo-Labeling for Unsupervised Domain Adaptation with Kernel GLMs

Authors: Nathan Weill, Kaizheng Wang

Organizations: Department of IEOR, Columbia University. · Department of IEOR and Data Science Institute, Columbia University.

Abstract

We propose a principled framework for unsupervised domain adaptation under covariate shift in kernel Generalized Linear Models (GLMs), encompassing kernelized linear, logistic, and Poisson regression with ridge regularization. Our goal is to minimize prediction error in the target domain by leveraging labeled source data and unlabeled target data, despite differences in covariate distributions. We partition the labeled source data into two batches: one for training a family of candidate models, and the other for building an imputation model. This imputation model generates pseudo-labels for the target data, enabling robust model selection. We establish non-asymptotic excess-risk bounds that characterize adaptation performance through an "effective labeled sample size", explicitly accounting for the unknown covariate shift. Experiments on synthetic and real datasets demonstrate consistent performance gains over source-only baselines.

Figures & tables

\min_{f\in\mathcal{F}}\bigg\{\frac{1}{n_{1}}\sum_{(\bm{x},y)\in\mathcal{D}_{1}}(a(f(\bm{x}))-yf(\bm{x}))+\frac{\lambda}{2}\|f\|_{\mathcal{F}}^{2}\bigg\}.

Algorithm 1 Pseudo-Labeling for Kernel GLM

Model	$a(z)$	$k$	$K$	$L$
Linear	$\frac{z^{2}}{2}$	$1$	$1$	$0$
Logistic	$\log(1+e^{z})$	$\frac{e^{2C}}{(1+e^{2C})^{2}}$	$1/4$	$0.1$
Poisson	$e^{z}$	$e^{-2C}$	$e^{2C}$	$e^{2C}$

Table 1: Properties of the log-partition function

a

[-2C,2C]

Figure 1: Comparisons of three approaches on a log-log plot.

x

-axis:

n

y

-axis: excess risk. Red crosses: pseudo-labeling. Cyan triangles: the oracle method. Blue circles: the naive method.

Figure 2: Baseline logistic regression performance on ID and OOD test data, when trained only on ID data, as a function of the ridge regularization parameter $\lambda$

Method	Mean	95% CI	SE
Naive	0.502	[0.463, 0.541]	0.019
Pseudo-labeling	0.428	[0.412, 0.444]	0.008
Oracle	0.373	[0.361, 0.385]	0.006

Table 2: Risk mean estimate across seeds with confidence interval (CI) and standard error (SE).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3:

B=n^{0.45}

\{1,\dots,n\}=\bigsqcup_{k=1}^{K}F_{k}^{(r)},\qquad F_{k}^{(r)}\cap F_{k^{\prime}}^{(r)}=\emptyset.

Appendix

Algorithm 2 Repeated

K

-Fold Cross-Validated Pseudo-Labeling (Practical Variant)

Method	Mean	95% CI	SE
Naive	0.444	[0.414, 0.474]	0.015
Pseudo-labeling	0.385	[0.373, 0.396]	0.006
Oracle	0.376	[0.363, 0.390]	0.007

Appendix

Table 3: Risk mean estimate across seeds with confidence interval (CI) and standard error (SE).

Figure 4: Selection risk curves of the three methods.

Unsupervised Domain Adaptation

Unsupervised domain adaptation transfers models from labeled source data to unlabeled target domains with different data distributions.

See all 183 papers

A General Representation-Based Approach to Multi-Source Domain Adaptation
Apr 26, 2026Ignavier Ng, Yan Li, Zijian Li +3Multi-Source Domain Adaptation Unsupervised Domain Adaptation
Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence
LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation
Jul 20, 2026Jing Li, Pan Liu, Meng Zhao +7Source-Free Domain Adaptation Unsupervised Domain Adaptation
School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China · Engineering Research Center of Learning-Based Intelligent System, Ministry of Education of the People’s Republic of China, Tianjin University of Technology, Tianjin 300384, China · School of Artificial Intelligence, Tianjin University, Tianjin 300350, China +1
Convergence Analysis of Nyström Subsampling in Covariate Shift Adaptation for Misspecified case
Jun 20, 2026Hanna Myleiko, Sergei Solodky, Vasyl SemenovUnsupervised Domain Adaptation Domain Adaptation
Institute of Mathematics NAS of Ukraine, 3 Tereschenkivska st., Kyiv, Ukraine. · Institute of Mathematics NAS of Ukraine, 3 Tereschenkivska st., Kyiv, Ukraine · University of Giessen, Department of Mathematics, Giessen, Germany. +1

Apr 26, 2026J/K move · Enter open · S save

A General Representation-Based Approach to Multi-Source Domain Adaptation

Ignavier Ng, Yan Li, Zijian Li, Yujia Zheng+2

Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence

Multi-Source Domain Adaptation Unsupervised Domain Adaptation Domain Adaptation Predictive Representation Learning

Open paper AskPDF arXiv

A central problem in unsupervised domain adaptation is determining what to transfer from labeled source domains to an unlabeled target domain. To handle high-dimensional observations (e.g., images), a line of approaches use deep learning to learn latent representations of the observations, which facilitate knowledge transfer in the latent space. However, existing approaches often rely on restrictive assumptions to establish identifiability of the joint distribution in the target domain, such as independent latent variables or invariant label distributions, limiting their real-world applicability. In this work, we propose a general domain adaptation framework that learns compact latent representations to capture distribution shifts relative to the prediction task and address the fundamental question of what representations should be learned and transferred. Notably, we first demonstrate that learning representations based on all the predictive information, i.e., the label's Markov blanket in terms of the learned representations, is often underspecified in general settings. Instead, we show that, interestingly, general domain adaptation can be achieved by partitioning the representations of Markov blanket into those of the label's parents, children, and spouses. Moreover, its identifiability guarantee can be established. Building on these theoretical insights, we develop a practical, nonparametric approach for domain adaptation in a general setting, which can handle different types of distribution shifts.

Abstract

Unsupervised Domain Adaptation

Explore similar work

A General Representation-Based Approach to Multi-Source Domain Adaptation

LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation

Convergence Analysis of Nyström Subsampling in Covariate Shift Adaptation for Misspecified case