We introduce a transfer learning framework for regression that leverages heterogeneous source domains to improve predictive performance in a data-scarce target domain. Our approach fits a conditional generative model to each source domain and calibrates a shared linear combination of the generated responses via conditional quantile matching at the observed target covariates. The calibration aligns the conditional response distributions while accommodating simultaneous shifts in both the covariate and conditional response distributions. The calibrated responses then serve as pseudo-labels to augment the target data for a broad class of downstream learning tasks. Theoretically, we derive an excess risk bound for an empirical risk minimizer (ERM) trained on the augmented data, separating the benefit of an enlarged training sample from errors due to density ratio estimation, source generator estimation, conditional calibration, and irreducible transfer bias. Empirically, simulations and real-data applications show that the proposed method can improve target-domain prediction when the transfer bias is small.
Figures & tables
Figure 1: Illustration of conditional quantile matching at a fixed covariate value, where the intercept β0 is omitted from the schematic.
Figure 2: Prediction performance of different ML models under the simulated training scenarios and competing transfer learning methods.
Target-Only
TLCQM
TKRR
CDAR
DARC
XGBoost
NN
KRR
XGBoost
NN
KRR
Platform I
9.180
5.717
21.794
4.099 (0.014)
4.211 (0.012)
4.073 (0.014)
6.981
139.327 (1.093)
18.220 (1.156)
Platform II
8.998
5.005
32.218
3.785 (0.065)
3.840 (0.046)
3.767 (0.068)
4.967
117.149 (1.194)
11.542 (0.921)
Table 1: Average MSEs of different ML models trained on target-only and TLCQM-augmented data, together with those of competing transfer learning methods, for the MAU prediction task across two device platforms. Standard errors of the MSEs based on 30 Monte Carlo experiments are reported in parenthesis, where those values smaller than 0.001 are omitted for brevity.
Function Class
Upper Bound
References
Finite hypothesis class F with supx∈X∣f(x)∣≤Bf for any f∈F .
O(Bfnlog(2∣F∣))
Theorem 3.7 in Mohri et al. [2018] ; see also Massart [2000] .
Linear predictor class F={x↦ΥTx:∥Υ∥2≤Bγ,∥x∥2<Bx} .
O(nBγBx)
Theorem 3 in Kakade et al. [2008] ; see also Theorem 1 in Awasthi et al. [2020] .
Sparse linear predictor F={x↦ΥTx:x∈Rd,∥Υ∥0≤s,∥Υ∥2≤Bγ,∥x∥2<Bx} .
O(BγBxnslogd)
Section 3.1 in Kakade et al. [2008] ; see also Lemma 7.1 in Koltchinskii [2011] .
Reproducing kernel Hilbert space F={f∈H:∥f∥H≤Bf} with supxk(x,x)≤κ
O(nBfκ)
Lemma 22 in Bartlett and Mendelson [2002] .
Binary decision tree class with L nodes and d real-valued features
O(nLlog(2Ld)log(2n))
Corollary 10 in Leboeuf et al. [2020] .
Neural network with the ReLU activation function, W weights, L layers, and outputs clipped to [−Bf,Bf] .
O(nWLlog(2W)log(2n))
Theorem 6 in Bartlett et al. [2019] ; see also Theorem 5 in Yin et al. [2019] , Theorem 14 in Truong [2022] , and Theorem 3.3 in Bartlett et al. [2017] .
Table 2: Upper bounds on the Rademacher complexity Radn(F) for common function classes
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Ratio n0nk
Source 1 only
Source 2 only
Target only
Naive pooling
“Mixup” (best α )
TLCQM
XGBoost
1
4.317
4.323
0.307
0.619
0.354
0.315
2
4.320
4.270
0.307
1.221
0.354
0.314
5
4.357
4.158
0.307
2.054
0.355
0.308
10
4.485
4.226
0.307
2.421
0.354
0.304
15
4.666
4.397
0.307
2.494
0.354
0.303
Appendix
Table 3: Prediction performance of machine learning models under different training strategies and source-to-target ratios. For each model and data scenario, the “mixup” column reports the best-performing choice of α in terms of MSE.
Source-to-Target Ratio n0nk
1
2
5
10
15
20
30
Mean Squared Error (XGBoost)
Engression Only
0.880
2.420
2.965
3.209
3.319
3.374
3.557
Conditional MM
0.307
0.309
0.307
0.305
0.303
0.303
0.302
OT Calibration
0.357
0.379
0.421
0.470
0.507
0.537
0.587
TLCQM
0.315
0.314
0.307
0.304
0.303
0.302
0.302
Mean Squared Error (KRR)
Appendix
Table 4: Results for ablation studies comparing engression only, conditional mean matching, OT calibration, and the proposed TLCQM framework across different source-to-target ratios.
Figure 3: Simulation results under the “ reversed conditional relation ” scenario. The top row reports ΔMSE=MSETLCQM−MSETarget−only ; negative values indicate favorable transfer. The bottom row reports marginal and conditional Wasserstein-1 distances from the first source and TLCQM-calibrated distributions to the target one.
Figure 4: Simulation results under the “ outside affine span ” scenario. The top row reports ΔMSE=MSETLCQM−MSETarget−only ; negative values indicate favorable transfer. The bottom row reports marginal and conditional Wasserstein-1 distances from the first source and TLCQM-calibrated distributions to the target one.
Figure 5: Performances of different machine learning models applied to target-only and TLCQM scenarios as well as other competing transfer learning methods on the “Apartment” data.
n0
100
200
300
500
XGBoost
11.99%
5.49%
1.70%
-2.70%
KRR
78.49%
66.00%
57.89%
47.81%
NN
80.01%
65.86%
49.55%
34.10%
Appendix
Table 5: Percentage reduction in MSE achieved by machine learning models trained on TLCQM-augmented data relative to their target-only counterparts on the “Apartment” data.