We introduce a transfer learning framework for regression that leverages heterogeneous source domains to improve predictive performance in a data-scarce target domain. Our approach fits a conditional generative model to each source domain and calibrates a shared linear combination of the generated responses via conditional quantile matching at the observed target covariates. The calibration aligns the conditional response distributions while accommodating simultaneous shifts in both the covariate and conditional response distributions. The calibrated responses then serve as pseudo-labels to augment the target data for a broad class of downstream learning tasks. Theoretically, we derive an excess risk bound for an empirical risk minimizer (ERM) trained on the augmented data, separating the benefit of an enlarged training sample from errors due to density ratio estimation, source generator estimation, conditional calibration, and irreducible transfer bias. Empirically, simulations and real-data applications show that the proposed method can improve target-domain prediction when the transfer bias is small.
Figures & tables
Figure 1: Illustration of conditional quantile matching at a fixed covariate value, where the intercept β0 is omitted from the schematic.
Figure 2: Prediction performance of different ML models under the simulated training scenarios and competing transfer learning methods.
Target-Only
TLCQM
TKRR
CDAR
DARC
XGBoost
NN
KRR
XGBoost
NN
KRR
Platform I
9.180
5.717
21.794
4.099 (0.014)
4.211 (0.012)
4.073 (0.014)
6.981
139.327 (1.093)
18.220 (1.156)
Platform II
8.998
5.005
32.218
3.785 (0.065)
3.840 (0.046)
3.767 (0.068)
4.967
117.149 (1.194)
11.542 (0.921)
Table 1: Average MSEs of different ML models trained on target-only and TLCQM-augmented data, together with those of competing transfer learning methods, for the MAU prediction task across two device platforms. Standard errors of the MSEs based on 30 Monte Carlo experiments are reported in parenthesis, where those values smaller than 0.001 are omitted for brevity.
Function Class
Upper Bound
References
Finite hypothesis class F with supx∈X∣f(x)∣≤Bf for any f∈F .
O(Bfnlog(2∣F∣))
Theorem 3.7 in Mohri et al. [2018] ; see also Massart [2000] .
Linear predictor class F={x↦ΥTx:∥Υ∥2≤Bγ,∥x∥2<Bx} .
O(nBγBx)
Theorem 3 in Kakade et al. [2008] ; see also Theorem 1 in Awasthi et al. [2020] .
Sparse linear predictor F={x↦ΥTx:x∈Rd,∥Υ∥0≤s,∥Υ∥2≤Bγ,∥x∥2<Bx} .
O(BγBxnslogd)
Section 3.1 in Kakade et al. [2008] ; see also Lemma 7.1 in Koltchinskii [2011] .
Reproducing kernel Hilbert space F={f∈H:∥f∥H≤Bf} with supxk(x,x)≤κ
O(nBfκ)
Lemma 22 in Bartlett and Mendelson [2002] .
Binary decision tree class with L nodes and d real-valued features
O(nLlog(2Ld)log(2n))
Corollary 10 in Leboeuf et al. [2020] .
Neural network with the ReLU activation function, W weights, L layers, and outputs clipped to [−Bf,Bf] .
O(nWLlog(2W)log(2n))
Theorem 6 in Bartlett et al. [2019] ; see also Theorem 5 in Yin et al. [2019] , Theorem 14 in Truong [2022] , and Theorem 3.3 in Bartlett et al. [2017] .
Table 2: Upper bounds on the Rademacher complexity Radn(F) for common function classes
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Ratio n0nk
Source 1 only
Source 2 only
Target only
Naive pooling
“Mixup” (best α )
TLCQM
XGBoost
1
4.317
4.323
0.307
0.619
0.354
0.315
2
4.320
4.270
0.307
1.221
0.354
0.314
5
4.357
4.158
0.307
2.054
0.355
0.308
10
4.485
4.226
0.307
2.421
0.354
0.304
15
4.666
4.397
0.307
2.494
0.354
0.303
Appendix
Table 3: Prediction performance of machine learning models under different training strategies and source-to-target ratios. For each model and data scenario, the “mixup” column reports the best-performing choice of α in terms of MSE.
Source-to-Target Ratio n0nk
1
2
5
10
15
20
30
Mean Squared Error (XGBoost)
Engression Only
0.880
2.420
2.965
3.209
3.319
3.374
3.557
Conditional MM
0.307
0.309
0.307
0.305
0.303
0.303
0.302
OT Calibration
0.357
0.379
0.421
0.470
0.507
0.537
0.587
TLCQM
0.315
0.314
0.307
0.304
0.303
0.302
0.302
Mean Squared Error (KRR)
Appendix
Table 4: Results for ablation studies comparing engression only, conditional mean matching, OT calibration, and the proposed TLCQM framework across different source-to-target ratios.
Figure 3: Simulation results under the “ reversed conditional relation ” scenario. The top row reports ΔMSE=MSETLCQM−MSETarget−only ; negative values indicate favorable transfer. The bottom row reports marginal and conditional Wasserstein-1 distances from the first source and TLCQM-calibrated distributions to the target one.
Figure 4: Simulation results under the “ outside affine span ” scenario. The top row reports ΔMSE=MSETLCQM−MSETarget−only ; negative values indicate favorable transfer. The bottom row reports marginal and conditional Wasserstein-1 distances from the first source and TLCQM-calibrated distributions to the target one.
Figure 5: Performances of different machine learning models applied to target-only and TLCQM scenarios as well as other competing transfer learning methods on the “Apartment” data.
n0
100
200
300
500
XGBoost
11.99%
5.49%
1.70%
-2.70%
KRR
78.49%
66.00%
57.89%
47.81%
NN
80.01%
65.86%
49.55%
34.10%
Appendix
Table 5: Percentage reduction in MSE achieved by machine learning models trained on TLCQM-augmented data relative to their target-only counterparts on the “Apartment” data.
Conditional generative modeling with limited target data can benefit from representations learned in a related source population. We develop a transfer learning framework for conditional flow matching that accommodates distinct source and target response distributions and dimensions. Two procedures exploit a representation learned from source data: TFM-Freeze estimates a target velocity head with the representation fixed, whereas TFM-Adapt jointly learns the head and a representation adaptation under a functional anchoring penalty. For explicit neural network classes, we establish bounds on conditional squared Wasserstein risk that account for source estimation, target fitting, representation mismatch, and numerical integration. When the source representation admits a smooth correction, adaptation replaces the frozen mismatch contribution by the cost of learning the correction and a controllable regularization bias. An information lower bound identifies regimes in which adaptation attains a strictly faster conditional squared Wasserstein risk rate than any generator restricted to the frozen context information. We also develop independent validation procedure for selecting among fitted conditional generators. We show that the selection procedure satisfies an oracle inequality, and the switching rule guarantees, with high probability, no increase in conditional energy risk relative to a target-only generator. Simulations and real data applications illustrate the gains from our proposed methods and the effectiveness of target validation in guarding against negative transfer.
Transfer learning, in regression or classification, concerns problems where one aims to use imperfect data from a source distribution P to improve prediction w.r.t. a target distribution Q. While there is now much theoretical understanding of the unsupervised setting of this problem (where no labeled target data is available), the supervised setting with labeled target data remains under-studied. We identify and characterize an interesting dichotomy in the rates achievable in the supervised setting: there is a weak transfer regime where optimal rates are determined by information in either the source or target dataset but not both, and a strong transfer regime where much faster target rates are achievable, accounting for complementary information in source and target data. Furthermore, while previous theoretical tools for the unsupervised setting can be extended to capture the weak transfer regime, we argue that they cannot capture the strong regime due to limitations inherent in the various notions of relatedness they rely on. Instead, we provide an appropriate refinement of relatedness notions---termed weak and strong moduli of transfer---that reveal the two regimes and lead to a nontrivial characterization of which pairs of distributions P and Q adhere to which regime. Finally, the above dichotomy reveals a new categorization of existing algorithmic approaches in terms of their optimality or lack-thereof across supervised regimes, and we provide a generic approach that remains nearly optimal in all regimes without prior distributional knowledge.
Conformal prediction converts point predictions into set-valued predictions with coverage guarantees under exchangeability between calibration and deployment data. We study conformal calibration transfer, where this requirement fails because labeled calibration is available only in a source space, while prediction sets are needed in a target space linked to the source through unlabeled paired observations (e.g., paired modalities or sensor changes). We propose Transported Conformal Calibration (TCC): we transport labeled source calibration into the target space using the paired data, and then correct residual post-transport mismatch using only unlabeled target inputs. We instantiate this correction with two complementary methods: TCC-KS, which uses a label-free uncertainty surrogate to detect mismatch and adjust calibration conservatively, and weighted-TCC, which reweights transported calibration toward the target domain for improved efficiency when weights are stable. We provide finite-sample target-domain coverage guarantees that adapt to an observable measure of mismatch. Across CIFAR-100-C, Tiny-ImageNet-C, and SEN12MS, we show reliable target-domain coverage transfer without labeled target calibration data, with label-free diagnostics that predict when correction is needed.
Achref Doula
Technical University of Darmstadt, Darmstadt, Germany · Tongji University, Shanghai, China.