stat.MLJul 31, 2025

Formal Bayesian Transfer Learning via the Total Risk Prior

Authors: Nathan Wycoff, Ali Arab, Lisa O. Singh

Abstract

Existing methods for transfer learning struggle to deal with situations where the source datasets are limited and not guaranteed to be well-aligned with the target dataset. A typical strategy is to use the empirical loss minimizer on the source data as a prior mean for the target parameters. Our key conceptual contribution is to use a risk minimizer conditional on source parameters instead. This allows us to construct a single joint prior distribution for all parameters from the source datasets as well as the target dataset. As a consequence, we benefit from full Bayesian uncertainty quantification and can perform model averaging via Gibbs sampling over indicator variables governing the inclusion of each source dataset. We show how a particular instantiation of our prior leads to a Bayesian Lasso in a transformed coordinate system and discuss computational techniques to scale our approach to moderately sized datasets. We discuss connections between the Maximum a Posteriori estimate associated with our approach and the recently proposed Trans-Lasso method and demonstrate that the MAP estimator MSE-dominates the Trans-Lasso in the normal means setting when there is no regularization on the source datasets. Finally, we perform numerical experiments finding that full Bayesian inference provides superior predictive performance relative to Trans-Lasso on a genetics application, especially when the source data are limited.

Explore similar work

CardsList
Jun 11, 2026cs.LG

Loss-Shift Transfer via Bayes Quotients

Transfer learning is usually studied as a consequence of distribution shift. This paper identifies an orthogonal failure mode in which the data distribution is fixed and the loss changes. This setting is called \emph{loss shift}. A loss determines which information in XX is Bayes-relevant, and two losses may therefore require different representations even under the same joint law P(X,Y)P(X,Y). The idea is formalized using Bayes quotients, which allow losses to be ordered by refinement. In the Bayes-quotient formulation, strict refinement gives an immediate qualitative obstruction. A source-minimal representation for a coarser loss is insufficient for a strictly finer target loss. For finite-output log loss, this obstruction becomes an exact quantitative identity. The excess risk is the conditional information about YY discarded by the representation. Experiments in controlled, learned, synthetic-image, and real-image settings show the predicted effect, i.e., classification-equivalent representations can have different optimal log-loss performance under a fixed data distribution.
Vasileios Sevetlidis
Distribution ShiftsLoss FunctionTransfer LearningBayesian Inference Approaches
AskPDF
Jul 10, 2026cs.LG

Pitfalls and Remedies for Multi-Task Bayesian Optimization

Bayesian optimization routinely warm-starts a target experiment with data from related source tasks, and the multi-task Gaussian process is the textbook surrogate for the job. We revisit this default in a controlled setting and find that it misestimates the cross-task correlation even in the simplest non-trivial case, affinely related source and target tasks, where a working transfer learning method should obviously succeed. We trace the failure to two independent structural mechanisms. Per-task standardization, the textbook fix for the affine slice ambiguity, propagates a finite-sample alignment error into the recovered correlation. The marginal likelihood itself identifies the correlation only at a per-sample rate that a Gaussian process at non-overlapping designs further dilutes. We propose three conservative remedies that follow from the analysis: promoting per-task means and scales to model parameters, restricting the task covariance to non-negative correlations, and co-locating part of the source and target designs. Across synthetic multi-task problems and surrogate-based hyperparameter tuning transfer, these remedies recover the target-only baseline on the simple instances, while the broader failure persists on harder instances and across most rank-based and latent-context variants.
Carl Hvarfner, Sam Daulton, Max Balandat +1
Bayesian OptimizationGaussian ProcessHyperparameterSurrogate GradientCross-Task InterferenceRemediation
AskPDF
Jul 3, 2026cs.LG

Transfer Learning in High-dimensional Ising Models

In high-dimensional Ising model estimation, target sample sizes are often limited, and effectively using auxiliary binary datasets of unknown relevance remains challenging. To address this, we propose Trans-Ising, a transfer learning method that combines a loss-based source screening rule with a two-stage estimation procedure. The method first identifies informative auxiliary sources using held-out target pseudolikelihood to prevent negative transfer. It then computes an initial estimator via pooled nodewise ℓ1\ell_1-regularized logistic regression, followed by a target-only correction step using a folded-concave penalty. Theoretically, we establish fixed-node ℓ2\ell_2 and ℓ1\ell_1 error bounds, exact graph selection consistency, and the conditional consistency of the screening rule. Through extensive simulations and real-data analyses, we demonstrate that Trans-Ising achieves lower estimation errors than both target-only estimation and naive data pooling.
Joonho Kim, Seyoung Park
Transfer LearningTwo-Dimensional Ising ModelHigh-DimensionalScreening
AskPDF