Reweighting for Transfer Classification: Exact Risk Conversion and Minimax Adaptation

Aoran Chen; Yang Feng

math.STNov 6, 2026

Reweighting for Transfer Classification: Exact Risk Conversion and Minimax Adaptation

Authors: Aoran Chen, Yang Feng

Organizations: Department of Statistics, The Pennsylvania State University · Department of Biostatistics, New York University

Abstract

A source can make a training mixture easier to learn while making its accuracy less valuable for the target. We study how the training problem a source creates and the finite-sample error of learning it jointly determine target-risk guarantees under reweighting. For binary classification, for each pair of laws and each positive target weight, we derive the sharpest target excess hinge-risk bound implied by mixture excess alone, uniformly over clipped predictors. This exact conversion curve is determined by density-weighted local votes. A two-sample oracle inequality for weighted support vector machines supplies the mixture excess, and the composition becomes explicit for Gaussian kernels. On a tilted-boundary class, we establish the all-procedure minimax classification rate $[n+s_+^2/(m^{-1}+τ^2)]^{-1}$ in the transferable strength $s_+$ and the tilt $τ$ , separating source sampling uncertainty ( $m^{-1}$ ) from relation uncertainty ( $τ^2$ ). A target-checked Gaussian-SVM selector adapts to unknown $s_+$ and $τ$ up to a logarithmic factor for fixed dimension, and retains the target-only rate up to the same factor against arbitrary independent sources under the stated target model. On a separate smooth opposition class, the minimax risk within the family of exact Gaussian SVMs fitted to the original labels with nonnegative weights is exponentially larger than the unrestricted minimax risk whenever the target sample is at most half the source sample, despite arbitrary data-dependent weighting and tuning. The value of a source thus depends on the training problem it creates, on what accuracy there means for the target, and on whether the permitted use of the source can implement the needed correction.

Figures & tables

Panel of Figure 1 — Figure 1: Each opposed location flips at its own source weight, and the flip profile is the largest target hinge excess the tied and flipped locations can incur at zero mixture excess. (a) A location with local vote ratio $r$ flips once the source weight passes $u_{c}(r)=(1+r)^{-1}$ (blue curve: ties on the curve, flips above it, surviving opposition below it). Raising $u$ moves a location up its vertical line; below the curve it keeps its label while its target-directed vote weakens. At a fixed weight $u_{0}$ , the horizontal line picks out the locations with $r\geq u_{0}^{-1}-1$ , the tie-or-flip set $\mathcal{C}(u_{0})$ of ( 2.5 ) (red segment); raising $u_{0}$ moves the line up and the segment to the left. (b) The flip profile $H_{\rm flip}(u)$ of ( 2.13 ) is the mass $2\nu_{T}$ of $\mathcal{C}(u)$ under the target-margin measure $\nu_{T}$ of ( 2.12 ) (blue). It never decreases as $u$ grows, it is flat where the locations reaching their thresholds carry zero target-margin mass, it is defined for $u<1$ , and it stops at the mass $2\nu_{T}(\Omega_{-})$ of all opposed locations (dotted). Population quantities; the profile is a schematic shape.

Panel of Figure 2 — Figure 2: A mixture bound $t$ is worth exactly $\Psi_{\alpha}(t)$ on the target, and the shape of $\Psi_{\alpha}$ sets the rate that survives the conversion. (a) One law pair and one weight, with $M_{\alpha}<\infty$ . The curve $\Psi_{\alpha}$ (blue, Theorem 1 ) starts at the flip cost $H_{\rm flip}(u)$ , rises concavely, and saturates at $2J_{T}$ once the budget reaches the vote capacity $2\nu_{\alpha}(\mathcal{D}_{\alpha}^{+})$ . Each sharp affine bound $ct+\mathcal{B}_{\alpha}(c)$ touches the curve at one budget (red, tangent at the dot); the line of slope $M_{\alpha}$ through $H_{\rm flip}(u)$ (yellow) has the smallest intercept any affine bound can have; by ( 3.6 ) the curve is the lower envelope of all these lines. (b) The three conversion shapes of Corollary 1 . Without opposition, or with opposition held below the flip threshold, the conversion is linear through the origin with slope $M_{\alpha}$ (yellow). For the constant vote $Q_{\alpha}\equiv q_{0}$ of Example 1 the exact curve is the line $t/q_{0}$ up to its saturation level $2J_{T}$ (blue). Near cancellation the bound is Hölder, $H_{\rm flip}(u)+Ct^{\chi}$ (red), and it starts at the flip cost. At one budget $t_{a}$ the three shapes give three target guarantees (dots); a smaller mixture bound ( $t_{b}\to t_{a}$ ) moves along one curve, a better source–target relation moves down to another curve, and a different weight changes the curve. In both panels the hatched band below the level $H_{\rm flip}(u)=\Psi_{\alpha}(0)$ is the largest cost the tied and flipped locations can still carry at zero mixture excess; the two linear relations have no such cost. A mixture bound of order $N_{\rm eff}^{-\omega}$ , with $N_{\rm eff}$ the effective sample size of ( 3.10 ), converts to $O(N_{\rm eff}^{-\omega})$ under the linear shapes and to $H_{\rm flip}(u)+O(N_{\rm eff}^{-\omega\chi})$ under the Hölder shape. Colors keep one meaning in both panels: blue is the exact curve, yellow the slope- $M_{\alpha}$ line, red a bound that pays a flip intercept. Schematic population quantities.

Figure 3: Target risk and selected borrowing along the three perturbation paths. The top row reports mean target classification excess $\mathcal{E}^{01}_{T}(\widehat{h})$ (log scale) with pointwise Monte Carlo standard errors for target-only ( $r=0$ ), proportional pooling ( $r=1$ ), fixed $\alpha=1/2$ ( $r=n/m$ ), the CV-adaptive rule, and the replicate-wise oracle. The bottom row reports the median selected source-to-target per-observation weight ratio $r$ with its interquartile range. Ribbons show population vote regions at the reference weight $\alpha=1/2$ : agreement (blue), surviving opposition (orange), and thresholded tie-or-flip (red). Dotted vertical lines in panel (c) mark the angles examined in Figure 4 . The path grids are specified in the Supplement (Section S.5.3 ).

Figure 4: Prediction maps behind Figure 3 (c), at $0^{\circ}$ , $90^{\circ}$ , and $135^{\circ}$ , from $100$ additional training replications per angle ( $n=50$ , $m=1{,}000$ ) on a shared reference sample of $20{,}000$ covariates. The first row shows population vote regions at $\alpha=1/2$ ; tie/flip ∗ uses $m_{T}m_{\alpha}\leq 10^{-3}$ on opposing points, including small positive products, and therefore differs from the exact theoretical set. The thin labeled contours in this row show the cancellation fraction $1-|m_{\alpha}|/(\alpha|m_{T}|+u|m_{S}|)$ , which equals $1-|s_{\alpha}|/(\alpha|s_{T}|+u|s_{S}|)$ under the common design, at $0.25$ , $0.50$ , $0.75$ , and $0.90$ ; it is $0$ where the domains agree and $1$ at a tie. The remaining rows show equal-domain, pooling, and CV-adaptive fits, colored by the target-aligned consensus $S_{M}$ of ( S.5.1 ): blue indicates repeated target-sign agreement, red persistent disagreement, and near-white unstable predictions. Contours show $V_{M}=1-S_{M}^{2}$ at $0.5$ , $0.8$ , and $0.95$ ; reported unstable and stable-wrong shares use $V_{M}>0.8$ and $S_{M}<-0.8$ . Solid and dashed curves mark the target and population mixture boundaries; for the CV-adaptive row the mixture boundary is drawn at the median selected $\alpha$ over the replications, and the dashed curve is omitted when this median equals $1$ , as at $135^{\circ}$ , where every replication selects target-only training and the population mixture is the target law. Gray masks delimit the displayed region. Definitions and the shared-reference protocol are in the Supplement (Section S.5.2 ).

Figure S1: Mixture learning and target performance at fixed weight $\alpha=1/2$ . For the same fitted SVM, the panels report the mixture hinge excess $\widehat{\mathcal{E}}_{\alpha}(\widehat{f})$ (solid, circles), the target hinge excess $\widehat{\mathcal{E}}_{T}(\widehat{f})$ (dashed, squares), and the conversion envelope $\Psi_{\alpha}\{\widehat{\mathcal{E}}_{\alpha}(\widehat{f})\}$ (dotted, triangles) evaluated at the fitted mixture excess. The horizontal dashed line is the target-only reference. Panels vary (a) source size under equal laws, (b) source margin power at zero rotation, and (c) source rotation at unit margin power. Curves are means over $1{,}000$ replications with pointwise Monte Carlo standard errors; in panel (a) the three curves coincide because $P_{S}=P_{T}$ . Panel grids and the full data-generating protocol are given in Section S.5.3 .

Cross-Domain Transfer Learning

Cross-domain transfer learning reuses knowledge across datasets, populations, devices, or tasks when their distributions and feature spaces differ.

See all 212 papers

Aug 29, 2024

Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning

Transfer learning, in regression or classification, concerns problems where one aims to use imperfect data from a source distribution P to improve prediction w.r.t. a target distribution Q. While there is now much theoretical understanding of the unsupervised setting of this problem (where no labeled target data is available), the supervised setting with labeled target data remains under-studied. We identify and characterize an interesting dichotomy in the rates achievable in the supervised setting: there is a weak transfer regime where optimal rates are determined by information in either the source or target dataset but not both, and a strong transfer regime where much faster target rates are achievable, accounting for complementary information in source and target data. Furthermore, while previous theoretical tools for the unsupervised setting can be extended to capture the weak transfer regime, we argue that they cannot capture the strong regime due to limitations inherent in the various notions of relatedness they rely on. Instead, we provide an appropriate refinement of relatedness notions---termed weak and strong moduli of transfer---that reveal the two regimes and lead to a nontrivial characterization of which pairs of distributions P and Q adhere to which regime. Finally, the above dichotomy reveals a new categorization of existing algorithmic approaches in terms of their optimality or lack-thereof across supervised regimes, and we provide a generic approach that remains nearly optimal in all regimes without prior distributional knowledge.

Steve Hanneke, Samory Kpotufe

Purdue University · Columbia University

Transfer Learning Cross-Domain Transfer Learning Sample Complexity

Ask PDF

Jun 11, 2026

Loss-Shift Transfer via Bayes Quotients

Transfer learning is usually studied as a consequence of distribution shift. This paper identifies an orthogonal failure mode in which the data distribution is fixed and the loss changes. This setting is called loss shift. A loss determines which information in

X

is Bayes-relevant, and two losses may therefore require different representations even under the same joint law

P(X,Y)

. The idea is formalized using Bayes quotients, which allow losses to be ordered by refinement. In the Bayes-quotient formulation, strict refinement gives an immediate qualitative obstruction. A source-minimal representation for a coarser loss is insufficient for a strictly finer target loss. For finite-output log loss, this obstruction becomes an exact quantitative identity. The excess risk is the conditional information about

Y

discarded by the representation. Experiments in controlled, learned, synthetic-image, and real-image settings show the predicted effect, i.e., classification-equivalent representations can have different optimal log-loss performance under a fixed data distribution.

Vasileios Sevetlidis

Athena Research Center, Kimmeria Campus, Xanthi, Greece · Democritus University of Thrace, Vas. Sofias Campus, Xanthi, Greece · International Hellenic University, Serres, Greece

Representation Learning Transfer Learning

Ask PDF

Sep 17, 2026

Extreme classification: beating chance with one training example from each class

We study a minimal classification problem: Given independent labeled observations

X\sim P

and

Z\sim Q

from two unknown distributions

P,Q

, and given an independent target

Y

drawn with equal probability from

P

Q

, can one classify

Y

strictly better than chance whenever

P\neq Q

? The one-nearest-neighbor rule succeeds for every pair of multivariate Gaussian distributions with distinct means and a common positive-definite covariance matrix but can perform strictly worse than chance even for smooth densities on the real line. We construct a fixed randomized kernel rule whose expected accuracy is exactly

1/2+\operatorname{MMD}_k^2(P,Q)/4

, and obtain characteristic kernels on countably generated measurable spaces from countable families of measurable binary questions. We also prove that a deterministic order rule on

\mathbb R

beats chance for every pair of distinct Borel probability measures. A measurable encoding then gives a deterministic distribution-free rule which beats chance on every countably generated measurable space, in particular every separable metric space. Finally, we show that no rule works for every distinct pair of distributions and every unknown unbalanced class prior; under adaptive target-class selection, every rule other than a fair coin is strictly worse than chance for some finitely supported pair.

Kevin Bleakley, Aaditya Ramdas

LMO, CELESTE · Inria and Laboratoire de Mathématiques d’Orsay, France · Department of Statistics, Stanford University, Stanford, CA, USA

Classification Binary Classification

Ask PDF

Abstract

Cross-Domain Transfer Learning

Explore similar work

Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning

Loss-Shift Transfer via Bayes Quotients

Extreme classification: beating chance with one training example from each class