When Does Dynamic Ensembling Pay Off? Diagnosing Regionwise Gains in Regression under Distribution Shift

Tianxin Zhou; Ruixi Lin

cs.LGAug 18, 2026

When Does Dynamic Ensembling Pay Off? Diagnosing Regionwise Gains in Regression under Distribution Shift

Authors: Tianxin Zhou, Ruixi Lin

Abstract

Whether input-dependent ("dynamic") combination of a regression model pool beats the best static blend depends on the shift and is rarely known before deployment. Can a small labeled target-domain probe tell us when reallocating trust across regions of the input space will pay off? We answer this with $\widehat{D}_{\mathrm{CF5}}$ , which estimates from the probe the cross-fitted gain of the regionwise convex combination over the best static convex blend: the realizable value of deciding, region by region, whom to trust. Across a frozen suite of 12 dataset-shift pairs (spatial, temporal, domain, feature-cluster), $\widehat{D}_{\mathrm{CF5}}$ predicts realized regionwise test gains with dataset-level Spearman $+0.98$ (95% CI $[+0.83, +1.00]$ ; $p=5\times10^{-5}$ ), including two cases overturning preregistered expectations. The relationship holds in a 16-pair sensitivity analysis (Spearman $+0.83$ ), whereas alternative probe diagnostics reach at most $+0.66$ . This contrast isolates regional trust reallocation: correlation is $+0.98$ for regionwise-convex gain, but $+0.01$ for smooth covariate-dependent stacking after affine correction. A controlled generator shows dynamic gains arise from the interaction of shift heterogeneity and local competence, increase with shift severity, and become realizable between 128 and 256 probe labels in the tested grid. The Probe-Validated Ensemble Selector chooses among a static affine stacker and dynamic realizers, deploying a candidate only when a held-out lower confidence bound clears the static-convex floor. In a preregistered prospective batch, it matched or improved the floor in all 12 runs; two deployments reduced test risk by 11% and 16%, while the gate rejected a candidate whose un-gated deployment incurred $>30\times$ the static loss. We release OpenRegShift, a reproducible evaluation harness for regression ensembles under distribution shift.

Figures & tables

Panel of Figure 1 — Figure 1: Data flow. The pool is trained on source data only and is evaluated at the target probe points. The probe is split once into a diagnostic sample (blue), a combiner sample and a gate sample (orange). The diagnostic branch yields $\widehat{D}_{\mathrm{CF5}}$ ; the selector branch fits candidates and validates them against the static convex floor. The test set is not used for fitting or selection. The two branches are complementary modules: the selector does not take $\widehat{D}_{\mathrm{CF5}}$ as input. The dashed blue arrow denotes the final C1 characterization analysis; test observations do not enter the diagnostic.

Data component	Information available	Permitted use
Source training set	Covariates and labels	Fit preprocessing and the frozen model pool
Source validation set	Covariates and labels	Fit residual variance heads and select the source-best model
Full target probe	Covariates only	Fit the partition used by the regional mechanism
Diagnostic sample $\mathcal{P}_{D}$	Covariates and labels	Estimate $\widehat{D}_{\mathrm{CF5}}$ by cross-fitting, then refit the static and regionwise convex combiners for C1 test evaluation
Combiner sample $\mathcal{P}_{C}$	Covariates and labels	Fit the static floor and candidate combiners
Gate sample $\mathcal{P}_{G}$	Covariates and labels	Validate candidates against the static floor

Table 1: Frozen information-access protocol. Target test observations are unavailable until final evaluation.

Figure 2: $\widehat{D}_{\mathrm{CF5}}$ versus realized regionwise gain on the frozen 12-pair suite. Large markers show dataset-level means and faded markers show the three individual seeds. The dotted line is the identity reference. Dataset-level Spearman correlation is $+0.98$ (bootstrap 95% CI $[+0.83,+1.00]$ ; permutation $p<10^{-4}$ ). Per-dataset 95% $t_{2}$ intervals appear in Fig. A.1 .

Diagnostic	Spearman	LOO-MAE
$\widehat{D}_{\mathrm{CF5}}$	$+0.979$	$0.0158$
Shift severity	$+0.657$	$0.0432$
Heterogeneity summary	$+0.545$	$0.0371$
Heterogeneity $\times$ competence	$+0.503$	$0.0342$
Local-competence summary	$+0.469$	$0.0441$
In-sample headroom	$+0.357$	$0.0259$

Table 2: Probe diagnostics on the frozen 12-pair suite. Spearman correlation is computed against realized regional gain. LOO-MAE is the leave-one-dataset-out calibration error in standardized-MSE units.

Figure 3: Controlled mechanism experiments. (a) Realized gain over the $(h,\kappa)$ grid. (b) Gain as shift severity increases. (c) Gain under nested probe budgets with a fixed test set. (d) Oracle loss as generic base-pool diversity changes. Results average three target splits within each of 10 independently seeded model pools; bands show pool-level 95% $t_{9}$ confidence intervals.

Panel of Figure 4 — Figure 4: Diagnostic and deployment workflow. Probe labels have disjoint uses: $\mathcal{P}_{D}$ estimates $\widehat{D}_{\mathrm{CF5}}$ , $\mathcal{P}_{C}$ fits the static floor and six candidates, and $\mathcal{P}_{G}$ validates the deployment decision. The partition is learned from all probe covariates without using their labels. The diagnostic characterizes the regional mechanism but does not enter the selector rule.

Stage	Decisions	Test outcome
Retrospective, $12\times 3$ runs	11 affine, 10 CDST-RBF, 1 regional, 14 floor	mean $\Delta L=-0.184$ ; worst $\Delta L=+0.000$
Preregistered prospective, $4\times 3$ runs	2 CDST-RBF, 10 floor	mean $\Delta L=-0.0035$ ; worst $\Delta L=+0.000$

Table 3: Evaluation of the Probe-Validated Ensemble Selector.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: The C1 association with per-dataset 95% $t_{2}$ confidence intervals over three seeds. Large markers show dataset means and faded markers show the individual seeds. Statistical inference in the primary analysis treats the 12 dataset/shift pairs, rather than the 36 seed runs, as the independent units.

Figure A.2: Sensitivity analysis across all 16 dataset/shift pairs. Markers show three-seed dataset means. Blue circles denote the frozen 12-pair C1 suite and red stars denote the four pairs from the prospective selector batch. The broken vertical axis preserves the extreme News result while resolving the remaining 15 pairs.

$J$	mean $J_{\mathrm{eff}}$	pooled Spearman	dataset-level Spearman
2	1.8	$+0.377$	$+0.600$
4	3.3	$+0.575$	$+0.700$
8	7.1	$+0.807$	$+1.000$
16	13.5	$+0.886$	$+1.000$
32	19.7	$+0.761$	$+0.900$

Appendix

Table A.1: Development-only sensitivity to partition resolution. Spearman correlations compare

\widehat{D}_{\mathrm{CF5}}

with the realized test gain of the regionwise convex combiner constructed at the same

J

Dataset	Source	Outcome	Role
California	sklearn/StatLib	median house value	Dev
Bike	UCI 275	$\log(1+\text{rental count})$	Dev
Superconduct	UCI 464	critical temperature	Dev
PM2.5	UCI 501	$\log(1+\mathrm{PM}_{2.5})$	Dev
Wine	UCI 186	wine quality	Dev
Parkinsons	UCI 189	total UPDRS	H1

Appendix

Table B.1: Dataset sources, prediction outcomes, and experimental roles. Dev denotes development; H1 and H2 denote the two held-out C1 batches; P3 denotes the prospective selector batch. Outcome transformations precede standardization by source-training statistics.

Dataset	Source $\rightarrow$ target	$n_{S}$	$n_{T}$	Type
California	latitude $<36$ $\rightarrow$ latitude $\geq 36$	11,814	8,826	spatial
Bike	2011 $\rightarrow$ 2012	8,645	8,734	temporal
Superconduct	remaining clusters $\rightarrow$ isolated cluster	16,583	4,680	feature-cluster
PM2.5	8 urban $\rightarrow$ 4 suburban/rural stations	60,000	50,000	spatial
Wine	white $\rightarrow$ red wine	4,898	1,599	domain
Parkinsons	28 patients $\rightarrow$ 14 held-out patients	3,932	1,943	domain

Appendix

Table B.2: Shift construction and experimental sample sizes. The 512-point target probe is drawn from the

n_{T}

observations; the remaining target observations form the test set.

Subset	Size	Permitted use
$\mathcal{P}_{D}$	205	estimate $\widehat{D}_{\mathrm{CF5}}$ by cross-fitting static and regionwise convex fits, then refit both C1 combinations on all 205 observations for test evaluation
$\mathcal{P}_{C}$	215	fit the static convex floor and the six selector candidates
$\mathcal{P}_{G}$	92	compare the fitted candidates with the floor and make the deployment decision
Total	512	labeled target-probe budget

Appendix

Table B.3: Allocation of the fixed 512-point labeled target probe.

Member	Estimator	Main settings
Gradient boosting	HistGradientBoostingRegressor	library defaults; seeded
Random forest	RandomForestRegressor	200 trees; minimum leaf size 2
Ridge regression	Ridge	penalty $\alpha=1$
Multilayer perceptron	MLPRegressor	hidden layers $(128,64)$ ; early stopping; at most 400 iterations
Nearest neighbors	KNeighborsRegressor	15 neighbors; distance weighting

Appendix

Table B.4: Models in the primary heterogeneous regression pool.

Stage	Frozen	Dataset/shift pairs	Role
H1	Aug. 3, 2026	Parkinsons, KC House, Appliances, Gas Turbine, ACS Income	First held-out test of the diagnostic and its sign-based decision rules.
H2	Aug. 4, 2026	Flights, Electricity	Second held-out test under the unchanged diagnostic protocol.
P3	Aug. 4, 2026	Protein, News, Traffic, Seoul Bike	Prospective evaluation of the final Probe-Validated Ensemble Selector. C1 quantities are reported only as a sensitivity analysis.

Appendix

Table B.5: Preregistration stages and their roles in the analysis. H1 and H2 belong to the frozen C1 suite. P3 evaluates the final selector, its diagnostic results enter only the 16-pair sensitivity analysis.

Experiment	Varied values	Fixed values
Factorial surface	$h,\kappa\in\{0,0.25,0.5,0.75,1\}$	$d=0.5$ , $\rho=0$ , $n=512$
Shift severity	$d\in\{0.125,0.25,0.5,0.75,1\}$	$h=\kappa=1$ , $\rho=0$ , $n=512$
Probe budget	$n\in\{64,128,256,512,1024\}$	$h=\kappa=1$ , $d=0.5$ , $\rho=0$
Pool homogenization	$\rho\in\{0,0.25,0.5,0.75,1\}$	$h=\kappa=1$ , $d=0.5$ , $n=512$

Appendix

Table C.1: Parameter grid for the controlled experiments.

Ensemble Learning

Ensemble learning combines multiple predictors to improve accuracy, robustness, calibration, or uncertainty estimates.

See all 272 papers

Aug 27, 2026

RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing

When machine learning classifiers are retrained, inputs correctly classified by the previous model version may be misclassified by the updated version, creating regression faults that are costly to detect because verifying predictions against ground truth may require human annotation, expert review, or expensive simulation rather than inexpensive model inference. Test input prioritization addresses this problem by ranking inputs so that a limited verification budget reveals as many regression faults as possible. Existing approaches rely predominantly on single-model confidence scores and do not exploit how predictions, decision boundaries, and local neighborhoods change between model versions. We propose RiskBlend, a classifier-agnostic prioritization framework that combines four complementary risk signals: historical failure patterns, prediction shift, decision-boundary shift, and neighborhood change. These signals are combined using validation-learned APFD-squared weighting. Across four datasets, five classifiers, four regression-update scenarios, and 15 random seeds, totaling 1,200 experimental configurations, RiskBlend achieves the highest average APFD in all 80 dataset-classifier-scenario combinations, with improvements of up to 0.32 APFD over the strongest baseline. Confidence-based methods remain competitive primarily for linear classifiers on sparse categorical features, which we attribute to feature-space geometry. The results show that cross-version behavioral signals provide important complementary information for prioritizing regression faults in machine learning systems.

Madhusudan Srinivasan, Namith Nishal Raphae

Department of Computer Science East Carolina University Greenville, USA

Automated Software Testing

Ask PDF

Jun 22, 2026

Target-Aware Linear Regression Under Distribution Shift

Distribution shift between training and deployment is a pervasive challenge for modern AI systems. In many cases, the target marginals of covariates and response are known or specified through population-level observations, boundary conditions, properties of simulator configurations, or alignment-time distributional constraints. Such knowledge may provide valuable side information for regression estimation. We study this problem in the multivariate linear regression setting with a stable conditional mean

E[Y\mid X]

across source and target, and identify the hybrid-loss estimator, which jointly incorporates both target marginals, as a benchmark target-aware estimator. Its direct computation, however, requires solving a coupled nonlinear optimization that is expensive at scale. Our main contribution is to develop and evaluate two computationally tractable alternatives: a constrained moment-matching estimator and a two-stage estimator that augments ordinary least squares with a calibration step. For all three estimators, we derive and compare closed-form asymptotic mean squared errors, yielding conditions under which the tractable alternatives match or closely approximate the hybrid benchmark, and regimes in which they do not. Monte Carlo experiments across three controlled shift regimes validate the theoretical results, investigate the accuracy-runtime tradeoffs among the three estimators, and translate into guidance on estimator choice. In particular, the two-stage estimator nearly matches the hybrid benchmark in the high signal-to-noise regime at essentially no additional cost, providing theoretical grounding for empirical observations in nonlinear settings.

Zhewen Hou, Tian Zheng

Department of Statistics Columbia University New York, NY

Distribution Shift Robustness Distribution Matching Linear Regression

Ask PDF

Jun 12, 2026

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study pre-deployment model evaluation under the joint presence of covariate shift and labeling of outcomes selectively based on observed features. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general loss function. We show identification of this estimand under standard assumptions and derive a bias-corrected estimator based on the influence function of the target risk. Finally, we evaluate our estimator through experiments using the eICU electronic health records database, showing that it tracks the true target risk more accurately than methods that address either selective labels or covariate shift alone, as well as baselines that combine standard plug-in approaches.

Annie Ulichney, Amanda Coston

Department of Statistics, University of California, Berkeley

Double ML Distribution Shift Robustness Distribution Shift Covariate Shift Selection Bias Doubly Robust Estimation+1

Ask PDF

Abstract

Ensemble Learning

Explore similar work

RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing

Target-Aware Linear Regression Under Distribution Shift

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias