cs.LGNov 3, 2026

Ensembles of Random Partition-Based Models under Covariate Shift

Authors: Mary Versa Clemens-Sewall, Eliza O'Reilly

Abstract

Distribution shifts, including covariate shifts, are ubiquitous in real-world applications of machine learning. Empirical evidence suggests that, across a variety of models with and without downstream adaptation, ensembles tend to be more robust to covariate shift than single models. We extend the theory of covariate shift severity from single models to ensembles, providing theoretical justification for the empirical trends. We introduce new discrepancy measures of covariate shift severity to control the risk of partition-based kernel estimators. We then show that with sufficient ensemble size, these models can achieve minimax-optimal risk convergence rates under covariate shift. Our results suggest that modelers can expect to need larger ensemble sizes for more severe covariate shifts.

Figures & tables

Explore similar work

Sep 10, 2026

General Quantification of Covariate and Concept Shifts

Generalization under distribution shift remains a core challenge in modern machine learning, yet existing learning bound theory is limited to narrow, idealized settings and is non-estimable from samples. In this paper, we bridge the gap between theory and practical applications. We first show that existing definition of concept shift breaks when the source and target supports mismatch. Leveraging entropic optimal transport, we propose a key notion: γ∗ ⁣γ^{*}\!-concept shifts, and derive a general error bound unifying covariate and γ∗ ⁣γ^{*}\!-concept shifts, which applies to broad loss functions, label spaces, and stochastic labeling. We further develop estimators for these shifts with concentration guarantees, and the DataShifts algorithm, which can quantify distribution shifts and estimate the error bound in most applications - a rigorous and general tool for analyzing learning error under distribution shift.
Jul 28, 2026

PIcsC: Partitioning-Induced Covariate Shift Correction

Covariate shift across training-data partitions biases model selection and parameter estimation in cross-validation, lifelong learning, and federated learning. We propose Partition-Induced Covariate-shift Correction (PIcsC), a Fisher information-based regularization framework that mitigates distribution mismatch between data partitions and a reference distribution. PIcsC approximates partition divergence using the Fisher Information Matrix (FIM) and incorporates the resulting statistic as a regularizer during optimization. The same formulation applies to both centrally partitioned datasets (batches or cross-validation folds) and inherently distributed data (federated clients or decentralized nodes), requiring only partition-local gradient statistics rather than raw data. We further introduce a conditional adaptation mechanism that combines FIM shift with KL divergence to detect significant distribution shifts and activates regularization only when necessary. Experiments on more than 40 datasets demonstrate consistent improvements under both natural and synthetic covariate shift. On fragmented batch and fold settings, PIcsC reduces fragmentation-induced performance degradation by more than 20% and 25%, respectively. On seven federated learning benchmarks, it consistently outperforms FedAvg, FedProx, and SCAFFOLD by 3 -5 percentage points without requiring client-specific personalization. These results demonstrate that Fisher information provides an effective and unified mechanism for mitigating partition-induced covariate shift across both centralized and distributed learning.
Aug 18, 2026

When Does Dynamic Ensembling Pay Off? Diagnosing Regionwise Gains in Regression under Distribution Shift

Whether input-dependent ("dynamic") combination of a regression model pool beats the best static blend depends on the shift and is rarely known before deployment. Can a small labeled target-domain probe tell us when reallocating trust across regions of the input space will pay off? We answer this with D^CF5\widehat{D}_{\mathrm{CF5}}, which estimates from the probe the cross-fitted gain of the regionwise convex combination over the best static convex blend: the realizable value of deciding, region by region, whom to trust. Across a frozen suite of 12 dataset-shift pairs (spatial, temporal, domain, feature-cluster), D^CF5\widehat{D}_{\mathrm{CF5}} predicts realized regionwise test gains with dataset-level Spearman +0.98+0.98 (95% CI [+0.83,+1.00][+0.83, +1.00]; p=5×10−5p=5\times10^{-5}), including two cases overturning preregistered expectations. The relationship holds in a 16-pair sensitivity analysis (Spearman +0.83+0.83), whereas alternative probe diagnostics reach at most +0.66+0.66. This contrast isolates regional trust reallocation: correlation is +0.98+0.98 for regionwise-convex gain, but +0.01+0.01 for smooth covariate-dependent stacking after affine correction. A controlled generator shows dynamic gains arise from the interaction of shift heterogeneity and local competence, increase with shift severity, and become realizable between 128 and 256 probe labels in the tested grid. The Probe-Validated Ensemble Selector chooses among a static affine stacker and dynamic realizers, deploying a candidate only when a held-out lower confidence bound clears the static-convex floor. In a preregistered prospective batch, it matched or improved the floor in all 12 runs; two deployments reduced test risk by 11% and 16%, while the gate rejected a candidate whose un-gated deployment incurred >30×>30\times the static loss. We release OpenRegShift, a reproducible evaluation harness for regression ensembles under distribution shift.