TrustMix: Graceful Degradation for Pretrained Models under Distribution Shift
Authors: Juliusz Ziomek, Maike Osborne, Yee Whye Teh
Organizations: Machine Learning Research Group, University of Oxford, Oxford OX1 2JD, United Kingdom · Department of Statistics, University of Oxford, Oxford OX1 2JD, United Kingdom
Predicting beyond the training domain necessarily requires assumptions about how the ground truth extrapolates, yet in most neural networks these assumptions are made only implicit through the architecture. We propose TrustMix, a lightweight post-hoc method that makes a pretrained model's extrapolative behaviour explicit and controllable. TrustMix combines the predictive distribution produced by a pretrained model with a user-specified fallback distribution p0(y∣x), such that as the out-of-distribution (OOD) deviation statistic grows, we collapse towards p0(y∣x). TrustMix provides graceful degradation: the predictive distribution converges toward the fallback distribution as the OOD signal grows. The fallback distribution p0(y∣x) can reflect domain knowledge, if available, or rely on a domain-agnostic stationarity assumption. TrustMix does not require retraining of the base model and can be applied to both regression and classification problems. We evaluated TrustMix on several synthetic and real-world distribution shifts, including tabular, image, and natural language processing tasks. We tested TrustMix on datasets such as CIFAR-10/100 and ImageNet and on numerous architectures including Convolutional Neural Networks (CNNs) and Transformers. TrustMix remains competitive in-distribution while improving predictive uncertainty under distribution shift, with gains in predictive likelihood for regression and calibration for classification. Finally, we showed that TrustMix can provide useful predictive distributions for decision-making applications by utilizing it as a surrogate for Bayesian Optimisation.
Figures & tables
Figure 1: Illustration of TrustMix on a toy regression problem. Top-left: predictions of a base MLP with a variance head, showing overconfident extrapolation outside the training region. Top-right: Mahalanobis deviation score D(x) , which increases sharply away from the training distribution. Bottom-right: resulting mixture weight Tψ(x) , which decreases with deviation. Bottom-left: TrustMix predictions obtained by mixing the base predictor with the fallback distribution, producing increased uncertainty under distribution shift.
Figure 2: Predictive fit on the 1 - D synthetic GP (non - stationary kernel, prior variance shrinking out - of - distribution). Base and σ - scaling extrapolate a confident trend and diverge; MC - Dropout is erratic; TrustMix reverts the mean to its fitted fallback with a flat uncertainty band; TrustMix + oracle reverts to the true prior.
10D, stationary RBF
10D, non-stationary
NLL ( ↓ )
NLL ( ↓ )
Base
18.99 [17.20, 37.15]
11.29 [5.61, 16.23]
σ -scaling
12.07 [9.76, 15.65]
11.94 [9.56, 15.83]
Dropout
2.22 [2.07, 2.59]
1.34 [1.25, 1.54]
σ -scaling + Dropout
2.45 [2.22, 2.68]
1.38 [1.28, 1.57]
Dist.-Aware σ -scaling
2.18 [2.11, 2.41]
1.72 [1.52, 1.87]
Table 1: Synthetic GP experiments on 10D space, models are given data from [−1,1]10 and tested on [−2,2]10 . Displayed values are predictive NLL ( ↓ ), median [Q1, Q3] over 10 seeds. Best method (except for oracle) in bold.
Dataset
VPower s
Weather
Metric
NLL ( ↓ )
OOD NLL ( ↓ )
NLL ( ↓ )
OOD NLL ( ↓ )
Base
-0.35 [-0.58, -0.33]
-0.28 [-0.29, -0.20]
-0.09 [-0.15, -0.07]
1.64 [0.69, 2.50]
σ -scaling
-0.37 [-0.59, -0.34]
-0.28 [-0.30, -0.11]
-0.11 [-0.16, -0.09]
2.25 [0.92, 3.44]
Dropout
0.08 [0.07, 0.10]
0.12 [0.09, 0.15]
0.42 [0.40, 0.49]
0.60 [0.52, 0.65]
σ -scaling + Dropout
-0.28 [-0.30, -0.27]
-0.36 [-0.36, -0.34]
-0.01 [-0.03, 0.02]
0.26 [0.24, 0.30]
Dist.-Aware σ -scaling
-0.37 [-0.59, -0.34]
-0.28 [-0.30, -0.12]
-0.11 [-0.16, -0.09]
2.25 [0.93, 3.45]
Table 2: Median NLL values ( ↓ ) with first and third quartiles in brackets for different methods on datasets over 10 seeds. Values in bold correspond to best method and ties w.r.t. Wilcoxon test.
Dataset
Amazon
Google Play
Trip Advisor
Yelp
Metric
NLL ( ↓ )
OOD NLL ( ↓ )
OOD NLL ( ↓ )
OOD NLL ( ↓ )
Base
0.66 [0.64, 0.69]
1.54 [1.50, 1.64]
1.57 [1.42, 1.70]
1.72 [1.59, 1.82]
σ -scaling
0.63 [0.62, 0.65]
1.42 [1.40, 1.46]
1.38 [1.31, 1.45]
1.53 [1.44, 1.56]
Dropout
0.65 [0.64, 0.66]
1.47 [1.44, 1.56]
1.54 [1.37, 1.58]
1.62 [1.50, 1.63]
σ -scaling + Dropout
0.64 [0.62, 0.65]
1.41 [1.38, 1.45]
1.40 [1.30, 1.43]
1.47 [1.41, 1.53]
Dist.-Aware σ -scaling
0.63 [0.62, 0.65]
1.40 [1.38, 1.44]
1.36 [1.29, 1.42]
1.50 [1.41, 1.53]
Table 3: Median NLL values ( ↓ ) with first and third quartiles in brackets for different methods on review score prediction over 10 seeds. Bold values are best method and ties w.r.t. Wilcoxon test. Base model here is DistilBERT.
Dataset
UTK-Face
Metric
NLL ( ↓ )
OOD NLL ( ↓ )
Base
0.39 [0.33, 0.46]
8.36 [3.94, 12.75]
Dropout
0.38 [0.32, 0.44]
7.22 [3.60, 10.13]
σ -scaling
0.36 [0.32, 0.44]
7.63 [5.26, 12.16]
σ -scaling + Dropout
0.34 [0.30, 0.41]
7.29 [4.85, 11.12]
Dist.-Aware σ -scaling
0.34 [0.31, 0.43]
2.12 [1.96, 2.40]
Table 4: Median NLL values ( ↓ ) with first and third quartiles in brackets for different methods on UTK-Face datasets over 10 seeds. Bold values are best method and ties w.r.t. Wilcoxon test. Base model here is ResNet-18.
Figure 3: Results on BO experiments. Plots show average rank (lower is better) across all 7 tested functions and all 25 seeds. The plots show how each method’s rank, based on the best value found by each iteration, evolves over time. Best method on last step and ties according to two-sided z-test are in bold.
10× ECE
NLL
Method
ID
OOD
ID
OOD
Base
0.63 ± 0.08
2.88 ± 0.61
0.80 ± 0.01
3.61 ± 0.36
DAC
0.39 ± 0.04
2.73 ± 0.31
0.77 ± 0.01
3.47 ± 0.13
HTS
0.39 ± 0.05
2.69 ± 0.35
0.76 ± 0.00
3.40 ± 0.15
IOP-IO
0.12 ± 0.02
2.40 ± 0.35
0.72 ± 0.01
3.81 ± 0.26
IROVA
0.47 ± 0.05
2.50 ± 0.40
1.21 ± 0.01
7.90 ± 0.64
Table 5: ECE ( ↓ ) and NLL ( ↓ ) on ID and OOD (corruption severity 5) datasets on CIFAR100 with the WideResNet 28-10 architecture. Reported values are means over 3 seeds, with values after ± being 1.96 times the standard error (half-width of the 95% confidence interval). Best values and ties w.r.t. two-sided z-test are in bold, second best underlined.
10× ECE
NLL
Method
ID
OOD
ID
OOD
Base
0.58 ± 0.02
0.28 ± 0.02
0.84 ± 0.01
3.25 ± 0.01
DAC
0.37 ± 0.03
0.74 ± 0.07
0.79 ± 0.01
3.24 ± 0.02
HTS
0.40 ± 0.03
1.19 ± 0.04
0.79 ± 0.01
3.28 ± 0.02
IOP-IO-Clip
0.14 ± 0.03
0.39 ± 0.14
0.73 ± 0.01
3.31 ± 0.03
IROVA
0.22 ± 0.04
1.13 ± 0.04
1.06 ± 0.05
9.60 ± 0.06
Table 6: ECE ( ↓ ) and NLL ( ↓ ) on ID and OOD (corruption severity 5) datasets on ImageNet with the ViT-B/16 architecture. Reported values are means over 3 seeds, with values after ± being 1.96 times the standard error (half-width of the 95% confidence interval). Best values and ties w.r.t. two-sided z-test are in bold, second best underlined.
Problem
Mean Spearman ρ
# positive
# significant ( p<0.05 )
CIFAR10 (WideResNet 28-10)
0.40
3/3
3/3
CIFAR100 (WideResNet 28-10)
0.32
3/3
3/3
ImageNet (ViT-B/16)
0.39
3/3
3/3
UTK
0.49
20/20
20/20
Vpower s
0.08
10/10
10/10
Weather
0.22
10/10
10/10
Table 7: Spearman rank correlation between TrustMix’s OOD deviation D(x) and NLL for the base model distribution pf(y∣x) used by TrustMix. Middle column: how many of the evaluations had a positive correlation; right column: how many were significant.
Figure 4: Plots showing the effect of removing validation datapoints on TrustMix. Blue curve shows random removal, where we randomly drop a given fraction of datapoints, whereas the red curve shows the effects of removing a fraction of points with highest OOD signal.
Figure 5: Behaviour of TrustMix on various OOD dataset. Top row: predictive NLL of the base model ( pf(y∣x) ), fallback ( p0(y∣x) ), and resulting TrustMix mixture p(y∣x) . Middle row: learned trust T(x) . Bottom row: distribution of D(x) over validation and OOD evaluation datasets. Lines and shaded areas in middle plot correspond to mean and standard errors over seeds. Top and Middle plots are made by binning the deviation signal on OOD datasets into 5 bins, plots show respective summary values over each bin.
ViT-B/16
10× ECE
NLL
Method
ID
OOD
ID
OOD
TrustMix (ours)
0.18 ± 0.04
0.21 ± 0.07
0.74 ± 0.01
3.28 ± 0.03
TrustMix (Energy)
0.18 ± 0.04
0.23 ± 0.07
0.74 ± 0.01
3.29 ± 0.03
ResNet 152
10× ECE
NLL
Table 8: ECE ( ↓ ) and NLL ( ↓ ) on ID and OOD (corruption severity 5) datasets on ImageNet with the ViT-B/16 and ResNet152 architectures, comparing TrustMix’s default Mahalanobis OOD deviation signal against an Energy signal. Reported values are means over 3 seeds, with values after ± being 1.96 times the standard error (half-width of the 95% confidence interval). Best values and ties w.r.t. two-sided z-test are in bold.
ViT-B/16
10× ECE
NLL
Method
ID
OOD
ID
OOD
TrustMix (Logit Space)
0.18 ± 0.04
0.21 ± 0.07
0.74 ± 0.01
3.28 ± 0.03
TrustMix (Prob. Space)
0.35 ± 0.02
1.04 ± 0.06
0.79 ± 0.01
3.27 ± 0.02
ResNet 152
10× ECE
NLL
Table 9: ECE ( ↓ ) and NLL ( ↓ ) on ID and OOD (corruption severity 5) datasets on ImageNet with the ViT-B/16 and ResNet152 architectures, comparing TrustMix’s logit-space fallback (default) against probability-space mixing. Reported values are means over 3 seeds, with values after ± being 1.96 times the standard error (half-width of the 95% confidence interval). Best values and ties w.r.t. two-sided z-test are in bold.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Architecture
Feature representations used
CIFAR10/100
WRN-28-10
Outputs after the three residual groups
ResNet-110
Outputs after the three residual groups
VGG16BN
Outputs after convolutional blocks 2, 3, 4, and 5,
together with the final convolutional feature map
ImageNet
ResNet-152
Outputs after residual stages 1–4
ViT-B/16
Representations after all 12 transformer encoder blocks
Appendix
Table 10: Intermediate representations used for Mahalanobis distance computations across architectures and experimental settings. For convolutional networks, we extract features after major stages or blocks of the network. For transformer-based models, we use the hidden representations after each transformer block together with the final normalization layer.
Dataset
VPower s
Weather
Metric
Coverage
OOD Coverage
Coverage
OOD Coverage
Base
0.96 ± 0.01
0.93 ± 0.02
0.97 ± 0.00
0.89 ± 0.02
σ -scaling
0.95 ± 0.00
0.91 ± 0.02
0.95 ± 0.00
0.87 ± 0.02
Dropout
0.99 ± 0.00
1.00 ± 0.00
1.00 ± 0.00
0.99 ± 0.00
σ -scaling + Dropout
0.95 ± 0.00
0.97 ± 0.00
0.95 ± 0.00
0.93 ± 0.01
Dist.-Aware σ -scaling
0.95 ± 0.00
0.91 ± 0.02
0.95 ± 0.00
0.87 ± 0.02
Appendix
Table 11: 95% predictive interval coverage and width for all methods on VPower s and Weather datasets. Displayed values are averages over 10 seeds with standard errors after ± sign. Best values and ties according to z-test in bold, second best are underlined.
Dataset
Amazon
Google Play
Trip Advisor
Yelp
Metric
Coverage
OOD Coverage
OOD Coverage
OOD Coverage
Base
0.93 ± 0.00
0.86 ± 0.02
0.77 ± 0.03
0.79 ± 0.02
σ -scaling
0.95 ± 0.00
0.90 ± 0.01
0.85 ± 0.02
0.85 ± 0.01
Dropout
0.94 ± 0.00
0.88 ± 0.01
0.80 ± 0.03
0.82 ± 0.02
σ -scaling + Dropout
0.95 ± 0.00
0.90 ± 0.01
0.84 ± 0.02
0.85 ± 0.01
Dist.-Aware σ -scaling
0.95 ± 0.00
0.91 ± 0.01
0.86 ± 0.01
0.86 ± 0.01
Appendix
Table 12: 95% predictive interval coverage and width for all methods on NLP tasks. Displayed values are averages over 10 seeds with standard errors after ± sign. Best values and ties according to z-test in bold, second best are underlined.
Dataset
UTK
Metric
ID Coverage
OOD Coverage
ID Width ( ↓ )
OOD Width ( ↓ )
Base
0.95 ± 0.01
0.60 ± 0.05
1.60 ± 0.11
2.04 ± 0.46
σ -scaling
0.95 ± 0.00
0.59 ± 0.04
1.55 ± 0.06
1.98 ± 0.45
Dropout
0.96 ± 0.01
0.61 ± 0.04
1.63 ± 0.11
2.06 ± 0.45
σ -scaling + Dropout
0.95 ± 0.00
0.59 ± 0.04
1.52 ± 0.06
1.93 ± 0.42
Dist.-Aware σ -scaling
0.95 ± 0.00
0.85 ± 0.02
1.51 ± 0.05
18.64 ± 5.83
Appendix
Table 13: 95% predictive interval coverage and width for all methods on UTK-Face. Displayed values are averages over 20 seeds with standard errors after ± sign. Best values and ties according to z-test in bold, second best are underlined.
10× ECE
NLL
Method
ID
OOD
ID
OOD
Base
0.23 ± 0.01
2.50 ± 0.20
0.16 ± 0.00
1.78 ± 0.07
DAC
0.09 ± 0.03
1.30 ± 0.26
0.13 ± 0.00
1.34 ± 0.07
HTS
0.10 ± 0.01
1.70 ± 0.22
0.14 ± 0.00
1.44 ± 0.06
IOP-IO
0.04 ± 0.02
1.44 ± 0.19
0.13 ± 0.00
1.47 ± 0.04
IROVA
0.11 ± 0.02
1.56 ± 0.23
0.20 ± 0.01
2.43 ± 0.55
Appendix
Table 14: ECE ( ↓ ) and NLL ( ↓ ) on ID and OOD (corruption severity 5) datasets on CIFAR10 with the WideResNet 28-10 architecture. Reported values are means over 3 seeds, with values after ± being 1.96 times the standard error (half-width of the 95% confidence interval). Best values and ties w.r.t. two-sided z-test are in bold, second best underlined.
10× ECE
NLL
Method
ID
OOD
ID
OOD
Base
0.32 ± 0.02
2.53 ± 0.22
0.30 ± 0.01
1.86 ± 0.11
DAC
0.09 ± 0.01
1.71 ± 0.21
0.28 ± 0.01
1.54 ± 0.06
HTS
0.09 ± 0.02
1.90 ± 0.06
0.28 ± 0.01
1.58 ± 0.03
IOP-IO
0.11 ± 0.03
1.62 ± 0.19
0.28 ± 0.01
1.61 ± 0.13
IROVA
0.13 ± 0.00
1.91 ± 0.07
0.33 ± 0.02
2.18 ± 0.18
Appendix
Table 15: ECE ( ↓ ) and NLL ( ↓ ) on ID and OOD (corruption severity 5) datasets on CIFAR10 with the VGG16BN architecture. Reported values are means over 3 seeds, with values after ± being 1.96 times the standard error (half-width of the 95% confidence interval). Best values and ties w.r.t. two-sided z-test are in bold, second best underlined.
10× ECE
NLL
Method
ID
OOD
ID
OOD
Base
1.14 ± 0.03
3.32 ± 0.13
1.23 ± 0.01
4.19 ± 0.13
DAC
0.24 ± 0.03
1.64 ± 0.24
1.12 ± 0.01
3.36 ± 0.09
HTS
0.26 ± 0.02
1.99 ± 0.15
1.12 ± 0.01
3.47 ± 0.07
IOP-IO
0.16 ± 0.09
1.90 ± 0.17
1.13 ± 0.01
3.54 ± 0.07
IROVA
0.54 ± 0.05
2.29 ± 0.24
1.61 ± 0.05
7.16 ± 0.42
Appendix
Table 16: ECE ( ↓ ) and NLL ( ↓ ) on ID and OOD (corruption severity 5) datasets on CIFAR100 with the VGG16BN architecture. Reported values are means over 3 seeds, with values after ± being 1.96 times the standard error (half-width of the 95% confidence interval). Best values and ties w.r.t. two-sided z-test are in bold, second best underlined.
10× ECE
NLL
Method
ID
OOD
ID
OOD
Base
1.32 ± 0.01
0.40 ± 0.01
0.87 ± 0.01
3.49 ± 0.01
DAC
0.36 ± 0.02
1.32 ± 0.09
0.73 ± 0.01
3.51 ± 0.01
HTS
0.40 ± 0.00
2.05 ± 0.01
0.73 ± 0.01
3.66 ± 0.03
IOP-IO-Clip
0.10 ± 0.04
0.88 ± 0.09
0.67 ± 0.01
3.55 ± 0.04
IROVA
0.19 ± 0.02
1.89 ± 0.02
0.97 ± 0.03
10.84 ± 0.09
Appendix
Table 17: ECE ( ↓ ) and NLL ( ↓ ) on ID and OOD (corruption severity 5) datasets on ImageNet with the ResNet152 architecture. Reported values are means over 3 seeds, with values after ± being 1.96 times the standard error (half-width of the 95% confidence interval). Best values and ties w.r.t. two-sided z-test are in bold, second best underlined.
Figure 6: Detailed plots showing the average best value, with confidence intervals, achieved by each algorithm at each iteration (lower is better). Where a method performs substantially differently from the others, an arrow indicates that it lies outside the y-axis limits
Figure 7: Detailed plots showing average based value with confidence intervals achieved by each algorithm by each iteration for each algorithm (lower the better). Where a method has vastly different performance to others, the arrow indicates that it lies outside of the y-axis limits.
We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift. We propose Audited Conformal Prediction (ACP), a method that leverages a small labeled dataset from the target population to train an auxiliary audit model identifying inputs where the legacy model is likely to fail. By integrating the audit model's outputs into the conformal prediction framework, ACP produces prediction sets that guarantee marginal coverage while achieving substantially higher conditional coverage in practice than existing approaches. We develop and analyze two complementary integration strategies -- one targeting marginal coverage with improved conditional performance, the other providing explicit group-conditional coverage guarantees -- and establish theoretical guarantees for both. Experiments on synthetic and real-world datasets validate the method and illustrate trade-offs between prediction set size and conditional coverage.
Yanfei Zhou, Rizal Fathony, Nam H. Nguyen +1
Department of Data Sciences and Operations University of Southern California Los Angeles, CA 90089 · AI Foundations, Capital One New York, NY 10017 · Department of Data Sciences and Operations Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90089
Distribution shift poses a significant challenge to the robustness of machine learning models, but the current solutions only aim to detect out-of-distribution (OOD) samples and predict uncertainty levels. We introduce a problem setting for failure attribution under distribution shift, which enables the models not only to detect OOD samples, but also to find out the reason for their failure. The solution we propose is called self-diagnosing models, which are capable of jointly learning predictive output, predictive uncertainty, and a failure attribution signal. In particular, we use the failure attribution vector, produced by a neural network, which provides a structured representation of predictive unreliability by distinguishing four different types of failures: covariance shift, semantic shift, noise corruption, and adversarial perturbation. In other words, we move from scalar uncertainty towards failure identification. For training the model, we introduce a consistency regularizer that encourages consistency between uncertainty and failure attribution predictions. Moreover, to be able to evaluate the model on its ability to find the reasons for failure, we construct several distribution shift benchmarks with predefined mechanisms for generating distribution shifts.
Performative prediction studies feedback loops that arise when predictive models are deployed in consequential domains. In these settings, deploying a model can change the population whose patterns the model aims to predict, inducing a distribution shift that is endogenous to the learning system. This perspective departs from classical treatments of distribution shift, where shifts are typically modeled as exogenous changes in the data-generating process. Yet, in practice, distribution shift is rarely one or the other. Predictive models may influence future data through the decisions they support, while the world itself continues to drift for reasons beyond the learner's control. We study partially performative prediction, a framework that captures both endogenous and exogenous sources of distribution shift. The framework generalizes performative prediction by allowing the data distribution to evolve both in response to the deployed model and according to an external, time-varying process. We extend the central notions of performative stability and performative optimality to this setting by defining their online analogues that track the evolving partially performative environment. We analyze practical learning heuristics, including repeated retraining, and characterize when they successfully adapt to partially performative environments.
Jaewook Lee, Tijana Zrnic
Department of Electrical Engineering, Stanford University · Departments of Statistics and Management Science & Engineering, Stanford University