Prior-Data Fitted Networks (PFNs) replace the hyperparameter refitting of Gaussian Process (GP) surrogates in Bayesian optimisation with a single forward pass, at the cost of a prior that is fixed at meta-training time. We show empirically that a PFN's predictive log-likelihood falls increasingly behind that of a refitted GP as the objective function's lengthscale grows beyond the mode of the PFN's meta-training prior, and that this deficit carries over to optimisation performance. To correct the mismatch without retraining the network, we introduce Anisotropic Trust Region (ATR) warping. Per-dimension lengthscales are estimated by kernel target alignment, and an affine rescaling of the input space maps the objective onto the lengthscale the PFN expects, with a trust region around the anchor determining which observations are retained, leaving the PFN's O(d⋅N2) inference cost unchanged. On functions drawn from a Matérn-3/2 kernel, ATR warping improves optimisation performance of the PFN at both extremes of the lengthscale range. Our implementation is publicly available at https://github.com/youngers2006/Trust-Region-Warping-For-PFNs.
Figures & tables
Figure 1: Median predictive log-likelihood of the PFN minus that of a refitted GP. Positive values (red) mean the PFN fits better, negative (blue) the GP; the colour bar is symmetric log.
Figure 2: 1D and 10D regression performance of the GP and PFN across a range of lengthscales.
Figure 3: 2D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN. The centre panel is the prior mode ( l=0.1385 ), the outer panels the tested extremes.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: 2D normalised regret curves (median and IQR over 21 repeats) for GP, PFN and random search across 15 lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 5: 5D normalised regret curves (median and IQR over 21 repeats) for GP, PFN and random search across 15 lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 6: 10D normalised regret curves (median and IQR over 21 repeats) for GP, PFN and random search across 15 lengthscales. At this budget only the GP makes clear progress, and only at the longer lengthscales, consistent with the d=10 onset near l=0.5 in Figure 2 ; at the shorter ones no method separates from random search. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 7: 2D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN across 15 lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 8: 5D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN across 15 lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 9: 10D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN across 15 lengthscales. Both methods make limited progress at this budget up until the larger lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 10: 2D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN across 7 lengthscales in [0.021,0.1] , run to a budget of 150 iterations rather than the 50 used elsewhere.
Figure 11: 2D ablation study. The centre panel is at the PFN’s prior mode ( l=0.1385 ), the left and right panels are the extremes of the tested range.
Table 12
Figure 12: KTA lengthscale estimates against the true lengthscale, taken at iteration 20 of the optimisation loop in 2D and averaged across the 2 axes (median and IQR over 21 repeats). The identity line is also plotted. Estimates are monotone but shrunk toward the centre of the range, so the warping factor undercorrects at both extremes.