Prior-Data Fitted Networks (PFNs) replace the hyperparameter refitting of Gaussian Process (GP) surrogates in Bayesian optimisation with a single forward pass, at the cost of a prior that is fixed at meta-training time. We show empirically that a PFN's predictive log-likelihood falls increasingly behind that of a refitted GP as the objective function's lengthscale grows beyond the mode of the PFN's meta-training prior, and that this deficit carries over to optimisation performance. To correct the mismatch without retraining the network, we introduce Anisotropic Trust Region (ATR) warping. Per-dimension lengthscales are estimated by kernel target alignment, and an affine rescaling of the input space maps the objective onto the lengthscale the PFN expects, with a trust region around the anchor determining which observations are retained, leaving the PFN's O(d⋅N2) inference cost unchanged. On functions drawn from a Matérn-3/2 kernel, ATR warping improves optimisation performance of the PFN at both extremes of the lengthscale range. Our implementation is publicly available at https://github.com/youngers2006/Trust-Region-Warping-For-PFNs.
Figures & tables
Figure 1: Median predictive log-likelihood of the PFN minus that of a refitted GP. Positive values (red) mean the PFN fits better, negative (blue) the GP; the colour bar is symmetric log.
Figure 2: 1D and 10D regression performance of the GP and PFN across a range of lengthscales.
Figure 3: 2D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN. The centre panel is the prior mode ( l=0.1385 ), the outer panels the tested extremes.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: 2D normalised regret curves (median and IQR over 21 repeats) for GP, PFN and random search across 15 lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 5: 5D normalised regret curves (median and IQR over 21 repeats) for GP, PFN and random search across 15 lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 6: 10D normalised regret curves (median and IQR over 21 repeats) for GP, PFN and random search across 15 lengthscales. At this budget only the GP makes clear progress, and only at the longer lengthscales, consistent with the d=10 onset near l=0.5 in Figure 2 ; at the shorter ones no method separates from random search. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 7: 2D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN across 15 lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 8: 5D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN across 15 lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 9: 10D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN across 15 lengthscales. Both methods make limited progress at this budget up until the larger lengthscales. The PFN’s prior mode is l=0.1385 , panels to the left of it are increasingly rough relative to the prior, and those to the right increasingly smooth.
Figure 10: 2D normalised regret curves (median and IQR over 21 repeats) for PFN and ATR-PFN across 7 lengthscales in [0.021,0.1] , run to a budget of 150 iterations rather than the 50 used elsewhere.
Figure 11: 2D ablation study. The centre panel is at the PFN’s prior mode ( l=0.1385 ), the left and right panels are the extremes of the tested range.
Table 12
Figure 12: KTA lengthscale estimates against the true lengthscale, taken at iteration 20 of the optimisation loop in 2D and averaged across the 2 axes (median and IQR over 21 repeats). The identity line is also plotted. Estimates are monotone but shrunk toward the centre of the range, so the warping factor undercorrects at both extremes.
Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in rare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.
Manuel Hülskamp, Julius Kobialka, Emanuel Sommer +2
We study Bayesian optimization (BO) through the lens of information geometry. Pulling back the Fisher information metric through the surrogate posterior map yields a local sensitivity tensor on the input space, which leads to an upper bound on the gradient of reparameterizable acquisition functions. This view explains vanishing-gradient behavior in high-dimensional BO and provides a common interpretation of heuristics such as RAASP and dimension-scaled lengthscales. Building on this analysis, we propose FITR, a trust-region-based BO method that replaces lengthscale-based scaling by local pullback-Fisher weights. FITR is not restricted to GP kernels with explicit lengthscales. On GP benchmarks with an SE kernel, experiments show competitive performance using FITR. The proposed method also easily generalizes to non-isotropic surrogates, although the gains are more task-dependent in that setting.
Saksham Kiroriwal, Julius Pfrommer, Jürgen Beyerer
Cognitive Industrial Systems, Fraunhofer IOSB, Karlsruhe, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
Gaussian-process Bayesian optimization (GP-BO) excels at black-box optimization of costly functions, e.g., hyperparameter optimization (HPO) and multi-agent system (MAS) design. Convergence-rate guarantees exist for select methods, notably GP upper confidence bound (GP-UCB), but require a fixed kernel. Critically, the kernel encodes how input proximity affects objective value similarity. When raw coordinates poorly match this geometry - as with log-scaled hyperparameters or localized peaks - input warping can greatly improve sample efficiency, yet known GP-UCB proofs require a fixed kernel. We propose Finite-Library Input-Warped Bayesian Optimization (FLIWBO), which selects warps from a finite library of smooth input maps by any history-dependent rule. It adapts the input geometry to accelerate learning while retaining high-probability convergence guarantees under mild hypotheses, with an explicit (Nε) library-size cost. Controlled diagnostics show that finite-library warping repairs planted geometry mismatches and identify FLIWBO failure cases. Across four repeated benchmarks - warped synthetic objectives, a confidence-fence trap, and Fashion-MNIST HPO - FLIWBO-UCB beats raw-coordinate GP-UCB under misspecified geometry, escapes traps that defeat even oracle-warp expected improvement, and recovers much of the gain from manual log scaling, while leading the tested methods that admit a matching regret guarantee. A 20-dimensional MAS design study further shows feasibility under costly noisy evaluations. Code for experiments is available: https://github.com/edvin-ketabati/bogp-paper-experiments.