ICICLE: In-Context Inference for Curve-Learning Extrapolation
Organizations: Department of Applied Mathematics and Computer Science, Technical University of Denmark
Abstract
Freeze-thaw Bayesian optimization allocates training budget by predicting how partially observed learning curves will continue. LC-PFN first applied prior-data fitted networks (PFNs) to learning-curve extrapolation, and FT-PFN extended this approach to configuration-aware freeze-thaw optimization with a ten-slot hyperparameter encoder. We introduce In-Context Inference for Curve-Learning Extrapolation (ICICLE), an in-context surrogate that accepts a variable number of hyperparameters through a shared set encoder and is trained on a synthetic prior covering typed hyperparameters, heavy-tailed noise, quantization, and divergence. ICICLE has 1.28 million parameters (8.7% of FT-PFN's count) and produces similar mean-regret trajectories to FT-PFN on LCBench, PD1, and TaskSet. On the 86-task Meta-Album benchmark with 33 hyperparameters, ICICLE reaches lower mean normalized regret than Quick-Tune overall and on extended, with no clear difference on mini or micro. It does so without pre-training or meta-training on Meta-Album tasks, whereas Quick-Tune is meta-trained on related Meta-Album tasks. Our release samples synthetic training tasks on the fly and provides a freeze-thaw scheduler for Ray Tune and an ONNX model for inference without JAX or PyTorch.
Figures & tables
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | FT-PFN | ICICLE prior |
|---|---|---|
| BNN depth | ||
| BNN core | Published weight scale, pre-activation and output noise, and 0.145 DropConnect | Unchanged |
| Active HP count | ; | |
| HP types | All active slots continuous | Per-task rates , , and ; each slot draws continuous, ordinal, or nominal in proportion to these rates |
| Discrete levels | Not applicable | ; . Ordinal values are evenly spaced on ; nominal values are i.i.d. |
| Observation noise | Clipped Gaussian with per-configuration | Clipped, variance-standardized Student-t with ; the same is multiplied by a per-task |
Hyperparameter Optimization
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
See all 184 papers