cs.LGNov 18, 2026

ICICLE: In-Context Inference for Curve-Learning Extrapolation

Authors: Laurits Fredsgaard, Michael Riis Andersen, Mikkel N. Schmidt

Organizations: Department of Applied Mathematics and Computer Science, Technical University of Denmark

Abstract

Freeze-thaw Bayesian optimization allocates training budget by predicting how partially observed learning curves will continue. LC-PFN first applied prior-data fitted networks (PFNs) to learning-curve extrapolation, and FT-PFN extended this approach to configuration-aware freeze-thaw optimization with a ten-slot hyperparameter encoder. We introduce In-Context Inference for Curve-Learning Extrapolation (ICICLE), an in-context surrogate that accepts a variable number of hyperparameters through a shared set encoder and is trained on a synthetic prior covering typed hyperparameters, heavy-tailed noise, quantization, and divergence. ICICLE has 1.28 million parameters (8.7% of FT-PFN's count) and produces similar mean-regret trajectories to FT-PFN on LCBench, PD1, and TaskSet. On the 86-task Meta-Album benchmark with 33 hyperparameters, ICICLE reaches lower mean normalized regret than Quick-Tune overall and on extended, with no clear difference on mini or micro. It does so without pre-training or meta-training on Meta-Album tasks, whereas Quick-Tune is meta-trained on related Meta-Album tasks. Our release samples synthetic training tasks on the fly and provides a freeze-thaw scheduler for Ray Tune and an ONNX model for inference without JAX or PyTorch.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Meta-Frieren: Beyond Single Level Online Learning

    Nov 4, 2026Vahan Arsenyan, Arnak DalalyanMulti-Armed BanditsThompson Sampling

  2. Multi-Task Bayesian In-Context Learning

    Jun 18, 2026Qingyang Zhu, Eric Karl Oermann, Kyunghyun ChoAmortized InferenceBayesian Inference