\toprule Final loss (ID) Final loss (OOD) Curve (ID) Curve (OOD) \cmidrule (lr)2-4 \cmidrule (lr)5-7 \cmidrule (lr)8-9 \cmidrule (l)10-11 Method MAE ↓ \downarrow ↓ ↓ \downarrow ↓ ρ ↑ \rho\uparrow ρ ↑ ↓ \downarrow ↓ ↓ \downarrow ↓ ρ ↑ \rho\uparrow ρ ↑ ↓ \downarrow ↓ ↓ \downarrow ↓ ↓ \downarrow ↓ ↓ \downarrow ↓ NCPL (paper): values reported by Zhang et al. (2026) ; NCPL (repro): our evaluation of released checkpoints. ∗ MoE uses our one-seed fine-tune; DataDecide uses the released model without adaptation. Neither has a paper-reported counterpart. TabPFN uses zero-shot variants only: frozen pretrained weights with labeled training runs as context, excluding the doc-guided tuning ladder. Settings and selection details: Appendix B.2.1 .
A separate joint-context benchmark answers 1,905 final-loss queries in 1.79 1.79 1.79 2.1 Predicting differences between training configurations
At a fixed model size and training budget, can a meta-model distinguish better training configurations from worse ones? We test this capability on optimizer settings and data mixtures, withholding the outcomes of the queried configurations.
Delphi’s 3,737 3{,}737 3 , 737 0.967 0.967 0.967 0.875 0.875 0.875 The second task predicts how training-data proportions affect evaluation loss. At fixed scale and progress, we withhold one mixture’s losses and predict them from its proportions and other mixtures’ records. Targets include domain losses and, where defined, total loss. On mixtures of five training domains from Ye et al. (2025) , TabPFN’s MAE is approximately 20 20 20 40 % 40\% 40% Figure 2: Predicting configuration differences at fixed scale and budget. (a) Delphi: each dot is the within-group rank correlation for one of all 24 held-out training-condition groups; diamonds are means. Losses are clipped at 6.0 6.0 6.0 1 1 1 2.2 Predicting the course of training
Can a meta-model forecast how a training run will unfold before observing its losses? We provide historical runs as context and query TabPFN with a configuration and training progress; successive queries reconstruct a loss trajectory. All evaluations withhold the queried run’s losses and reuse the same frozen backbone, with corpus-specific features and context.
Figure 3: One frozen backbone forecasts configuration-dependent training trajectories. (a–b) StepLaw learning-rate sweeps; (c) trajectory differences beyond fixed offsets. (d–e) Marin optimizer configurations; (f) an early-phase OOD weakness. (g) MoE trajectory and aggregate OOD MAE, compared with MoE-fine-tuned NCPL. (h) Data recipes at a held-out 750M scale; (i) recipe ablation and XGBoost comparison. Colors distinguish configurations in a–b, d–e, h; line styles distinguish predictors. Metadata-selected curves display progress > 5 % >5\% > 5% Configuration-dependent dynamics. TabPFN captures how learning-rate-dependent loss gaps narrow during StepLaw training (Figure 3 a–b). These forecasts go beyond shifting a common decay curve by a fixed offset: their error on time-varying OOD gaps is 38 % 38\% 38% Extending to expert configurations. Adding expert count and routing settings lets the same backbone forecast Marin-MoE trajectories from historical MoE runs (Figure 3 g). Across held-out OOD runs, curve MAE is approximately 70 % 70\% 70% Transferring data recipes across scales. On DataDecide, records from smaller models support trajectory forecasts for held-out 750M–1B models. TabPFN distinguishes the direction and approximate magnitude of recipe differences at a common scale (Figure 3 h), with lower error than recipe-aware XGBoost (i). Removing recipe identity increases OOD error nearly eightfold: the forecasts use recipe information rather than scale alone. All evaluated recipes occur in context; the transfer is to larger models, not unseen recipes.
2.3 Predicting outcomes beyond loss
Internal training observables. Can a meta-model predict how training evolves beyond the loss curve? We use Marin records to predict parameter and gradient norms from configuration and training progress, without observing the queried run’s weights or trajectory. Figure 4 a–b compares predicted and recorded trajectories for the same two held-out runs. TabPFN captures the rise and subsequent decline of parameter norms and the early decay of gradient norms, rather than imposing a shared decreasing curve on both targets. Peak timing and magnitude remain imperfect, and the forecasts do not recover individual gradient spikes. These predictions reuse the frozen backbone with observable-specific contexts and target processing.
Downstream performance. We next ask whether configuration–accuracy records from smaller models can forecast the benchmark scores of larger models. We supply historical checkpoint evaluations as context and query DataDecide’s held-out 750M–1B models, without observing their losses or benchmark scores. Figure 4 c shows that TabPFN distinguishes downstream performance across configurations, although it still underestimates the strongest 1B outcomes. Ten-task OLMES ( Gu et al., 2024 ) macro-accuracy MAE is 1.62 1.62 1.62 3.40 3.40 3.40 Figure 4: Predicting internal dynamics and downstream outcomes. (a–b) Recorded (solid) and predicted (dashed) norms for the same ID and OOD runs; norms sum per-tensor L2 norms, with a log axis in (b). (c) Each point is a held-out DataDecide size–recipe group; the dotted line is perfect prediction. Targets are the last available benchmark evaluations, not necessarily completed training. All panels average two TabPFN seeds. Selection rules and scoring: Appendix A.3 . 2.4 Adapting predictions and guiding experiments
Updating forecasts during training. We supply historical training records as context and query losses at future steps for a target configuration. As target-run measurements arrive, we append these configuration–step–loss records and query the remaining trajectory again, without updating backbone weights. On Delphi’s seven larger target runs, observing the first 20 % 20\% 20% 56 % 56\% 56% Choosing training configurations. Before launching a run, we query candidate configurations and select the one with the lowest predicted final loss. Historical configuration–loss pairs supply the context; candidate losses are hidden during selection and used afterward to score the recommendation against the best measured candidate. StepLaw’s learning-rate and batch-size sweeps illustrate this procedure (Figure 5 b). It also applies to Delphi’s seven-dimensional optimizer search: across 23 held-out condition groups, mean excess loss is 0.0038 0.0038 0.0038 0.0254 0.0254 0.0254 80 % 80\% 80% Using observations to guide subsequent experiments. In offline StepLaw replay, we reveal each selected candidate’s measured loss, append it to context, and use updated predictions to choose the next untested configuration. All policies start from the same two random evaluations. We compare history-only predictions, conditioning on the two initial outcomes but freezing context thereafter, and continual context updates. After five total evaluations, mean OOD best-found regret is 0.1398 % 0.1398\% 0.1398% 0.0473 % 0.0473\% 0.0473% 0.0394 % 0.0394\% 0.0394% Figure 5: Revise forecasts and guide training decisions. (a) All seven Delphi runs; diamonds mark means. (b) A StepLaw sweep; contours interpolate measured loss. (c) All 23 Delphi conditions; diamonds mark means, boxes the interquartile range. (d) Same-backbone context controls; 95% bootstrap bands over twelve conditions. Budgets include initialization; regret uses measured minima. Log axis in (a), symmetric-log regret in (c–d). Protocols: Appendix A.3 . We next ask whether learned configuration coordinates can improve this predictor.
3 TabPFN could be better
Good hyperparameters change with training scale. We use Kolmogorov–Arnold Networks (KANs; Liu et al., 2024 ) to learn scale-dependent learning-rate and batch-size references, making a configuration’s position relative to favorable choices explicit.
Learning a representation of training choices. Two small KANs learn reference values from the best-performing runs within each StepLaw training group, as functions of model size and token budget (Figure 6 ). We append log-ratios to these references to the original configuration fields. TabPFN’s backbone stays frozen, and no queried-run outcomes enter the KAN fit.
Figure 6: Learned coordinates improve prediction and selection at larger scales. (a,b) Archived seed-0 KAN edge functions, centered for display; each pair sums with an intercept to a reference value. Bottom: StepLaw results against the default, untuned TabPFN route, which is Table 2 ’s TabPFN predictor. MAE is in nats, regret in percent. TabPFN pools corpora; the KAN route uses StepLaw context. Matched-context controls and uncertainty estimates are in Appendix C.1 . Improving predictions and training decisions. Against the default TabPFN route, larger-model final-loss and curve errors decrease by 29 % 29\% 29% 16 % 16\% 16% 80 % 80\% 80% Inspecting what the KAN learns. The learning-rate reference decreases with model size and increases with token budget; batch size varies nonlinearly with token budget. These inspectable relationships are fitted, not universal training laws.
We next test the backbone against prospectively arriving outcomes.
4 A prospective forecast of the Marin Hero Run
Can the same meta-model incorporate observations from an ongoing run and forecast what follows? We test this on the Marin Hero Run, a public MoE model with approximately 535B total and 23B active parameters ( Hall, 2026 ) . On 25 August 2026, we froze a completion forecast (V0) using four smaller ladder runs and five hero observations through 3.84 % 3.84\% 3.84% Our reference is Marin’s official scaling law, which extrapolates the same four ladder runs and was archived alongside V0. On all 14 subsequent checkpoints available in the 24 September public-log check, V0’s MAE is 0.0216 0.0216 0.0216 0.0576 0.0576 0.0576 90 % 90\% 90% 0.0788 0.0788 0.0788 Figure 7: Forecast first, score as observations arrive. (a) The V0 forecast frozen on 25 August: open/filled points are supplied/subsequent observations; the blue band is TabPFN’s nominal 90 % 90\% 90% 14.61 % 14.61\% 14.61% 90 % 90\% 90% The forecast was preserved and never overwritten by later predictions. V0 preceded subsequent evaluations, but scoring rules were finalized after the first two arrived: predictions were frozen prospectively, not the entire evaluation protocol. The two predictors also use different information, since the official law does not see the prefix. This case extends a frozen backbone from offline queries to forecasts conditioned on an ongoing run; it does not isolate a benefit from ladder records beyond the prefix alone.
5 Discussion and limitations
Training questions can share a predictive backbone. Configuration fields, prediction targets, and contextual observations can change without new TabPFN weights. This extends meta-modeling beyond a fixed configuration-to-loss mapping, while retaining task-specific representations and target processing.
From reuse to reliable decisions. Three challenges remain: representing genuinely new regimes, such as unseen data recipes; recognizing unreliable extrapolation; and demonstrating better experimental decisions. Our offline selection results motivate prospective, equal-budget comparisons counting pilot runs, search, and adaptation. Lower prediction error alone does not establish compute savings.
6 Related work
Scaling laws and training dynamics. Scaling laws extend beyond model and data size to mixtures ( Ye et al., 2025 ) , sparsity ( Abnar et al., 2025 ) , and learning-rate schedules ( Tissue et al., 2024 ; Luo et al., 2025 ) . Hyperparameter transfer ( Yang et al., 2021 ; Wang and Aitchison, 2024 ) and scale-dependent tuning ( Porian et al., 2024 ) further qualify these predictions. Our numerical response functions complement, rather than replace, such structured laws.
Learned predictors of training outcomes. OmniPred trains language models as general regressors ( Song et al., 2024 ) ; NCPL specializes an LLM to training outcomes ( Zhang et al., 2026 ) . Our question is whether a frozen numerical backbone can serve changing configuration fields, targets, and contexts, not whether text-based regression is inherently incapable of doing so.
Learning curves and numerical in-context learning. Bayesian neural surrogates ( Klein et al., 2017 ) and Deep Power Laws ( Kadra et al., 2023 ) connect curve prediction to optimization. PFNs amortize inference through pretraining ( Müller et al., 2022 ) ; LC-PFN, FT-PFN, and Adam-PFN specialize it to curves and hyperparameter search ( Adriaensen et al., 2023 ; Rakotoarison et al., 2024 ; Athanasiadis et al., 2025 ) . We instead reuse TabPFN’s general tabular prior ( Hollmann et al., 2023 ; Hollmann et al., 2025 ) across training questions, without pretraining a dedicated training-curve model.
Downstream prediction. Model ladders predict through intermediate losses ( Bhagia et al., 2025 ) ; observational laws use a low-dimensional capability space ( Ruan et al., 2024 ) , and NeuNeu learns from accuracy trajectories and token-level losses ( Hu et al., 2026 ) . Predictability remains setting-dependent ( Lourie et al., 2025 ) .
Inspectable adaptation. KANs learn univariate edge functions ( Liu et al., 2024 ) . We use them to learn inspectable configuration coordinates for a frozen predictor, rather than replace its backbone or establish causal training laws.
Reproducibility statement
Experiments use public run corpora and released checkpoints of Zhang et al. (2026) . Splits, information sets, selection rules, and leakage checks for the reported results are documented in § 2 –§ 4 and Appendices A – E . Code, cached predictions, and the timestamped forecast protocol will be released upon acceptance.
AI use statement
The authors used generative AI assistants (OpenAI ChatGPT, Codex-based coding agents, and Anthropic Claude via Claude Code) to help refine research hypotheses and experimental designs; implement and refactor code; run and monitor experiments; build audit procedures; analyze results; design figures; search the literature; and draft, translate, and edit the manuscript. The research questions were posed by the authors, and all final methodological choices and interpretations of results were made by the authors; every experiment was approved by the authors and executed under their supervision. The authors inspected raw outputs and reviewed the code, citations, figures, and claims in the submission. All reported measurements come from the deterministic evaluators and intervention procedures described in the paper, not from language-model judgments. The authors reviewed all AI-assisted content and take full responsibility for the final paper.
Appendix A gives experimental details; Appendices B – D supplement Sections 2 – 4 ; Appendix E covers transfer, field-order robustness, and training responses.