Modern software systems expose hundreds of configuration options whose combinations can substantially affect execution time, memory and energy usage. Measuring software performance resulting from a configuration set requires running a realistic workload. Since the configuration space grows exponentially, exhaustive benchmarking is infeasible. Prior work therefore trains models on a small set of measured configurations to predict the performance of the rest. Yet these models are specific to a particular software version, hardware platform, and workload, so changes require new measurements and retraining, and such changes are becoming more frequent with coding agents. We challenge this per-system training assumption with SNAPPERF (Software-aware, No-retraining, Adaptive Prediction), a sample-efficient tabular foundation model specialized for configurable software. SNAPPERF is pretrained once on synthetic data. Each new update only requires a few measured configurations as context to predict the rest without any per-system training or hyperparameter tuning. Across 39,866 configurations from 12 real-world systems, spanning 6 domains and 4 performance metrics, SNAPPERF reduces prediction error by 38% over the prior state of the art at the same measurement budget, and by 25% over the best general-purpose tabular foundation model. It matches the state of the art's accuracy with 54% fewer measurements on average. These results suggest that knowledge about software can be encoded in pretraining data rather than in per-system models, opening a new research direction in software-aware pretraining.
Figures & tables
Quality mode
Threads
Constant bitrate
…
Encoding time
best
1
no
…
60.8 s
good
2
yes
…
22.3 s
real-time
4
no
…
5.7 s
… 121 measured of 2,736 configurations
best
4
yes
…
?
good
1
no
…
?
TABLE I: The prediction task on the VP8 video encoder [ 12 ] : each row is a configuration and its measured performance, the encoding time of the same job. The performance of every unmeasured configuration must be predicted from a small measured sample.
Fig. 1: Overview of SNAPPerf. ① A transformer-based model is pretrained once, offline, on about 29 million synthetic configurable systems. ② For any software and any update, a few measured configurations become the context of the frozen model, which ③ predicts every unmeasured configuration with its uncertainty. Existing approaches instead train and tune a new model after every change (bottom).
Method
Median MRE
Avg. rank
Top group (of 60)
Specialized foundation model [(ours)], no per-system training
SNAPPerf
2.80
1.08
55
General-purpose foundation models, no per-system training
Nori
3.80
2.43
0 9
TabICL
4.43
2.72
0 8
TabPFN
4.13
3.15
0 4
TABLE II: Results over 12 systems, 5 budgets and 30 repetitions. Median MRE: median relative error (%). Avg. rank: mean Scott-Knott ESD rank over the 60 cases (lower is better). Top group: cases in which a method is statistically tied for best (99% confidence); several methods can share a case.
Fig. 2: Error given a growing share of the training sample, relative to DaL on the full sample. Below 1 × , the model is more accurate than DaL. Lines: median over 60 cases; band: interquartile range of SNAPPerf. In the median case, SNAPPerf matches DaL’s accuracy with 43% of the sample, and Nori, the best general-purpose model, with 76%.
Fig. 3: Error of each foundation model relative to the most accurate one in the same case, median over five budgets. Bold marks the best model per system.
Pretraining cost is a major bottleneck for research on tabular foundation models, slowing the iteration cycle for new architectures, priors, and optimization ideas. Yet the community lacks a simple way to compare and accumulate pretraining speedups. We introduce a community speedrun for nanoTabPFN: contributors modify a single-file training script and compete to reach a fixed downstream ROC AUC target on subsampled TabArena using one NVIDIA L40S GPU. The current best record reaches the target in 0.92 minutes, an 81x speedup over the 74.32 minute baseline while using 22x fewer synthetic datasets. The speedrun format provides a simple protocol for the community to add, verify, and stack pretraining improvements, with the leaderboard open to contributions. Code and records are available at https://github.com/borawhocodess/modded-nanotabpfn.
Salih Bora Ozturk, Alexander Pfefferle, Frank Hutter
University of Freiburg, Freiburg im Breisgau, Germany · ELLIS Institute Tübingen, Tübingen, Germany · Prior Labs, Freiburg im Breisgau, Germany
Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the software world model, and view current code-execution benchmarks as covering one well-studied slice of it -- control flow. In this paper, we take a step toward a broader evaluation by shifting the observable axis to execution resources: alongside test outcome and exception class, we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data to hold the test close to real-world software engineering tasks. All tested models, frontier ones included, show modest performance and brittle behaviour, suggesting a notable lack of understanding of how software is executed, as opposed to how its source code is written.
A coding agent may safely modify one repository while silently breaking downstream services, libraries, or datastores that depend on it. Exhaustively running integration tests after every agent action is impractical, so the agent must predict these failures before executing them. Existing software world models predict the agent's own observations, while static change-impact analysis only identifies where a change may propagate. We instead introduce the Software World Model (SWM), which models the broader system affected by a code change and predicts its blast set: the components that the change will break. SWM follows three stages: explore, learn, and act. Explore executes candidate changes from restored system states, prioritizing regions where observed failures contradict the dependency graph. Learn fine-tunes a language model on these execution outcomes to predict downstream breakage. Act converts sampled predictions into per-consumer break probabilities for change ranking, proactive migration, and deciding when another execution is worth its cost. On held-out synthetic systems, SWM improves blast-set F1 from 0.431 for static reachability to 0.571±0.037, more than halves ranking regret, and improves migration return at all nine evaluation checkpoints. The two methods are complementary: reachability is stronger on dependencies represented in the graph, while SWM recovers failures caused by couplings the graph misses. Experiments on held-out real libraries further show that predicting structured failure outcomes, rather than only scalar risk, is important for downstream decision quality.