stat.MLNov 9, 2026

Prediction with Foundation Models under Deployment Budgets

Authors: Zhaohui Xu, Xiaofei Xu, Doudou Zhou

Organizations: Department of Statistics and Data Science, National University of Singapore · School of Mathematics and Statistics, Wuhan University

Abstract

Foundation models can supply predictions for a downstream task that draw on knowledge from extensive pretraining, but they differ in task-specific accuracy and query cost. We develop a framework that improves prediction for the task under a budget on the average query cost per input, using training data to learn which models to query and how to combine their outputs with features available without querying. The first method queries the same models for every input; the second lets them vary across inputs and uses Q-learning to decide which model to query next and when to stop. For the first method, we show that several individually inaccurate outputs can jointly outperform the best single one. It attains the minimax rate when the regression function depends on the outputs only through models within the budget. Otherwise, any method within the budget can be forced to a slower rate, which the first method attains. For the second method, we show when varying the queried models across inputs strictly improves on querying the same models for every input, and bound its prediction risk and average cost in finite samples. In simulations and a multimodal laptop price application, both methods outperform existing methods.

Figures & tables

Explore similar work

Nov 6, 2026

Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

Routers typically choose a model per query by trading predicted correctness against cost. Most price each model at a fixed cost, as if every query used the same number of output tokens, but the output length of reasoning models varies by an order of magnitude across queries. When is per-query cost worth predicting? What factors drive cost? How cheaply can predictions be made? We predict both the success probability and the output length of candidate models from the prefill activations of a single small frozen LLM, the scout. With reasoning candidates, perfect per-query cost knowledge would save 29-61% of spending on eight of nine datasets, and much less with non-reasoning candidates on the same problems. On those eight datasets, our predicted costs save 9-33% at matched accuracy relative to median-length pricing, up to 69% of what perfect knowledge would save. They are significantly cheaper than each estimator we reimplemented or adapted from published work, including ZeroRouter and a MixLLM-style predictor, on a majority of datasets. Output length depends mostly on the problem. On most math and code datasets, harder problems are longer, so predicted success alone prices cost nearly as well. On knowledge and mixed-format datasets they are not, and predicted success falls well behind a probe trained on length. The probe keeps most of its saving with 20 labelled problems and transfers without labels to four of seven new benchmarks. On a mixture of all nine benchmarks, it saves 22% beyond pricing each benchmark at its own median length. Savings stop growing at about a 2B scout, so above 4B a larger scout costs more than it adds.
Jul 12, 2026

Learning to Fine-tune Foundation Models under Resource Limitations

We study the problem of optimal continual fine-tuning for a pre-trained Foundation Model deployed at a resource-limited device. At each time slot, a new batch of training data arrives, and the controller is faced with two options: either use the data to fine-tune the model and incur a compute cost, or do not fine-tune the model and discard the data. After the decision, the performance of the current model is measured in terms of an application-specific performance metric such as classification accuracy. Our objective is to learn an optimal policy that determines when to fine-tune the model on a single task (e.g., sentiment analysis), under a finite compute budget. We formulate this online decision-making problem as a constrained Markov Decision Process, where the system state captures three essential aspects: (i) model's performance, (ii) computational budget, and (iii) data distribution relevance to historic data encountered up to that point. The transition to the next state is stochastic and therefore, we propose a reinforcement learning-based method to solve this problem, namely the actor-critic algorithm. We also consider the special case where the performance of fine-tuning for a given model can be predicted or estimated prior to decision; in this case the problem becomes a Dynamic Programming one. Experiments with a large pre-trained model on a widely-used text classification dataset demonstrate that our method consistently outperforms fine-tuning approaches with the same compute budget by more than 4%4\% in terms of accuracy and achieves 97%97\% of full-parameter fine-tuning accuracy while requiring only 25%25\% of the fine-tuning steps.
Jun 24, 2026

Embedding Foundation Model Predictions in Discrete-Choice Models with Structural Guarantees

Tabular foundation models achieve strong accuracy on choice prediction tasks, but their predictions often violate the economic logic those tasks require: raising a price can increase predicted demand, implied estimates of willingness to pay are frequently negative or implausible, and unavailable alternatives receive nonzero probability. We propose TabFM-Logit, an adapter that takes a foundation model's predicted choice probabilities as a precomputed feature and embeds them inside a multinomial logit's utility. In Stage 1, we fit the multinomial logit's structural coefficients by maximum likelihood with sign constraints. In Stage 2, we freeze those coefficients and fit a small neural correction operating on the foundation model's predictions. We prove that this composition exactly preserves the multinomial logit's marginal rate of substitution, so the analytically computable value of time becomes a mathematical guarantee rather than an empirical accident, and that training the two stages jointly breaks identifiability of the structural coefficients. Counterfactual derivatives and response signs are therefore the multinomial logit's by construction, and the foundation model contributes accuracy on observed choices. On respondents and households unseen in training, the adapter gains 5.7-6.8 percentage points (pp) of test accuracy on Swissmetro, 1.5 pp on LPMC, and 3.3-3.6 pp on IoT-Wearables over the multinomial logit (4.2 pp on average), stays within 1.0 pp of the raw foundation model, maintains 100% cost monotonicity, and keeps the multinomial logit's values of time exactly. The gain is largest where the multinomial logit is weakest and smallest where it is already strong. In every case it comes with the multinomial logit's guarantees, which the raw foundation models lack.