Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model. Teams therefore evaluate candidates offline, on data produced by already-deployed models. This is off-policy evaluation (OPE), and it faces two distribution shifts: as a model is updated in post-training, its responses diverge from the logged ones (policy shift), and the reward definition under which it is judged changes with business requirements (reward shift). Classical OPE methods are ill-suited to this continual-deployment setting because they are defined per task and require fitting from scratch on every new logged dataset or reward definition. To address this, we propose PFN-OPE, a prior-data fitted network that amortizes OPE across a distribution of contextual-bandit tasks. We pretrain it once on tasks constructed from a pool of LLM responses scored by several reward functions, in which both shifts occur. At test-time it maps a logged dataset and one sampled target response per prompt to a value estimate in a single forward pass, with no per-task fitting. On HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies, PFN-OPE achieves 2.0 to 9.3 times lower error than the best baselines across all tested configurations in the reward-shifted settings.
Figures & tables
Figure 1: One checkpoint transfers to four rewards absent from pretraining. MSE at n=512 over the six pairs of seen LLMs, with 95% intervals. The ArmoRM aggregate is the seen reward; the other four are unseen. PFN-OPE’s MSE is 2.0 to 9.3 times lower than the best baseline’s.
Target: Llama-3.2-3B
Target: Gemma-3-1B
Behavior LLM
Qwen2.5 7B
Qwen2.5 1.5B
Llama-3.2 1B
Qwen2.5 7B
Qwen2.5 1.5B
Llama-3.2 1B
HelpSteer2
PFN-OPE (ours)
0.041 ± 0.006
0.87 ± 0.16
0.97 ± 0.23
1.54 ± 0.30
2.01 ± 0.21
2.75 ± 0.42
DM-ridge
0.130 ± 0.047
17.78 ± 1.10
8.45 ± 0.66
5.94 ± 0.56
30.35 ± 1.90
19.37 ± 1.80
DM-kernel
0.050 ± 0.055
16.57 ± 0.91
7.72 ± 0.87
4.91 ± 0.73
30.32 ± 1.70
18.98 ± 1.90
DM-MLP
1.26 ± 1.60
26.40 ± 6.60
7.99 ± 2.00
7.44 ± 3.20
39.57 ± 8.10
20.52 ± 5.40
Table 1: PFN-OPE achieves the lowest MSE in 11 of 12 behavior–target pairs when both the target LLM and reward model are absent from pretraining. MSE ( ×10−2 ) at n=512 under GRM; the small grey number is the larger half-width of the 95% interval. Bold marks the lowest MSE within each dataset and column, determined before rounding.
Figure 2: The advantage grows with the correction and survives 32 times more logged data than the baselines. Legend as in Figure 1 . (a) MSE against the standardized correction ∣Cstd∣ at n=512 : above 0.2 , every baseline’s MSE rises with the correction, PFN-OPE’s stays lower. (b, c) MSE against logged rows n , for seen LLMs under ArmoRM (b) and unseen LLMs under GRM (c).
Figure 3: Ablation of the prior. MSE of four PFNs pretrained in a 2×2 design over the reward and the policy prior.
Figure 4: The PFN fails where its prior has no tasks, and the shifted policies fill the gap. Each panel shows the pretraining tasks. Triangles are the evaluation tasks with an unseen target LLM under GRM. Left: one reward per task, direct policies. Right: affine reward combinations, full policy prior.
Reliable evaluation of large language models (LLMs) is essential for their development and deployment, yet is often costly, risky, and difficult to perform safely online. We study off-policy evaluation for LLMs, where limited human-labeled data from a behavior model are used to evaluate a newer target LLM. This setting is challenging because labels are scarce, behavior--target distribution shift is common, and response likelihoods are often unavailable for black-box LLMs. We propose the Optimal Transport-based Robust Off-Policy Evaluation (OTROPE), a likelihood-free evaluation that performs distributional correction in a semantic space via optimal transport to align labeled behavior-policy samples with unlabeled target-policy samples. OTROPE combines corrected human-labeled residuals with proxy predictors, yielding a doubly robust-style evaluation without behavior-policy modeling or density-ratio estimation. We theoretically characterize why baseline evaluators fail under LLM distribution shift, and establish consistency and convergence rates for OTROPE when either the reweighted behavior distribution or the proxy predictor converges. Experiments on synthetic and real LLM evaluation tasks show that OTROPE consistently outperforms baselines while enabling ensembles of weaker LLM evaluators to approach and sometimes surpass stronger evaluators. Code is available at https://github.com/LinerXiang/OTROPE.
Liner Xiang, Wenbo Zhang, Hengrui Cai
Department of Statistics, University of California, Irvine
LLMs are increasingly deployed in dynamic, real-world settings, where the distribution of user prompts can shift substantially over time as new tasks, prompts, and users are introduced to a deployed model. Such natural prompt distribution shift poses a major challenge to LLM reliability, particularly for specialized models designed for narrow domains or user populations. Despite attention to out-of-distribution robustness, there is very limited exploration of measuring natural prompt distribution shift in prior work, and its impact on deployed LLMs remains poorly understood. We introduce the LLM Evaluation under Natural prompt Shift (LENS) framework: a data-centric approach for quantifying natural prompt distribution shift and evaluating its effect on the performance of deployed LLMs. We perform a large-scale evaluation using 192 real-world post-deployment prompt shift settings over time, user group, and geographic axes, training a total of 81 models on 4.68M training prompts, and evaluating on 57.6k prompts. We find that even moderate shifts in user prompt behavior correspond with large performance drops (73% average loss) in deployed LLMs. This performance degradation is particularly prevalent when users from different latent groups and geographic regions interact with models and is correlated with natural prompt distribution shift over time. We systematically characterize how LLM instruction following ability degrades over time and between user groups. Our findings highlight the critical need for data-driven monitoring to ensure LLM performance remains stable across diverse and evolving user populations.
Parker Seegmiller, Sarah Masud Preum
Department of Computer Science Dartmouth College Hanover, NH, USA
Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.