Amortized Off-Policy Evaluation for LLMs

Younwoo Choi; Leo Feng; Vincent Liu; Haanvid Lee

cs.LGOct 7, 2026

Amortized Off-Policy Evaluation for LLMs

Authors: Younwoo Choi, Leo Feng, Vincent Liu, Haanvid Lee

Organizations: RBC Borealis · University of Toronto · Vector Institute

Abstract

Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model. Teams therefore evaluate candidates offline, on data produced by already-deployed models. This is off-policy evaluation (OPE), and it faces two distribution shifts: as a model is updated in post-training, its responses diverge from the logged ones (policy shift), and the reward definition under which it is judged changes with business requirements (reward shift). Classical OPE methods are ill-suited to this continual-deployment setting because they are defined per task and require fitting from scratch on every new logged dataset or reward definition. To address this, we propose PFN-OPE, a prior-data fitted network that amortizes OPE across a distribution of contextual-bandit tasks. We pretrain it once on tasks constructed from a pool of LLM responses scored by several reward functions, in which both shifts occur. At test-time it maps a logged dataset and one sampled target response per prompt to a value estimate in a single forward pass, with no per-task fitting. On HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies, PFN-OPE achieves 2.0 to 9.3 times lower error than the best baselines across all tested configurations in the reward-shifted settings.

Figures & tables

Figure 1: One checkpoint transfers to four rewards absent from pretraining. MSE at

n=512

over the six pairs of seen LLMs, with 95% intervals. The ArmoRM aggregate is the seen reward; the other four are unseen. PFN-OPE’s MSE is

2.0

9.3

times lower than the best baseline’s.

	Target: Llama-3.2-3B			Target: Gemma-3-1B
Behavior LLM	Qwen2.5 7B	Qwen2.5 1.5B	Llama-3.2 1B	Qwen2.5 7B	Qwen2.5 1.5B	Llama-3.2 1B
HelpSteer2
PFN-OPE (ours)	0.041 $\pm$ 0.006	0.87 $\pm$ 0.16	0.97 $\pm$ 0.23	1.54 $\pm$ 0.30	2.01 $\pm$ 0.21	2.75 $\pm$ 0.42
DM-ridge	0.130 $\pm$ 0.047	17.78 $\pm$ 1.10	8.45 $\pm$ 0.66	5.94 $\pm$ 0.56	30.35 $\pm$ 1.90	19.37 $\pm$ 1.80
DM-kernel	0.050 $\pm$ 0.055	16.57 $\pm$ 0.91	7.72 $\pm$ 0.87	4.91 $\pm$ 0.73	30.32 $\pm$ 1.70	18.98 $\pm$ 1.90
DM-MLP	1.26 $\pm$ 1.60	26.40 $\pm$ 6.60	7.99 $\pm$ 2.00	7.44 $\pm$ 3.20	39.57 $\pm$ 8.10	20.52 $\pm$ 5.40

Table 1: PFN-OPE achieves the lowest MSE in 11 of 12 behavior–target pairs when both the target LLM and reward model are absent from pretraining. MSE (

\times 10^{-2}

) at

n=512

under GRM; the small grey number is the larger half-width of the 95% interval. Bold marks the lowest MSE within each dataset and column, determined before rounding.

Figure 2: The advantage grows with the correction and survives 32 times more logged data than the baselines. Legend as in Figure 1 . (a) MSE against the standardized correction

|C_{\mathrm{std}}|

n=512

: above

0.2

, every baseline’s MSE rises with the correction, PFN-OPE’s stays lower. (b, c) MSE against logged rows

n

, for seen LLMs under ArmoRM (b) and unseen LLMs under GRM (c).

Figure 3: Ablation of the prior. MSE of four PFNs pretrained in a

2\times 2

design over the reward and the policy prior.

Figure 4: The PFN fails where its prior has no tasks, and the shifted policies fill the gap. Each panel shows the pretraining tasks. Triangles are the evaluation tasks with an unseen target LLM under GRM. Left: one reward per task, direct policies. Right: affine reward combinations, full policy prior.

Explore similar work

OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models
Sep 28, 2026Liner Xiang, Wenbo Zhang, Hengrui CaiLLM Evaluation Off-Policy Evaluation
Department of Statistics, University of California, Irvine
Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance
Apr 19, 2026Parker Seegmiller, Sarah Masud PreumLLM Evaluation Instruction Following
Department of Computer Science Dartmouth College Hanover, NH, USA
Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
Apr 30, 2026Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2LLM Evaluation Language Model Generation Evaluation
SAP · Stanford University

Sep 28, 2026, cs.AIJ/K move · Enter open · S save

OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models

Liner Xiang, Wenbo Zhang, Hengrui Cai

Department of Statistics, University of California, Irvine

LLM Evaluation Off-Policy Evaluation

Open paper AskPDF arXiv

Reliable evaluation of large language models (LLMs) is essential for their development and deployment, yet is often costly, risky, and difficult to perform safely online. We study off-policy evaluation for LLMs, where limited human-labeled data from a behavior model are used to evaluate a newer target LLM. This setting is challenging because labels are scarce, behavior--target distribution shift is common, and response likelihoods are often unavailable for black-box LLMs. We propose the Optimal Transport-based Robust Off-Policy Evaluation (OTROPE), a likelihood-free evaluation that performs distributional correction in a semantic space via optimal transport to align labeled behavior-policy samples with unlabeled target-policy samples. OTROPE combines corrected human-labeled residuals with proxy predictors, yielding a doubly robust-style evaluation without behavior-policy modeling or density-ratio estimation. We theoretically characterize why baseline evaluators fail under LLM distribution shift, and establish consistency and convergence rates for OTROPE when either the reweighted behavior distribution or the proxy predictor converges. Experiments on synthetic and real LLM evaluation tasks show that OTROPE consistently outperforms baselines while enabling ensembles of weaker LLM evaluators to approach and sometimes surpass stronger evaluators. Code is available at https://github.com/LinerXiang/OTROPE.

Abstract

Explore similar work

OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models

Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading