Prior-Data Fitted Networks for In-Context Off-Policy Evaluation

Shuguang Yu; Ting Li; Fan Zhou

stat.MLOct 25, 2026

Prior-Data Fitted Networks for In-Context Off-Policy Evaluation

Authors: Shuguang Yu, Ting Li, Fan Zhou

Organizations: The Hong Kong Polytechnic University, Hong Kong, China · Shanghai University of Finance and Economics, Shanghai, China

Abstract

Off-policy evaluation (OPE) is essential for assessing decision policies before deployment when online experimentation is costly or unsafe. Accurate estimation remains difficult when behavior-policy data are limited and the target policy favors states and actions rarely observed in those data. We propose InCOPE, a prior-data fitted network that learns a shared policy-value prediction rule across synthetic OPE tasks. We construct an OPE task prior over Markov decision processes and behavior-target policy pairs, generating behavior-policy datasets and exact target-policy values for pretraining. For a new tabular decision problem, the pretrained network uses the behavior-policy dataset, target policy, and known task settings as context to predict the policy's expected return without task-specific parameter updates. Experiments across four tabular domains, including a semi-synthetic clinical setting, demonstrate InCOPE's effectiveness for policy-value estimation.

Figures & tables

Figure 1: Network architecture of InCOPE.

Method	GridWorld	Tumor	Dispatch	Clinical
	$n=4$ – $32$	$n=8$ – $48$	$n=4$ – $24$	$n=24$ – $160$
WPDIS	$1.519\pm 0.081$	$1.229\pm 0.062$	$0.566\pm 0.053$	$0.199\pm 0.008$
FQE	$0.982\pm 0.065$	$2.329\pm 0.030$	$13.248\pm 0.192$	$0.467\pm 0.008$
Tabular-MIS	$0.982\pm 0.065$	$2.293\pm 0.030$	$13.069\pm 0.199$	$0.461\pm 0.008$
MB-OPE	$0.953\pm 0.092$	$1.686\pm 0.042$	$1.368\pm 0.117$	$0.211\pm 0.017$
OPERA	$0.985\pm 0.068$	$2.279\pm 0.075$	$13.200\pm 0.200$	$0.482\pm 0.009$

Table 1: Comparison of InCOPE with existing OPE estimators across four domains.

Figure 2: OPE accuracy across trajectory counts and behavior–target policy mismatch levels.

(a) Fraction of original-reward source tasks
Setting	GridWorld	Tumor	Dispatch	Clinical
$0\%$	$1.679\pm 0.094$	$3.829\pm 0.106$	$1.718\pm 1.132$	$0.1387\pm 0.0022$
$50\%$	$0.841\pm 0.042$	$1.468\pm 0.383$	$0.336\pm 0.118$	$0.0962\pm 0.0367$
$100\%$	$\mathbf{0.764}\pm 0.040$	$\mathbf{0.660}\pm 0.191$	$\mathbf{0.318}\pm 0.070$	$\mathbf{0.0663}\pm 0.0126$
(b) Source-policy range with original rewards fixed
Narrow	$0.995\pm 0.043$	$1.380\pm 0.625$	$0.338\pm 0.083$	$0.0982\pm 0.0286$

Table 2: Effects of source rewards and source-policy range on InCOPE specialists.

(a) Retraining without log-derived information
Method	GridWorld	Tumor	Dispatch	Clinical
Full InCOPE	$\mathbf{0.785}\pm 0.045$	$\mathbf{0.680}\pm 0.190$	$\mathbf{0.323}\pm 0.069$	$\mathbf{0.0718}\pm 0.0185$
Log-free InCOPE	$0.812\pm 0.034$	$1.134\pm 0.085$	$1.054\pm 0.056$	$0.0934\pm 0.0491$
(b) Regression with complete task inputs
Full InCOPE	$\mathbf{0.755}\pm 0.040$	$\mathbf{0.659}\pm 0.187$	$\mathbf{0.316}\pm 0.073$	$\mathbf{0.0692}\pm 0.0175$
Ridge	$1.900\pm 0.268$	$2.676\pm 1.097$	$0.517\pm 0.026$	$0.1852\pm 0.0373$

Table 3: Effects of current-task observations and estimator design on OPE accuracy.

Figure 3: Comparison of InCOPE with target-policy-only predictors across trajectory counts.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Coordinates	Definition
1	$\log(1+N_{sa})/\log(1+nH)$
2–3	$(\widehat{r}_{sa},\,\widehat{\sigma}_{sa})$
4–5	$(\pi_{e}(a\mid s),\,\widehat{\pi}_{b}(a\mid s))$
6	$\log\!\left[\operatorname{clip}\!\left(\frac{\pi_{e}(a\mid s)}{\max\{\widehat{\pi}_{b}(a\mid s),10^{-4}\}};10^{-4},25\right)\right]/\log 25$
7–8	$(\rho_{0}(s),\,\gamma)$
9	$\mathcal{N}_{10^{-6}}\!\left((\pi_{e}(a\mid s)\log(1+N_{sa}))_{sa}\right)_{sa}$

Appendix

Table 4: The 25 state–action input features in

\mathbf{x}_{sa}

Regime	Proportion	Action preference	Concentration
Moderate	$0.22$	Higher scores ( $z$ )	$\mathrm{U}(0.30,0.42)$
Severe	$0.43$	Lower scores ( $-z$ )	$\mathrm{U}(0.30,0.44)$
Extreme	$0.35$	Lower scores ( $-z$ )	$\mathrm{U}(0.54,0.70)$

Appendix

Table 5: GridWorld source behavior-policy generation.

Mode	Probability	Slip $\epsilon$	Stay $\zeta$	Cost $c$	Bonus $b_{g}$	Penalty $b_{-}$
Sparse	$.31$	$[.02,.20]$	$[0,.04]$	$[0,.055]$	$[.45,1.25]$	$[.35,1.15]$
Cliff	$.25$	$[.02,.16]$	$[0,.04]$	$[.015,.065]$	$[.70,1.55]$	$[.70,1.75]$
Dense	$.20$	$[0,.18]$	$[0,.05]$	$[0,.035]$	$[.25,.95]$	$[.20,.85]$
Maze	$.24$	$[.02,.22]$	$[0,.06]$	$[0,.05]$	$[.40,1.20]$	$[.35,1.20]$

Appendix

Table 6: GridWorld test-mode probabilities and uniform parameter ranges.

Regime	Probability	Target policy	Behavior policy
Moderate	$.22$	$.90\pi_{c}+.10u$	$.55\pi_{w}+.45u$
Severe	$.43$	$.95\pi_{c}+.05u$	$.65\pi_{-}+.35u$
Extreme	$.35$	$.98\pi_{c}+.02u$	$.90\pi_{-}+.10u$

Appendix

Table 7: GridWorld test-policy mixtures.

Parameter	Range	Parameter	Range
Gompertz $r_{g}$	$[0.025,0.090]$	Capacity $K$	$[0.90,1.40]$
Exponential rate $r_{e}$	$[0.04,0.12]$	Linear cap $\ell$	$[0.015,0.060]$
Anabolic rate $\alpha_{g}$	$[0.04,0.12]$	Catabolic rate $\beta_{g}$	$[0.02,0.09]$
Growth power $p$	$[0.55,0.90]$	Chemotherapy dose $d_{c}$	$[0.35,1.00]$
Maximum effect $E_{\max}$	$[0.04,0.18]$	Half-effect dose $E_{50}$	$[0.15,0.75]$
Growth-response blend $\omega$	$[0.35,0.85]$	Radiotherapy dose $d_{r}$	$[0.50,2.50]$

Appendix

Table 8: Tumor source parameter ranges.

Mixture	Regime	Target policy $\pi_{e}$	Behavior policy $\pi_{b}$
Standard	Low	$0.75\pi^{Q}+0.25\pi^{C}$	$0.75\pi_{e}+0.25U$
	Moderate	$0.90\pi^{Q}+0.10U$	$0.55\pi^{C}+0.45U$
	Severe	$0.96\pi^{Q}+0.04U$	$0.80\pi^{C}+0.20U$
Broader	Low	$0.65\pi^{Q}+0.35\pi^{C}$	$0.60\pi_{e}+0.40U$
	Moderate	$0.82\pi^{Q}+0.18U$	$0.40\pi^{C}+0.60U$
	Severe	$0.90\pi^{Q}+0.10U$	$0.65\pi^{C}+0.35U$

Appendix

Table 9: Behavior–target policy pairs for Tumor tasks.

Parameter	Meaning	Distribution or value
$p$	Imbalance persistence	$\mathrm{U}(0.16,0.24)$
$w$	Market-pressure weight	$\mathrm{U}(0.03,0.08)$
$\kappa$	State-pressure strength	$\mathrm{U}(0.12,0.28)$
$\lambda$	Queue relaxation	$\mathrm{U}(0.44,0.70)$
$\varphi$	Fleet reallocation fraction	$\mathrm{U}(0.24,0.48)$
$A_{0}$	Unconstrained response amplitude	$\mathrm{U}(0.20,0.25)$

Appendix

Table 10: Dispatch source parameter distributions.

Parameter	Range
$(b_{+},c_{+},m_{+},k_{+})$	$([.004,.014],[.185,.300],[.24,.42],[7,12])$
$(b_{-},c_{-},m_{-},k_{-})$	$([.003,.014],[.105,.200],[.67,.84],[7,12])$
$(b_{o},c_{o},k_{o},c_{T})$	$([.002,.013],[.001,.018],[-1.3,1.3],[.28,.34])$
$(h_{0},\sigma_{h},T_{K},\lambda)$	$([.425,.485],[.010,.032],[.46,1.12],[.10,.38])$
$(q_{l},q_{h})$	$([.55,.74],[.84,.98])$
$(c_{\rho},b_{\rho},\tau,\alpha_{\rho})$	$([.28,.72],[.13,.34],[.02,.20],[.24,1.20])$

Appendix

Table 11: Clinical source dynamics parameters.

Parameter	Meaning	Range
$\kappa$	Severity-profile exponent	$[.45,3.20]$
$\beta$	Profile strength	$[.04,.48]$
$\sigma_{r}$	Log-profile noise SD	$[.015,.14]$
$c_{r}$	Terminal-reward concentration parameter	$[.82,.985]$
$b_{d},b_{i}$	Dose and interaction burdens	$[.05,.38],\ [0,.16]$

Appendix

Table 12: Clinical source reward-profile parameters.

Regime	$v_{*}$	$r_{*}$	$q_{*}$
Diffuse low	$[.28,.42]$	—	—
Diffuse ordinary	$[.60,.76]$	—	—
Concentrated ordinary	$[.58,.72]$	$[.52,.70]$	$[.12,.22]$
Concentrated high	$[.88,1.18]$	$[.50,.70]$	$[.14,.25]$
Competing ordinary	$[.52,.80]$	$[.36,.58]$	$[.10,.20]$

Appendix

Table 13: Clinical source outcome regimes.

Domain	Total queries	Queries per batch	Trajectories $n$	Horizon $H$
GridWorld	2,500	500	$4,8,12,16,24,32$	$12,18,24,32$
Tumor	1,500	300	$8,16,24,32,48$	$25$
Dispatch	1,500	300	$4,8,12,16,24$	$24,32,40$
Clinical	1,200	240	$24,48,96,160$	$5,10,20$

Appendix

Table 14: Main-comparison query counts, trajectory counts, and horizons.

Predictor	Settings
Ridge	Training-set feature standardization; penalty $\alpha\in\{0.1,1,10,100,1000\}$ ; lsqr solver.
kNN	Euclidean distance; inverse-distance weighting; $k\in\{5,25,100\}$ .
RBF-SVR	$C/s_{y}\in\{0.1,1,10\}$ , $\epsilon=0.05s_{y}$ , and kernel coefficient gamma=scale .
ExtraTrees	300 trees; minimum leaf size in $\{2,5,10,20\}$ ; $80\%$ of features considered per split.

Appendix

Table 15: Settings for target-policy-only regressors.

Method	GridWorld	Tumor	Dispatch	Clinical
Full InCOPE	$0.782\pm 0.046$	$0.686\pm 0.192$	$0.317\pm 0.068$	$0.0693\pm 0.0156$
Prior mean	$1.093\pm 0.007$	$3.757\pm 0.026$	$1.897\pm 0.005$	$0.1091\pm 0.0005$
Ridge	$1.105\pm 0.046$	$1.800\pm 1.117$	$1.897\pm 0.302$	$0.2137\pm 0.0456$
kNN	$1.128\pm 0.045$	$2.448\pm 0.791$	$1.503\pm 0.034$	$0.1143\pm 0.0044$
RBF-SVR	$1.016\pm 0.013$	$1.379\pm 0.473$	$1.713\pm 0.039$	$0.1119\pm 0.0035$
ExtraTrees	$0.995\pm 0.029$	$1.388\pm 0.326$	$1.680\pm 0.020$	$0.1126\pm 0.0026$

Appendix

Table 16: Comparison of InCOPE with target-policy-only predictors over all test queries.

Off-Policy Evaluation

Off-policy evaluation estimates a policy’s value from data collected by another policy, using weighting, models, and uncertainty-aware methods.

See all 74 papers

Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
Jul 24, 2026Yuta Natsubori, Masataka Ushiku, Yuta SaitoOff-Policy Evaluation Contextual Bandits
Hakuhodo DY Holdings, Inc. · Cornell University
Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling
Oct 6, 2026Nicolò Felicioni, Michael Benigni, Maurizio Ferrari Dacrema +1Causal Effect Estimation Off-Policy Evaluation
Spotify · Politecnico di Milano
Logging Policy Design for Off-Policy Evaluation
May 14, 2026Connor Douglas, Joel Persson, Foster ProvostOff-Policy Evaluation Policy Optimization
New York University · Spotify

Jul 24, 2026J/K move · Enter open · S save

Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits

Yuta Natsubori, Masataka Ushiku, Yuta Saito

Hakuhodo DY Holdings, Inc. · Cornell University

Off-Policy Evaluation Contextual Bandits Off-Policy RL Cross-Dataset Generalization

Open paper AskPDF arXiv

Off-Policy Evaluation and Learning (OPE/L) in contextual bandits is rapidly gaining popularity in real systems because new policies can be evaluated and learned securely using only historical logged data. However, existing methods in OPE/L cannot handle many challenging but prevalent scenarios such as few-shot data, deterministic logging policies, and new actions. In many applications, such as personalized medicine, content recommendations, education, and advertising, we need to evaluate and learn new policies in the presence of these challenges. Existing methods cannot evaluate and optimize effectively in these situations due to the notorious variance issue or limited exploration in the logged data. To enable OPE/L even under these unsolved challenges, we propose a new problem setup of Cross-Domain OPE/L, where we have access not only to the logged data from the target domain in which the new policy will be implemented but also to logged datasets collected from other domains. This novel formulation is widely applicable because we can often use historical data not only from the target hospital, country, device, or user segment but also from other hospitals, countries, devices, or segments. We develop a new estimator and policy gradient method to solve OPE/L by leveraging both target and source datasets, resulting in substantially enhanced OPE/L in the previously unsolved situations in our empirical evaluations.

Abstract

Off-Policy Evaluation

Explore similar work

Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits

Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling

Logging Policy Design for Off-Policy Evaluation