Generative Policy Improvement from Observational Data

Jonas Schweisthal; Dennis Frauen; Stefan Feuerriegel; Emma Brunskill

cs.AIOct 16, 2026

Generative Policy Improvement from Observational Data

Authors: Jonas Schweisthal, Dennis Frauen, Stefan Feuerriegel, Emma Brunskill

Organizations: LMU Munich · Munich Center for Machine Learning (MCML) · Department of Computer Science, Stanford University

Abstract

We study how to improve decision policies from observational data to generate better actions when actions are complex, structured objects, such as text messages or images. This task is subject to two major challenges: (i) action-level overlap is often too weak for reliable off-policy evaluation and improvement, and (ii) generating new structured actions often requires expensive training or fine-tuning (e.g., of an LLM) and can be ill-defined due to information loss in action representations. To address both, we propose PolicyLift, a three-stage framework for generative policy improvement with structured actions from observational data. In Stage 1, PolicyLift learns a small set of discrete action codes that capture high-level strategies in the observed action space and are tailored to support policy improvement under limited overlap. In Stage 2, it learns a context-dependent policy over these action codes. At deployment time, Stage 3 "lifts" the learned code-level policy back to the original action space by generating actions from the historical behavior system in a code-conditioned way. We show theoretically that improving the policy over action codes guarantees improvement of the induced policy in the original action space. We demonstrate PolicyLift across experiments in synthetic and real-world text generation tasks with consistent policy improvements over the behavior policy.

Figures & tables

Figure 2: Overview. Generative policy improvement from observational data with complex, structured actions.

Panel of Figure 2 — Algorithm 1 PolicyLift

Figure 3: Synthetic experiments. (a) Deployment; (b) predictiveness; (c1–c2) overlap. (a,b,c1) Policy improvement (higher better); (c2) RA estimation error (lower better). 5 seeds, std errors where available. Each diagnostic fixes other stages; see Appendix D.4 .

Variant	Synthetic	Headlines	Offline Prompts
Proposed: Pred.&bal. codes + Code-RA	$0.1579\pm 0.0083$	$0.0482\pm 0.0095$	$0.1332\pm 0.0034$
Stage 1: code replacement
Random	$0.0049\pm 0.0040$	$0.0400\pm 0.0084$	$0.0116\pm 0.0019$
$k$ -means	$0.0015\pm 0.0040$	$0.0219\pm 0.0000$	$0.0902\pm 0.0036$
Predictive only	$0.1547\pm 0.0076$	$0.0059\pm 0.0026$	$0.1498\pm 0.0280$
Stage 2: objective replacement

Table 1: PolicyLift ablations. Policy improvement (

\uparrow

); cells report mean

\pm

SE over 5 seeds. Green : proposed configuration. Red : mean

\pm

SE intervals entirely below proposed. Gray : overlapping intervals.

Method	Policy value
Behavior policy	$0.1031\pm 0.0000$
PolicyLift	$0.2363\pm 0.0034$
Information-advantaged references
Prompt-based regression	$0.2266\pm 0.0109$
DSO-style prompt policy	$0.2684\pm 0.0458$

Table 2: OfflinePrompts. Policy value (

\uparrow

), mean

\pm

SE over five seeds. Green : PolicyLift . Red : behavior. Gray : references using additional prompt information unavailable to PolicyLift ; their mean

\pm

SE intervals overlap its interval.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	Obs. outcomes	Structured actions	Learned repr.	Overlap controlled	Deploys in orig. space
Standard OPE/OPL ( Dudík et al., 2014 ; Swaminathan and Joachims, 2015 ; Kallus, 2021 )	✓	✗	–	( ✗ )	–
Action embeddings / clusters ( Saito and Joachims, 2022 ; Saito et al., 2023 ; Saito et al., 2025 )	✓	✓	( ✓ )	( ✗ )	✗
Learned action representations ( Peng et al., 2023 ; Cief et al., 2024 )	✓	✓	✓	( ✗ )	✗
Prompt-guided generation ( Kiyohara et al., 2025 )	✓	✓	✗	( ✗ )	( ✓ )
Decoder-based offline RL ( Fujimoto et al., 2019 ; Zhou et al., 2021 )	✓	( ✓ )	✓	( ✓ )	( ✓ )
Inference-time reranking / Best-of- $N$ ( Gao et al., 2023 ; Gui et al., 2024 ; Huang et al., 2025 )	✗	✓	✗	–	✓

Appendix

Table 3: Differences to selected related streams. We compare PolicyLift to the most relevant lines of work along the two challenges emphasized in this paper: overlap in structured action spaces and deployment of a representation-level policy as a policy over original structured actions.

Quantity	Value
Number of simulator codes and context types	$K=8$
Observed context dimension	$K+4=12$
Action block dimensions	$8+8+4+64+2$
Outcome noise	$\sigma_{y}=0.25$
Absolute-code block	$\lambda_{\mathrm{abs}}=0.8,\ \sigma_{\mathrm{abs}}=0.15$
Balanced-code block	$\lambda_{\mathrm{bal}}=0.6,\ \sigma_{\mathrm{bal}}=0.25$

Appendix

Table 4: Default parameters for the synthetic diagnostics.

Symbol	Meaning
$X$	observed context text, here an Upworthy article excerpt
$A$	observed action text, here a generated headline
$Y$	observed scalar reward
$Z^{\star}\in\mathcal{Z}$	hidden simulator context representation
$C^{\star}\in\mathcal{C}^{\star}$	hidden simulator action code
$\mu_{C^{\star}}(c\mid z)$	code-level logging policy

Appendix

Table 5: Core notation for the semi-synthetic benchmark.

Component	Meaning	Values
$T$	topic	public_affairs , science_health , entertainment , sports_human
$U$	tone	informational , emotional , surprising
$N$	specificity need	low , high

Appendix

Table 6: Hidden simulator context representation

Z^{\star}=(T,U,N)

Component	Meaning	Values
$S$	headline strategy	summary , curiosity , emotion , direct
$R$	tailoring level	generic , tailored

Appendix

Table 7: Hidden simulator action code

C^{\star}=(S,R)

Topic $T$	$U=\texttt{informational}$	$U=\texttt{emotional}$	$U=\texttt{surprising}$
public_affairs	summary $>$ direct $>$ emotion $>$ curiosity	direct $>$ emotion $>$ summary $>$ curiosity	curiosity $>$ direct $>$ summary $>$ emotion
science_health	summary $>$ curiosity $>$ direct $>$ emotion	direct $>$ emotion $>$ summary $>$ curiosity	curiosity $>$ summary $>$ direct $>$ emotion
entertainment	direct $>$ curiosity $>$ emotion $>$ summary	emotion $>$ curiosity $>$ direct $>$ summary	curiosity $>$ emotion $>$ direct $>$ summary
sports_human	summary $>$ emotion $>$ direct $>$ curiosity	emotion $>$ direct $>$ summary $>$ curiosity	curiosity $>$ emotion $>$ direct $>$ summary

Appendix

Table 8: Simulator ranking of headline strategies. Entries are ordered from best to worst.

Figure 4: Complete synthetic diagnostics. Top: policy improvement (higher is better). Bottom: deployment loss, code regret, and RA estimation error, respectively (lower is better).

Deployment rule	Policy improvement $V(\pi)-V(\mu)$	Value $V(\pi)$	Deployment loss $V_{C}(\pi_{C})-V(\pi_{\kappa})$	Abs. deployment error
BCL / Ours	0.620	0.938	0.000	0.000
Marginal	-2.188	-1.870	2.808	2.808
Learned dec.	-2.170	-1.853	2.790	2.790
Uniform	-2.010	-1.693	2.630	2.630

Appendix

Table 9: Deployment diagnostic. Default synthetic DGP at

\alpha_{\mathrm{dep}}=1

; analytically computed values.

Code representation	Value	Policy improvement $V(\pi)-V(\mu)$	Code regret $V_{C^{\star}}^{\star}-V_{G}^{\star}$	Prediction error $\mathbb{E}[(Y-m_{G}(X,G))^{2}]$
Oracle true codes	1.000 (0.000)	0.620 (0.000)	0.000 (0.000)	0.000 (0.000)
Oracle balanced codes	0.698 (0.000)	0.318 (0.000)	0.302 (0.000)	0.012 (0.000)
Pred.+bal. codes	0.668 (0.036)	0.288 (0.036)	0.332 (0.036)	0.088 (0.001)
Pred. codes	0.612 (0.015)	0.232 (0.015)	0.388 (0.015)	0.084 (0.003)
K-means	0.424 (0.006)	0.044 (0.006)	0.576 (0.006)	0.148 (0.001)
Random	0.424 (0.004)	0.044 (0.004)	0.576 (0.004)	0.148 (0.001)

Appendix

Table 10: Predictiveness diagnostic. Default synthetic DGP at

\alpha_{\mathrm{pred}}=1

; mean (standard error) over random seeds.

Code representation	Value	Policy improvement $V(\pi)-V(\mu)$	RA error $\|\widehat{V}_{\mathrm{RA}}-V\|$	Expected draws
Oracle true codes	0.649 (0.000)	0.044 (0.000)	0.005 (0.000)	1.10
Oracle balanced codes	0.653 (0.000)	0.048 (0.000)	0.009 (0.004)	2.16
Pred.+bal. codes	0.635 (0.008)	0.030 (0.008)	0.049 (0.008)	3.63
Pred. codes	0.633 (0.004)	0.028 (0.004)	0.047 (0.014)	3.09
K-means	0.607 (0.003)	0.002 (0.003)	0.044 (0.005)	7.96
Random	0.603 (0.002)	-0.002 (0.002)	0.052 (0.008)	8.17

Appendix

Table 11: Overlap diagnostic. Default synthetic DGP at

\kappa_{\mathrm{ov}}=0.9

; mean (standard error) over random seeds.

Panel of Figure 5 — Figure 5: Sensitivity to loss parameters on the semi-synthetic dataset. $\beta$ varies in the top row and $\gamma$ in the bottom row. Left: policy improvement; right: expected draws.

Component	Choice
Code space	$K=8$
Context/action dimensions	$12$ and $8+8+4+64+2$
Support threshold	$\tau=0.03$
Deployment diagnostic	oracle codes and oracle code policy; vary only deployment rule
Predictiveness diagnostic	BCL and oracle Stage 2; vary code representation and $\alpha_{\mathrm{pred}}$
Overlap diagnostic	learned RA Stage 2 with support rule; vary representation and $\kappa_{\mathrm{ov}}$

Appendix

Table 12: Implementation details for the synthetic diagnostics.

Component	Choice
Data	$905$ logged actions, $2513$ generated actions, $5$ seeds
Data generator model	GPT-5.4-mini
Code space and support	$K=8$ , $\tau=0.03$
Reward	strategy-rank lookup with $(1.00,0.65,0.15,-0.15)$ and context baseline scale $0.2$
Context representation	sentence-transformers/all-MiniLM-L6-v2; append $3.0\,e_{Z^{\star}}$
Action representation	sentence-transformers/all-MiniLM-L6-v2; append noisy $C^{\star}$ , strategy, nuisance, and random blocks

Appendix

Table 13: Implementation details for the semi-synthetic headline benchmark.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Iterative Policy Refinement through Semantic Rollout Analysis
Oct 1, 2026Feiyu Gavin Zhu, Qi Xu, Zhifei Deng +4Policy Optimization LLM-Guided RL
Carnegie Mellon University · Meta
Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies
May 31, 2026Hikmet Simsir, Ozgur S. OguzRobot Policy Adaptation Reinforcement Learning
Department of Computer Engineering, Bilkent University, Ankara, Türkiye.
You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector
Mar 16, 2026Omkar Patil, Ondrej Biza, Thomas Weng +9Robot Policy Adaptation Robotic Manipulation
Robotics and AI Institute (RAI) · Arizona State University (ASU) · Northeastern University

Oct 1, 2026J/K move · Enter open · S save

Iterative Policy Refinement through Semantic Rollout Analysis

Feiyu Gavin Zhu, Qi Xu, Zhifei Deng, Zhigang Hua+3

Carnegie Mellon University · Meta

Policy Optimization LLM-Guided RL Imitation Learning

Open paper AskPDF arXiv

Structured policies improve efficiency, robustness, and interpretability in imitation learning by introducing task-specific inductive bias, but existing structure generation methods rely either on extensive human input or on static domain knowledge encoded in LLMs, which may be inconsistent with the expert demonstrations. We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts. By logging rollouts as semantically meaningful tabular data and prompting the LLM to generate diagnostic analysis code, our method identifies suboptimalities in the policy structure and iteratively corrects them without requiring human instruction. Experiments on car racing and door opening tasks show that our approach improves imitation learning performance by up to 15% over zero-shot LLM-generated structures and requires 75% less compute to achieve the same reinforcement learning performance. These results demonstrate that tabular rollout analysis provides an effective feedback signal to align LLM-generated policy structures with expert demonstrations, and we can utilize it to generate good policy structures automatically.

Abstract

Offline RL

Explore similar work

Iterative Policy Refinement through Semantic Rollout Analysis

Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies

You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector