We study how to improve decision policies from observational data to generate better actions when actions are complex, structured objects, such as text messages or images. This task is subject to two major challenges: (i) action-level overlap is often too weak for reliable off-policy evaluation and improvement, and (ii) generating new structured actions often requires expensive training or fine-tuning (e.g., of an LLM) and can be ill-defined due to information loss in action representations. To address both, we propose PolicyLift, a three-stage framework for generative policy improvement with structured actions from observational data. In Stage 1, PolicyLift learns a small set of discrete action codes that capture high-level strategies in the observed action space and are tailored to support policy improvement under limited overlap. In Stage 2, it learns a context-dependent policy over these action codes. At deployment time, Stage 3 "lifts" the learned code-level policy back to the original action space by generating actions from the historical behavior system in a code-conditioned way. We show theoretically that improving the policy over action codes guarantees improvement of the induced policy in the original action space. We demonstrate PolicyLift across experiments in synthetic and real-world text generation tasks with consistent policy improvements over the behavior policy.
Figures & tables
Figure 2: Overview. Generative policy improvement from observational data with complex, structured actions.
Algorithm 1 PolicyLift
Figure 3: Synthetic experiments. (a) Deployment; (b) predictiveness; (c1–c2) overlap. (a,b,c1) Policy improvement (higher better); (c2) RA estimation error (lower better). 5 seeds, std errors where available. Each diagnostic fixes other stages; see Appendix D.4 .
Variant
Synthetic
Headlines
Offline Prompts
Proposed: Pred.&bal. codes + Code-RA
0.1579±0.0083
0.0482±0.0095
0.1332±0.0034
Stage 1: code replacement
Random
0.0049±0.0040
0.0400±0.0084
0.0116±0.0019
k -means
0.0015±0.0040
0.0219±0.0000
0.0902±0.0036
Predictive only
0.1547±0.0076
0.0059±0.0026
0.1498±0.0280
Stage 2: objective replacement
Table 1: PolicyLift ablations. Policy improvement ( ↑ ); cells report mean ± SE over 5 seeds. Green : proposed configuration. Red : mean ± SE intervals entirely below proposed. Gray : overlapping intervals.
Method
Policy value
Behavior policy
0.1031±0.0000
PolicyLift
0.2363±0.0034
Information-advantaged references
Prompt-based regression
0.2266±0.0109
DSO-style prompt policy
0.2684±0.0458
Table 2: OfflinePrompts. Policy value ( ↑ ), mean ± SE over five seeds. Green : PolicyLift . Red : behavior. Gray : references using additional prompt information unavailable to PolicyLift ; their mean ± SE intervals overlap its interval.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Obs. outcomes
Structured actions
Learned repr.
Overlap controlled
Deploys in orig. space
Standard OPE/OPL ( Dudík et al., 2014 ; Swaminathan and Joachims, 2015 ; Kallus, 2021 )
✓
✗
–
( ✗ )
–
Action embeddings / clusters ( Saito and Joachims, 2022 ; Saito et al., 2023 ; Saito et al., 2025 )
✓
✓
( ✓ )
( ✗ )
✗
Learned action representations ( Peng et al., 2023 ; Cief et al., 2024 )
✓
✓
✓
( ✗ )
✗
Prompt-guided generation ( Kiyohara et al., 2025 )
✓
✓
✗
( ✗ )
( ✓ )
Decoder-based offline RL ( Fujimoto et al., 2019 ; Zhou et al., 2021 )
✓
( ✓ )
✓
( ✓ )
( ✓ )
Inference-time reranking / Best-of- N ( Gao et al., 2023 ; Gui et al., 2024 ; Huang et al., 2025 )
✗
✓
✗
–
✓
Appendix
Table 3: Differences to selected related streams. We compare PolicyLift to the most relevant lines of work along the two challenges emphasized in this paper: overlap in structured action spaces and deployment of a representation-level policy as a policy over original structured actions.
Quantity
Value
Number of simulator codes and context types
K=8
Observed context dimension
K+4=12
Action block dimensions
8+8+4+64+2
Outcome noise
σy=0.25
Absolute-code block
λabs=0.8,σabs=0.15
Balanced-code block
λbal=0.6,σbal=0.25
Appendix
Table 4: Default parameters for the synthetic diagnostics.
Symbol
Meaning
X
observed context text, here an Upworthy article excerpt
A
observed action text, here a generated headline
Y
observed scalar reward
Z⋆∈Z
hidden simulator context representation
C⋆∈C⋆
hidden simulator action code
μC⋆(c∣z)
code-level logging policy
Appendix
Table 5: Core notation for the semi-synthetic benchmark.
Component
Meaning
Values
T
topic
public_affairs , science_health , entertainment , sports_human
Table 10: Predictiveness diagnostic. Default synthetic DGP at αpred=1 ; mean (standard error) over random seeds.
Code representation
Value
Policy improvement V(π)−V(μ)
RA error ∣VRA−V∣
Expected draws
Oracle true codes
0.649 (0.000)
0.044 (0.000)
0.005 (0.000)
1.10
Oracle balanced codes
0.653 (0.000)
0.048 (0.000)
0.009 (0.004)
2.16
Pred.+bal. codes
0.635 (0.008)
0.030 (0.008)
0.049 (0.008)
3.63
Pred. codes
0.633 (0.004)
0.028 (0.004)
0.047 (0.014)
3.09
K-means
0.607 (0.003)
0.002 (0.003)
0.044 (0.005)
7.96
Random
0.603 (0.002)
-0.002 (0.002)
0.052 (0.008)
8.17
Appendix
Table 11: Overlap diagnostic. Default synthetic DGP at κov=0.9 ; mean (standard error) over random seeds.
Figure 5: Sensitivity to loss parameters on the semi-synthetic dataset. β varies in the top row and γ in the bottom row. Left: policy improvement; right: expected draws.
Component
Choice
Code space
K=8
Context/action dimensions
12 and 8+8+4+64+2
Support threshold
τ=0.03
Deployment diagnostic
oracle codes and oracle code policy; vary only deployment rule
Predictiveness diagnostic
BCL and oracle Stage 2; vary code representation and αpred
Overlap diagnostic
learned RA Stage 2 with support rule; vary representation and κov
Appendix
Table 12: Implementation details for the synthetic diagnostics.
Structured policies improve efficiency, robustness, and interpretability in imitation learning by introducing task-specific inductive bias, but existing structure generation methods rely either on extensive human input or on static domain knowledge encoded in LLMs, which may be inconsistent with the expert demonstrations. We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts. By logging rollouts as semantically meaningful tabular data and prompting the LLM to generate diagnostic analysis code, our method identifies suboptimalities in the policy structure and iteratively corrects them without requiring human instruction. Experiments on car racing and door opening tasks show that our approach improves imitation learning performance by up to 15% over zero-shot LLM-generated structures and requires 75% less compute to achieve the same reinforcement learning performance. These results demonstrate that tabular rollout analysis provides an effective feedback signal to align LLM-generated policy structures with expert demonstrations, and we can utilize it to generate good policy structures automatically.
Behavior cloning with high-capacity generative policies achieves strong imitation performance, but is often limited by demonstration coverage and distribution shift. Direct reinforcement learning fine-tuning can improve performance, but updating large action decoders is frequently unstable and sample inefficient. We propose Lagrangian Perturbation Diffusion Steering (LP-DS), a lightweight adaptation method that improves a frozen generative policy by learning a compact noise-space perturbation before decoding. LP-DS optimizes this perturbation with a Lagrangian trust-region objective, improving downstream value while constraining deviation from the latent prior. Across RoboMimic manipulation, OpenAI Gym locomotion, and Adroit dexterous manipulation benchmarks, LP-DS improves sample efficiency, success, and return while maintaining higher action-space entropy than unconstrained noise-space steering, with return improvements of up to 25% over prior baselines. Additional evaluations with flow-matching backbones, a large vision-language-action model, and physical Franka deployment show that LP-DS is not limited to compact diffusion policies or simulated benchmarks. Project page: https://sites.google.com/view/lp-ds/home.
Hikmet Simsir, Ozgur S. Oguz
Department of Computer Engineering, Bilkent University, Ankara, Türkiye.
Generative robot policies trained on demonstrations using behavior cloning often learn actions that are sub-optimal or misaligned with respect to the downstream task. Policy improvement approaches aim to bridge this gap and improve the cumulative reward with minimal interventions on the pre-trained policy. However, an impediment to their practical deployment is the requirement of cumbersome hyperparameter tuning specific to model or task families. In this work, we present an episodic, derivative-free latent policy improvement approach that works with little to no changes across model scales from MLPs to VLAs. We demonstrate that the performance of a pretrained, frozen diffusion or flow matching policy can be improved with respect to a downstream reward by swapping the sampling of initial noise from the prior distribution (typically isotropic Gaussian) with a well-chosen, constant initial noise input---a golden ticket. We show the prevalence of golden tickets by improving the policy performance of 46 out of 51 tasks across manipulation benchmarks, with absolute improvements in success rate by up to 79% for simulated tasks, and 28% within 60 search episodes for real-world tasks. Further, we find that the versatility of our approach opens up new avenues such as simultaneous policy improvement for multiple downstream rewards. Project webpage: https://lottery-tickets.rai-inst.com/.
Omkar Patil, Ondrej Biza, Thomas Weng +9
Robotics and AI Institute (RAI) · Arizona State University (ASU) · Northeastern University