Iterative Policy Refinement through Semantic Rollout Analysis

Feiyu Gavin Zhu; Qi Xu; Zhifei Deng; Zhigang Hua; Luke Simon; Jean Oh; Reid Simmons

cs.LGOct 1, 2026

Iterative Policy Refinement through Semantic Rollout Analysis

Authors: Feiyu Gavin Zhu, Qi Xu, Zhifei Deng, Zhigang Hua, Luke Simon, Jean Oh, Reid Simmons

Organizations: Carnegie Mellon University · Meta

Abstract

Structured policies improve efficiency, robustness, and interpretability in imitation learning by introducing task-specific inductive bias, but existing structure generation methods rely either on extensive human input or on static domain knowledge encoded in LLMs, which may be inconsistent with the expert demonstrations. We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts. By logging rollouts as semantically meaningful tabular data and prompting the LLM to generate diagnostic analysis code, our method identifies suboptimalities in the policy structure and iteratively corrects them without requiring human instruction. Experiments on car racing and door opening tasks show that our approach improves imitation learning performance by up to 15% over zero-shot LLM-generated structures and requires 75% less compute to achieve the same reinforcement learning performance. These results demonstrate that tabular rollout analysis provides an effective feedback signal to align LLM-generated policy structures with expert demonstrations, and we can utilize it to generate good policy structures automatically.

Figures & tables

Figure 1: Overview of policy refinement. 1. Generate the initial structure by querying an LLM with the task description. 2. Update parameters through imitation learning on the demonstrations. 3. Generate analysis code based on the desired behavior of the policy and its structure topology. 4. Collect tabular trajectories , recording observations, actions, and named latent values during policy rollouts. 5. Execute analyses and revise the structure based on the analysis results. The updated program returns to parameter fitting and another refinement cycle. 6. Reinforcement learning after the structure aligns with the demonstrations to go beyond the demonstration performance.

Panel of Figure 2 — Figure 2: A sample speed-control subgraph and its corresponding PyTorch implementation. Solid nodes distinguish observations (gray), semantic latents (blue), action outputs (green), and trainable weights (orange); dashed nodes denote operation nodes (some collapsed due to space constraints). The returned speed_error exposes the latent value for further rollout analysis.

Panel of Figure 3 — Figure 3: Example of off-track recovery structure refinement. (a) The steering structure before refinement (other nodes omitted); (b) The refined structure adds stronger steering when the race car is off-track; (c) The desired behavior of re-centering, corresponding test, and the result; (d) Recorded lateral offset and steering.

Method	Demo	LLM calls $\downarrow$	Resets $\downarrow$	Return $\uparrow$
Human demonstrations	-	-	-	$871.0\pm 12.8$
MLP + IL	Yes	0	0	$636.6\pm 258.6$
MLP + RL, low compute	Yes	0	400	$591.8\pm 330.5$
MLP + RL, high compute	Yes	0	2000	$850.2\pm 101.4$
Initial structure + IL	Yes	1	0	$766.5\pm 100.4$
Initial structure + RL, low compute	Yes	1	400	$657.0\pm 37.3$

Table 1: Car Racing performance and resource usage. Returns are mean

\pm

standard deviation over ten random seeds. Resets include refinement and training interactions, excluding final evaluation.

Figure 4: Execution feedback improves policy refinement in Car Racing. Generated analysis avoids the large intermediate drop under direct table feedback. Shading shows standard deviations.

Panel of Figure 5 — Figure 5: Door Opening progresses through three revisions. The numbers below show the mean and std of distance from handle to target across 10 seeds; lower is better.

Feedback	Input tokens (k) $\downarrow$	Output tokens (k) $\downarrow$
Trajectory table	$110.1\pm 1.6$	$17.9\pm 1.3$
Analysis results	$26.0\pm 3.4$	$37.5\pm 2.9$

Table 2: Generated analysis uses fewer input tokens and more output tokens than direct trajectory-table feedback. Values summarize six refinement rounds, with population standard deviations.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	MLES	Initial	Refined
Speed Control
speed during straights	$\checkmark$	$\checkmark$	$\checkmark$
speed at track center	$\checkmark$	$\checkmark$	$\checkmark$
slow at corners	$\checkmark$	$\checkmark$	$\checkmark$
slow at sides / large offset	$\checkmark$	$\checkmark$	$\checkmark$
slow when outside track	$\checkmark$

Appendix

Table 3: Strategy categories identified by manual inspection of Car Racing policies.

Data Analysis	Iter 1	Iter 2	Iter 3
Sanity Checks
Weights stay constant	$\checkmark$	$\checkmark$	$\checkmark$
Signs consistent with assumptions	$\checkmark$	$\checkmark$	$\checkmark$
Summary statistics	$\checkmark$
Actions
Output reconstruction	$\checkmark$		$\checkmark$

Appendix

Table 4: Generated diagnostics adapt to the policy across three refinement rounds. Checks cover computational consistency, action behavior, and task-specific control relationships.

Round	Slip feature	Fitted $w_{q}$	Diagnosis and revision
0	Mean wheel reading	$+0.1729$	Positive coefficient violates the intended sign; replace wheel mean with a relative proxy.
1	$\bar{u}-v$	$+3.8367$	Positive coefficient persists; rectify the residual.
2	$\max(\bar{u}-v,0)$	$+6.6077$	Sign anomaly persists; constrain the coefficient to be nonpositive.
3	$p/(1+\alpha p)$	$-0.00390$	Compression and $w_{q}=-\|\eta\|$ preserve the branch’s speed-reducing role after fitting.

Appendix

Table 5: Successive revisions align the slip proxy’s learned contribution with its intended speed-reducing role.

Figure 6: Recorded slip and target-speed values before refinement (left) and after three revisions (right), with shared axis limits.

Panel of Figure 7 — Figure 7: Car Racing and Door Opening: continuous control from semantic observations.

Explore similar work

When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
May 29, 2026Stephane Hatgis-Kessell, Emma BrunskillLarge Language Model Policy Optimization Frictive Policy Optimization
Department of Computer Science, Stanford University
When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL
May 27, 2026Youting Wang, Yuan Tang, Bowen Liu +2Large Language Model Reinforcement Learning Refinement
Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias
May 8, 2026Rahaf Abu Hara, Vaibbhav Murarri, Claudio ZitoFrictive Policy Optimization
Laboratory of AI and Robotics Research (LAIRR) Heriot-Watt University Dubai, United Arab Emirates

May 29, 2026, cs.LGJ/K move · Enter open · S save

When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?

Stephane Hatgis-Kessell, Emma Brunskill

Department of Computer Science, Stanford University

Large Language Model Policy Optimization Frictive Policy Optimization Offline Reinforcement Learning Prompt Engineering

Open paper AskPDF arXiv

We study when large language models (LLMs) can serve as effective black-box policy optimizers for reinforcement learning (RL) tasks, i.e., when can we replace classical RL algorithms with an LLM? We explore this question by introducing Prompted Policy Optimization (PromptPO), an iterative method that prompts an LLM with Python descriptions of the state space, action space, and reward function, then has it generate and refine executable policies based on rollout feedback. Across hard exploration environments, Meta-World robotics tasks, and several real-world control problems, PromptPO often matches or exceeds the performance of standard RL baselines while using substantially fewer environment interactions. To maximize expected return, and without further explicit prompting, the policies PromptPO outputs range from tuned proportional controllers or rule-based plans to policies that run planning algorithms like value iteration. Our results demonstrate that LLM-based policy optimization is sufficient when the LLM can leverage prior knowledge about the environment or optimization strategy. PromptPO underperforms standard RL baselines in MuJoCo domains. This demonstrates possible limitations of LLM-based policy optimization to settings that requiring fine-grained continuous control.

Abstract

Explore similar work

When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?

When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL

Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias