Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

Christophe D. Hounwanou; John Emeka Eze; Yaé U. Gaba

cs.LGAug 18, 2026

Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

Authors: Christophe D. Hounwanou, John Emeka Eze, Yaé U. Gaba

Organizations: African Institute for Mathematical Sciences, Rwanda · AI Research and Innovation Nexus for Africa (AIRINA Labs), AI.Technipreneurs, Bénin · Sefako Makgatho Health Sciences University (SMU), South Africa · African Center for Advanced Studies (ACAS), Cameroon

Abstract

Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.

Figures & tables

Panel of Figure 1 — Figure 1: Framework overview. Solid arrows are per-step dataflow: the environment emits state $s_{t}$ and its caption $\phi(s_{t})$ ; the LLM planner produces the active subgoal $g_{t}$ and a bounded potential score $\Phi(s,g)$ ; the RL policy $\pi_{\theta}$ selects action $a_{t}$ conditioned on both $s_{t}$ and $g_{t}$ ; the shaping term $F=\gamma\Phi(s^{\prime},g)-\Phi(s,g)$ is added to the environment reward. Dashed arrows are periodic: the memory drives policy updates and, on the schedule of Section 4.4 , triggers plan revisions.

$\Phi$ configuration	Base $V^{\ast}(s_{0})$	Shaped $\tilde{V}^{\ast}(s_{0})$	Shaped optimal policy	Matches base?
$\Phi\equiv 0$	$0.9000$	$0.9000$	[ $s_{0}$ : forward, $s_{1}$ : forward]	yes
$\Phi\equiv 1$	$0.9000$	$0.7100$	[ $s_{0}$ : forward, $s_{1}$ : forward]	yes
sign-flip	$0.9000$	$1.4000$	[ $s_{0}$ : forward, $s_{1}$ : forward]	yes
adversarial large	$0.9000$	$-5.0500$	[ $s_{0}$ : forward, $s_{1}$ : forward]	yes

Table 1: Numerical verification of Proposition 4.1 . All four

\Phi

configurations, including one with per-state values 10–20

\times

the base reward magnitude, preserve the base optimal policy. The shaped value

\tilde{V}^{\ast}

shifts by the potential offset

-\Phi(s_{0})

plus a vanishing telescope tail, as the theorem predicts.

Metric	Value
Parse rate (outputs with $\geq 1$ subgoal)	$100.0\%$ ( $20/20$ )
Mean plan length (parsed only)	$8.00$ subgoals
Median plan length	$7$ subgoals
Mean ground-truth coverage	$54.8\%$
Mean extraneous subgoals per plan	$5.30$
Median tokens out per plan	$37$

Table 2: Planner-in-isolation audit for Qwen-2.5:14b (locally served) on 20 MiniGrid tasks. Prompt template: P1 (Appendix A ).

Config	n	Success rate	Final return	Mean steps/ep	LLM calls/seed
PPO baseline	3	$28.1\%$ [ $4.8$ , $65.2$ ]	$0.106$ [ $0.008$ , $0.262$ ]	$340.4$ [ $317.2$ , $352.4$ ]	$0$
Hybrid (Qwen)	3	$9.3\%$ [ $0.0$ , $14.3$ ]	$0.058$ [ $0.000$ , $0.095$ ]	$348.0$ [ $340.0$ , $358.5$ ]	$30$

Table 3: Pipeline validation results on MiniGrid-DoorKey-6x6. LLM budget across all runs: 92 Qwen-2.5:14b calls, 4,199 output tokens, 1.32 hours of Ollama CPU wall clock. Neither configuration converges to high success at this budget; DoorKey-6x6 typically requires roughly

5\times 10^{5}

environment steps for PPO convergence, whereas the pilot uses

3\times 10^{4}

LLM-Guided RL

Large language model-guided reinforcement learning uses language models to shape rewards, exploration, planning, and policy learning.

See all 140 papers

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
Jul 28, 2026Yunpeng ChuLLM Alignment Reward Shaping
Stony Brook University
When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
May 29, 2026Stephane Hatgis-Kessell, Emma BrunskillLanguage Model-Based Control Policy Optimization
Department of Computer Science, Stanford University
Regime-Conditional Stabilisation of LLM-Augmented Cooperative Multi-Agent Reinforcement Learning
Jul 5, 2026Faid Keddouri, Sohaib Houhou, Aissa Boulmerka +1Reward Shaping LLM-Guided RL
National School of Artificial Intelligence (ENSIA), Sidi Abdellah Campus, Algiers, Algeria · Cosys-Grettia, Univ Gustave Eiffel, F-77454 Marne-la-Vallee, France.

Jul 28, 2026J/K move · Enter open · S save

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

Yunpeng Chu

Stony Brook University

LLM Alignment Reward Shaping Reward Design for RL RL from Human Feedback

Open paper AskPDF arXiv

Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models. This mismatch leads to sparse learning signals and suboptimal alignment. We introduce MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function

Φ(x,y;φ)

across auxiliary tasks before RLHF training. The learned shaping produces a composite reward that preserves policy optimality while providing task-specific learning signals. Our meta-objective combines task discrimination, entropy regularization, and potential-based conservation for stable convergence. We provide theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization. Experiments on LLaMA-3-8B across four benchmarks show consistent improvements over PPO, DPO, GRPO, and DAPO, achieving a 90.8% length-controlled win rate on AlpacaEval 2.0 and a score of 9.14 on MT-Bench, with 41% less training instability. MeRLa retains its benefits when combined with process-based and rubric-based enhanced rewards.

Abstract

LLM-Guided RL

Explore similar work

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?

Regime-Conditional Stabilisation of LLM-Augmented Cooperative Multi-Agent Reinforcement Learning