Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation

Jiashu Yao; Heyan Huang; Zeming Liu; Yuhang Guo

cs.CLApr 13, 2026

Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation

Authors: Jiashu Yao, Heyan Huang, Zeming Liu, Yuhang Guo

Organizations: Beijing Institute of Technology · Beihang University

Abstract

To overcome the sparse reward challenge in reinforcement learning (RL) for agents based on large language models (LLMs), we propose Mutual Information Self-Evaluation (MISE), an RL paradigm that utilizes hindsight generative self-evaluation as dense reward signals while simultaneously calibrating them against the environmental feedbacks. Empirically, MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals. Theoretically, our work provides the first formal foundation for the paradigm of generative self-rewarding. We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy. This theoretical insight then informs and justifies our calibration step, which actively aligns these rewards with the optimal policy. Extensive experiments show that MISE outperforms strong baselines, enabling open-source LLMs about 7B parameters to achieve performance comparable to GPT-4o on validation without expert supervision.

Figures & tables

Figure 1: An LLM agent ( ) is tasked with interacting with a environment ( ) to complete a task ( ), getting intermediate rewards accumulated to a final score ( ). The environmental rewards are sparse, i.e., most environmental intermediate rewards are zero, so many strategically valuable actions are not properly rewarded.

Methods	AI-Feedback	Label-Free	Dense-Reward	Reinforce	Calibrate
SFT	✗	✗	✗	✗	✗
PPO Schulman et al. (2017)	✗	✔	✗	✔	✗
RFT Yuan et al. (2023)	✗	✔	✗	✗	✗
DPO Rafailov et al. (2024)	✗	✗	✗	✔	✗
ETO Song et al. (2024)	✗	✗	✗	✔	✗
StepAgent Deng et al. (2024)	✔	✗	✔	✔	✔

Table 1: Comparison of MISE with the relevant methods. MISE leverages self-evaluation feedback rather than expert supervision to generate informative dense rewards. Meanwhile, MISE calibrates the self-evaluation to keep the dense rewards consistent with environmental rewards, avoiding overfitting to potential evaluation bias.

Figure 2: Optimizing towards hindsight self-evaluation ideally enables the agent ( $\pi_{\theta}$ in the top) to make decisions as effective as those made with full future sight ( $\pi_{P}$ in the bottom).

Figure 3: MISE framework overview. Apart from environmental sparse rewards ( $r_{E}$ ), MISE generates dense process self-evaluation rewards ( $r_{P}$ ), which are calibrated by rewards ( $r_{C}$ ) to be consistent with actual performance. MISE adapts LLM agents with dense rewards without overfitting to self-evaluation biases.

Models	Average			FTWP		ScienceWorld		WebShop
Models	FF Rate	Suc (V)	Suc (T)	Suc (V)	Suc (T)	Suc (V)	Suc (T)	Suc (V)	Suc (T)
GPT-4o	82.13	40.40	38.29	43.66	51.08	40.65	32.41	36.88	31.39
GPT-4o-mini	72.54	32.55	29.64	27.26	31.63	38.60	31.98	31.79	25.30
LLaMA3-8B	67.83	28.92	23.06	21.73	17.24	36.12	31.79	28.90	20.14
+ ReAct	68.68	28.30	23.63	18.81	17.80	37.97	33.10	28.13	19.98
+ RFT	77.47	31.29	26.10	24.75	22.32	38.64	34.02	30.48	21.97

Table 2: Main experiments on three tasks. Bolded numbers denote the best results for each base model.

Figure 4: The overall experimental procedure. Each individual steps can be included or skipped.

Models	FF Rate	Suc(Val)	Suc (Test)
GPT-4o	82.13	43.66	51.08
GPT-4o-mini	72.54	27.26	31.63
SFT	86.46	66.54	58.53
ETO	84.80	67.04	59.81
StepAgent	84.66	67.21	59.69
MISE	84.79	68.45	62.66

Table 3: Experiment results on FTWP task in the scenario with training labels. All adaptations are based on LLaMA3-8B. Additional details for SFT are listed in Appendix E .

Models	Self-Evaluation Quality		Task Completion Quality
Models	Extractability	Accuracy	FF Rate	Suc Rate (Val)	Suc Rate (Test)
GPT-4o	97.73	85.22	82.13	43.66	51.08
LLaMA3	79.55	67.28	67.83	21.73	17.24
LLaMA3 + PPO	75.00	68.13	78.90	27.87	24.48
LLaMA3 + MISE	86.36	72.33	77.57	34.71	28.90
+ w/o $r_{m}$ (vanilla GRM)	68.18	65.91	80.86	28.45	25.61

Table 4: Analytic experiments with optional corruption of self-evaluation process. Results of both self-evaluation and task completion are assessed. Note that MISE without the reward for self-evaluation

r_{C}

is equivalent to the vanilla PRM approach. Best results based on LLaMA3 models are shown in bold.

Figure 5: The task completion rates and positive self-evaluation rates in three epochs on the valid set. MISE w/o denotes MISE without self-evaluation calibration rewards.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Prompts for agents (left) and self-evaluation (right).

Figure 7: The graph (left) and illustration (right) for reward for self-evaluation ( $r_{m}$ ). $r_{m}$ punishes $p_{\text{real}}$ and $p_{c}$ for being inconsistent, while not requiring them to be perfectly identical.

Hyper-Parameters	SFT-Based Methods	PPO-Based Methods	DPO-Based Methods
LoRA rank	$64$	$64$	$64$
LoRA $\alpha$	$64$	$64$	$64$
LoRA modules	all linear	all linear	all linear
LoRA dropout	$0.0$	$0.0$	$0.0$
learning rate	$3e-6$	$1e-5$	$3e-7$
lr scheduler	linear	constant	linear

Appendix

Table 5: Hyperparameters for the experiments. SFT-based methods include SFT, RFT; PPO-based methods include PPO, MISE; DPO-based methods include online DPO, ETO, StepAgent. The asterisk symbol denotes that, we use KL coefficient of

0.1

with LLaMA3 and Gemma2. With Qwen2, we use higher coefficient of

0.2

. This adjustment was made because Qwen2, being initially less capable, tends to diverge at an early training stage with the lower KL coefficient. Increasing the KL coefficient helps to stabilize the training process for the model.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation
Jul 6, 2026Muhammad Zain Amin, Kibele Sebnem YildirimReinforcement Learning RL for Language Model Reasoning
Department of Software Engineering and IT, ´Ecole de Technologie Sup´erieure, Montreal, Canada · Centre int´egr´e de traumatologie, Hˆopital du Sacr´e-Cœur de Montr´eal, Universit´e de Montr´eal, Montreal, Canada · Desautels Faculty of Management, McGill University, Montreal, Canada.
Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents
Aug 18, 2026Christophe D. Hounwanou, John Emeka Eze, Yaé U. GabaLLM-Guided RL Reward Shaping
African Institute for Mathematical Sciences, Rwanda · AI Research and Innovation Nexus for Africa (AIRINA Labs), AI.Technipreneurs, Bénin · Sefako Makgatho Health Sciences University (SMU), South Africa +1
Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving
Sep 29, 2026Yiming Wang, Yikang Liu, Qingyuan Tian +4RL for Language Models Reinforcement Learning
School of Computer Science, Shanghai Jiao Tong University · Tencent

Jul 6, 2026J/K move · Enter open · S save

Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

Muhammad Zain Amin, Kibele Sebnem Yildirim

Department of Software Engineering and IT, ´Ecole de Technologie Sup´erieure, Montreal, Canada · Centre int´egr´e de traumatologie, Hˆopital du Sacr´e-Cœur de Montr´eal, Universit´e de Montr´eal, Montreal, Canada · Desautels Faculty of Management, McGill University, Montreal, Canada.

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Memory-Augmented RL

Open paper AskPDF arXiv

Reinforcement Learning is commonly used to train large language models using environmental feedback. In applied settings, the environment usually provides sparse or delayed feedback. This makes it difficult for the model to pinpoint which actions in its reasoning led to success or failure. So, learning effectively from these signals is hard because the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations. We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode. When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt. Unlike inference-time reflection approaches, such as Reflexion, the framework optimizes self-review with policy gradients and internalizes improvements into the base policy via selective distillation, ensuring they persist across future episodes. A cross-episode memory keeps successful self-reviews for reuse when encountering similar tasks in future episodes during training. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3-4B and OLMo-3- 7B, on GSM8K benchmark. SRRL consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement.

Abstract

Reinforcement Learning

Explore similar work

Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving