HiRE: Hindsight Reward Editing for Policy Finetuning

Haoyi Niu; Zhengtao Han; Yufeng Ji; Zhongyu Li; Koushil Sreenath

cs.ROSep 22, 2026

HiRE: Hindsight Reward Editing for Policy Finetuning

Authors: Haoyi Niu, Zhengtao Han, Yufeng Ji, Zhongyu Li, Koushil Sreenath

Organizations: University of California, Berkeley · The Chinese University of Hong Kong

Abstract

Pre-trained robot policies always require finetuning to adapt to specific environments. Reinforcement Learning (RL) offers high performance potential because it improves action optimality rather than simply mimicking data. However, such potential depends heavily on reward quality. Sparse rewards lack process feedback, human-designed rewards are costly and biased, and semantic rewards from foundation representations are often not control-centric. We propose Hindsight Reward Editing (HiRE), a training-free framework to break this reward bottleneck. HiRE bridges the broad knowledge of foundation representation models with physical control awareness, by contrasting successful and failed trajectories in hindsight. It calibrates foundation representation models by identifying "trap states" that are predicted as high-rewarding states yet eventually result in failure, and vice versa. HiRE explicitly penalizes these traps while boosting rewards for critical successful states. This approach can be flexibly compatible with any foundation representations and RL algorithms. Experiments show that HiRE consistently outperforms other reward recipes by delivering dense, control-aware feedback that prevents value function collapse and reward hacking, thereby achieving superior sample efficiency, stable policy updates, and higher performance ceilings, e.g., at least 3x performance of the base policies. Qualitative results are at https://hire-project.github.io .

Figures & tables

Figure 1: Comparisons on RoboMimic and MimicGen tasks. Success rate versus online environment steps for different reward recipes integrated with DICE-RL policy learning backbone. Curves are averaged over three random seeds, with shaded regions denoting variability across seeds.

Figure 2: HiRE reflects task progress better than naïve visual similarity-based rewards. We visualize reward scores along two representative trajectories, with selected segments highlighted and shown as thumbnails above the plot. The shaded regions correspond to key behavioral phases, including target approach, target loss, target recovery, and direction correction. While DINO similarity-based potentials mainly reflect visual resemblance to the ultimate goal frame, the HiRE potential remains faithful to reflect task progress and penalizes visually plausible but failure-prone states.

Figure 3: Real-robot evaluation. Left: Representative successful HiRE rollouts for Bottle_Pick-and-Place (top) and Towers_of_Hanoi (bottom). Right: Success rates over online fine-tuning episodes. Starting from the same base policy ( $20\%$ ), HiRE reaches $67\%$ on Bottle_Pick-and-Place and $73\%$ on Towers_of_Hanoi , consistently outperforming sparse-reward and RoboMeter baselines.

Figure 4: Critic loss. We compare critic loss between HiRE and baselines during policy finetuning. HiRE maintains non-trivial critic updates, while sparse reward yields significantly lower critic loss due to its sparse signals, and Robometer probably produces inconsistent rewards that are hard for the critic to converge.

Figure 5: The negative buffer captures distinct failure behaviors during policy finetuning. We visualize positive and negative buffer samples in a shared t-SNE space across training steps. Online positive buffer samples correspond to successful rollouts whereas negative ones are the terminal frames of failure rollouts. Density quantile contours highlight high-density regions of the negative-buffer distribution, where selected failure frames are sampled as representative examples. Expert initial and goal frames are plotted as reference markers.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Value
Visual encoder	ResNet-18 + spatial-softmax, trained end-to-end
Keypoints / fusion	32 keypoints/cam, 64-D/cam $\rightarrow$ fused 128-D
Action head	1D UNet, base 128, multipliers $[1,2,2]$
Horizons	$T_{a}=8$ , $T_{o}=1$ , $T_{\mathrm{img}}=1$

Appendix

Table 1: Policy architecture.

Policy	P1	P2	P3	P4	P5	P6	Success	Rate
BC	2	0	0	2	0	2	6/30	20.0%
Online ep. 30	1	2	1	0	1	1	6/30	20.0%
Online ep. 40	4	4	2	1	1	0	12/30	40.0%
Online ep. 50	3	5	2	1	2	3	16/30	53.3%
Online ep. 60	3	5	3	3	3	3	20/30	66.7%
Online ep. 70	4	3	3	4	3	3	20/30	66.7%

Appendix

Table 2: Detailed Real-robot evaluation rollout success rates (HiRE) on Bottle_Pick-and-Place . Each policy is evaluated at six fixed initial positions (P1–P6) with five trials per position, for

30

trials in total. Each cell reports the number of successful attempts out of five, and the last two columns report the overall success count and success rate. BC denoted the pre-trained base policy; the remaining rows are checkpoints taken at increasing numbers of online finetuning episodes.

Figure 6: Six fixed initial bottle positions (P1–P6) used for real-robot evaluation. Each panel shows the base-view camera observation at the start of an episode, with the bottle placed at one of the six predefined table positions.

Figure 7: Wrist view, base view, and visualization of offline expert data.

Setting	Value
Residual actor	3-layer MLP, hidden 1 024, GELU, LayerNorm; 56-D noise
Action decoding	DDIM [ 33 ] , 10 steps
Critic	ensemble of 10; 16 candidates, max_q_min selection
Collection	100 steps/iter $\times$ 8 parallel envs
Updates per iter	20, batch 256
Actor : critic updates	1 : 2; target soft update $\tau=0.01$

Appendix

Table 3: DICE-RL finetuning hyperparameters.

Figure 8: Foundation representation ablations. We compare HiRE with frozen DINOv2 and SigLIP encoders. Curves show mean success rates, with shaded regions denoting variability across available random seeds.

Figure 9: Hyperparameter sensitivity across four simulation tasks. We compare contrastive weights $\lambda\in\{0.1,0.3,0.7,0.9\}$ and an additional configuration with negative-buffer capacity 512 ( buffer=512 ). Curves are averaged over three random seeds, with shaded regions denoting variability across seeds.

Figure 10: Reward-composition comparisons on Stack_Three and Tool_Hang . We compare full HiRE, direct addition of the HiRE potential (w/o PBRS), and direct addition of raw goal similarity (Raw similarity). Curves are averaged over three random seeds, with shaded regions denoting variability across seeds.

Figure 11: Buffer-design ablations on Stack_Three and Tool_Hang . We compare full HiRE with the positive-buffer-only variant, which removes the negative buffer and keeps only success-manifold attraction. Curves are averaged over three random seeds, with shaded regions denoting variability across seeds.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 261 papers

CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
Jul 2, 2026Hexian Ni, Tao Lu, Yinghao CaiRobotic RL Multimodal Reward Modeling
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
Mar 17, 2026Yanru Wu, Weiduo Yuan, Esteban Martinez Licon +4Long-Horizon Robotic Manipulation Robot Skill Learning
USC Physical Superintelligence Lab
Progress Reward Model for Reinforcement Learning via Large Language Models
Oct 15, 2026Xiuhui Zhang, Ning Gao, Xingyu Jiang +4Reward Design for RL LLM-Guided RL
Beihang University, 37 Xueyuan Road, Haidian District, Beijing

Jul 2, 2026J/K move · Enter open · S save

CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning

Hexian Ni, Tao Lu, Yinghao Cai

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.

Robotic RL Multimodal Reward Modeling Reward Decomposition Reward Design for RL Preference-Based RL

Open paper AskPDF arXiv

Reward design remains a central challenge in reinforcement learning (RL). Hand-crafted rewards are often difficult to specify and may lead to suboptimal policies, while learned rewards from preferences can suffer from inefficiency and unstable training. Inspired by the dual nature of human learning explored in cognitive science, we decompose rewards into two complementary components: Formal Rewards (FR), explicitly designed based on task knowledge, and Residual Rewards (RR), learned from observations to capture implicit and nuanced preferences. Based on this decomposition, we propose CoRe, a hybrid framework that integrates FR and RR with vision-language models (VLMs) feedback to achieve preference-aligned policies without human involvement. Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design and optimize FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capturing nuanced rewards that complement FR, ensuring alignment with human intent. Through the synergy of FRM and RRM, CoRe enables the automatic construction of reliable rewards that are efficient and preference-aligned. Extensive experiments demonstrate that CoRe outperforms existing approaches in terms of policy learning effectiveness and efficiency on ten robotic manipulation tasks in simulation and five real-worlds. Videos can be found on our project website: https://core-2026.github.io/

Abstract

Reward Design for RL

Explore similar work

CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

Progress Reward Model for Reinforcement Learning via Large Language Models