MVR: Multi-view Video Reward Shaping for Reinforcement Learning

Lirui Luo; Guoxi Zhang; Hongming Xu; Yaodong Yang; Cong Fang; Qing Li

cs.CVMar 2, 2026

MVR: Multi-view Video Reward Shaping for Reinforcement Learning

Authors: Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, Qing Li

Organizations: School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI

Abstract

Reward design is of great importance for solving complex tasks with reinforcement learning. Recent studies have explored using image-text similarity produced by vision-language models (VLMs) to augment rewards of a task with visual feedback. A common practice linearly adds VLM scores to task or success rewards without explicit shaping, potentially altering the optimal policy. Moreover, such approaches, often relying on single static images, struggle with tasks whose desired behavior involves complex, dynamic motions spanning multiple visually different states. Furthermore, single viewpoints can occlude critical aspects of an agent's behavior. To address these issues, this paper presents Multi-View Video Reward Shaping (MVR), a framework that models the relevance of states regarding the target task using videos captured from multiple viewpoints. MVR leverages video-text similarity from a frozen pre-trained VLM to learn a state relevance function that mitigates the bias towards specific static poses inherent in image-based methods. Additionally, we introduce a state-dependent reward shaping formulation that integrates task-specific rewards and VLM-based guidance, automatically reducing the influence of VLM guidance once the desired motion pattern is achieved. We confirm the efficacy of the proposed framework with extensive experiments on challenging humanoid locomotion tasks from HumanoidBench and manipulation tasks from MetaWorld, verifying the design choices through ablation studies.

Figures & tables

Figure 1: The proposed MVR computes visual guidance using a VLM and videos collected from multiple viewpoints. In this example, the task requires a humanoid robot to run forward. Being captured from different viewpoints, the image sequences encode complementary information and enable comprehensive evaluation of the agent’s behaviors. This example also illustrates the pitfall of using image-text similarity for dynamic tasks—running requires rhythmic alternation of legs, but optimizing image-text similarity leads to realizing the pose that best matches “running” repeatedly. The shaping term prioritizes states that establish alternating leg cadence; once a stable gait and the target forward speed are achieved, its influence automatically decreases and the task reward takes precedence (see Sec. 4 ).

Figure 2: The entire framework of the proposed MVR. MVR periodically samples state sequences and renders them into videos from different viewpoints (step 2). It then queries a VLM for the similarity scores and video embeddings of the videos and augments its dataset $\mathcal{D}$ with state sequences, video embeddings, and similarity scores (step 3). Additionally, it keeps the state sequences with top- $k$ similarity scores in a reference set $\mathcal{D}^{\text{ref}}$ . With $\mathcal{D}$ , MVR updates the state relevance model (step 4). Lastly, using the latest state relevance model and the reference set, MVR computes visual feedback $r^{\text{VLM}}$ for the online RL agent (step 1), which is combined with task rewards $r^{\text{task}}$ through state-dependent reward shaping that automatically decays as the agent’s behavior aligns with the reference set.

Task	MVR	TQC	VLM-RM	RoboCLIP	DreamerV3	Success / Max Return
Walk	$\mathbf{927.47\pm 1.83}$ ✓	$510.58\pm 299.16$	$535.35\pm 355.18$	$737.34\pm 194.62$ ✓	$800.2\pm 158.7$ ✓	700 / 1000
Run	$\mathbf{749.23\pm 56.82}$ ✓	$647.87\pm 186.98$	$14.93\pm 1.11$	$501.15\pm 179.71$	$633.8\pm 222.4$	700 / 1000
Stair	$208.60\pm 166.22$	$\mathbf{282.95\pm 120.54}$	$44.96\pm 4.12$	$211.33\pm 56.25$	$131.1\pm 43.6$	700 / 1000
Slide	$\mathbf{735.03\pm 142.85}$ ✓	$514.91\pm 106.36$	$163.13\pm 41.22$	$494.20\pm 21.66$	$436.5\pm 200.1$	700 / 1000
Stand	$\mathbf{918.55\pm 29.30}$ ✓	$576.59\pm 371.0$	$728.69\pm 102.19$	$849.73\pm 108.86$ ✓	$622.7\pm 404.8$	800 / 1000
Sit_Simple	$861.07\pm 19.35$ ✓	$822.07\pm 101.28$ ✓	$293.72\pm 65.46$	$553.90\pm 244.51$	$\mathbf{891.4\pm 38.4}$ ✓	750 / 1000

Table 1: MVR outperforms VLM-RM and RoboCLIP for HumanoidBench tasks. The first four tasks are dynamic tasks and the last five are static tasks. We compare MVR with TQC ( Kuznetsov et al., 2020 ) , DreamerV3 ( Hafner et al., 2023 ) , VLM-RM ( Rocamonde et al., 2024 ) , and RoboCLIP ( Sontakke et al., 2024 ) . The DreamerV3 originate from the TD-MPC2 benchmark suite ( Hansen et al., 2024 ) . MVR performs the best for five tasks and has the best rank averaged over all tasks. VLM-RM performs the best for two tasks. RoboCLIP cannot perform well for these tasks. 3 3 3 Success thresholds follow the official HumanoidBench specification ( Sferrazza et al., 2024 ) ; the maximum returns correspond to the theoretical upper bounds defined in the benchmark.

Task	MVR	TQC	VLM-RM	RoboCLIP	DreamerV3
hammer	$0.20\pm 0.24$	$0.20\pm 0.40$	$\mathbf{0.40\pm 0.49}$	$0.00\pm 0.00$	$0.10\pm 0.17$
push-wall	$\mathbf{0.60\pm 0.49}$	$0.10\pm 0.20$	$0.40\pm 0.37$	$\mathbf{0.60\pm 0.37}$	$0.00\pm 0.00$
faucet-close	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$0.13\pm 0.15$
push-back	$0.50\pm 0.45$	$\mathbf{0.60\pm 0.37}$	$0.30\pm 0.40$	$0.20\pm 0.40$	$0.37\pm 0.38$
stick-pull	$\mathbf{0.20\pm 0.24}$	$0.10\pm 0.20$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$
handle-press-side	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$0.77\pm 0.17$

Table 2: MVR achieves higher success rate on Metaworld tasks. We use the results for DreamerV3 reported by TD-MPC2 benchmark suite ( Hansen et al., 2024 ) . The three methods that incorporate visual guidance, MVR, VLM-RM, and RoboCLIP, all outperform TQC and DreamerV3. Within them, MVR achieves the highest success rate, which is followed by RoboCLIP. These results support our claim for the importance of using visual feedback and derive the feedback from multi-view videos.

Panel of Figure 3 — Figure 3: Method ablation and the influence of the number of views.

Task	$0^{\circ}$	$90^{\circ}$	$180^{\circ}$	$270^{\circ}$	MVR
Stand	$914.17\pm 32.54$	$929.12\pm 10.58$	$912.20\pm 26.66$	$\mathbf{931.10\pm 32.08}$	$918.55\pm 29.30$
Run	$611.43\pm 294.44$	$651.23\pm 164.96$	$\mathbf{875.33\pm 6.44}$	$567.91\pm 159.47$	$749.23\pm 56.82$
Slide	$559.59\pm 0.00$	$639.28\pm 131.48$	$647.18\pm 154.73$	$657.64\pm 205.81$	$\mathbf{735.03\pm 142.85}$
Sit_Hard	$528.14\pm 176.60$	$659.10\pm 163.77$	$452.15\pm 210.89$	$723.72\pm 112.11$	$\mathbf{756.67\pm 108.79}$

Table 3: MVR can aggregate information from multiple views. This table compares the task performance of using four different viewpoints only and random viewpoints. While the best viewpoint is task-dependent, using random viewpoints (MVR) has the best performance for Slide and Sit_Hard and second-best performance for Run, indicating that MVR can aggregate multi-view information.

Figure 4: MVR identifies suboptimal states through reward shaping. We visualize states generated by a TQC agent for the Sit_Hard task and annotate them with $r^{\text{task}}$ (left) and the shaped reward $r^{\text{MVR}}=r^{\text{task}}+wr^{\text{VLM}}$ (right) computed by a trained MVR agent. While task rewards are high when the agent is close to the chair, MVR’s visual guidance component assigns low values to improper sitting poses (sitting on chair’s leg, leaning, or sitting at the edge), effectively shaping the reward landscape to discourage these visually suboptimal but task-rewarding states.

Task	ViCLIP-L (428M)	ViCLIP-B (150M)
Stand	$\mathbf{918.55}\pm\mathbf{29.30}$	$917.97\pm\mathbf{30.71}$
Run	$\mathbf{749.23}\pm\mathbf{56.82}$	$517.25\pm\mathbf{383.12}$
Slide	$735.03\pm\mathbf{142.85}$	$\mathbf{778.84}\pm\mathbf{24.80}$
Sit_Hard	$\mathbf{756.67}\pm\mathbf{108.79}$	$689.50\pm\mathbf{234.37}$

Table 4: MVR with small and large VLMs.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt	Description
Stand	a humanoid robot standing
Walk	a humanoid robot walking
Run	a humanoid robot running
Stair	a humanoid robot walking
Slide	a humanoid robot walking
Sit_Simple	a humanoid robot sitting on a chair

Appendix

Table A1: Text instructions for humanoid robot tasks.

Prompt	Description
hammer	a robot gripper hammering a screw on the wall
push-wall	a robot gripper bypassing a wall and pushing a cylinder
faucet-close	a robot gripper rotating the faucet clockwise
push-back	a robot gripper pushing the wooden block backward
stick-pull	a robot gripper pulling a stick
handle-press-side	a robot gripper pressing a handle down sideways

Appendix

Table A2: Text instructions for MetaWorld robot tasks.

Category	Hyperparameter	Value
TQC Network Architecture
	Policy Layers	[256, 256, 256]
	Critic Layers	[256, 256, 256]
	Quantile Count	50
Reward Model
	Backbone	[512,512]

Appendix

Table A3: Hyperparameters of MVR

Panel of Figure 12 — Figure A1: Top-/bottom-ranked frames by $r^{\text{task}}$ and $r^{\text{MVR}}$ in the Sit_Hard task. The first two rows show the top and bottom frames under the task reward $r^{\text{task}}$ , while the last two rows show the corresponding frames under the shaped reward $r^{\text{MVR}}$ . For both rewards, the highest-ranked frames coincide with visually unambiguous successful sitting poses, confirming that MVR preserves the optimal behavior emphasized by the task reward. The bottom-ranked frames are dominated by clearly failed attempts under both signals, serving mainly as a sanity check that obviously bad states receive low values. Together with the qualitative analysis in Fig. 4 , this suggests that the main effect of MVR is to reshuffle intermediate-reward states, down-weighting visually unstable but task-rewarding poses while leaving the optimal behavior unchanged.

Algorithm	Model
MVR	ViCLIP
VLM-RM	CLIP-ViT-H-14-laion2B-s32B-b79K
RoboCLIP	ViCLIP

Appendix

Table A4: Models Used in MVR, VLM-RM, and RoboCLIP

Task	MVR-SAC	SAC	MVR-TQC	TQC
Stand	$704.24\pm 329.17$	$581.77\pm 371.68$	$\mathbf{918.55\pm 29.30}$	$576.59\pm 371.0$
Run	$280.24\pm 103.91$	$299.39\pm 314.65$	$\mathbf{749.23\pm 56.82}$	$647.87\pm 186.98$
Slide	$219.79\pm 176.13$	$94.43\pm 52.89$	$\mathbf{735.03\pm 142.85}$	$514.91\pm 106.36$
Sit_Hard	$172.46\pm 124.35$	$263.87\pm 97.12$	$\mathbf{756.67\pm 108.79}$	$511.85\pm 155.45$

Appendix

Table A5: MVR yields greater performance improvements when integrated with the TQC algorithm compared to SAC on complex humanoid tasks. This table presents average returns for MVR-SAC and MVR-TQC against their SAC and TQC baselines across selected HumanoidBench tasks.

Task	Simba	Simba+MVR
Stand	$\mathbf{906.94}$	$814.53$
Run	$741.16$	$\mathbf{817.69}$
Slide	$577.87$	$\mathbf{814.53}$
Sit_Hard	$783.95$	$\mathbf{785.23}$

Appendix

Table A6: HumanoidBench returns for Simba with and without MVR shaping. Simba numbers are taken from the original publication; Simba+MVR averages three seeds under the same horizon.

Task	TQC	MVR-ViCLIP	MVR-S3D
Run	$647.87\pm 186.98$	$749.23\pm 56.82$	$\mathbf{793.61\pm 57.27}$
Slide	$514.91\pm 106.36$	$\mathbf{735.03\pm 142.85}$	$696.51\pm 32.39$
Stand	$576.59\pm 371.00$	$918.55\pm 29.30$	$\mathbf{937.20\pm 14.28}$
Sit_Hard	$511.85\pm 155.45$	$\mathbf{756.67\pm 108.79}$	$678.53\pm 106.32$

Appendix

Table A7: Effect of VLM backbone on MVR performance. Returns on four HumanoidBench tasks for TQC, MVR with ViCLIP, and MVR with S3D.

Task	TQC	MVR
Pole	$601.62\pm 192.34$	$\mathbf{956.43\pm 13.02}$

Appendix

Table A8: HumanoidBench Pole task under heavy occlusion. Returns for TQC and MVR.

Task	TQC	MVR	MVR-HandCamera
Hammer	$0.20\pm 0.40$	$0.20\pm 0.24$	$0.20\pm 0.24$
Push-Wall	$0.10\pm 0.20$	$0.60\pm 0.49$	$0.80\pm 0.40$
Faucet-Close	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$
Push-Back	$0.60\pm 0.37$	$0.50\pm 0.45$	$0.20\pm 0.24$
Stick-Pull	$0.10\pm 0.20$	$0.20\pm 0.24$	$0.00\pm 0.00$
Handle-Press-Side	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$

Appendix

Table A9: MetaWorld success rates with a wrist-mounted camera. TQC uses only task rewards. MVR uses exocentric videos, while MVR-HandCamera additionally incorporates a wrist-mounted hand camera in the reward pipeline.

Panel of Figure 19 — Figure A2: Success rate vs. training step on two MetaWorld tasks. Across both (a) push-wall and (b) peg-unplug-side , MVR consistently outperforms TQC throughout training, with success rates already higher in the early stages, indicating that multi-view visual shaping accelerates learning rather than only improving final performance.

Method	Stand	Run	Slide	Sit_Hard
MVR (only $r^{\text{VLM}}$ )	$7.73\pm 0.68$	$1.51\pm 0.05$	$0.83\pm 0.02$	$2.60\pm 0.19$
MVR (full)	$918.55\pm 29.30$	$749.23\pm 56.82$	$735.03\pm 142.85$	$756.67\pm 108.79$
TQC (only $r^{task}$ )	$576.59\pm 371.00$	$647.87\pm 186.98$	$514.91\pm 106.36$	$511.85\pm 155.45$

Appendix

Table A10: Ablating the reward components. Using only the VLM reward fails to solve the humanoid tasks, whereas combining it with

r^{task}

recovers the full performance reported in Footnote 3 .

Task	MVR (Temporal Avg.)	MVR (Attention)
Stand	$918.55\pm 29.30$	$940.80\pm 11.11$
Run	$749.23\pm 56.82$	$519.83\pm 366.45$
Slide	$735.03\pm 142.85$	$683.57\pm 243.58$
Sit_Hard	$756.67\pm 108.79$	$822.27\pm 18.04$

Appendix

Table A11: Temporal averaging versus attention-based pooling. Attention can help static tasks but remains unstable on dynamic locomotion tasks, so we adopt simple averaging in the main paper.

Task	TQC	MVR-ALL	MVR
Stand	$576.59\pm 371.00$	$879.06\pm 54.27$	$\mathbf{918.55\pm 29.30}$
Run	$647.87\pm 186.98$	$672.53\pm 115.21$	$\mathbf{749.23\pm 56.82}$
Slide	$514.91\pm 106.36$	$596.27\pm 59.93$	$\mathbf{735.03\pm 142.85}$
Sit_Hard	$511.85\pm 155.45$	$510.78\pm 115.49$	$\mathbf{756.67\pm 108.79}$

Appendix

Table A12: HumanoidBench returns for the all-view baseline. MVR-ALL renders four videos simultaneously and feeds them jointly to the VLM. Means and standard deviations are computed over three seeds.

Signal	Pearson vs success	Pearson vs $r^{task}$
Raw VLM similarity $f^{\text{VLM}}$	0.91	0.21
Learned relevance $f^{\text{MVR}}$	0.96	0.22
Shaped reward $r^{\text{MVR}}$	0.98	0.85

Appendix

Table A13: Reward-signal correlations on Sit_Hard.

Backbone	$\text{corr}(f^{\text{VLM}},r^{task})$	Negative traj.	$\text{corr}(f^{\text{MVR}},r^{task})$
ViCLIP-L (428M)	0.21	50%	0.22
ViCLIP-B (150M)	0.51	10%	0.23

Appendix

Table A14: Correlation with

r^{task}

and fraction of negatively correlated trajectories (Sit_Hard).

Backbone	$\text{corr}(f^{\text{VLM}},\,\text{success})$	$\text{corr}(f^{\text{MVR}},\,\text{success})$	$\text{corr}(r^{\text{MVR}},\,\text{success})$
ViCLIP-L (428M)	0.91	0.96	0.98
ViCLIP-B (150M)	0.90	0.95	0.97

Appendix

Table A15: Correlation with success across VLM backbones (Sit_Hard).

Figure A3: The optimal value of

w

in Eq. 6 is task-dependent . Smaller values (

w=0.01

) appear beneficial for tasks requiring rapid motion like ’Run’, allowing the agent to prioritize the task reward. Conversely, tasks emphasizing stability like ’Stand’ benefit from larger values (

w=0.1

w=0.5

), giving more weight to the VLM guidance. ’Sit_Hard’ shows peak performance at

w=0.5

, while ’Slide’ performs best at

w=0.1

. This indicates that tuning

w

based on task requirements may be necessary for optimal results.

Task	MVR	FuRL
push	$\mathbf{0.10\pm 0.20}$	$0.06\pm 0.08$
window-close	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
Average	$\mathbf{0.55\pm 0.10}$	$0.53\pm 0.04$

Appendix

Table A16: MVR demonstrates competitive or superior performance compared to FuRL on the selected MetaWorld tasks for which FuRL results are available. Both methods combine task-specific rewards (

r^{\text{task}}

) with VLM-based rewards. FuRL results are sourced from ( Fu et al., 2024 ) .

Task	MVR	FuRL
Walk	$\mathbf{927.47\pm 1.83}$	$1.64\pm 0.31$
Run	$\mathbf{749.23\pm 56.82}$	$13.32\pm 12.79$
Stair	$\mathbf{208.60\pm 166.22}$	$4.41\pm 2.32$
Slide	$\mathbf{735.03\pm 142.85}$	$19.74\pm 11.83$
Stand	$\mathbf{918.55\pm 29.30}$	$30.15\pm 8.31$
Sit_Simple	$\mathbf{861.07\pm 19.35}$	$196.15\pm 154.17$

Appendix

Table A17: MVR significantly outperforms FuRL on all tested dynamic and static HumanoidBench tasks. For this comparison, FuRL, which combines task rewards (

r^{\text{task}}

) with VLM-based rewards, was assessed using a VLM guidance mechanism updated at a frequency similar to MVR’s for scalability. These results further underscore the effectiveness of MVR’s approach in leveraging multi-view video guidance from VLMs for complex humanoid control.

Category	Task	MVR	RoboCLIP
Dynamic	Walk	$\mathbf{927.47\pm 1.83}$	$25.31\pm 5.98$
	Run	$\mathbf{581.33\pm 210.35}$	$8.19\pm 3.09$
	Stair	$\mathbf{164.74\pm 132.96}$	$19.25\pm 8.63$
	Slide	$\mathbf{622.05\pm 195.11}$	$27.37\pm 8.28$
Static	Stand	$\mathbf{925.55\pm 25.10}$	$36.67\pm 7.66$
	Sit_Simple	$\mathbf{757.34\pm 143.96}$	$16.95\pm 1.10$

Appendix

Table A18: MVR substantially outperforms the original RoboCLIP (PPO, S3D) across all HumanoidBench tasks. The original RoboCLIP combines task rewards (

r^{\text{task}}

) with VLM-based rewards from its S3D model.

Task	TQC (vision)	MVR (vision)
hammer	$0.17\pm 0.24$	$\mathbf{0.80\pm 0.28}$
push-wall	$\mathbf{0.33\pm 0.47}$	$0.17\pm 0.24$
faucet-close	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
push-back	$0.00\pm 0.00$	$0.00\pm 0.00$
stick-pull	$0.00\pm 0.00$	$0.00\pm 0.00$
handle-press-side	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$

Appendix

Table A19: Pixel-conditioned MetaWorld success rates. Both agents operate on proprioception concatenated with R3M-ViT features. Returns are averaged over five seeds.

Task	TQC-Dense	MVR-Dense	MVR-Sparse
hammer	$0.20\pm 0.40$	$0.20\pm 0.24$	$\mathbf{0.90\pm 0.20}$
push-wall	$0.10\pm 0.20$	$\mathbf{0.60\pm 0.49}$	$0.30\pm 0.24$
faucet-close	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
push-back	$\mathbf{0.60\pm 0.37}$	$0.50\pm 0.45$	$0.00\pm 0.00$
stick-pull	$0.10\pm 0.20$	$\mathbf{0.20\pm 0.24}$	$0.00\pm 0.00$
handle-press-side	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$

Appendix

Table A20: Sparse-reward MetaWorld experiments. MVR-Sparse uses only a sparse success signal from the environment plus visual shaping, yet achieves competitive performance with the dense-reward baseline TQC-Dense. Results are averaged over five seeds.

Figure A4: The two objectives for learning state relevance . The state relevance function $f^{\text{MVR}}$ comprises a state encoder $g^{\text{state}}$ and a relevance vector $g^{\text{rel}}$ . $L_{\text{matching}}$ aligns the relative ranking of state sequences (derived from $f^{\text{MVR}}$ ) with the ranking induced by VLM video-text similarity scores. $L_{\text{reg}}$ regularizes the state sequence representations by enforcing consistency with the similarity structure observed in the VLM’s video embedding space. This dual approach allows MVR to learn state relevance effectively while mitigating noise from visual details and viewpoint variations.

Reward Shaping

Reward shaping designs intermediate or modified feedback to make reinforcement learning more informative, efficient, and aligned with task goals.

See all 142 papers

Automating Potential-based Reward Shaping with Vision Language Model Guidance
Jun 25, 2026Henrik Müller, Daniel KudenkoReward Shaping Sparse-Reward RL
L3S Research Center Leibniz University Hannover Hannover, 30167, Germany
Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
Mar 17, 2026Yanru Wu, Weiduo Yuan, Esteban Martinez Licon +4Long-Horizon Robotic Manipulation Robot Skill Learning
USC Physical Superintelligence Lab
RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards
Jun 24, 2026Anıl Can Ateş, Orhan Kahraman, Cihan TopalRL for Robotics Long-Horizon Robotic Manipulation
Istanbul Technical University

Jun 25, 2026J/K move · Enter open · S save

Automating Potential-based Reward Shaping with Vision Language Model Guidance

Henrik Müller, Daniel Kudenko

L3S Research Center Leibniz University Hannover Hannover, 30167, Germany

Reward Shaping Sparse-Reward RL Policy Learning Reward Design for RL

Open paper AskPDF arXiv

Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision language model (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.

Abstract

Reward Shaping

Explore similar work

Automating Potential-based Reward Shaping with Vision Language Model Guidance

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards