Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation

Yijie Xu; Haopeng Jin; Run Zhou; Shengbang Liu; Sixiang Chen; Hongyang Cheng; Sicheng Hu; Peterson Co

cs.ROAug 16, 2026

Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation

Authors: Yijie Xu, Haopeng Jin, Run Zhou, Shengbang Liu, Sixiang Chen, Hongyang Cheng, Sicheng Hu, Peterson Co,

Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · EvoPhys AI · Beijing University of Posts and Telecommunications · Tencent ·

Abstract

Vision-language-action (VLA) models improve robotic manipulation but remain vulnerable to compounding errors, scene changes, and off-trajectory states. Reinforcement learning can refine pretrained VLA policies, yet sparse success signals hinder exploration, while engineered dense rewards are costly and task-specific. Existing learned visual reward models often rely on static before-after observations, causing temporal ambiguity and weak discrimination between robustness-preserving variations and task-invalid failures under out-of-distribution (OOD) execution. We introduce Robo-Dopamine 2.0, a history- and OOD-aware process reward model with a pairwise prediction interface. It combines (1) history-conditioned pairwise rewards that use source-aligned reference panels for synthetic OOD queries and observed rollout history for online queries, while preserving the queried endpoints, and (2) an OOD-aware signed progress space that represents valid progress, robustness, failure, and recovery. A Signed-Hop Curriculum with transition-aware replay learns coarse execution ordering before fine-grained progress calibration. We also construct an OOD trajectory dataset and a five-family benchmark. Reference panels improve mean visual order consistency (VOC) from 0.967 to 0.986 and OOD-robust VOC from 0.906 to 0.958. With the same 400K pairwise-reward budget, Signed-Hop training with 25% replay reaches 0.9872 mean VOC, compared with 0.9858 for a matched-pool shuffled control. In downstream reinforcement learning, the full model achieves 86.8% mean RoboTwin success and 71/80 successful real-world insertions.

Figures & tables

Figure 1: Overview of Robo-Dopamine 2.0. Standard queries use same-episode expert history, online queries use the observed rollout history, and synthetic OOD queries use source-aligned successful-reference panels while preserving their edited endpoints. All context modes share the same pairwise GRM interface.

Model	Input	ID	Temp.	Neg.	Rob.	OOD-T	Avg.
GRM	static	0.974	0.977	0.951	0.939	0.948	0.958
Robo-Dopamine 2.0	static	0.981	0.984	0.985	0.906	0.979	0.967
Robo-Dopamine 2.0	panel	0.991	0.994	0.994	0.958	0.991	0.986

Table 1: Trajectory-level VOC across five benchmark families. GRM denotes the GRM-8B-Pro baseline; Robo-Dopamine 2.0 is one QA200K-mixed Qwen3-VL-8B checkpoint evaluated with static or prebuilt

3{\times}3

panel inputs. ID, Temp., Neg., Rob., and OOD-T denote in-distribution, temporal-memory, OOD-negative, OOD-robust, and OOD-temporal evaluation. Avg. is the unweighted mean; higher is better.

Figure 2: Task-completion classification accuracy on 60 episodes. We compare GPT-5.5, Gemini-3-Pro, Claude-Opus-4.8, Qwen3.5-397B-A17B-fp8, RoboBrain 2.0, and Robo-Dopamine 2.0-8B. This discrete label metric is distinct from VOC.

Base	Training condition	Static	Panel	Gain
GRM	baseline	0.958	0.965	+0.007
Robo-Dopamine 2.0	$3{\times}3$ +QA	0.967	0.986	+0.019
GRM	OOD+no panel+QA	0.967	0.977	+0.010
GRM	OOD+ $2{\times}2$	0.957	0.985	+0.028
GRM	OOD+ $3{\times}3$	0.955	0.988	+0.033
Robo-Dopamine 2.0	OOD+no panel	0.965	0.974	+0.009

Table 2: Temporal-context and OOD ablations in average VOC. Static and Panel query the same checkpoint;

+\mathrm{QA}

denotes QA200K. Full results are supplementary.

Figure 3: Per-family VOC gains of Signed-Hop over its matched-pool shuffled control. Replay sensitivity is provided in the supplementary material.

Reward condition	OOD	Hist.	Cup	Plate	Hand.	Avg.
RLinf Sparse	–	–	94.2	95.0	68.2	85.8
C00 Controlled	$\times$	$\times$	77.5	82.1	57.0	72.2
C01 History	$\times$	$\checkmark$	79.5	88.6	57.1	75.1
C10 OOD	$\checkmark$	$\times$	85.2	89.5	67.1	80.6
C11 Full	$\checkmark$	$\checkmark$	96.2	94.6	69.7	86.8

Table 3: RoboTwin success rate (%). Sparse uses 1,000 epochs; C00–C11 use 300. Avg. is the unweighted task mean.

Figure 4: Dual-Franka repeated Insert Square setup with ConRFT. The numbered callouts identify the four-hole square, target pegs, and acting arm. Each trial has $K=4$ attempts, creating visually similar states at different cumulative progress.

Reward	OOD	Hist.	Inserts / 80	Full / 20
Event Sparse	–	–	48	8
C00 Controlled	$\times$	$\times$	46	7
C01 History	$\times$	$\checkmark$	67	13
C10 OOD	$\checkmark$	$\times$	50	9
C11 Full	$\checkmark$	$\checkmark$	71	15

Table 4: Repeated Insert Square . Event Sparse uses only the per-insertion event reward; C00–C11 additionally use GRM shaping. Inserts counts successes over 80 attempts; Full counts 20 trials completing all four attempts.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: OOD provenance case study. All task-reference and temporal-context images are drawn from the aligned successful source episode, while only the queried before–after views contain the OOD edits.

Figure 6: Logged progress predictions from matched static and rollout-history-conditioned executions of repeated Insert Square . Tick

I_{k}

k\in\{1,\ldots,4\}

, denotes the

k

-th insertion attempt, and the gray dashed line denotes execution order. Static predictions exhibit non-monotonic jumps at recurring states, whereas rollout-history conditioning produces more order-consistent estimates.

Hyperparameter	Symbol	Value
Global batch size	–	256
Learning rate	–	$3{\times}10^{-4}$
Reward discount	$\gamma$	0.98
Demonstrations	–	40
Insertion attempts / trial	$K$	4
Online episodes (C00–C11)	–	200

Appendix

Table 5: Optimization settings for the real-world ConRFT experiments.

Strategy	Step	Temp.	Neg.	Rob.	OOD-T
No replay	200K	0.9860	0.9945	0.9510	0.9870
No replay	400K	0.9950	0.9937	0.9606	0.9924
25% replay	200K	0.9860	0.9945	0.9510	0.9870
25% replay	400K	0.9953	0.9945	0.9618	0.9931

Appendix

Table 6: Stage 2 VOC for the two large-to-fine curricula. Moderate replay yields slightly higher final scores on the reported temporal and OOD families.

Figure 7: Sensitivity to the fraction $\rho$ of Stage 2 samples replayed from the large-Hop pool. The shaded neighborhood marks the moderate-replay regime around the selected $\rho=0.25$ setting.

Panel of Figure 8 — Figure 8: Episode-level progress traces on two held-out cases. Top: progress curves for a RoboCasa sink-spout episode with foreground occlusion and an in-distribution LIBERO middle-drawer episode. Bottom left: the source-reference pre-query panel for the sink-spout case. Bottom right: the same-episode pre-query panel for the drawer case. Gray cells denote fixed padding.

Figure 9: $3{\times}3$ memory-panel sweep over between-state panel count. Both backbones peak with one or two between-state panels; additional panels do not monotonically improve VOC.

Process Reward Models

Process reward models assess intermediate reasoning steps and actions to guide learning, search, and evaluation.

See all 141 papers

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
Mar 17, 2026Yanru Wu, Weiduo Yuan, Esteban Martinez Licon +4Long-Horizon Robotic Manipulation Robot Skill Learning
USC Physical Superintelligence Lab
SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
Jun 9, 2026Qianzhong Chen, Hau Zheng, Justin Yu +8RL for Robotics Reward Modeling
Stanford University · UC Berkeley · Shanghai Jiao Tong University
What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models
May 13, 2026Yuanfang Peng, Jingjing Fu, Chuheng Zhang +6VLM Robustness Domain Generalization
1Hong Kong University of Science and Technology · 2Microsoft Research Asia · 3Zhejiang University

Mar 17, 2026J/K move · Enter open · S save

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

Yanru Wu, Weiduo Yuan, Esteban Martinez Licon, Ang Qi+3

USC Physical Superintelligence Lab

Long-Horizon Robotic Manipulation Robot Skill Learning Robotic RL Multimodal Reward Modeling Visual RL VLMs for Robotics+1

Open paper AskPDF arXiv

Reinforcement Learning (RL) has shown strong potential for improving robotic manipulation policies, yet its practical use remains bottlenecked by the difficulty of specifying reward functions that are both semantically meaningful and reusable across tasks. In this paper, we propose Large Reward Models (LRMs), a framework that adapts foundation VLMs into frame-level reward generators for robot policy refinement. We specialize a state-of-the-art VLM on a multi-source dataset spanning real-world robot trajectories, human-object interactions, and simulated manipulation environments. Unlike prior approaches that mainly evaluate trajectories post-hoc, LRMs expose multiple reward interfaces from visual observations: progress estimation, task completion, and temporal contrastive comparison. Starting from an imitation-learned policy, we use these VLM-derived rewards to guide PPO refinement on held-out long-horizon manipulation tasks. Our experiments show that LRM progress rewards provide the strongest non-privileged online refinement signal, improving the IL baseline and narrowing the gap to privileged environment rewards. We further deploy progress rewards for progress-weighted behavioral cloning on four real-world manipulation tasks spanning two robot platforms, improving over SFT on all four tasks. These results suggest that modality-specific specialization of foundation VLMs can provide practical visual reward signals for both simulated policy refinement and physical robot self-improvement without hand-coded task rewards.

Abstract

Process Reward Models

Explore similar work

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation

What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models