Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation

Haoyu Zhang; Zecui Zeng; Bin Wang; Lusong Li; Liang Lin; Long Cheng

cs.ROAug 19, 2026

Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation

Authors: Haoyu Zhang, Zecui Zeng, Bin Wang, Lusong Li, Liang Lin, Long Cheng

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · JD Explore Academy, China

Abstract

Learning robotic policies requires dense rewards that remain informative when behavior departs from successful demonstrations. Progress-based rewards estimate how far an observation has advanced along a nominal successful trajectory, but may remain high after an incorrect transition. We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations. Given the visual history up to a transition start, the model predicts the latent displacement associated with successful execution and scores the observed displacement through signed directional and symmetric magnitude agreement. This transition-level comparison penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress. Dream2Reward requires no failure annotations, progress labels, or synthetic negatives, and produces a dense causal reward. Across mechanism diagnostics and shared-trajectory evaluations, it provides stronger success-failure separation and more informative feedback on low-quality behavior than progress-based alternatives. Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance, including in real-robot manipulation. These results show that comparing realized motion with predicted successful change provides an effective way to convert positive demonstrations into dense rewards for robot learning.

Figures & tables

Fig. 1: Illustration of the Dream2Reward framework. From a language-conditioned visual history, a latent transition model trained only on positive demonstration data predicts the successful latent displacement $\widehat{\Delta z}^{+}_{t}$ . At inference, a candidate transition is encoded into its own displacement $\Delta z_{t}^{\mathrm{cand}}$ , and the reward is the angular alignment and magnitude agreement between the two: transitions along the success direction with an appropriate step size score high, whereas wrong-direction, overshooting, or stagnant transitions score low.

Fig. 2: Latent transition alignment diagnostics for successful, wrong-language, and failed-episode transitions: (a) raw reward

r_{t}^{\mathrm{raw}}

, (b) absolute log magnitude ratio

|\log\rho_{t}|

, and (c) angular score

r_{t}^{\mathrm{ang}}

. Significance is computed on per-trajectory aggregates: paired correct- versus wrong-language scores use a two-sided Wilcoxon signed-rank test, while successful- versus failed-episode scores use a two-sided Mann–Whitney

U

test;

\ast\ast\ast

p<0.001

\ast\ast

p<0.01

\ast

p<0.05

, n.s.:

p\geq 0.05

Fig. 3: Failure-detection ROC-AUC on identical real-robot trajectories, using negative episode-mean native rewards.

Model	AUC $\uparrow$	Gap $\uparrow$
Dream2Reward	0.95	+0.376
ROBOMETER	0.93	+0.253
Robo-Dopamine	0.41	$-$ 0.107

TABLE I: Failure-sensitive reward validation on the real-robot scoring set.

Fig. 4: Evaluation-normalized success–failure gap on identical trajectories; significance uses a two-sided Mann–Whitney

U

test on per-trajectory scores. D2R denotes Dream2Reward, RBM denotes ROBOMETER, and R-Dopa denotes Robo-Dopamine.

Fig. 5: Representative real-robot failure and recovery traces. Each block shows task snapshots on the top and synchronized reward traces below, comparing Dream2Reward with the progress-based baselines ROBOMETER and Robo-Dopamine on the same rollout. Green labels indicate nominal task progress or successful recovery, red labels indicate failure events or post-failure states, and the shaded intervals are qualitative annotations for the displayed traces. The examples cover stalled grasping, paper-bag cleanup failure, plate-edge interference during grasping, and recovery after an initial grasp miss.

Variant	Removed / changed component	AUC $\uparrow$
Full Dream2Reward	–	0.95
Angle only	magnitude term	0.92
Magnitude only	angular term	0.92
No language conditioning	task text $g$	0.80

TABLE II: Ablation results on failure-sensitive validation. All variants use the same evaluation protocol; AUC is failure-detection ROC-AUC.

Panel of Figure 8 — Fig. 6: Online RL comparison between Dream2Reward and Robo-Dopamine under the same SAC pipeline. (a) True task success and evaluation-normalized model reward on two manipulation tasks over five seeds; $\widetilde{r}_{t}\in[0,1]$ is used only for visualization. (b) Model reward versus true success across evaluation checkpoints. The line shows a least-squares fit, and the shaded upper-left region indicates high predicted reward despite low task success.

Property	D2R	ROBOMETER	Robo-Dopamine
Locally-flat step fraction $\downarrow$	0.024	0.032	0.577
Signal coverage $\uparrow$	0.942	0.925	0.422
Low-qual. neg.-reward frac. $\uparrow$	0.745	0.413	0.061
Reward–time MI $I(R;T)$ $\downarrow$	0.011	0.083	0.042

TABLE III: Per-step reward signal quality on robo_arena. Rewards are normalized per method and task using the median and MAD, and results are macro-averaged across tasks. Best values are bold.

Panel of Figure 10 — Fig. 7: Real-robot offline RL with Q-guided flow on four tabletop manipulation tasks: the task suite (top; panels (a)–(d) show the four tasks) and per-task offline RL success rates (bottom).

Property	D2R	ROBOMETER	Robo-Dopamine
Effective-credit step fraction $\uparrow$	0.801	0.703	0.404
Failure-drop min-AUC $\uparrow$	0.661	0.580	0.333
End-segment success–fail gap $\uparrow$	0.098	0.056	0.009
Failure end-segment reward $\downarrow$	$-0.091$	$-0.001$	$+0.004$

TABLE IV: Offline-relevant transition-label statistics on robo_arena. Rewards are normalized per method and task using the median and MAD, and results are macro-averaged over 40 tasks. Best values are bold.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 252 papers

DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation
Jul 14, 2026Yu Fang, Wanxi Dong, Jiaqi Liu +7RL for Robotics Reward Modeling
University of North Carolina at Chapel Hill · Carnegie Mellon University · Shanghai Jiao Tong University +1
Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models
Mar 17, 2026Pierre Krack, Tobias Jülg, Wolfram Burgard +1Reward Modeling Reward Design for RL
Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg, Germany. · TUM School of Computation, Information and Technology, Technical University of Munich, Germany
Disentangling Task-Relevant Latent Dynamics for Reward Modeling
Oct 9, 2026Dayou Li, Jionghao Wang, Peihao Wang +9Dense Reward RL Multimodal Reward Modeling
Texas A&M University · University of Texas at Austin · Google DeepMind +1

Jul 14, 2026J/K move · Enter open · S save

DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

Yu Fang, Wanxi Dong, Jiaqi Liu, Yue Yang+6

University of North Carolina at Chapel Hill · Carnegie Mellon University · Shanghai Jiao Tong University+1

RL for Robotics Reward Modeling Multimodal Reward Modeling Robotic Manipulation

Open paper AskPDF arXiv

Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning. However, its practical adoption remains bottlenecked by the lack of reliable vision-language reward models that provide dense and informative feedback. Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels. Collecting failure trajectories typically requires laborious human effort, while pseudo-failures constructed by relabeling successful demonstrations fail to capture the diverse physical failure modes that arise during robot execution. Meanwhile, existing reward models often predict sparse binary or trajectory-level rewards, which provide limited guidance for efficient policy optimization. We introduce DenseReward, a dense robotic reward model that addresses both challenges. To train DenseReward, we develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. DenseReward predicts dense frame-level reward scores from visual observations and language instructions, enabling fine-grained estimation of task progress throughout an episode. Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. We further demonstrate that DenseReward provides effective reward guidance for downstream model predictive control and reinforcement learning. We release the dataset, trained reward models, and evaluation suite to support the development of failure-aware dense reward modeling for robot learning.

Abstract

Reward Design for RL

Explore similar work

DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models

Disentangling Task-Relevant Latent Dynamics for Reward Modeling