TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

Shirui Chen; Cole Harrison; Ying-Chun Lee; Angela Jin Yang; Zhongzheng Ren; Lillian J. Ratliff; Jiafei Duan; Dieter Fox

cs.ROFeb 22, 2026

TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

Authors: Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, +1 more

Organizations: University of Washington · Allen Institute for AI · Amazon · University of North Carolina at Chapel Hill

Abstract

General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior. Yet obtaining such feedback at scale remains difficult, since existing approaches often rely on manual progress annotations, task-specific demonstrations, or reward models trained on curated robot datasets. We introduce TOPReward, a training-free progress reward method that probes pretrained Video-Language Models (VLMs) through their internal token probabilities rather than asking them to generate numerical progress values. Given a video prefix and a language instruction, TOPReward measures the model's likelihood that the instructed task has been completed, converting latent video-language understanding into a dense reward signal without task-specific reward-model training or manually annotated progress labels. We evaluate TOPReward on ManiRewardBench, our real-world manipulation benchmark spanning 130 unique tasks and four robot platforms, as well as on Open X-Embodiment datasets. Across these settings, TOPReward substantially outperforms prior training-free VLM reward methods on open-source models and is competitive with a trained reward-model baseline on progress-estimation metrics, while requiring no reward-model training. Additional analyses show that the reward is sensitive to the specified instruction and is not explained by time index alone. Finally, TOPReward supports downstream applications including success detection and offline reward-weighted behavior cloning.

Figures & tables

Figure 1: Result highlights . TOPReward enables effective zero-shot estimation of task progress across diverse and challenging real-world manipulation tasks, without task-specific training. Across several VLM backbones, TOPReward provides a temporally consistent visual reward signal that supports multiple downstream applications, including success detection, policy improvement, and evaluation on our in-house benchmark, ManiRewardBench .

Method	Model	VOC	Kendall	Pearson	MAE
GVL	Molmo2-8B	-0.019	0.000	0.001	0.500
GVL	Qwen3-VL-8B	0.218	0.192	0.443	0.396
GVL	Gemini-2.5-Pro	0.572	0.534	0.669	0.218
TOPReward	Molmo2-8B	0.525	0.427	0.503	0.308
TOPReward	Qwen3-VL-8B	0.874	0.771	0.856	0.163
TOPReward	Qwen3-VL-32B	0.890	0.799	0.870	0.191

Table 1: Results on the filtered Open X-Embodiment dataset. VOC reports mean dataset-level VOC over the same 33 filtered datasets and 20 episodes per dataset. Higher is better except MAE.

Method	Model	VOC	Kendall	Pearson	MAE
GVL	Molmo2-8B	-0.003	-0.002	-0.003	0.514
GVL	Qwen3-VL-8B	0.316	0.261	0.326	0.392
TOPReward	Molmo2-8B	0.619	0.484	0.569	0.316
TOPReward	Qwen3-VL-8B	0.942	0.858	0.915	0.126
TOPReward	Qwen3-VL-32B	0.864	0.771	0.824	0.194
Robometer	4B	0.858	0.768	0.861	0.160

Table 2: Results on ManiRewardBench . We report metrics averaged over the 497 successful-evaluation episodes. This progress-evaluation subset contains 113 tasks across four robot platforms: LeRobot, Franka, Bimanual YAM, and Single-arm YAM. VOC and Kendall tau-b measure rank agreement, Pearson measures linear agreement, and MAE measures normalized progress error. Higher is better except MAE.

Figure 2: Instruction sensitivity. Instruction-grounding matrices for TOPReward and Robometer-4B. Rows are terminal trajectory videos and columns are candidate task instructions. For TOPReward, each cell is the terminal TOPReward score for that video–instruction pair; for Robometer-4B, each cell is its terminal reward prediction. We apply Sinkhorn normalization for visualization and report mean diagonal mass, so both rows and columns sum to one and bright diagonal cells indicate strong one-to-one matching between trajectories and their intended instructions.

Method	Score	ROC-AUC	PR-AUC
GVL (Qwen3-VL-8B)	VOC	0.728	0.893
TOPReward (Qwen3-VL-8B)	success score	0.939	0.952
TOPReward (Qwen3-VL-32B)	success score	0.965	0.973
Robometer-4B	trained success head	0.930	0.951

Table 3: Success detection results. We evaluate 279 cleaned LeRobot episodes from ManiRewardBench (150 successful, 129 failed). TOPReward uses the training-free prefix-margin plus full-video Yes/No score defined in Appendix C ; GVL uses per-trajectory VOC; Robometer-4B uses its trained success head.

Task	Pretrained	BC	TOP-RWBC (Ours)
Place toy car in box	1	2	3
Stack red cube on green cube	1.33	1	2.33
Put pen into cup	1.67	5.67	6.33
Place doll in box	0	7	10
Pick up cube	4	7	10
Put cube in cup	4	6	9

Table 4: Real-world experiments. Partial success score out of 10 trials for reward-weighted behavior cloning on single-arm SO-100 tasks.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Top 10 tokens by absolute difference in mean final-step token probability between successful and failed trajectories. The affirmative token True shows the largest separation, motivating its use as the completion token in TOPReward . Left: mean token probability by group; right: absolute difference in mean token probability.

Figure 4: VOC comparison across datasets. Mean dataset-level VOC for GVL (0-shot) and TOPReward across two evaluation sets: filtered OXE (33 datasets, 20 episodes each) and the ManiRewardBench successful progress-evaluation subset (4 datasets, 113 tasks, 497 episodes). Error bars denote standard deviation across datasets within each evaluation set.

ManiRewardBench
Dataset	Qwen3-VL-8B			Gemini-2.5-Pro			Molmo2-8B
	GVL	TOPR	$\Delta$	GVL	TOPR	$\Delta$	GVL	TOPR	$\Delta$
ManiRewardBench_bimanual_yam	0.164	0.947	0.783	0.566	0.546	-0.021	0.007	0.565	0.558
ManiRewardBench_franka	0.242	0.942	0.700	0.695	0.448	-0.247	0.000	0.662	0.662
ManiRewardBench_lerobot	0.332	0.954	0.622	0.620	0.578	-0.041	-0.001	0.595	0.597
ManiRewardBench_single_yam	0.544	0.945	0.401	0.752	0.488	-0.264	-0.017	0.642	0.659

Appendix

Table 5: Filtered datasets (alphabetical) comparing GVL (0-shot) vs TOPReward (TOPR) and their difference, broken down by model backbone. Bold indicates the higher VOC between the two methods for that dataset/model.

Figure 5: Per-episode VOC distributions, broken down by evaluation set ( ManiRewardBench vs Open X-Embodiment) and model backbone.

Figure 6: Distribution of dataset-level $\Delta$ VOC $=$ VOC( TOPReward ) $-$ VOC(GVL), shown separately for each model backbone. Positive values indicate TOPReward outperforms GVL; the dashed line marks the per-model mean.

Figure 7: Progress traces for ManiRewardBench. Example progress traces predicted by TOPReward (orange) compared to stage-aware ground-truth completion (dashed) from ManiRewardBench , computed from annotated subtask boundaries. We also overlay Gemini-GVL (blue) on the same episodes when available.

Figure 8: Qualitative example of “Fold the Towel”: Instruction-conditioned progress estimation on a real trajectory. The curve shows TOPReward ’s predicted completion value over time, with annotated values at selected frames corresponding to semantic subtasks.

Figure 9: The six real-world single-arm SO-100 manipulation tasks used for reward-weighted behavior cloning evaluation.

Figure 10: Qualitative comparison on “Place doll in box.” The pretrained policy and behavior cloning (BC) both fail, while TOP-RWBC, fine-tuned with reward weights from TOPReward , succeeds consistently. Frames are uniformly sampled from evaluation rollouts.

	Qwen3-VL-8B		Molmo2-8B
Dataset	Base	+Chat	Base	+Chat
Bimanual YAM	0.947	0.269	0.570	0.408
Franka	0.943	0.528	0.696	0.615
Single-arm YAM	0.946	0.703	0.691	0.546
Mean	0.945	0.500	0.652	0.523
$\Delta$	$-47.1\%$		$-19.8\%$

Appendix

Table 6: Effect of Chat Template on TOPReward VOC. Chat template degrades Qwen3-VL-8B performance by nearly 50% and Molmo2-8B by 20%, demonstrating that the logit-based formulation is sensitive to prompt formatting.

Dataset	Episodes	Tasks
Lerobot	150	22
Lerobot failed	156	23
Franka	150	51
Bimanual YAM	97	20
Single-arm YAM	100	20
Total	653	136

Appendix

Table 7: Summary of ManiRewardBench datasets. 6 tasks appear in both the Lerobot and Lerobot failed splits, giving 130 unique tasks across the full benchmark.

Figure 11: Counts of different example tasks in the single-arm YAM dataset.

Figure 12: Counts of different example tasks in the bimanual YAM dataset.

Figure 13: Counts of different example tasks in the SO-100 dataset.

Figure 14: Counts of different example tasks in the Franka dataset.

Panel of Figure 16 — Figure 16: Screenshot of the Annotation Tool.

Panel of Figure 17 — Figure 17: Expert demonstration with annotation for the task ”Clean the table”.

Explore similar work

From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
May 22, 2026Christian Gumbsch, Leonardo Barcellona, Lennard Schünemann +7RL for Robotics Multimodal Reward Modeling
University of Amsterdam · 2Catholic University of Leuven · 3Toyota Research Institute +1
TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models
Aug 9, 2026Yidong Wang, Yan Zhan, Ziteng Feng +16Reward Modeling Pairwise Preference Evaluation
Peking University · University of Science and Technology of China · Southeast University +6
SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
Mar 30, 2026Philip Schroeder, Thomas Weng, Karl Schmeckpeper +3RL for Robotics Multimodal Reward Modeling
MIT · RAI Institute

May 22, 2026J/K move · Enter open · S save

From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models

Christian Gumbsch, Leonardo Barcellona, Lennard Schünemann, Platon Karageorgis+6

University of Amsterdam · 2Catholic University of Leuven · 3Toyota Research Institute+1

RL for Robotics Multimodal Reward Modeling Reward Design for RL Test-Time Prompt Tuning

Open paper AskPDF arXiv

Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior policy training. We propose Demo2Reward a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3-10 trajectories) to reduce false positives while preserving true positives. Crucially, this requires no additional model training or computation resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function.

Abstract

Explore similar work

From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models

TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models

SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning