General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations, providing only local, frame-level supervision. While effective for expert demonstrations, this paradigm scales poorly to large-scale robotics datasets where failed and suboptimal trajectories are abundant and assigning dense progress labels is ambiguous. We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision. Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints across trajectories of the same task, enabling effective learning from both real and augmented failed trajectories. To support this formulation at scale, we curate RBM-1M, a reward-learning dataset comprising over one million trajectories spanning diverse robot embodiments and tasks, including substantial suboptimal and failure data. Across benchmarks and real-world evaluations, Robometer learns more generalizable reward functions than prior methods and improves robot learning performance across a diverse set of downstream applications. Code, model weights, and videos at https://robometer.github.io/.
Figures & tables
Fig. 2 : Robometer is a VLM-based reward model that predicts dense, per-frame progress-based rewards and success labels for the first of two video trajectories. To scalably train with failure data, we also predict which of the two video trajectories better completes the task (preference). We use three strategies for curating training examples from our given datasets, which are further detailed in Section III-D with model architecture shown in Appendix Figure 11 . At inference, the model can input either up to two trajectories: with a single trajectory, it outputs per-frame progress & success predictions; with two, it additionally predicts a preference for which trajectory best performs the task.
Fig. 3 : Video-Language Reward Confusion Matrix. For each task sampled at random from self-collected, unseen data from RBM-EVAL-OOD , we compute rewards for all combinations of demonstration videos and language descriptions. Robometer produces the most diagonal-heavy confusion matrix, indicating strong alignment between unseen demos and instructions. We also report the column-normalized diagonal mean under each model, which represents the fraction of the model’s total reward for aligned task and video pairs.
Fig. 4 : Qualitative Analysis of Failure, Suboptimal and Successful Trajectories. We visualize the progress predictions for three trajectories of different quality for the same task. Notably, for the suboptimal trajectory, Robometer predicts steadily increasing progress as the robot approaches the pen holder, but sharply reduces its progress estimate when the marker is dropped, correctly reflecting regression in task completion. In contrast, RoboReward continues to assign high progress despite the task failure. Finally, Robometer is the only model that correctly predicts task success for the successful trajectory (i.e., high final progress value and explicit success prediction).
Fig. 5 : RL w/ Ablation Models in LIBERO-90 tasks from scratch, corresponding to ablations trained only on LIBERO-10/Object/Goal/Spatial data from Table III . We report the average success rate ± standard deviation across 5 seeds.
Fig. 6 : Automatic online RL with DSRL [ 123 ] on a DROID [ 65 ] setup with Robometer improves π0 from 20% to 85% on a single-stage task and 20% to 70% on a two-stage task, outperforming RoboReward ’s overall success rate by 2.5× . DSRL with Robometer learns to avoid base π0 errors such as collisions or moving the wrong object. The setup is deemed “automatic” because success detection and stage advancement are handled automatically by the reward model, requiring human intervention only for physical scene resets.
Fig. 7 : Offline RL results using IQL on a mixture of Noisy and Expert trajectories. Robometer rewards consistently outperform both RoboReward and sparse rewards: 2.4× average success rate improvement over the best baseline for each task.
Fig. 8 : (a): Proportion of task-relevant subtrajectories out of 100 retrieval queries. Our method consistently retrieves a high number of relevant subtrajectories using either the preference or progress objective. (b): Success rates of LoRA-finetuned π0.5 policies using the retrieved trajectories from each method. Small amounts of suboptimal & unrelated data retrieved by other baselines degrade policy-learning performance: Robometer -retrieval attains an average 4.5× success rate improvement over the best baseline.
Fig. 9 : Insufficient-progress failures. Although the robot continues to move, predicted progress stagnates or oscillates without converging to success, enabling detection of failures caused by stalling or premature termination.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 10 : Pie chart of RBM-1M dataset types. Full table with individual dataset details in Table VIII .
Dataset
Embodiment
# Trajectories
Citation
OXE Eval Suite
Franka, Google Robot, Jaco 2, WidowX
14,398
[ 24 ]
RACER (Eval)
Franka Panda (simulation)
7,227
[ 27 ]
Metaworld Eval
Sawyer
151
[ 134 ]
LIBERO Eval
Franka Panda (simulation)
3,950
–
Total
5 unique robot embodiments
25,726
–
Appendix
TABLE VI : RBM-EVAL-ID - In distribution evaluation datasets overview. # Trajectories is computed by aggregating all evaluation splits per dataset, counting unique annotated trajectories.
Dataset
Embodiment
# Trajectories
USC Franka
Franka Panda
24
USC Koch
Bimanual Kochv1.1
150
USC Trossen
Trossen
27
USC xArm
xArm
36
MIT Franka
Franka Panda
304
UTD SO101
SO101
30
Appendix
TABLE VII : RBM-EVAL-OOD - Out-of-distribution evaluation datasets overview. # Trajectories is computed by aggregating all evaluation splits per dataset, counting unique annotated trajectories.
Dataset
Embodiment
# Trajectories
Citation
Expert Demonstration Datasets
OXE Mix
11 mixed embodiments
449,475
[ 24 ]
AGIBotWorld-Alpha Subset
AgiBot G1 Bimanual Mobile Manipulator
216,911
[ 25 ]
Galaxea Open World
Galaxea R1 Lite Bimanual Mobile Manipulator
108,118
[ 64 ]
RoboReward OXE Mix
13 mixed embodiments
45,072
[ 76 ]
MolmoACT
Franka Panda
15,546
[ 74 ]
Appendix
TABLE VIII : RBM-1M Training datasets overview. # Trajectories is determined by counting every unique video-language annotation (possibly across multiple views when available).
Parameter
Value
Base Model
Qwen/Qwen3-VL-4B-Instruct
Number of frames
8
Per-device batch size
16
Learning rate
2×10−5
Weight decay
0.01
Total training steps
6500
Appendix
TABLE IX : Training Configuration for Robometer
Dataset
Success Cutoff
Open-X Embodiment (OXE)
Aloha Mobile
0.95
Austin BUDS
0.95
Berkeley Cable Routing
0.95
Berkeley FANUC Manipulation
0.98
Bridgev2
0.95
Appendix
TABLE X: Dataset-specific success cutoffs used to correct for delayed episode termination in teleoperated data. Datasets not contained here use a default success cutoff of 1.0.
Pref. Loss
VOC r↑
Kendall τ↑
Succ–Fail Diff. ↑
BT
0.862
0.325
0.242
BCE
0.945
0.643
0.320
Appendix
TABLE XI: Bradley–Terry (BT) versus preference label from a dedicated ⟨∣pref_token∣⟩ (BCE) given both videos in a single forward pass, evaluated on RBM-EVAL-OOD .
Fig. 11 : Robometer model architecture. A VLM backbone processes a language instruction with one or two trajectories. For both single and paired inputs, per-frame hidden states of the first trajectory are passed to MLP heads predicting dense task progress and binary success. For paired trajectories, a preference token is used with a preference head to select the trajectory that better satisfies the instruction. The model is trained with joint progress, success, and preference objectives.
Dataset
RL-VLM-F (%)
Robometer (%)
USC Franka
52.1
75.0
USC Koch
54.4
79.4
USC Trossen
66.7
76.2
USC xArm
48.6
88.9
MIT Franka
54.4
85.4
UTD SO-101
56.7
90.0
Appendix
TABLE XIV: RL-VLM-F vs Robometer on Trajectory Quality . Different quality trajectory pairwise preference accuracy on 500 comparisons for each of the RBM-EVAL-OOD datasets.
Dataset
RL-VLM-F (%)
Robometer (%)
USC Franka
70.7
100.0
USC Koch
54.7
89.8
USC Trossen
64.0
99.0
USC xArm
73.3
98.2
MIT Franka
55.3
98.4
UTD SO-101
72.7
100.0
Appendix
TABLE XV: RL-VLM-F vs Robometer on Different Tasks . Different task trajectory pairwise preference accuracy on 500 comparisons for each of the RBM-EVAL-OOD datasets.
Fig. 12 : Qualitative analysis of Robometer on a long-horizon task. With sufficiently detailed task instructions, Robometer correctly assigns increasing rewards across multiple stages of the task.
Model Type
Model
RoboRewardBench MAE ( ↓ )
Qwen3-4B Models
Robometer (only RoboReward data)
0.75
RoboReward -4B
0.85
Qwen3-VL-4B-Instr.
1.03
Closed / Larger
RoboReward -8B
0.67
GPT-5-mini
0.69
Qwen3-VL-8B-Instr.
0.89
Appendix
TABLE XVII : Evaluation on the RoboRewardBench benchmark [ 76 ] .
VOC r↑
Kendall τ↑
Succ–Fail Diff. ↑
Weights (λpref,λprog)
LIBERO (10)
LIBERO (90)
LIBERO (10)
LIBERO (90)
LIBERO (10)
LIBERO (90)
(2,1)
0.983
0.964
0.981
0.898
0.347
0.352
(1,2)
0.986
0.946
0.982
0.875
0.423
0.367
(1,1)
0.994
0.976
0.986
0.919
0.483
0.455
Appendix
TABLE XIX: Loss weight ablation results. Hyperparameter sweep over preference/progress loss weights (λpref,λprog) on LIBERO-10 and LIBERO-90. Each model is trained on LIBERO-(10, Spatial, Object, Goal) and LIBERO-90 is heldout for evaluation only.
LIBERO-90
RBM-EVAL-OOD
Model
VOC r↑
Kend. τ↑
S-F ↑
VOC r↑
Kend. τ↑
S-F ↑
Robometer (-succ)
0.97
0.89
0.40
0.94
0.60
0.27
Robometer
0.98
0.92
0.46
0.94
0.64
0.32
Appendix
TABLE XX : Success prediction head ablation results. Reward alignment, trajectory ranking, and final reward difference between successful and failure trajectories on RBM-EVAL-OOD and LIBERO-90.
Hyperparameter
Value
Batch size
128
Target update rate τ
0.005
Discount factor γ
0.99
Target update interval
1
Number of critics
5
Critics sampled per update
2
Appendix
TABLE XXI: RL with Ablated Reward Models. SAC hyperparameters.
TABLE XXIII : Automatic Online RL reward relabeling. True positives (TP) and false positives (FP) for reward predictions during online RL.
Hyperparameter
Value
Batch size
256
Target update rate τ
0.005
Discount factor γ
{0.9,0.95,0.99}
Target update interval
1
Advantage temperature
2
Expectile
0.7
Appendix
TABLE XXIV: SO-101 offline RL experiment. Hyperparameters used for Implicit Q-Learning (IQL) in offline policy training.
Component
Actor
Critic
Transformer Encoder
Model dimension ( dmodel )
256
256
Number of heads
8
8
Encoder layers
6
6
Transformer dropout
0.0
0.0
Transformer activation
GELU
GELU
Appendix
TABLE XXV: SO-101 offline RL model architectures. Transformer-based actor and critic architectures used for offline IQL training. Both networks share a lightweight Transformer encoder, followed by task-specific MLP heads.
Fig. 13 : Data filtering & retrieval scene configuration from all cameras.
Fig. 14 : Failure Detection OOD confusion matrices with ternary ground truth and binary prediction. Rows indicate ground-truth execution outcomes ( failure , suboptimal , success ), while columns indicate binary predictions ( predicted failure vs. predicted success ). Suboptimal trajectories correspond to executions that make partial progress but do not complete the task. Suboptimal trajectories are treated as failures in our quantitative evaluation, as emphasized with the horizontal orange divider in the above confusion matrices. Color intensity reflects the number of trajectories.
Fig. 15 : Irreversible failures. Terminal events such as drops or spills cause a sharp regression in predicted task progress, which our model reliably flags as failures shortly after the event.
Fig. 16 : Semantic failures. The robot executes smooth and physically plausible trajectories but violates the task instruction, resulting in persistently low predicted progress and failure detection without an abrupt terminal event.
Fig. 17 : Model-Based RL with Robometer integrated into DreamZero [ 133 ] . In this cluttered scene, Robometer improves DreamZero’s performance from 20% success rate to 70%.
Reward models are a bottleneck for reinforcement learning in embodied AI. Long-horizon robotic manipulation requires scalable vision feedback beyond handcrafted rewards or task-specific annotations. Existing open-source VLM reward judges like RoboReward adopt simple 1--5 trajectory progress scoring, lacking pairwise preferences for RLHF, DPO and Bradley-Terry frameworks, while failing to optimize video scene understanding. Augmenting RoboReward with pairwise comparison and video-QA supervision causes inconsistency between pairwise preferences and pointwise scores, introducing training noise and hurting downstream performance---an issue aggregation methods such as TrustJudge cannot resolve. To address this, we propose TrustRoboReward, a multi-paradigm reward modeling framework equipped with Preference-Ordered Isotonic Score Editing (POISE). We construct a unified four-paradigm dataset with trajectory progress scoring (Score-A), video-QA answer quality scoring (Score-B), and their pairwise counterparts (Pair-A, Pair-B). Pairwise labels align better with human judgment than pointwise scores, inspiring us to calibrate pointwise scores to avoid score-pair reversals against pairwise preferences. POISE rectifies pointwise scores and eliminates cross-paradigm reversal conflicts unresolved by TrustJudge. Theoretically, POISE reduces score-pair reversal conflicts from 20.15% to 0%, whereas TrustJudge retains 20.46% conflicts on the same corpus. Evaluated on our benchmark, Qwen3-VL-4B trained with POISE achieves an overall reward score of 77.96%, nearly matching GPT-5-mini (78.09%, gap 0.13%) and outperforming the strongest RoboReward-4B baseline by 10.13%. It also lifts test-time score-pair consistency to 71.90%, exceeding RoboReward-4B (57.26%) and GPT-5-mini (68.09%). Integrating TrustJudge aggregation during inference boosts the overall score to 78.57%, surpassing the GPT-5-mini teacher model.
Yidong Wang, Yan Zhan, Ziteng Feng +16
Peking University · University of Science and Technology of China · Southeast University +6
Robotic learning takes place in dynamic environments with large behavior spaces. A terminal success signal only tells the robot whether the task is completed. It does not explain whether the current behavior is making progress, remaining unchanged, or undoing earlier progress. For this reason, recent studies have increasingly explored progress rewards that provide feedback during task execution. However, the current literature lacks a shared framework. Existing methods use different observations, goal specifications, output signals, supervision sources, and evaluation protocols. This makes it difficult to compare them and understand what their results actually validate. In this survey, we provide a unified view of progress reward modeling for robotic learning. We organize the field in three connected steps. We first study the interface of a progress model. This defines the problem from the outside by asking what information the model receives and what form of progress signal it produces. We then move inside the model and study the methods used to construct this signal. This reveals the different assumptions and mechanisms behind progress estimation and reward generation. Finally, we examine the data and benchmarks that support these methods. This shows how progress supervision is obtained and what different evaluations actually measure. Together, these three perspectives connect what a progress model is, how it is built, and how its quality is validated. We further summarize the main limitations of current approaches and discuss future research directions.
Jianshu Zhang, Keliang Wu, Haoran Lu +8
Northwestern University · Carnegie Mellon University · University of Wisconsin–Madison +2
General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior. Yet obtaining such feedback at scale remains difficult, since existing approaches often rely on manual progress annotations, task-specific demonstrations, or reward models trained on curated robot datasets. We introduce TOPReward, a training-free progress reward method that probes pretrained Video-Language Models (VLMs) through their internal token probabilities rather than asking them to generate numerical progress values. Given a video prefix and a language instruction, TOPReward measures the model's likelihood that the instructed task has been completed, converting latent video-language understanding into a dense reward signal without task-specific reward-model training or manually annotated progress labels. We evaluate TOPReward on ManiRewardBench, our real-world manipulation benchmark spanning 130 unique tasks and four robot platforms, as well as on Open X-Embodiment datasets. Across these settings, TOPReward substantially outperforms prior training-free VLM reward methods on open-source models and is competitive with a trained reward-model baseline on progress-estimation metrics, while requiring no reward-model training. Additional analyses show that the reward is sensitive to the specified instruction and is not explained by time index alone. Finally, TOPReward supports downstream applications including success detection and offline reward-weighted behavior cloning.
Shirui Chen, Cole Harrison, Ying-Chun Lee +6
University of Washington · Allen Institute for AI · Amazon +1