Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons

Anthony Liang; Yigit Korkmaz; Jiahui Zhang; Minyoung Hwang; Abrar Anwar; Sidhant Kaushik; Aditya Shah; Alex S. Huang

cs.ROMar 2, 2026

Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons

Authors: Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, +9 more

Organizations: Univ. of Southern California · UT Dallas · MIT · Indep. Researcher · Univ. of Washington · Ai2 · NVIDIA

Abstract

General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations, providing only local, frame-level supervision. While effective for expert demonstrations, this paradigm scales poorly to large-scale robotics datasets where failed and suboptimal trajectories are abundant and assigning dense progress labels is ambiguous. We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision. Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints across trajectories of the same task, enabling effective learning from both real and augmented failed trajectories. To support this formulation at scale, we curate RBM-1M, a reward-learning dataset comprising over one million trajectories spanning diverse robot embodiments and tasks, including substantial suboptimal and failure data. Across benchmarks and real-world evaluations, Robometer learns more generalizable reward functions than prior methods and improves robot learning performance across a diverse set of downstream applications. Code, model weights, and videos at https://robometer.github.io/.

Figures & tables

Fig. 2 : Robometer is a VLM-based reward model that predicts dense, per-frame progress-based rewards and success labels for the first of two video trajectories. To scalably train with failure data, we also predict which of the two video trajectories better completes the task (preference). We use three strategies for curating training examples from our given datasets, which are further detailed in Section III-D with model architecture shown in Appendix Figure 11 . At inference, the model can input either up to two trajectories: with a single trajectory, it outputs per-frame progress & success predictions; with two, it additionally predicts a preference for which trajectory best performs the task.

Fig. 3 : Video-Language Reward Confusion Matrix. For each task sampled at random from self-collected, unseen data from RBM-EVAL-OOD , we compute rewards for all combinations of demonstration videos and language descriptions. Robometer produces the most diagonal-heavy confusion matrix, indicating strong alignment between unseen demos and instructions. We also report the column-normalized diagonal mean under each model, which represents the fraction of the model’s total reward for aligned task and video pairs.

Fig. 4 : Qualitative Analysis of Failure, Suboptimal and Successful Trajectories. We visualize the progress predictions for three trajectories of different quality for the same task. Notably, for the suboptimal trajectory, Robometer predicts steadily increasing progress as the robot approaches the pen holder, but sharply reduces its progress estimate when the marker is dropped, correctly reflecting regression in task completion. In contrast, RoboReward continues to assign high progress despite the task failure. Finally, Robometer is the only model that correctly predicts task success for the successful trajectory (i.e., high final progress value and explicit success prediction).

Fig. 5 : RL w/ Ablation Models in LIBERO-90 tasks from scratch, corresponding to ablations trained only on LIBERO-10/Object/Goal/Spatial data from Table III . We report the average success rate

\pm

standard deviation across 5 seeds.

Fig. 6 : Automatic online RL with DSRL [ 123 ] on a DROID [ 65 ] setup with Robometer improves $\pi_{0}$ from 20% to 85% on a single-stage task and 20% to 70% on a two-stage task, outperforming RoboReward ’s overall success rate by $2.5\times$ . DSRL with Robometer learns to avoid base $\pi_{0}$ errors such as collisions or moving the wrong object. The setup is deemed “automatic” because success detection and stage advancement are handled automatically by the reward model, requiring human intervention only for physical scene resets.

Fig. 7 : Offline RL results using IQL on a mixture of Noisy and Expert trajectories. Robometer rewards consistently outperform both RoboReward and sparse rewards: $2.4\times$ average success rate improvement over the best baseline for each task.

Fig. 8 : (a): Proportion of task-relevant subtrajectories out of 100 retrieval queries. Our method consistently retrieves a high number of relevant subtrajectories using either the preference or progress objective. (b): Success rates of LoRA-finetuned $\pi_{0.5}$ policies using the retrieved trajectories from each method. Small amounts of suboptimal & unrelated data retrieved by other baselines degrade policy-learning performance: Robometer -retrieval attains an average $4.5\times$ success rate improvement over the best baseline.

Panel of Figure 8 — Fig. 9 : Insufficient-progress failures. Although the robot continues to move, predicted progress stagnates or oscillates without converging to success, enabling detection of failures caused by stalling or premature termination.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 10 : Pie chart of RBM-1M dataset types. Full table with individual dataset details in Table VIII .

Dataset	Embodiment	# Trajectories	Citation
OXE Eval Suite	Franka, Google Robot, Jaco 2, WidowX	14,398	[ 24 ]
RACER (Eval)	Franka Panda (simulation)	7,227	[ 27 ]
Metaworld Eval	Sawyer	151	[ 134 ]
LIBERO Eval	Franka Panda (simulation)	3,950	–
Total	5 unique robot embodiments	25,726	–

Appendix

TABLE VI : RBM-EVAL-ID - In distribution evaluation datasets overview. # Trajectories is computed by aggregating all evaluation splits per dataset, counting unique annotated trajectories.

Dataset	Embodiment	# Trajectories
USC Franka	Franka Panda	24
USC Koch	Bimanual Kochv1.1	150
USC Trossen	Trossen	27
USC xArm	xArm	36
MIT Franka	Franka Panda	304
UTD SO101	SO101	30

Appendix

TABLE VII : RBM-EVAL-OOD - Out-of-distribution evaluation datasets overview. # Trajectories is computed by aggregating all evaluation splits per dataset, counting unique annotated trajectories.

Dataset	Embodiment	# Trajectories	Citation
Expert Demonstration Datasets
OXE Mix	11 mixed embodiments	449,475	[ 24 ]
AGIBotWorld-Alpha Subset	AgiBot G1 Bimanual Mobile Manipulator	216,911	[ 25 ]
Galaxea Open World	Galaxea R1 Lite Bimanual Mobile Manipulator	108,118	[ 64 ]
RoboReward OXE Mix	13 mixed embodiments	45,072	[ 76 ]
MolmoACT	Franka Panda	15,546	[ 74 ]

Appendix

TABLE VIII : RBM-1M Training datasets overview. # Trajectories is determined by counting every unique video-language annotation (possibly across multiple views when available).

Parameter	Value
Base Model	Qwen/Qwen3-VL-4B-Instruct
Number of frames	8
Per-device batch size	16
Learning rate	$2\times 10^{-5}$
Weight decay	0.01
Total training steps	6500

Appendix

TABLE IX : Training Configuration for Robometer

Dataset	Success Cutoff
Open-X Embodiment (OXE)
Aloha Mobile	0.95
Austin BUDS	0.95
Berkeley Cable Routing	0.95
Berkeley FANUC Manipulation	0.98
Bridgev2	0.95

Appendix

TABLE X: Dataset-specific success cutoffs used to correct for delayed episode termination in teleoperated data. Datasets not contained here use a default success cutoff of 1.0.

Pref. Loss	VOC $r\uparrow$	Kendall $\tau\uparrow$	Succ–Fail Diff. $\uparrow$
BT	0.862	0.325	0.242
BCE	0.945	0.643	0.320

Appendix

TABLE XI: Bradley–Terry (BT) versus preference label from a dedicated

\langle|\text{pref\_token}|\rangle

(BCE) given both videos in a single forward pass, evaluated on RBM-EVAL-OOD .

Fig. 11 : Robometer model architecture. A VLM backbone processes a language instruction with one or two trajectories. For both single and paired inputs, per-frame hidden states of the first trajectory are passed to MLP heads predicting dense task progress and binary success. For paired trajectories, a preference token is used with a preference head to select the trajectory that better satisfies the instruction. The model is trained with joint progress, success, and preference objectives.

Dataset	RL-VLM-F (%)	Robometer (%)
USC Franka	52.1	75.0
USC Koch	54.4	79.4
USC Trossen	66.7	76.2
USC xArm	48.6	88.9
MIT Franka	54.4	85.4
UTD SO-101	56.7	90.0

Appendix

TABLE XIV: RL-VLM-F vs Robometer on Trajectory Quality . Different quality trajectory pairwise preference accuracy on 500 comparisons for each of the RBM-EVAL-OOD datasets.

Dataset	RL-VLM-F (%)	Robometer (%)
USC Franka	70.7	100.0
USC Koch	54.7	89.8
USC Trossen	64.0	99.0
USC xArm	73.3	98.2
MIT Franka	55.3	98.4
UTD SO-101	72.7	100.0

Appendix

TABLE XV: RL-VLM-F vs Robometer on Different Tasks . Different task trajectory pairwise preference accuracy on 500 comparisons for each of the RBM-EVAL-OOD datasets.

Fig. 12 : Qualitative analysis of Robometer on a long-horizon task. With sufficiently detailed task instructions, Robometer correctly assigns increasing rewards across multiple stages of the task.

Model Type	Model	RoboRewardBench MAE ( $\downarrow$ )
Qwen3-4B Models	Robometer (only RoboReward data)	0.75
	RoboReward -4B	0.85
	Qwen3-VL-4B-Instr.	1.03
Closed / Larger	RoboReward -8B	0.67
	GPT-5-mini	0.69
	Qwen3-VL-8B-Instr.	0.89

Appendix

TABLE XVII : Evaluation on the RoboRewardBench benchmark [ 76 ] .

	VOC $r\uparrow$		Kendall $\tau\uparrow$		Succ–Fail Diff. $\uparrow$
Weights $(\lambda_{\text{pref}},\lambda_{\text{prog}})$	LIBERO (10)	LIBERO (90)	LIBERO (10)	LIBERO (90)	LIBERO (10)	LIBERO (90)
$(2,\,1)$	0.983	0.964	0.981	0.898	0.347	0.352
$(1,\,2)$	0.986	0.946	0.982	0.875	0.423	0.367
$(1,1)$	0.994	0.976	0.986	0.919	0.483	0.455

Appendix

TABLE XIX: Loss weight ablation results. Hyperparameter sweep over preference/progress loss weights

(\lambda_{\text{pref}},\lambda_{\text{prog}})

on LIBERO-10 and LIBERO-90. Each model is trained on LIBERO-(10, Spatial, Object, Goal) and LIBERO-90 is heldout for evaluation only.

	LIBERO-90			RBM-EVAL-OOD
Model	VOC $r\uparrow$	Kend. $\tau\uparrow$	S-F $\uparrow$	VOC $r\uparrow$	Kend. $\tau\uparrow$	S-F $\uparrow$
Robometer (-succ)	0.97	0.89	0.40	0.94	0.60	0.27
Robometer	0.98	0.92	0.46	0.94	0.64	0.32

Appendix

TABLE XX : Success prediction head ablation results. Reward alignment, trajectory ranking, and final reward difference between successful and failure trajectories on RBM-EVAL-OOD and LIBERO-90.

Hyperparameter	Value
Batch size	128
Target update rate $\tau$	0.005
Discount factor $\gamma$	$0.99$
Target update interval	1
Number of critics	5
Critics sampled per update	2

Appendix

TABLE XXI: RL with Ablated Reward Models. SAC hyperparameters.

Hyperparameter	Value
Batch size	64
Target update rate $\tau$	0.005
Discount factor $\gamma$	$0.995$
Target update interval	1
Number of critics	4
Critics sampled per update	2

Appendix

TABLE XXII: Automatic Online RL. DSRL SAC hyperparameters.

Method	True Positives	False Positives
RoboReward	6	45
Robometer	18	0

Appendix

TABLE XXIII : Automatic Online RL reward relabeling. True positives (TP) and false positives (FP) for reward predictions during online RL.

Hyperparameter	Value
Batch size	256
Target update rate $\tau$	0.005
Discount factor $\gamma$	$\{0.9,0.95,0.99\}$
Target update interval	1
Advantage temperature	2
Expectile	0.7

Appendix

TABLE XXIV: SO-101 offline RL experiment. Hyperparameters used for Implicit Q-Learning (IQL) in offline policy training.

Component	Actor	Critic
Transformer Encoder
Model dimension ( $d_{\text{model}}$ )	256	256
Number of heads	8	8
Encoder layers	6	6
Transformer dropout	0.0	0.0
Transformer activation	GELU	GELU

Appendix

TABLE XXV: SO-101 offline RL model architectures. Transformer-based actor and critic architectures used for offline IQL training. Both networks share a lightweight Transformer encoder, followed by task-specific MLP heads.

Fig. 13 : Data filtering & retrieval scene configuration from all cameras.

Panel of Figure 29 — Fig. 14 : Failure Detection OOD confusion matrices with ternary ground truth and binary prediction. Rows indicate ground-truth execution outcomes ( failure , suboptimal , success ), while columns indicate binary predictions ( predicted failure vs. predicted success ). Suboptimal trajectories correspond to executions that make partial progress but do not complete the task. Suboptimal trajectories are treated as failures in our quantitative evaluation, as emphasized with the horizontal orange divider in the above confusion matrices. Color intensity reflects the number of trajectories.

Panel of Figure 30 — Fig. 15 : Irreversible failures. Terminal events such as drops or spills cause a sharp regression in predicted task progress, which our model reliably flags as failures shortly after the event.

Panel of Figure 31 — Fig. 16 : Semantic failures. The robot executes smooth and physically plausible trajectories but violates the task instruction, resulting in persistently low predicted progress and failure detection without an abrupt terminal event.

Fig. 17 : Model-Based RL with Robometer integrated into DreamZero [ 133 ] . In this cluttered scene, Robometer improves DreamZero’s performance from 20% success rate to 70%.

Explore similar work

TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models
Aug 9, 2026Yidong Wang, Yan Zhan, Ziteng Feng +16Reward Modeling Pairwise Preference Evaluation
Peking University · University of Science and Technology of China · Southeast University +6
Progress Reward Modeling for Robotic Learning: A Comprehensive Survey
Jul 22, 2026Jianshu Zhang, Keliang Wu, Haoran Lu +8Reward Modeling Process Reward Models
Northwestern University · Carnegie Mellon University · University of Wisconsin–Madison +2
TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
Feb 22, 2026Shirui Chen, Cole Harrison, Ying-Chun Lee +6Robot Learning Multimodal Reward Modeling
University of Washington · Allen Institute for AI · Amazon +1

Aug 9, 2026J/K move · Enter open · S save

TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models

Yidong Wang, Yan Zhan, Ziteng Feng, Zhenyu Cui+15

Peking University · University of Science and Technology of China · Southeast University+6

Reward Modeling Pairwise Preference Evaluation Multimodal Reward Modeling Isotonic Regression

Open paper AskPDF arXiv

Reward models are a bottleneck for reinforcement learning in embodied AI. Long-horizon robotic manipulation requires scalable vision feedback beyond handcrafted rewards or task-specific annotations. Existing open-source VLM reward judges like RoboReward adopt simple 1--5 trajectory progress scoring, lacking pairwise preferences for RLHF, DPO and Bradley-Terry frameworks, while failing to optimize video scene understanding. Augmenting RoboReward with pairwise comparison and video-QA supervision causes inconsistency between pairwise preferences and pointwise scores, introducing training noise and hurting downstream performance---an issue aggregation methods such as TrustJudge cannot resolve. To address this, we propose TrustRoboReward, a multi-paradigm reward modeling framework equipped with Preference-Ordered Isotonic Score Editing (POISE). We construct a unified four-paradigm dataset with trajectory progress scoring (Score-A), video-QA answer quality scoring (Score-B), and their pairwise counterparts (Pair-A, Pair-B). Pairwise labels align better with human judgment than pointwise scores, inspiring us to calibrate pointwise scores to avoid score-pair reversals against pairwise preferences. POISE rectifies pointwise scores and eliminates cross-paradigm reversal conflicts unresolved by TrustJudge. Theoretically, POISE reduces score-pair reversal conflicts from 20.15% to 0%, whereas TrustJudge retains 20.46% conflicts on the same corpus. Evaluated on our benchmark, Qwen3-VL-4B trained with POISE achieves an overall reward score of 77.96%, nearly matching GPT-5-mini (78.09%, gap 0.13%) and outperforming the strongest RoboReward-4B baseline by 10.13%. It also lifts test-time score-pair consistency to 71.90%, exceeding RoboReward-4B (57.26%) and GPT-5-mini (68.09%). Integrating TrustJudge aggregation during inference boosts the overall score to 78.57%, surpassing the GPT-5-mini teacher model.

Abstract

Explore similar work

TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models

Progress Reward Modeling for Robotic Learning: A Comprehensive Survey

TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics