Long-horizon robotic manipulation remains challenging for reinforcement learning (RL) because sparse rewards provide limited guidance for credit assignment. Practical policy improvement thus relies on richer intermediate supervision, such as dense progress rewards, which are costly to obtain and ill-suited to non-monotonic behaviors such as backtracking and recovery. To address this, we propose Advantage Reward Modeling (ARM), a framework that shifts from hard-to-quantify absolute progress to estimating relative advantage. We introduce a cost-effective tri-state labeling strategy -- Progressive, Regressive, and Stagnant -- that reduces human cognitive overhead while ensuring high cross-annotator consistency. By training on these intuitive signals, ARM enables automated progress annotation for both complete demonstrations and fragmented DAgger-style data. Integrating ARM into an offline RL pipeline allows for adaptive action-reward reweighting, effectively filtering suboptimal samples. Our approach achieves a 99.4% success rate on a challenging long-horizon towel-folding task, demonstrating improved stability and data efficiency over current VLA baselines with near-zero human intervention during policy training.
Figures & tables
Figure 1: Overview of our proposed framework. The system consists of three main components: (1) The Advantage Reward Model (ARM) with its MIMO-based Temporal Transformer, supervised by a lightweight tri-state labeling strategy; (2) An automated pipeline for global progress reconstruction; and (3) The Advantage-Weighted Behavior Cloning (AW-BC) algorithm, which optimizes the policy using length-invariant relative gains extracted from the reconstructed progress.
Figure 2: Comparison between MISO and MIMO architectures. MISO stands for Multi-Input Single-Output, and MIMO stands for Multi-Input Multi-Output.
Figure 3: Illustration of the tri-state labeling strategy applied to a demonstration episode.
Figure 4: Overview of the long-horizon towel-folding task. The sequence includes extracting a towel from clutter, placing and flattening it on the table, executing a precise multi-stage folding strategy, and transporting the folded towel into the target box.
Metrics
SARM
ARM (Ours)
MSE ↓
0.0059
0.0014
Success Identification Accuracy (%)
Standard (SE)
83.3 (10/12)
100.0 (12/12)
Failure (FE)
91.6 (11/12)
100.0 (12/12)
Table 1: Quantitative Evaluation of Reward Models. All models are evaluated on a validation set of 50 trajectories . “MSE” measures the trajectory reconstruction fidelity against GT progress (normalized to [0,1] ). The bottom section reports the Success Identification Accuracy , assessing the Completion Head’s ability to correctly classify the final state of Standard ( SE , 12 successful episodes), and Failure ( FE , 12 failed episodes) trajectories. Best performances are highlighted in bold .
Figure 5: Qualitative comparison of progress reconstruction. We visualize the progress curves of SARM and ARM against the GT for a representative episode. While SARM struggles with non-monotonic behaviors, ARM reconstructs a smooth, high-fidelity curve that closely tracks the GT, even during regressive adjustments.
Method
Success Rate
Task Throughput
Folding Precision
(%)
(Episodes / hr)
(Score)
BC-Baseline (GR00T N1.5)
62.1
18
2.2
RA-BC (GR00T + SARM)
78.5
24
2.7
AW-BC (GR00T + ARM)
99.4
32
3.6
Table 2: Quantitative Comparison of Downstream Policy Performance. We report the success rate, operational task throughput (episodes completed per hour), and folding precision (final edge alignment score; detailed annotation protocol provided in the Supplementary Material) on the long-horizon towel-folding task. Our proposed AW-BC (ARM) framework significantly outperforms both standard Behavior Cloning and prior reward-aware baselines across all metrics.
Annotation Protocol
Rate (Samples/8h)
Human Baseline (Seg.) †
100
Human Tri-state (Ours) †
250
VLM (Qwen3-VL) ‡
∼3000
Auto Tri-state (Ours) ‡
>400,000
Table 3: Labeling Efficiency Comparison. Annotation throughput comparison between human and automated labeling protocols per 8-hour shift.
Figure 6: Qualitative comparison of progress reconstruction. Our tri-state approach generates smoother, more consistent dense progress signals compared to the stepped curves of manual segmentation and VLM methods.
Method
Architecture
Throughput (it/s)
Qwen3-VL
MISO
1.03
SARM Baseline
SISO
3.9
ARM (Ours)
MIMO
14.1
Table 4: MIMO Architecture Efficiency Comparison. We evaluate the inference throughput across different reward modeling approaches. ARM is evaluated with its MIMO design handling 5 parallel outputs per input, VLM labeling represents traditional single-input approaches, and SARM serves as the baseline. All measurements are conducted on a single NVIDIA A100 GPU under comparable conditions.
Method
Task Seg.
Tri-state
RA-BC
AW-BC
Success Rate (%)
SARM
✓
–
✓
–
78.5
ARM
–
✓
✓
–
92.3
ARM
–
✓
–
✓
99.4
Table 5: Ablation Study. We systematically evaluate the contributions of tri-state labeling and AW-BC training through three key configurations.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Vision Encoder
CLIP ViT-B/32
Sequential Aggregator Window
5 frames (1Hz sampling)
Training Epochs
2
Hardware Configuration
2 × NVIDIA A100 GPUs
Effective Batch Size
64
Optimization Configuration
Appendix
Table 6: ARM Training Hyperparameters. Complete hyperparameter settings for training the Advantage Reward Model.
Parameter
Value
Foundation Model
GR00T-N1.5 (3B parameters)
Policy Head
Diffusion Transformer (DiT) Flow Matching
Action Dimension
14D bimanual actions
Action Horizon ( H )
32
Camera Views
3 × 224×224 (head + wrists)
Training Epochs
7
Appendix
Table 7: Policy Training Hyperparameters. Complete hyperparameter settings for Advantage-Weighted Behavior Cloning using the GR00T-N1.5 foundation model.
Figure 7: Visualization of ARM Inference Results. The left panels show the third-person view of the bimanual towel-folding task at t=69s and t=70s . The right panels display the corresponding progress curves: predicted progress Ppred (blue) and ground truth Pgt (green). ARM accurately captures the non-monotonic progress “dip” caused by a regressive adjustment, with the Multi-frame Advantage head correctly outputting Δpred=−1 .
Figure 8: Hardware setup for real-world experiments. The system features a 6-DoF bimanual robot configuration controlled via an AgileX master-slave teleoperation interface. It is equipped with a global base camera and two wrist-mounted cameras to capture comprehensive visual observations alongside the 14-dimensional proprioceptive data.
Reinforcement learning for robot manipulation is often bottlenecked by reward design, especially in long-horizon tasks: sparse success rewards provide weak supervision, while hand-crafted dense rewards are tedious to design and generalize poorly across tasks. Progress-based reward models offer a promising alternative by estimating how far an observation has advanced toward task completion, but existing approaches often require task-specific demonstrations or progress labels, and can assign high rewards to visually plausible but physically incorrect states. We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward. RARM is trained once on general-purpose videos with a contrastive temporal objective, requiring no robot-specific data, task-specific reward labels, or per-task reward engineering. At deployment, RARM matches rollout clips to reference clips and rewards only confident forward progress, suppressing uncertain matches that may otherwise produce false-positive rewards. Across 9 simulated manipulation tasks from LIBERO and MetaWorld and 4 real-world tasks, RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding, where unreliable progress estimates are especially harmful.
Pengzhi Yang, Xinyu Wang, Pengyu Jing +7
NUS Human-Centered Robotic Lab · ‡ Booking.com · University of Cambridge +1
Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still relies heavily on behavior cloning, which requires costly high-quality demonstrations and keeps policies near the demonstration distribution. Reward models can reduce this dependence by reweighting demonstrations and providing dense supervision for on-robot reinforcement learning (RL), but they must be dense, accurate, and general. Existing methods fall short: task-specific stage-aware models are accurate but require per-task annotations, while general vision-language-model (VLM) reward models are broadly applicable but too coarse for fine-grained long-horizon progress. We introduce RM, a multi-task stage-aware reward model that combines an action-primitive-based stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head to produce dense per-step rewards across manipulation tasks. Building on RM, we further propose SPIRAL (Self-Policy Improvement via Reward-Aligned Learning), an on-policy reward-guided framework that improves VLA policies from cheap autonomous rollouts. On a 10-task benchmark, RM reduces value-estimation MSE by 80% over the strongest baselines; when used in SPIRAL, it improves task success from around 50% to near-perfect performance on Folding Shorts (58% to 100%) and Cleaning Whiteboard (50% to 90%), showing that high-quality dense rewards are key to a stable robot data flywheel. Project website: https://qianzhong-chen.github.io/sarm2.github.io/.
Qianzhong Chen, Hau Zheng, Justin Yu +8
Stanford University · UC Berkeley · Shanghai Jiao Tong University
Progress reward modeling is the problem of estimating how a robot's behavior changes task progress over time. Reliable estimation requires distinguishing meaningful state changes from failed attempts and task-irrelevant actions. We introduce the Agentic Reward System (ARS), an inference framework for progress reward modeling with general-purpose vision-language models (VLMs), without additional reward-model training. Given an offline trajectory and a task instruction, ARS uses adaptive visual inspection for both event proposal and verification. A subagent proposes a task-relevant event timeline, which a primary agent verifies and revises before estimating per-frame progress. ARS can incorporate optional terminal outcome labels and visual references to inform its judgments. It can also audit progress estimates from external reward models. We evaluate ARS with a 27B VLM on a controlled semantic-mismatch benchmark and downstream policy learning in simulation and on a real robot. The benchmark reveals that several evaluated reward baselines assign spurious progress to wrong-object manipulation even in simple pick-and-place scenes. ARS better suppresses these errors and outperforms these baselines in simulation policy learning. We further demonstrate that ARS supports long-horizon policy learning from mixed-quality offline experience on real-robot multi-screw fastening in a full-scale laboratory replica of an industrial washing-machine assembly line. These results suggest that structured inference and verification can improve the usefulness of general-purpose VLMs for robot reward modeling. Code is at https://github.com/midea-ai/ars