Long-horizon robotic manipulation remains challenging for reinforcement learning (RL) because sparse rewards provide limited guidance for credit assignment. Practical policy improvement thus relies on richer intermediate supervision, such as dense progress rewards, which are costly to obtain and ill-suited to non-monotonic behaviors such as backtracking and recovery. To address this, we propose Advantage Reward Modeling (ARM), a framework that shifts from hard-to-quantify absolute progress to estimating relative advantage. We introduce a cost-effective tri-state labeling strategy -- Progressive, Regressive, and Stagnant -- that reduces human cognitive overhead while ensuring high cross-annotator consistency. By training on these intuitive signals, ARM enables automated progress annotation for both complete demonstrations and fragmented DAgger-style data. Integrating ARM into an offline RL pipeline allows for adaptive action-reward reweighting, effectively filtering suboptimal samples. Our approach achieves a 99.4% success rate on a challenging long-horizon towel-folding task, demonstrating improved stability and data efficiency over current VLA baselines with near-zero human intervention during policy training.
Figures & tables
Figure 1: Overview of our proposed framework. The system consists of three main components: (1) The Advantage Reward Model (ARM) with its MIMO-based Temporal Transformer, supervised by a lightweight tri-state labeling strategy; (2) An automated pipeline for global progress reconstruction; and (3) The Advantage-Weighted Behavior Cloning (AW-BC) algorithm, which optimizes the policy using length-invariant relative gains extracted from the reconstructed progress.
Figure 2: Comparison between MISO and MIMO architectures. MISO stands for Multi-Input Single-Output, and MIMO stands for Multi-Input Multi-Output.
Figure 3: Illustration of the tri-state labeling strategy applied to a demonstration episode.
Figure 4: Overview of the long-horizon towel-folding task. The sequence includes extracting a towel from clutter, placing and flattening it on the table, executing a precise multi-stage folding strategy, and transporting the folded towel into the target box.
Metrics
SARM
ARM (Ours)
MSE ↓
0.0059
0.0014
Success Identification Accuracy (%)
Standard (SE)
83.3 (10/12)
100.0 (12/12)
Failure (FE)
91.6 (11/12)
100.0 (12/12)
Table 1: Quantitative Evaluation of Reward Models. All models are evaluated on a validation set of 50 trajectories . “MSE” measures the trajectory reconstruction fidelity against GT progress (normalized to [0,1] ). The bottom section reports the Success Identification Accuracy , assessing the Completion Head’s ability to correctly classify the final state of Standard ( SE , 12 successful episodes), and Failure ( FE , 12 failed episodes) trajectories. Best performances are highlighted in bold .
Figure 5: Qualitative comparison of progress reconstruction. We visualize the progress curves of SARM and ARM against the GT for a representative episode. While SARM struggles with non-monotonic behaviors, ARM reconstructs a smooth, high-fidelity curve that closely tracks the GT, even during regressive adjustments.
Method
Success Rate
Task Throughput
Folding Precision
(%)
(Episodes / hr)
(Score)
BC-Baseline (GR00T N1.5)
62.1
18
2.2
RA-BC (GR00T + SARM)
78.5
24
2.7
AW-BC (GR00T + ARM)
99.4
32
3.6
Table 2: Quantitative Comparison of Downstream Policy Performance. We report the success rate, operational task throughput (episodes completed per hour), and folding precision (final edge alignment score; detailed annotation protocol provided in the Supplementary Material) on the long-horizon towel-folding task. Our proposed AW-BC (ARM) framework significantly outperforms both standard Behavior Cloning and prior reward-aware baselines across all metrics.
Annotation Protocol
Rate (Samples/8h)
Human Baseline (Seg.) †
100
Human Tri-state (Ours) †
250
VLM (Qwen3-VL) ‡
∼3000
Auto Tri-state (Ours) ‡
>400,000
Table 3: Labeling Efficiency Comparison. Annotation throughput comparison between human and automated labeling protocols per 8-hour shift.
Figure 6: Qualitative comparison of progress reconstruction. Our tri-state approach generates smoother, more consistent dense progress signals compared to the stepped curves of manual segmentation and VLM methods.
Method
Architecture
Throughput (it/s)
Qwen3-VL
MISO
1.03
SARM Baseline
SISO
3.9
ARM (Ours)
MIMO
14.1
Table 4: MIMO Architecture Efficiency Comparison. We evaluate the inference throughput across different reward modeling approaches. ARM is evaluated with its MIMO design handling 5 parallel outputs per input, VLM labeling represents traditional single-input approaches, and SARM serves as the baseline. All measurements are conducted on a single NVIDIA A100 GPU under comparable conditions.
Method
Task Seg.
Tri-state
RA-BC
AW-BC
Success Rate (%)
SARM
✓
–
✓
–
78.5
ARM
–
✓
✓
–
92.3
ARM
–
✓
–
✓
99.4
Table 5: Ablation Study. We systematically evaluate the contributions of tri-state labeling and AW-BC training through three key configurations.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Vision Encoder
CLIP ViT-B/32
Sequential Aggregator Window
5 frames (1Hz sampling)
Training Epochs
2
Hardware Configuration
2 × NVIDIA A100 GPUs
Effective Batch Size
64
Optimization Configuration
Appendix
Table 6: ARM Training Hyperparameters. Complete hyperparameter settings for training the Advantage Reward Model.
Parameter
Value
Foundation Model
GR00T-N1.5 (3B parameters)
Policy Head
Diffusion Transformer (DiT) Flow Matching
Action Dimension
14D bimanual actions
Action Horizon ( H )
32
Camera Views
3 × 224×224 (head + wrists)
Training Epochs
7
Appendix
Table 7: Policy Training Hyperparameters. Complete hyperparameter settings for Advantage-Weighted Behavior Cloning using the GR00T-N1.5 foundation model.
Figure 7: Visualization of ARM Inference Results. The left panels show the third-person view of the bimanual towel-folding task at t=69s and t=70s . The right panels display the corresponding progress curves: predicted progress Ppred (blue) and ground truth Pgt (green). ARM accurately captures the non-monotonic progress “dip” caused by a regressive adjustment, with the Multi-frame Advantage head correctly outputting Δpred=−1 .
Figure 8: Hardware setup for real-world experiments. The system features a 6-DoF bimanual robot configuration controlled via an AgileX master-slave teleoperation interface. It is equipped with a global base camera and two wrist-mounted cameras to capture comprehensive visual observations alongside the 14-dimensional proprioceptive data.