ARS: Agentic Reward System for Robot Learning

Sheng Hu; Weiyi Lu; Lingbing Zeng; Gan Weng; Weiwei Zhang; Kai Xie; Xiaofeng Mou; Yi Xu

cs.ROSep 28, 2026

ARS: Agentic Reward System for Robot Learning

Authors: Sheng Hu, Weiyi Lu, Lingbing Zeng, Gan Weng, Weiwei Zhang, Kai Xie, Xiaofeng Mou, Yi Xu

Organizations: AIRC, Midea Group

Abstract

Progress reward modeling is the problem of estimating how a robot's behavior changes task progress over time. Reliable estimation requires distinguishing meaningful state changes from failed attempts and task-irrelevant actions. We introduce the Agentic Reward System (ARS), an inference framework for progress reward modeling with general-purpose vision-language models (VLMs), without additional reward-model training. Given an offline trajectory and a task instruction, ARS uses adaptive visual inspection for both event proposal and verification. A subagent proposes a task-relevant event timeline, which a primary agent verifies and revises before estimating per-frame progress. ARS can incorporate optional terminal outcome labels and visual references to inform its judgments. It can also audit progress estimates from external reward models. We evaluate ARS with a 27B VLM on a controlled semantic-mismatch benchmark and downstream policy learning in simulation and on a real robot. The benchmark reveals that several evaluated reward baselines assign spurious progress to wrong-object manipulation even in simple pick-and-place scenes. ARS better suppresses these errors and outperforms these baselines in simulation policy learning. We further demonstrate that ARS supports long-horizon policy learning from mixed-quality offline experience on real-robot multi-screw fastening in a full-scale laboratory replica of an industrial washing-machine assembly line. These results suggest that structured inference and verification can improve the usefulness of general-purpose VLMs for robot reward modeling. Code is at https://github.com/midea-ai/ars

Figures & tables

Figure 1: Overview of ARS. ARS uses adaptive visual inspection for both event proposal and verification. A subagent proposes task-relevant events from an instruction and offline trajectory. The primary agent verifies and refines these proposals, then estimates per-frame task progress. ARS supports optional terminal outcomes and multimodal references, plus external reward auditing.

Figure 2: Progress estimates for a partially successful LIBERO trajectory under the instruction “turn on the stove and put the moka pot on it.” Insets show selected frames and ARS progress values. ARS captures the regression when the stove is unintentionally switched off and credits subsequent progress toward pot placement. Baseline curves appear below (Appendix C ), with ARS repeated as a light gray dashed line for reference. The event proposal and primary-agent verification for this trajectory are detailed in Appendix B .

Group	Method	VOC $\uparrow$	FPSR $\uparrow$	CMPG $\uparrow$
Reward-trained	LRM	0.828	0.893	0.373
	R 2 VLM	0.916	0.184	-0.050
	SOLE-R1	0.902	0.361	0.266
	Robometer	0.983	0.240	0.366
Audit	ARS w/ Robometer proposal	0.988	0.915	0.891
Reward-training-free	TOPReward + Qwen3-VL-8B	0.930	0.368	0.685

Table 1: Results on the semantic-mismatch benchmark. The Audit configuration uses Robometer predictions as an external proposal, audited and revised by ARS with Qwen3.6-27B.

Group	Method	Success (%) $\uparrow$	vs. Vanilla $\uparrow$	Retained (%)
Baseline	Vanilla BC	44.2 $\pm$ 0.3	—	100.0
Reward-trained	LRM	38.6 $\pm$ 0.5	-5.6	13.7
	R 2 VLM	41.5 $\pm$ 0.4	-2.7	13.6
	SOLE-R1	44.8 $\pm$ 0.4	+0.6	52.6
	Robometer	46.8 $\pm$ 0.9	+2.6	89.5
Reward-training-free	TOPReward + Qwen3-VL-8B	43.1 $\pm$ 0.5	-1.1	77.0

Table 2: Policy-learning results on LIBERO-Long. Retained is the micro-averaged percentage of action chunks kept from the original dataset; gains are in percentage points.

Configuration	VOC $\uparrow$	FPSR $\uparrow$	CMPG $\uparrow$
Robometer -4B	0.983	0.240	0.366
ARS + Qwen3.5-4B	0.986	0.625	0.633
GVL-style + Qwen3.6-27B	0.637	0.900	0.354
ARS + Qwen3.6-27B	0.995	0.918	0.904

Table 3: Base-model and inference-procedure comparisons on the semantic-mismatch benchmark.

Configuration	Success (%) $\uparrow$	Retained (%)	Queries	Input (K)	Output (K)
TOPReward + Qwen3-VL-32B	47.6 $\pm$ 0.7	67.9	390.3	2,866.3	0.0
ARS w/o verification & inspection	45.4 $\pm$ 0.3	78.1	7.3	242.7	10.1
ARS w/o verification	52.3 $\pm$ 0.8	70.3	10.2	413.2	9.9
ARS w/o context separation	51.8 $\pm$ 0.5	68.7	19.0	1,097.2	15.2
Full ARS	57.9 $\pm$ 0.4	66.9	24.9	980.6	21.1

Table 4: Workflow ablations on LIBERO-Long. Input/output token counts are in thousands (K).

Configuration	Success (%) $\uparrow$
Demo + all rollouts BC	50.5
Demo-only BC	58.0
ARS	72.0
ARS + visual reference	81.5

Table 5: Real-world policy learning.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Main instructions
Shared instructions	Ground visual claims in inspected trajectory media. Use the task instruction to identify relevant entities and goal relations. Describe visible state changes and localize events using original frame indices. Request focused temporal windows and spatial crops when additional evidence could change the judgment. Express uncertainty through confidence scores and concise explanations.
Event proposal	Construct an event timeline covering the entire trajectory. Organize intervals around meaningful changes in task state, including preparation, control acquisition, sustained manipulation, completion, recovery, and periods without progress. Associate each event with its frame interval, supporting visual evidence, and confidence.
Independent verification	Treat proposed events as provisional claims. Acquire visual evidence during verification to examine each event’s description, temporal boundaries, and confidence. Inspect neighboring frames to assess boundary placement. Revise, split, or merge events when necessary, and confirm each resulting event before proceeding.
Progress estimation	For sequential tasks, infer subgoals and estimate total progress from their currently achieved contributions, including partial completion. Assign limited credit to useful preparation, localized increases to verified achievements, and gradual increases to continued advancement. Use plateaus for preserved states and decreases for deterioration. Later observations may verify an earlier event; assign credit to the supported event boundary. Preserve meaningful partial achievement in incomplete trajectories.
Candidate-curve auditing	When an external progress curve is supplied, inspect its increases, decreases, and terminal values against the visual evidence. Retain supported estimates and revise unsupported intervals. Without a candidate curve, generate progress values for the complete trajectory.
Optional auxiliary context	Use terminal outcome labels to constrain completion judgments while determining event timing from visual evidence. Use reference examples to interpret task states and goal relations. Apply reference progress values only when the corresponding state is visibly supported in the evaluated trajectory.

Appendix

Table 6: Main instructions used by ARS, summarized by component.

Tool	Stage	Function
read_robot_instruction	E, V, P	Retrieve the task instruction and any explicitly supplied task clarification.
read_metadata	E, V, P	Retrieve technical video metadata, including frame count, frame dimensions, presentation frame rate, and frame-indexing conventions. This tool provides no additional task annotations or simulator state.
inspect_visual	E, V, P	Retrieve a selected frame interval as a video clip or image tile, with configurable temporal sampling and an optional spatial crop. Return the media with its evidence identifier and frame indices.
read_observations	E, V, P	Retrieve the current event timeline and coverage diagnostics.
add_observations revise_observations	E, V	Create or revise event intervals, descriptions, evidence links, and confidence scores.
merge_observations	V	Combine events that describe one continuous phase.

Appendix

Table 7: Tools available in the default ARS workflow.

Proposed event	Frame interval
Initial state and approach to the stove knob	$[0,94]$
Knob grasping and turning	$[95,127]$
Burner switching off and withdrawal	$[128,180]$
Repositioning and approach to the moka pot	$[181,269]$
Moka pot grasp acquisition	$[270,295]$
Lifting and transporting the moka pot	$[296,339]$

Appendix

Table 8: Subagent-proposed event timeline for the LIBERO trajectory in Figure 2 , before primary-agent verification.

Method	Exposure	Evidence
LRM	N	ManiSkill is described as zero-shot to the reward models ( Wu et al., 2026 ) .
R 2 VLM	N	No ManiSkill training is reported for the evaluated checkpoint ( Zhang et al., 2026d ) .
SOLE-R1	N	No ManiSkill training source is identified in the published training mixture ( Schroeder et al., 2026 ) .
Robometer	$\mathbb{E}$	RBM-1M includes FAILSafe trajectories collected in ManiSkill ( Liang et al., 2026 ) .
TOPReward	N	No ManiSkill training is reported in the Qwen model cards; no additional reward-model training is performed ( Chen et al., 2026b ) .
ARS	N	No ManiSkill training is reported in the Qwen model cards; no additional reward-model training is performed.

Appendix

Table 9: Training-data exposure of reward models to ManiSkill.

Method	$0$	$10^{-4}$	$10^{-3}$	$0.005$	$0.01$
LRM	$0.893\,/\,0.159$	$0.893\,/\,0.159$	$0.893\,/\,0.159$	$0.893\,/\,0.159$	$0.893\,/\,0.159$
R 2 VLM	$0.184\,/\,0.791$	$0.184\,/\,0.791$	$0.184\,/\,0.791$	$0.184\,/\,0.791$	$0.190\,/\,0.786$
SOLE-R1 , $[-1,1]$	$0.361\,/\,0.731$	$0.361\,/\,0.731$	$0.361\,/\,0.731$	$0.361\,/\,0.731$	$0.371\,/\,0.729$
Robometer	$0.240\,/\,0.978$	$0.242\,/\,0.977$	$0.246\,/\,0.976$	$0.273\,/\,0.971$	$0.306\,/\,0.965$
TOPReward -8B, linear	$0.368\,/\,0.656$	$0.383\,/\,0.642$	$0.390\,/\,0.622$	$0.452\,/\,0.560$	$0.508\,/\,0.509$
TOPReward -32B, linear	$0.383\,/\,0.723$	$0.412\,/\,0.698$	$0.447\,/\,0.660$	$0.585\,/\,0.526$	$0.693\,/\,0.406$

Appendix

Table 10: Tolerance sensitivity on the semantic-mismatch benchmark. Each entry is FPSR / EPCR, both on a

[0,1]

scale, using native predictions. TOPReward-8B and -32B use Qwen3-VL. ARS and Audit use Qwen3.6-27B. TOPReward probability outputs are reported in the diagnostics section.

Method	Exposure	Evidence
LRM	$\mathbb{E}$	LIBERO is an explicit source in the training mixture ( Wu et al., 2026 ) .
R 2 VLM	N	No LIBERO training is reported for the evaluated checkpoint ( Zhang et al., 2026d ) .
SOLE-R1	$\mathbb{E}$	Supervised fine-tuning includes 504,544 ecot_libero_all examples ( Schroeder et al., 2026 ) .
Robometer	$\mathbb{E}$	RBM-1M includes LIBERO-{Long, Object, Spatial, Goal} and generated failures ( Liang et al., 2026 ) .
TOPReward	N	No LIBERO training is reported in the Qwen model cards; no additional reward-model training is performed ( Chen et al., 2026b ) .
ARS	N	No LIBERO training is reported in the Qwen model cards; no additional reward-model training is performed.

Appendix

Table 11: Training-data exposure of reward models to LIBERO.

Original instruction	Appended clarification
put both the alphabet soup and the tomato sauce in the basket	alphabet soup: blue-and-orange can with alphabet-letter soup graphics. tomato sauce: red/green/white can with tomato pictures.
put both the alphabet soup and the cream cheese box in the basket	alphabet soup: blue-and-orange can with alphabet-letter soup graphics. cream cheese: white rectangular box with a large purple-and-blue label.
put both the cream cheese box and the butter in the basket	cream cheese: white rectangular box with a large purple-and-blue label. butter: red-orange rectangular box with a blue-and-white panel.
put the white mug on the left plate and put the yellow and white mug on the right plate	Plate sides are robot-relative: white mug goes to robot-left (image right); yellow-and-white mug goes to robot-right (image left).

Appendix

Table 12: Task instruction clarifications used on LIBERO-Long. The same additions are provided to ARS and every compared baseline. Each addition is appended to the original instruction under the heading “Task-specific hints:”.

Method	Success (%) $\uparrow$
Vanilla BC	40.1 $\pm$ 1.8
LRM	36.8 $\pm$ 1.5
R 2 VLM	34.1 $\pm$ 1.1
SOLE-R1	40.4 $\pm$ 2.8
Robometer	42.1 $\pm$ 2.3
TOPReward + Qwen3-VL-8B	37.9 $\pm$ 2.5

Appendix

Table 13: Policy-learning results on LIBERO-Long at the final 30k-step checkpoint. Entries report mean success rate

\pm

standard deviation over the same three training runs per method as in Table 2 .

Configuration	Queries	Input (K)	Output (K)	Input / query (K)
Full ARS : subagent	5.8	187.0	3.9	32.5
Full ARS : primary agent	19.2	793.6	17.1	41.4
Full ARS : total	24.9	980.6	21.1	39.3
ARS w/o context separation	19.0	1,097.2	15.2	57.6

Appendix

Table 14: Per-agent inference usage of full ARS on LIBERO-Long, with the single-context control for comparison. Queries and token counts are means per trajectory. Input per query is aggregate input tokens divided by model queries.

Configuration	VOC $\uparrow$	FPSR $\uparrow$	CMPG $\uparrow$
ARS w/o verification & inspection	0.991	0.915	0.895
ARS w/o verification	0.995	0.916	0.910
ARS w/o context separation	0.993	0.913	0.910
Full ARS	0.995	0.918	0.904

Appendix

Table 15: Additional workflow ablations on the semantic-mismatch benchmark using Qwen3.6-27B.

Frame order	Success (%) $\uparrow$	Retained (%)
Chronological	0.0	3.58
Shuffled	$34.1\pm 1.1$	14.80

Appendix

Table 16: GVL-style single-prompt inference with Qwen3.6-27B on LIBERO-Long, using all trajectory frames. Retained is the percentage of action chunks kept for policy training.

Thinking mode	Semantic-mismatch			LIBERO-Long
Thinking mode	VOC $\uparrow$	FPSR $\uparrow$	CMPG $\uparrow$	Success (%) $\uparrow$	Retained (%)
Disabled	0.990	0.915	0.877	55.4 $\pm$ 0.7	62.0
Enabled	0.995	0.918	0.904	57.9 $\pm$ 0.4	66.9

Appendix

Table 17: Effect of the base model’s thinking mode under the same ARS workflow with Qwen3.6-27B. LIBERO-Long uses no outcome conditioning; Retained is the percentage of chunks kept from the original dataset.

Explore similar work

RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation
Jun 20, 2026Pengzhi Yang, Xinyu Wang, Pengyu Jing +7Robotic Manipulation Progress Reward Modeling
† NUS Human-Centered Robotic Lab · ‡ Booking.com · § University of Cambridge +1
Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
Mar 17, 2026Yanru Wu, Weiduo Yuan, Esteban Martinez Licon +4Robotic Manipulation Policies Robot Systems
USC Physical Superintelligence Lab
Progress Reward Modeling for Robotic Learning: A Comprehensive Survey
Jul 22, 2026Jianshu Zhang, Keliang Wu, Haoran Lu +8Progress Reward Modeling Scalable Robot Learning
Northwestern University · Carnegie Mellon University · University of Wisconsin–Madison +2

Jun 20, 2026, cs.ROJ/K move · Enter open · S save

RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation

Pengzhi Yang, Xinyu Wang, Pengyu Jing, Kehan Wen+6

† NUS Human-Centered Robotic Lab · ‡ Booking.com · § University of Cambridge+1

Robotic Manipulation Progress Reward Modeling

Open paper AskPDF arXiv

Reinforcement learning for robot manipulation is often bottlenecked by reward design, especially in long-horizon tasks: sparse success rewards provide weak supervision, while hand-crafted dense rewards are tedious to design and generalize poorly across tasks. Progress-based reward models offer a promising alternative by estimating how far an observation has advanced toward task completion, but existing approaches often require task-specific demonstrations or progress labels, and can assign high rewards to visually plausible but physically incorrect states. We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward. RARM is trained once on general-purpose videos with a contrastive temporal objective, requiring no robot-specific data, task-specific reward labels, or per-task reward engineering. At deployment, RARM matches rollout clips to reference clips and rewards only confident forward progress, suppressing uncertain matches that may otherwise produce false-positive rewards. Across 9 simulated manipulation tasks from LIBERO and MetaWorld and 4 real-world tasks, RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding, where unreliable progress estimates are especially harmful.

Abstract

Explore similar work

RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

Progress Reward Modeling for Robotic Learning: A Comprehensive Survey