cs.ROOct 1, 2026

CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization

Authors: Jiaxuan Luo, Xingguo Xu, Shanshan Wang, Yuhan Zhou, Zhen Zhang

Organizations: Johns Hopkins University · Dalian University of Technology · Rutgers University · Baidu · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Learned visual reward models are increasingly used to optimize robot policies, yet a reward model can score an execution that acts on the wrong object as highly as one that completes the task. We show that optimizing such a reward can amplify these wrong-object failures while reward and task success both rise, so the signals a practitioner would normally monitor look healthy. We fine-tune every denoiser parameter of a diffusion policy against Robometer on a drawer task. Starting from a supervised policy with no prior reward exposure, five training runs raise task success by 10.2 percentage points and wrong-object failures by 10.9 points on 512 evaluation seeds, whereas five runs trained on the simulator's task-completion signal raise success without amplifying wrong-object failures (difference 9.2 points, 95% CI 5.6 to 13.0). The amplification recurs from a policy previously optimized against learned rewards, under the policy's native diffusion sampler, at matched distance from the initial policy, and across constrained-policy experiments with two critics and two optimizers. A tilt model explains when it occurs: under KL-regularized optimization, an outcome becomes more frequent whenever its expected reward under the initial policy exceeds the population average. Robometer separates successes from failures well overall (AUROC .81) but scores wrong-object failures slightly above successes (AUROC .37), so optimization raises both. The same model predicts the outcome shifts across 26 constrained settings (Spearman .89), including those in which task success falls, and Robometer's own published success-termination recipe inherits the error. A frozen outcome verifier redirects the same optimization toward the requested task.

Figures & tables

Figure 1: High visual reward can accompany the wrong object. Critic-only training opens the upper distractor while the target stays closed; critic–verifier training opens the requested middle drawer. Both receive high scores from the original Robometer, shown on the same zero-to-one scale. Frames are the final critic inputs for the first prespecified evaluation seed in constrained-policy PPO (Section 4.2 ), chosen independently of outcome; counts and intervals are in Section 5.2 .
ComparisonΔ\Delta Success (pp)Δ\Delta Wrong-object (pp)Δ\Delta Robometer
Supervised initialization: five trainings per reward
Robometer+10.2[6.9, 13.6]+10.9[7.3, 14.4]+.137[.107, .167]
Terminal completion+13.9[9.5, 18.3]+1.6[ −- 1.9, 5.2]+.062[.029, .096]
Robometer −- completion−- 3.7[ −- 7.5, 0.1]+9.2[5.5, 13.0]+.075[.043, .107]
Warm start: five trainings per reward, three dense-completion oracles
Robometer (critic)+8.4[4.9, 11.9]+3.8[0.8, 6.8]+.061[.036, .086]
Table 1: Full-denoiser evaluation on 512 shared seeds per study. Entries are pooled iteration-40 changes from each study’s common initialization, or differences between rewards, with 95% crossed bootstrap intervals over trainings and evaluation seeds; contrasts resample the two families independently. A wrong-object failure leaves the requested drawer closed while opening the distractor by more than 20 mm. Per-training results are in Appendices B.2 and C.2 .
Figure 2: Reward gain and wrong-object amplification develop together. Mean Robometer score, task success and wrong-object failure across five warm-start training runs per objective on the drawer task, at five checkpoints evaluated on the same 512 seeds. Shading shows 95% pointwise crossed-bootstrap intervals over runs and seeds.
Figure 3: A semantic inversion hidden by other failures. Score distributions over all 20,992 warm-start executions on the 512-seed bank (the initial policy and iterations 10–40 of the ten critic and Product runs), grouped by physical outcome. Panel annotations give the success–wrong-object AUROC; bars mark medians.
Figure 4: One tilt predicts outcome shifts across protocols. Each colored marker is one constrained stratum: the selector’s KL-regularized optimum over the 24 noise sequences, with per-sequence reward and outcome rates estimated from executions outside the critic arm and the tilt strength matched to the reward gain the critic arm reached, against the measured critic-minus-uniform change. Stars are the full-denoiser studies (supervised Robometer and completion, warm-start critic), from reweighting the initial policy’s 512 episodes. The gray line is equality.
Figure 5: Real-robot outcome composition after simulation fine-tuning. Stacked bars show the percentage of 40 physical trials per policy ending in success, wrong-object failure or other failure. Intervals and Fisher tests are in Table 8 .
Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

ClaimEvidence and controlScope
Useful learning and wrong-object amplification coexistFive critic trainings, 512 shared seeds; Table 1 . Oracle controls in Appendix C .Full-denoiser optimization from a reward-exposed drawer policy under the fixed evaluation protocol.
Prior critic exposure is not requiredThree trainings per objective from supervised initialization, 512 seeds; Appendix C.2 .A layout selected by a fixed capability screen.
Verification redirects behaviorFive product trainings versus critic, 512 shared seeds; verifier-only controls in Appendices C and M.1 .Product–critic contrasts under the fine-tuning sampler and matched verifier-only controls with task-specific outcome supervision.
The pattern recurs across protocolsDrawer Robometer and distinct-color LIV stacking under PPO and GRPO; Appendix D .Fixed noise supports and task-specific verifiers; complete task–critic grid in Appendix K .
Aggregate AUROC can mask semantic misrankingCollected-trajectory audit; Section 5.3 and Appendix B.5 .Descriptive reward ordering on collected trajectories from the evaluated policies.
Appendix
Table 2: Evidence supporting the main claims. Controls and diagnostics answer different questions and are not pooled into the primary endpoint.
Figure 6: Task improvement can coexist with wrong-object amplification. Each small marker is one full-denoiser fine-tuning run on the occluded drawer task, placed by its change in task success and in wrong-object failure from the shared initialization to iteration 40, in percentage points on the same 512 evaluation seeds. A wrong-object failure opens the upper distractor drawer without completing the requested middle drawer. Critic runs (circles) train on the Robometer critic alone; Product runs (triangles) train on the critic multiplied by a frozen success verifier. Open diamonds mark the mean of the five runs of each reward. Their horizontal and vertical bars are marginal 95% intervals for the success and wrong-object changes respectively, resampling runs and evaluation seeds; together they do not form a joint confidence region.
Figure 7: All ten fine-tuned policies, each with its own intervals. Each row compares one full-denoiser fine-tuning run at iteration 40 with the shared initialization on the same 512 occluded-drawer seeds; parentheses give its successes out of 512 at initialization and iteration 40. Panels show the changes in task success and wrong-object failure, in percentage points, and in Robometer score. A wrong-object failure opens the upper distractor drawer without completing the requested middle drawer. Critic runs train on the Robometer critic alone; Product runs train on the critic multiplied by a frozen success verifier. The last three rows pool the five runs of each reward and contrast the two. Bars are 95% intervals, over evaluation seeds for single runs and over runs and seeds for pooled rows.
Figure 8: Individual full-denoiser training histories. Robometer score, task success and wrong-object failure on the same fixed 512-seed evaluation bank. C1–C5 are the five critic-only histories; P1–P5 use critic ×\times frozen success verifier (Product). Each line averages one history’s 512 executions at each checkpoint; all histories share the initial policy. Lines connect evaluated checkpoints without smoothing. The primary endpoint is iteration 40.
  RobometerVerifierProduct
PanelkkSWOSWOSWO
All executions16313016001600
 642341063106400
 256100156025420246100
Final policies1688016001600
 642737063106220
Appendix
Table 3: Physical outcomes among the highest-scored executions. Counts of successes (S), wrong-object failures (W) and other failures (O) among the top kk executions ranked by each score. The upper panel uses all 20,992 warm-start executions on the 512-seed bank; the lower panel the 5,120 executions of the ten iteration-40 critic and Product policies.
PolicyΔ\Delta Success (pp)Δ\Delta Wrong-object (pp)Δ\Delta Completion
Dense oracle A+14.5[10.5, 18.4]−- 0.2[ −- 3.3, 2.9]+.127[.092, .161]
Dense oracle B+13.1[9.4, 16.8]−- 2.1[ −- 5.1, 0.8]+.134[.101, .168]
Dense oracle C+12.3[8.4, 16.2]−- 2.3[ −- 5.3, 0.6]+.123[.089, .157]
Terminal oracle+19.1[15.0, 23.2]−- 1.0[ −- 4.5, 2.3]+.176[.140, .213]
Verifier only (V1)+15.2[11.1, 19.3]−- 1.6[ −- 4.7, 1.6]+.146[.109, .183]
Verifier only, six trainings+15.8[12.2, 19.4]−- 1.2[ −- 3.8, 1.4]+.146[.112, .180]
Appendix
Table 4: Learnability and verifier-only controls on the warm-start 512-seed bank. Each control is the iteration-40 policy of one training, compared with the shared initialization on the same 512 seeds, renderer and scorer as Table 1 . Single trainings use paired seed intervals; the pooled rows use crossed intervals over trainings and seeds.
TrainingΔ\Delta Success (pp)Δ\Delta Wrong-object (pp)Δ\Delta Robometer
Robometer 1+9.4[5.5, 13.3]+13.5[9.4, 17.6]+.123[.092, .153]
Robometer 2+9.8[6.1, 13.5]+10.2[6.4, 14.1]+.167[.137, .198]
Robometer 3+9.6[5.9, 13.5]+11.9[8.2, 15.8]+.129[.098, .160]
Robometer 4+11.7[7.6, 15.8]+9.6[5.7, 13.7]+.138[.105, .171]
Robometer 5+10.7[6.6, 14.6]+9.2[5.3, 13.1]+.130[.099, .162]
Completion 1+17.6[12.9, 22.3]+3.9[0.4, 7.4]+.094[.061, .126]
Appendix
Table 5: All sixteen trainings from the supervised initialization on its 512-seed bank. Five trainings per reward form the primary comparison (Table 1 ); an independent repetition of three trainings per reward was evaluated on the same bank. Single trainings use paired seed intervals; pooled rows use crossed intervals, and the contrast resamples the two rewards independently.
Figure 9: Semantic amplification and the verification intervention. Critic minus initialization (orange circles) and product minus critic (blue squares), with 95% paired execution intervals in percentage points. Rows are grouped by task, critic and color condition. These 13 strata contain all nine repeated-error trainings and both trained stacking colors; the near-color strata are retained even when induction is unresolved. All 29 strata, including Robometer stacking and drawer LIV boundaries, appear in Figure 13 . Training numbers identify histories within each task–critic–optimizer group; the two stacking colors share histories. Original IDs and complete counts are in Supplementary Tables 28 and 16 .
TrainingnnInitializationOracleCriticProduct
(a) Original critic score
Drawer / Robometer
PPO 1320.40500.55170.71410.6594
PPO 2320.39030.48170.81020.6249
PPO 3320.33930.42730.77090.6456
GRPO 1640.30080.61520.79370.6314
Appendix
Table 6: Oracle, exploitation and repair across nine training histories. Panel (a) reports mean original critic scores: Robometer success probability for drawer and native LIV cosine for stacking. Panels (b) and (c) report physical task successes and wrong-object failures out of nn evaluation executions per policy. Initialization is the uniform noise-selection policy; Oracle, Critic and Product optimize physical completion, the visual critic, and critic ×\times frozen verifier, respectively. Oracle policies are evaluated with the same visual critic as the other policies. Stacking uses the 2,536-label verifier. Original run IDs are mapped in Table 28 .
  Reward changeTask-success change (count of nn )
TrainingnnCritic −- Init.Oracle −- Init.Critic −- Init.Product −- Critic
Drawer / Robometer
PPO 132++ .3091[ ++ .1915, ++ .4252]++ 20[ ++ 14, ++ 26]−- 1[ −- 5, ++ 3]++ 22[ ++ 17, ++ 27]
PPO 232++ .4199[ ++ .3139, ++ .5218]++ 8[ ++ 3, ++ 14]−- 9[ −- 14, −- 4]++ 23[ ++ 18, ++ 28]
PPO 332++ .4316[ ++ .3184, ++ .5410]++ 12[ ++ 6, ++ 18]−- 4[ −- 8, −- 1]++ 23[ ++ 18, ++ 28]
GRPO 164++ .4929[ ++ .4252, ++ .5601]++ 42[ ++ 34, ++ 50]−- 8[ −- 14, −- 3]++ 52[ ++ 45, ++ 58]
Appendix
Table 7: Oracle learnability, reward exploitation and repair: paired changes. Each row is one training history; entries are paired differences with 95% intervals in brackets. Init. is the uniform noise-selection policy. The reward change is the change in mean original critic score under critic-only training: Robometer success probability for drawer and native LIV cosine for stacking. Success changes are counts out of nn evaluation executions. Seven of nine trainings have a success-loss interval strictly below zero; drawer PPO 1 and GRPO 2 establish amplified wrong-object behavior but not a reliable binary-success loss. Native reward and repair success increase in all nine.
Figure 10: Real-robot deployment
 Outcome rate (%)Change from Initialization (pp)
PolicySuccessWrong-objectOtherSuccessWrong-object
Initialization30.0[18.1, 45.4]22.5[12.3, 37.5]47.5[32.9, 62.5]––
Critic-trained25.0[14.2, 40.2]37.5[24.2, 53.0]37.5[24.2, 53.0]−- 5.0[ −- 23.8, ++ 14.3]++ 15.0[ −- 5.0, ++ 33.5]
Completion-trained40.0[26.3, 55.4]17.5[8.7, 31.9]42.5[28.5, 57.8]++ 10.0[ −- 10.6, ++ 29.5]−- 5.0[ −- 22.4, ++ 12.7]
Appendix
Table 8: Real-robot outcomes. Each policy executes 40 physical drawer trials. Rates are percentages of trials with Wilson 95% intervals; changes relative to Initialization are in percentage points with Newcombe 95% intervals. No change is statistically resolved (two-sided Fisher exact p=.80p=.80 and .22.22 for Critic-trained success and wrong-object, .48.48 and .78.78 for Completion-trained). Success opens the requested middle drawer by at least 6 cm; a wrong-object failure pulls the upper distractor more than 2 cm beyond its initial opening without success. Initialization is the shared warm-start initialization of the full-denoiser study; Critic-trained is Critic 5 at iteration 40; Completion-trained is the terminal-completion oracle of Appendix C.1 at iteration 40.
TaskCritic, decoy colorAmplified strata
DrawerRobometer5/5
StackingLIV, distinct color4/4
StackingLIV, near color0/4
StackingRobometer, near color2/5
StackingRobometer, distinct color0/5
DrawerLIV0/6
Appendix
Table 9: Wrong-object amplification across the constrained grid. A stratum counts as amplified when the paired 95% interval for the critic-minus-uniform change in wrong-object failure lies above zero. The 29 strata come from 20 training histories; the two stacking colors of a training share its history.
   Change in count of nn   
TrainingDecoynnWrong-objectSuccessCritic scoreAmplified
Drawer / Robometer
PPO 1–32++ 15[ ++ 9, ++ 21]−- 1[ −- 5, ++ 3]++ .3091[ ++ .1915, ++ .4252]✓
PPO 2–32++ 24[ ++ 19, ++ 29]−- 9[ −- 14, −- 4]++ .4199[ ++ .3139, ++ .5218]✓
PPO 3–32++ 22[ ++ 17, ++ 27]−- 4[ −- 8, −- 1]++ .4316[ ++ .3184, ++ .5410]✓
GRPO 1–64++ 46[ ++ 39, ++ 53]−- 8[ −- 14, −- 3]++ .4929[ ++ .4252, ++ .5601]✓
Appendix
Table 10: All 29 strata of the constrained grid. Each row is one training history evaluated on one task and critic and, for stacking, one decoy color; the two stacking colors of a training share its history. Entries are critic-minus-uniform changes after optimization with paired 95% intervals: wrong-object failures and successes as counts out of nn evaluation executions, and the original critic score in its native units (Robometer success probability; LIV cosine). A check marks the 11 strata whose wrong-object interval lies above zero.
TrainingΔ\Delta Success (pp)Δ\Delta Wrong-object (pp)Δ\Delta Robometer
Critic 1+8.0[3.7, 12.3]+6.6[2.5, 10.7]+.092[.060, .124]
Critic 2+13.1[8.4, 18.0]+4.9[0.8, 9.0]+.097[.062, .132]
Critic 3+10.4[5.9, 14.8]+7.2[3.1, 11.3]+.075[.042, .108]
Critic 4+13.3[8.8, 17.8]+1.2[ −- 2.7, 5.1]+.085[.052, .118]
Critic 5+9.6[5.5, 13.7]+5.1[1.2, 9.0]+.069[.038, .101]
Critic, pooled+10.9[6.6, 15.2]+5.0[1.0, 8.9]+.084[.054, .114]
Appendix
Table 11: Critic-only policies under the policy’s native sampler. The shared initialization and the five iteration-40 critic policies of Table 1 , decoded with the checkpoint’s 100-step DDPM sampler (standard-deviation floor .05 at the final step) instead of the five-step fine-tuning sampler, on the same 512 seeds, renderer and scorer. The initialization succeeds in 31.4% and fails on the wrong object in 19.3% of these executions.
Family and checkpointTrainingsKL per decisionΔ\Delta Success (pp)Δ\Delta Wrong-object (pp)
Warm start, critic, iteration 1050.46 [0.42, 0.50]+1.1+0.7
Warm start, critic, iteration 2051.01 [0.98, 1.05]+3.1+2.0
Warm start, critic, iteration 3051.62 [1.57, 1.67]+6.2+2.7
Warm start, critic, iteration 4052.15 [2.03, 2.25]+8.4+3.8
Warm start, product, iteration 1050.55 [0.51, 0.60]+3.2−- 0.4
Warm start, product, iteration 2051.20 [1.17, 1.23]+8.6−- 0.7
Appendix
Table 12: Distance from the initial policy and behavior. The KL divergence of each policy’s five-step denoising chain from the initial policy’s, per decision, estimated on that policy’s own 512 evaluation episodes (mean over trainings, range in brackets), beside the changes in success and wrong-object failure on the same seeds.
ThresholdWarm-start criticCritic −- productSupervised RobometerRobometer −- completion
5 mm+3.9[0.7, 7.0]+5.2[2.6, 8.0]+12.1[8.4, 15.9]+10.9[7.0, 14.9]
10 mm+3.7[0.6, 6.8]+5.0[2.4, 7.7]+10.9[7.1, 14.7]+9.6[5.9, 13.6]
15 mm+4.1[1.0, 7.2]+4.8[2.2, 7.6]+10.4[6.7, 14.1]+9.6[6.0, 13.4]
20 mm+3.8[0.8, 6.8]+4.4[1.8, 7.1]+10.9[7.3, 14.4]+9.2[5.5, 13.0]
25 mm+3.9[1.2, 6.8]+4.2[1.8, 6.8]+10.9[7.5, 14.5]+9.0[5.5, 12.5]
30 mm+4.3[1.5, 7.1]+4.5[1.9, 7.1]+10.7[7.3, 14.3]+8.8[5.3, 12.5]
Appendix
Table 13: Wrong-object changes across distractor-opening thresholds. Each entry is the pooled change (pp) in failures that leave the target closed while opening the distractor by more than the threshold, with crossed bootstrap intervals. The paper uses 20 mm.
FamilyReward gainPredicted Δ\Delta SMeasuredPredicted Δ\Delta WMeasured
Supervised, Robometer+.124+9.7[0.1, 0.1]+10.0+7.2[0.1, 0.1]+10.2
Supervised, completion+.163+14.3[0.1, 0.2]+14.5−- 2.5[ −- 0.0, −- 0.0]+1.9
Warm start, critic+.061+4.1[0.0, 0.0]+8.4+3.9[0.0, 0.0]+3.8
Warm start, product+.203+27.7[0.3, 0.3]+15.5−- 5.0[ −- 0.1, −- 0.0]−- 0.6
Warm start, verifier only+.339+33.3[0.3, 0.4]+15.9−- 5.8[ −- 0.1, −- 0.0]−- 1.1
Appendix
Table 14: Tilt predictions for the full-denoiser studies. The initial policy’s 512 evaluation episodes are reweighted by exp⁡(λr)\exp(\lambda r) with λ\lambda chosen so that the reweighted mean of the optimized reward equals the gain each family reached at iteration 40. Intervals resample the initial episodes. Changes are in percentage points.
 SuccessWrong-objectOther failure
Executions2,0091,1301,325
Detected as success (probability >> .6)73.4%89.4%1.6%
Mean detection step (of 300)295236297
 AUROC: S vs. W / other / all failures
Terminal success probability0.195 / 0.951 / 0.603
Dense progress with success termination0.125 / 0.935 / 0.562
Appendix
Table 15: Robometer’s published online-RL recipe on the drawer selector. All 4,464 executed drawer episodes of the five Robometer selector trainings, rescored with the recipe: every step pays −1-1 plus the predicted progress, the step whose success probability exceeds .6 pays 00 and ends the episode. The last block tilts the uniform selector by each reward to the KL that the trained terminal-reward selectors reached.
Figure 11: R82 fixed final PPO evaluation: the prespecified first paired seed, all 32 paired scores and complete physical behavior counts. Wrong-object counts include failures regardless of original reward.
Figure 12: All eight R82 PPO feedback batches, 24 executions per objective before each update, sharing the first uniform batch. Reward always denotes original Robometer, including on the product-trained trajectory. Training curves are not substituted for the new-seed final evaluation.
Run / conditionnnΔrC−U\Delta r_{C-U} [95% CI]UUOOCCPP
Drawer / Robometer / PPO
R8232+.309 [+.192,+.425]+.309\,[+.192,+.425]4/924/23/2425/5
R8732+.420 [+.314,+.522]+.420\,[+.314,+.522]11/419/32/2825/3
R8832+.432 [+.318,+.541]+.432\,[+.318,+.541]6/618/42/2825/5
Drawer / Robometer / GRPO
R11464+.493 [+.425,+.560]+.493\,[+.425,+.560]12/854/14/5456/0
Appendix
Table 16: Physical repair in drawer Robometer and stacking LIV. Each policy entry is success/wrong-object count out of nn ; Δr\Delta r is original native reward change. U/O/C/PU/O/C/P denote uniform/oracle/critic/product. Drawer uses 384 labels; stacking uses 2,536 (2,080 beyond the smaller head). Both trained colors and all nine main trainings are retained. The verifier-only ablation appears in Section M.1 ; all budgets and frozen physical intervals appear in Section K .
Figure 13: All 29 task/color strata of the constrained-policy grid, grouped as main repair panels, Robometer stacking and drawer LIV. Orange: critic minus uniform; blue: product minus critic; frozen 95% paired intervals. Robometer stacking and drawer LIV largely lack induction, and product does not uniformly help there (for example the distinct-color R166 row).
Figure 14: Absolute outcomes of every arm in all 29 constrained-policy strata. Each bar partitions all fresh executions (32 for R82/87/88, 64 otherwise) into success, wrong-object failure and other failure; widths are proportions, not uncertainty. N/D: near/distinct trained color; *: normalized PPO; NA: no matched verifier-only control. The last column marks whether the critic-minus-uniform native-reward interval is strictly positive (+) or unresolved (?); it is not a hacking verdict.
Figure 15: Product minus verifier-only in the 26 strata with a matched verifier-only control, with frozen 95% paired intervals. Retaining the original reward raises success in several distinct-color Robometer-stacking strata, lowers it in R166 near and in drawer LIV R182 and R178, and is indistinguishable elsewhere; NA marks the three drawer PPO trainings without that control.
Run / conditionnnLabelsUUOOCCPPVV
Drawer / Robometer / PPO
R82323844/924/23/2425/5–
R873238411/419/32/2825/3–
R88323846/618/42/2825/5–
Drawer / Robometer / GRPO
R1146438412/854/14/5456/055/0
Appendix
Table 17: Every final common-protocol endpoint and supervision budget. Cells are success / total wrong-object counts. N/D denote near/distinct cube colors, both trained. VV is the identical-head verifier-only control; – denotes a missing matched control. Expanded cube heads add 2,080 labels.
Figure 16: All Robometer physical contrasts with their original paired 95% intervals. Success and total wrong-object changes are separate; ++ indicates a native-reward-gain interval above zero. NA denotes a missing identical-budget only control. Both cube colors are trained. The rows retain near-color failures and both signs of distinct-color GRPO effects; row count is not independent-training count.
Figure 17: All LIV physical contrasts and original paired 95% intervals, including both cube label budgets and all six drawer panels. Distinct-color cube repair repeats, but an added original-LIV contribution over the identical verifier is not established. Near-color and drawer negative inductions remain visible. Drawer wrong-opening excludes upper closing; continuous movement is reported in Section L.3 .
Static metric (initial policy)Full panel ρ\rhoStrataDisjoint halves ρ\rho (combined)Strata (A →\to B / B →\to A)
S1 AUROC, success vs all failures−.34 [−.63,+.05]-.34\,[-.63,+.05]29−.31 [−.58,+.08]-.31\,[-.58,+.08]28 / 28
S2 AUROC, success vs wrong-object−.82 [−.91,−.68]-.82\,[-.91,-.68]29−.74 [−.82,−.61]-.74\,[-.82,-.61]25 / 27
S3 standardized margin+.72 [+.51,+.87]+.72\,[+.51,+.87]29+.63 [+.41,+.76]+.63\,[+.41,+.76]29 / 29
S4 top-quartile enrichment+.87 [+.72,+.95]+.87\,[+.72,+.95]29+.83 [+.70,+.90]+.83\,[+.70,+.90]29 / 29
ρ\rho (S2) −- ρ\rho (S1), common strata−.48 [−.80,−.20]-.48\,[-.80,-.20]29−.34 [−.64,−.14]-.34\,[-.64,-.14]–
Appendix
Table 18: Spearman correlation between each initial-policy static metric and the wrong-object change after critic optimization, across induction strata, with 95% bootstrap intervals over 20 training clusters. The full panel uses the same initial-policy episodes for the metric and the change; the disjoint-halves column computes them on label-blind halves of each panel in both directions and averages the two. The difference row is not pre-declared in the original analysis plan.
Figure 18: Wrong-object change after critic optimization against the initial-policy generic AUROC (left) and semantic AUROC (right), one point per stratum, marked by task and critic.
Figure 19: Training feedback of the whole-network arms: raw values of each 126-episode batch, without smoothing. Grey lines are the four oracle histories and appear only in the success panel, because oracle feedback was neither scored by the critic nor fully logged for wrong-object failures; the grey column marks the two value-only iterations.
ArmIter.Δ\Delta success (pp) [95% CI]Δ\Delta completion [95% CI]
History A010+0.0 [−6.2,+6.2]+0.0\,[-6.2,+6.2]+.037 [−.009,+.085]+.037\,[-.009,+.085]
 020+8.6 [+2.3,+14.8]+8.6\,[+2.3,+14.8]+.092 [+.035,+.151]+.092\,[+.035,+.151]
 030+7.8 [+0.8,+14.8]+7.8\,[+0.8,+14.8]+.132 [+.061,+.204]+.132\,[+.061,+.204]
 040+10.9 [+3.1,+19.5]+10.9\,[+3.1,+19.5]+.152 [+.080,+.226]+.152\,[+.080,+.226]
History B010+10.9 [+5.5,+17.2]+10.9\,[+5.5,+17.2]+.095 [+.050,+.144]+.095\,[+.050,+.144]
 020+19.5 [+12.5,+27.3]+19.5\,[+12.5,+27.3]+.180 [+.121,+.241]+.180\,[+.121,+.241]
Appendix
Table 19: Physical-feedback arms of the whole-network fine-tuning study, each iteration against that arm’s own iteration-0 weights on its 128 paired evaluation seeds. Only the iteration-40 row enters the pre-declared verdict.
Figure 20: Every execution in the separate checkpoint-diagnostic panel: the shared initial-policy set and iterations 10–40 of each learned arm, 128 seeds each. No execution is dropped or reordered by outcome.
TrainingUUOOCCPPInterpretation
R853094053Induction and partial repair
R93319443Induction; repair fails
R9428696784Rewarded errors coexist with task learning
Appendix
Table 20: All three initial RoboReward NPG trainings, success counts out of 96. Two improve under the product; the failure is retained. These are separate protocols from the common PPO/GRPO matrix.
Run / conditionnnΔrC−U\Delta r_{C-U} [95% CI]UUOOCCPP
Drawer / LIV / PPO
R182 / original64+.0023 [+.0002,+.0049]+.0023\,[+.0002,+.0049]14/1158/217/1053/5
R178 / original64−.0001 [−.0008,+.0006]-.0001\,[-.0008,+.0006]12/1345/011/1548/5
R187 / norm.64+.0222 [+.0194,+.0250]+.0222\,[+.0194,+.0250]14/1060/27/063/0
R188 / norm.64+.0220 [+.0193,+.0245]+.0220\,[+.0193,+.0245]14/643/315/158/4
Drawer / LIV / GRPO
Appendix
Table 21: All six drawer LIV panels, using the same 384-label verifier. Entries are success/wrong-opening counts out of 64. R178 has no established native gain. Normalized PPO (norm.) differs from original-scale PPO. Closing is excluded from wrong-opening counts; continuous motion and exact intervals are retained in Section L.3 .
RunProtocolΔSC−U\Delta S_{C-U} /64ΔqT\Delta q_{T} (mm)ΔqD\Delta q_{D} (mm)
R182PPO+3 [−1,+7]+3\,[-1,+7]+3.92 [−2.11,+10.22]+3.92\,[-2.11,+10.22]−4.55 [−10.49,+1.09]-4.55\,[-10.49,+1.09]
R178PPO−1 [−3,+0]-1\,[-3,+0]−1.26 [−4.26,+.73]-1.26\,[-4.26,+.73]+2.47 [−.08,+6.21]+2.47\,[-.08,+6.21]
R187Norm. PPO−7 [−13,−1]-7\,[-13,-1]+13.38 [+3.09,+23.44]+13.38\,[+3.09,+23.44]−45.21 [−53.85,−36.76]-45.21\,[-53.85,-36.76]
R188Norm. PPO+1 [−9,+11]+1\,[-9,+11]+21.02 [+10.40,+31.10]+21.02\,[+10.40,+31.10]−36.57 [−45.58,−28.44]-36.57\,[-45.58,-28.44]
R183GRPO−4 [−12,+3]-4\,[-12,+3]+10.02 [−.80,+20.53]+10.02\,[-.80,+20.53]−47.31 [−56.07,−38.72]-47.31\,[-56.07,-38.72]
R179GRPO+2 [−6,+11]+2\,[-6,+11]+21.96 [+14.77,+29.41]+21.96\,[+14.77,+29.41]−39.79 [−46.82,−32.96]-39.79\,[-46.82,-32.96]
Appendix
Table 22: All six drawer LIV movement contrasts and original paired 95% intervals. Positive target change is more opening; negative upper net-motion change is more closing, which is excluded from the wrong-opening event. R183 target-opening uncertainty crosses zero despite its positive mean.
Development familyOutcome and boundary
Reward-weighted denoiser regressionInitial mitigation does not persist across the second training. It does not establish neural-decoder RL repair.
Noise-policy SACOracle also degrades; the control does not validate a learnable family in which learned reward is specifically to blame.
RoboReward cube PPO/GRPOComplete later panels do not induce the targeted original high-score failures. Product task improvement is not hacking repair.
Terminal-feature replacementDoes not reliably repair identity errors; late policy observations and final critic images are different inputs.
Unbalanced-color trainingNegative transfer and supervision effects are retained as development boundaries. Final balanced-color evidence does not establish unseen-color or unseen-task transfer.
Appendix
Table 23: Negative findings relevant to interpreting the final method. The current manuscript does not substitute these protocols for the complete common PPO/GRPO endpoints.
Run / colorUUOOCCPPVVΔSP−V\Delta S_{P-V}
Stacking / Robometer / PPO
R164 / near20/1938/2126/3238/1830/25+8 [+2,+14]+8\,[+2,+14]
R164 / distinct20/1759/360/161/048/16+13 [+9,+16]+13\,[+9,+16]
R169 / near27/2343/1737/2541/1936/20+5 [−1,+11]+5\,[-1,+11]
R169 / distinct27/2263/161/362/245/18+17 [+13,+21]+17\,[+13,+21]
R175 / near24/1540/1330/2832/2039/17−7 [−14,+0]-7\,[-14,+0]
Appendix
Table 24: Robometer stacking with the 456-label head: success/wrong-object counts out of 64. The last column is the paired product-minus-only success-count difference and its 95% interval. Both colors participate in each training. Only controls for R164/R166/R184 come from matched R168/R170/R189 follow-ups; they do not add independent replications. Expanded-head results are retained in Section K .
WindowTest AUROCBrierTP / 24FP / 88
Full episode1.000.0022240
Final quarter.718.2225918
Appendix
Table 25: Ballot visual-window confirmation on new object positions and terminal postures. Each head is fitted only on the 112 training executions; the acceptance threshold is .5.
Figure 21: Original ballot-control frames from the first test position/posture group, using release cells selected by training-only critic (top) and product (bottom) reward. By withdrawal, the successful ball is inside; failure leaves a visible terminal ball. Final-quarter verification misses this success. This pair illustrates window-dependent prediction, not terminal unobservability or online PPO performance; Tables 25 and 26 report the complete panels.
 Training 1Training 2
ObjectiveCritic rewardSuccess / 64Critic rewardSuccess / 64
Uniform UU.039012.038312
Oracle OO.047163.046764
Original reward CC.06472.06815
Full-history product PP.046863.046664
Verifier-only VV.046863.046964
Appendix
Table 26: New-seed endpoints for online ballot-release PPO. This controlled extension trains a 14-action scripted release selector, with two independent histories per objective; it is separate from the common frozen-decoder noise-policy matrix. The last row is a paired follow-up changing only the verifier window and its corresponding frozen head.
ParentMeasureUUChC_{h}CCPPVVOO
ID opening .475
R82Success / wrong10/104/583/6061/061/054/6
 Original reward0.3260.8380.8560.6890.6840.662
R87Success / wrong15/104/573/5752/449/554/1
 Original reward0.3300.8130.8090.6130.6020.602
R88Success / wrong10/163/564/5416/3022/2059/2
Appendix
Table 27: Checkpoint continuation: all three prespecified parents, both test openings. UU is uniform, ChC_{h} the frozen hacked parent, and C/P/V/OC/P/V/O the original-reward/product/verifier/oracle continuations. Success/wrong are success and wrong-object counts out of 64 paired new seeds; rewards are original Robometer scores. Both conditions use the same final checkpoint 008; no labels or policy updates are added for the held-out opening.
Figure 22: Repair after exploitation and the value of process evidence. (a) All three exploited parents, with fixed endpoints at training opening .475 and opening .375 held out from continuation; no new verifier labels. (b) Two ballot PPO histories, using the same 112 labels and 64-frame count. Full-history verification improves repair over a final-quarter head. Verifier-only is retained in both panels; full controls and paired intervals are in Sections M.5 and M.4 .
SettingTraining IDOriginal run ID
Drawer / RobometerPPO 1R82
Drawer / RobometerPPO 2R87
Drawer / RobometerPPO 3R88
Drawer / RobometerGRPO 1R114
Drawer / RobometerGRPO 2R151
Stacking / LIVPPO 1R167
Appendix
Table 28: Reader-facing identifiers and original experiment records. Training numbers are local to each task–critic–optimizer group. Both trained stacking colors and all verifier budgets from one training share its ID; they are not independent replications. The two development runs supply data for the expanded stacking verifier.
More in this topic

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 241 papers

Explore similar work

CardsList
Jun 19, 2026

Robot Critics that Sweat the Small Stuff

Large vision-language models contain several priors about the world and object interactions, making them useful critics during inference to steer robot policies towards success. However, closed-loop robot manipulation requires judging small visual differences between success and failure, which remains a challenge for current VLMs. We introduce a method to fine-tune critics by constructing pairwise progress supervision using success and failure rollouts obtained from a policy. Our fine-tuned critic excels at fine-grained progress reasoning and subtle failure detection, outperforming prior progress reasoning baselines. Additionally, we use an action-conditioned video model to predict the visual effect of several candidate actions sampled from a policy, and show that our critic can correctly identify successful candidates to execute, improving the average policy success rate by 11% across real-world tasks and 5.9% across simulation tasks.

Sruthi Sudhakar, Junbang Liang, Sreehari Rammohan +3

Columbia University · Toyota Research Institute

Robot Failure DetectionVLM AdaptationRobotic Policy EvaluationRobotic ManipulationVLMs for Robotics
AskPDF
May 22, 2026

From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models

Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior policy training. We propose Demo2Reward a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3-10 trajectories) to reduce false positives while preserving true positives. Crucially, this requires no additional model training or computation resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function.

Christian Gumbsch, Leonardo Barcellona, Lennard Schünemann +7

University of Amsterdam · 2Catholic University of Leuven · 3Toyota Research Institute +1

RL for RoboticsMultimodal Reward ModelingReward Design for RLTest-Time Prompt Tuning
AskPDF
Aug 19, 2026

Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation

Learning robotic policies requires dense rewards that remain informative when behavior departs from successful demonstrations. Progress-based rewards estimate how far an observation has advanced along a nominal successful trajectory, but may remain high after an incorrect transition. We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations. Given the visual history up to a transition start, the model predicts the latent displacement associated with successful execution and scores the observed displacement through signed directional and symmetric magnitude agreement. This transition-level comparison penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress. Dream2Reward requires no failure annotations, progress labels, or synthetic negatives, and produces a dense causal reward. Across mechanism diagnostics and shared-trajectory evaluations, it provides stronger success-failure separation and more informative feedback on low-quality behavior than progress-based alternatives. Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance, including in real-robot manipulation. These results show that comparing realized motion with predicted successful change provides an effective way to convert positive demonstrations into dense rewards for robot learning.

Haoyu Zhang, Zecui Zeng, Bin Wang +3

School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · JD Explore Academy, China

Reward Design for RLDense Reward RLReinforcement LearningRobotic ManipulationLearning from Demonstration
AskPDF