cs.ROOct 1, 2026
SaveCriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
Organizations: Johns Hopkins University · Dalian University of Technology · Rutgers University · Baidu ·
Abstract
Learned visual reward models are increasingly used to optimize robot policies, yet a reward model can score an execution that acts on the wrong object as highly as one that completes the task. We show that optimizing such a reward can amplify these wrong-object failures while reward and task success both rise, so the signals a practitioner would normally monitor look healthy. We fine-tune every denoiser parameter of a diffusion policy against Robometer on a drawer task. Starting from a supervised policy with no prior reward exposure, five training runs raise task success by 10.2 percentage points and wrong-object failures by 10.9 points on 512 evaluation seeds, whereas five runs trained on the simulator's task-completion signal raise success without amplifying wrong-object failures (difference 9.2 points, 95% CI 5.6 to 13.0). The amplification recurs from a policy previously optimized against learned rewards, under the policy's native diffusion sampler, at matched distance from the initial policy, and across constrained-policy experiments with two critics and two optimizers. A tilt model explains when it occurs: under KL-regularized optimization, an outcome becomes more frequent whenever its expected reward under the initial policy exceeds the population average. Robometer separates successes from failures well overall (AUROC .81) but scores wrong-object failures slightly above successes (AUROC .37), so optimization raises both. The same model predicts the outcome shifts across 26 constrained settings (Spearman .89), including those in which task success falls, and Robometer's own published success-termination recipe inherits the error. A frozen outcome verifier redirects the same optimization toward the requested task.
Figures & tables
Figure 1: High visual reward can accompany the wrong object. Critic-only training opens the upper distractor while the target stays closed; critic–verifier training opens the requested middle drawer. Both receive high scores from the original Robometer, shown on the same zero-to-one scale. Frames are the final critic inputs for the first prespecified evaluation seed in constrained-policy PPO (Section 4.2 ), chosen independently of outcome; counts and intervals are in Section 5.2 .
| Comparison | Success (pp) | Wrong-object (pp) | Robometer | |||
| Supervised initialization: five trainings per reward | ||||||
| Robometer | +10.2 | [6.9, 13.6] | +10.9 | [7.3, 14.4] | +.137 | [.107, .167] |
| Terminal completion | +13.9 | [9.5, 18.3] | +1.6 | [ 1.9, 5.2] | +.062 | [.029, .096] |
| Robometer completion | 3.7 | [ 7.5, 0.1] | +9.2 | [5.5, 13.0] | +.075 | [.043, .107] |
| Warm start: five trainings per reward, three dense-completion oracles | ||||||
| Robometer (critic) | +8.4 | [4.9, 11.9] | +3.8 | [0.8, 6.8] | +.061 | [.036, .086] |
Table 1: Full-denoiser evaluation on 512 shared seeds per study. Entries are pooled iteration-40 changes from each study’s common initialization, or differences between rewards, with 95% crossed bootstrap intervals over trainings and evaluation seeds; contrasts resample the two families independently. A wrong-object failure leaves the requested drawer closed while opening the distractor by more than 20 mm. Per-training results are in Appendices B.2 and C.2 .
Figure 2: Reward gain and wrong-object amplification develop together. Mean Robometer score, task success and wrong-object failure across five warm-start training runs per objective on the drawer task, at five checkpoints evaluated on the same 512 seeds. Shading shows 95% pointwise crossed-bootstrap intervals over runs and seeds.
Figure 3: A semantic inversion hidden by other failures. Score distributions over all 20,992 warm-start executions on the 512-seed bank (the initial policy and iterations 10–40 of the ten critic and Product runs), grouped by physical outcome. Panel annotations give the success–wrong-object AUROC; bars mark medians.
Figure 4: One tilt predicts outcome shifts across protocols. Each colored marker is one constrained stratum: the selector’s KL-regularized optimum over the 24 noise sequences, with per-sequence reward and outcome rates estimated from executions outside the critic arm and the tilt strength matched to the reward gain the critic arm reached, against the measured critic-minus-uniform change. Stars are the full-denoiser studies (supervised Robometer and completion, warm-start critic), from reweighting the initial policy’s 512 episodes. The gray line is equality.
Figure 5: Real-robot outcome composition after simulation fine-tuning. Stacked bars show the percentage of 40 physical trials per policy ending in success, wrong-object failure or other failure. Intervals and Fisher tests are in Table 8 .
Appendix figures & tables44 assets
Supplementary material from the paper’s appendix.
Appendix
| Claim | Evidence and control | Scope |
|---|---|---|
| Useful learning and wrong-object amplification coexist | Five critic trainings, 512 shared seeds; Table 1 . Oracle controls in Appendix C . | Full-denoiser optimization from a reward-exposed drawer policy under the fixed evaluation protocol. |
| Prior critic exposure is not required | Three trainings per objective from supervised initialization, 512 seeds; Appendix C.2 . | A layout selected by a fixed capability screen. |
| Verification redirects behavior | Five product trainings versus critic, 512 shared seeds; verifier-only controls in Appendices C and M.1 . | Product–critic contrasts under the fine-tuning sampler and matched verifier-only controls with task-specific outcome supervision. |
| The pattern recurs across protocols | Drawer Robometer and distinct-color LIV stacking under PPO and GRPO; Appendix D . | Fixed noise supports and task-specific verifiers; complete task–critic grid in Appendix K . |
| Aggregate AUROC can mask semantic misranking | Collected-trajectory audit; Section 5.3 and Appendix B.5 . | Descriptive reward ordering on collected trajectories from the evaluated policies. |
Table 2: Evidence supporting the main claims. Controls and diagnostics answer different questions and are not pooled into the primary endpoint.
Figure 6: Task improvement can coexist with wrong-object amplification. Each small marker is one full-denoiser fine-tuning run on the occluded drawer task, placed by its change in task success and in wrong-object failure from the shared initialization to iteration 40, in percentage points on the same 512 evaluation seeds. A wrong-object failure opens the upper distractor drawer without completing the requested middle drawer. Critic runs (circles) train on the Robometer critic alone; Product runs (triangles) train on the critic multiplied by a frozen success verifier. Open diamonds mark the mean of the five runs of each reward. Their horizontal and vertical bars are marginal 95% intervals for the success and wrong-object changes respectively, resampling runs and evaluation seeds; together they do not form a joint confidence region.
Figure 7: All ten fine-tuned policies, each with its own intervals. Each row compares one full-denoiser fine-tuning run at iteration 40 with the shared initialization on the same 512 occluded-drawer seeds; parentheses give its successes out of 512 at initialization and iteration 40. Panels show the changes in task success and wrong-object failure, in percentage points, and in Robometer score. A wrong-object failure opens the upper distractor drawer without completing the requested middle drawer. Critic runs train on the Robometer critic alone; Product runs train on the critic multiplied by a frozen success verifier. The last three rows pool the five runs of each reward and contrast the two. Bars are 95% intervals, over evaluation seeds for single runs and over runs and seeds for pooled rows.
Figure 8: Individual full-denoiser training histories. Robometer score, task success and wrong-object failure on the same fixed 512-seed evaluation bank. C1–C5 are the five critic-only histories; P1–P5 use critic frozen success verifier (Product). Each line averages one history’s 512 executions at each checkpoint; all histories share the initial policy. Lines connect evaluated checkpoints without smoothing. The primary endpoint is iteration 40.
| Robometer | Verifier | Product | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Panel | S | W | O | S | W | O | S | W | O | |
| All executions | 16 | 3 | 13 | 0 | 16 | 0 | 0 | 16 | 0 | 0 |
| 64 | 23 | 41 | 0 | 63 | 1 | 0 | 64 | 0 | 0 | |
| 256 | 100 | 156 | 0 | 254 | 2 | 0 | 246 | 10 | 0 | |
| Final policies | 16 | 8 | 8 | 0 | 16 | 0 | 0 | 16 | 0 | 0 |
| 64 | 27 | 37 | 0 | 63 | 1 | 0 | 62 | 2 | 0 | |
Table 3: Physical outcomes among the highest-scored executions. Counts of successes (S), wrong-object failures (W) and other failures (O) among the top executions ranked by each score. The upper panel uses all 20,992 warm-start executions on the 512-seed bank; the lower panel the 5,120 executions of the ten iteration-40 critic and Product policies.
| Policy | Success (pp) | Wrong-object (pp) | Completion | |||
| Dense oracle A | +14.5 | [10.5, 18.4] | 0.2 | [ 3.3, 2.9] | +.127 | [.092, .161] |
| Dense oracle B | +13.1 | [9.4, 16.8] | 2.1 | [ 5.1, 0.8] | +.134 | [.101, .168] |
| Dense oracle C | +12.3 | [8.4, 16.2] | 2.3 | [ 5.3, 0.6] | +.123 | [.089, .157] |
| Terminal oracle | +19.1 | [15.0, 23.2] | 1.0 | [ 4.5, 2.3] | +.176 | [.140, .213] |
| Verifier only (V1) | +15.2 | [11.1, 19.3] | 1.6 | [ 4.7, 1.6] | +.146 | [.109, .183] |
| Verifier only, six trainings | +15.8 | [12.2, 19.4] | 1.2 | [ 3.8, 1.4] | +.146 | [.112, .180] |
Table 4: Learnability and verifier-only controls on the warm-start 512-seed bank. Each control is the iteration-40 policy of one training, compared with the shared initialization on the same 512 seeds, renderer and scorer as Table 1 . Single trainings use paired seed intervals; the pooled rows use crossed intervals over trainings and seeds.
| Training | Success (pp) | Wrong-object (pp) | Robometer | |||
|---|---|---|---|---|---|---|
| Robometer 1 | +9.4 | [5.5, 13.3] | +13.5 | [9.4, 17.6] | +.123 | [.092, .153] |
| Robometer 2 | +9.8 | [6.1, 13.5] | +10.2 | [6.4, 14.1] | +.167 | [.137, .198] |
| Robometer 3 | +9.6 | [5.9, 13.5] | +11.9 | [8.2, 15.8] | +.129 | [.098, .160] |
| Robometer 4 | +11.7 | [7.6, 15.8] | +9.6 | [5.7, 13.7] | +.138 | [.105, .171] |
| Robometer 5 | +10.7 | [6.6, 14.6] | +9.2 | [5.3, 13.1] | +.130 | [.099, .162] |
| Completion 1 | +17.6 | [12.9, 22.3] | +3.9 | [0.4, 7.4] | +.094 | [.061, .126] |
Table 5: All sixteen trainings from the supervised initialization on its 512-seed bank. Five trainings per reward form the primary comparison (Table 1 ); an independent repetition of three trainings per reward was evaluated on the same bank. Single trainings use paired seed intervals; pooled rows use crossed intervals, and the contrast resamples the two rewards independently.
Figure 9: Semantic amplification and the verification intervention. Critic minus initialization (orange circles) and product minus critic (blue squares), with 95% paired execution intervals in percentage points. Rows are grouped by task, critic and color condition. These 13 strata contain all nine repeated-error trainings and both trained stacking colors; the near-color strata are retained even when induction is unresolved. All 29 strata, including Robometer stacking and drawer LIV boundaries, appear in Figure 13 . Training numbers identify histories within each task–critic–optimizer group; the two stacking colors share histories. Original IDs and complete counts are in Supplementary Tables 28 and 16 .
| Training | Initialization | Oracle | Critic | Product | |
|---|---|---|---|---|---|
| (a) Original critic score | |||||
| Drawer / Robometer | |||||
| PPO 1 | 32 | 0.4050 | 0.5517 | 0.7141 | 0.6594 |
| PPO 2 | 32 | 0.3903 | 0.4817 | 0.8102 | 0.6249 |
| PPO 3 | 32 | 0.3393 | 0.4273 | 0.7709 | 0.6456 |
| GRPO 1 | 64 | 0.3008 | 0.6152 | 0.7937 | 0.6314 |
Table 6: Oracle, exploitation and repair across nine training histories. Panel (a) reports mean original critic scores: Robometer success probability for drawer and native LIV cosine for stacking. Panels (b) and (c) report physical task successes and wrong-object failures out of evaluation executions per policy. Initialization is the uniform noise-selection policy; Oracle, Critic and Product optimize physical completion, the visual critic, and critic frozen verifier, respectively. Oracle policies are evaluated with the same visual critic as the other policies. Stacking uses the 2,536-label verifier. Original run IDs are mapped in Table 28 .
| Reward change | Task-success change (count of ) | ||||||||
| Training | Critic Init. | Oracle Init. | Critic Init. | Product Critic | |||||
| Drawer / Robometer | |||||||||
| PPO 1 | 32 | .3091 | [ .1915, .4252] | 20 | [ 14, 26] | 1 | [ 5, 3] | 22 | [ 17, 27] |
| PPO 2 | 32 | .4199 | [ .3139, .5218] | 8 | [ 3, 14] | 9 | [ 14, 4] | 23 | [ 18, 28] |
| PPO 3 | 32 | .4316 | [ .3184, .5410] | 12 | [ 6, 18] | 4 | [ 8, 1] | 23 | [ 18, 28] |
| GRPO 1 | 64 | .4929 | [ .4252, .5601] | 42 | [ 34, 50] | 8 | [ 14, 3] | 52 | [ 45, 58] |
Table 7: Oracle learnability, reward exploitation and repair: paired changes. Each row is one training history; entries are paired differences with 95% intervals in brackets. Init. is the uniform noise-selection policy. The reward change is the change in mean original critic score under critic-only training: Robometer success probability for drawer and native LIV cosine for stacking. Success changes are counts out of evaluation executions. Seven of nine trainings have a success-loss interval strictly below zero; drawer PPO 1 and GRPO 2 establish amplified wrong-object behavior but not a reliable binary-success loss. Native reward and repair success increase in all nine.
Figure 10: Real-robot deployment
| Outcome rate (%) | Change from Initialization (pp) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Policy | Success | Wrong-object | Other | Success | Wrong-object | |||||
| Initialization | 30.0 | [18.1, 45.4] | 22.5 | [12.3, 37.5] | 47.5 | [32.9, 62.5] | – | – | ||
| Critic-trained | 25.0 | [14.2, 40.2] | 37.5 | [24.2, 53.0] | 37.5 | [24.2, 53.0] | 5.0 | [ 23.8, 14.3] | 15.0 | [ 5.0, 33.5] |
| Completion-trained | 40.0 | [26.3, 55.4] | 17.5 | [8.7, 31.9] | 42.5 | [28.5, 57.8] | 10.0 | [ 10.6, 29.5] | 5.0 | [ 22.4, 12.7] |
Table 8: Real-robot outcomes. Each policy executes 40 physical drawer trials. Rates are percentages of trials with Wilson 95% intervals; changes relative to Initialization are in percentage points with Newcombe 95% intervals. No change is statistically resolved (two-sided Fisher exact and for Critic-trained success and wrong-object, and for Completion-trained). Success opens the requested middle drawer by at least 6 cm; a wrong-object failure pulls the upper distractor more than 2 cm beyond its initial opening without success. Initialization is the shared warm-start initialization of the full-denoiser study; Critic-trained is Critic 5 at iteration 40; Completion-trained is the terminal-completion oracle of Appendix C.1 at iteration 40.
| Task | Critic, decoy color | Amplified strata |
|---|---|---|
| Drawer | Robometer | 5/5 |
| Stacking | LIV, distinct color | 4/4 |
| Stacking | LIV, near color | 0/4 |
| Stacking | Robometer, near color | 2/5 |
| Stacking | Robometer, distinct color | 0/5 |
| Drawer | LIV | 0/6 |
Table 9: Wrong-object amplification across the constrained grid. A stratum counts as amplified when the paired 95% interval for the critic-minus-uniform change in wrong-object failure lies above zero. The 29 strata come from 20 training histories; the two stacking colors of a training share its history.
| Change in count of | |||||||||
| Training | Decoy | Wrong-object | Success | Critic score | Amplified | ||||
| Drawer / Robometer | |||||||||
| PPO 1 | – | 32 | 15 | [ 9, 21] | 1 | [ 5, 3] | .3091 | [ .1915, .4252] | ✓ |
| PPO 2 | – | 32 | 24 | [ 19, 29] | 9 | [ 14, 4] | .4199 | [ .3139, .5218] | ✓ |
| PPO 3 | – | 32 | 22 | [ 17, 27] | 4 | [ 8, 1] | .4316 | [ .3184, .5410] | ✓ |
| GRPO 1 | – | 64 | 46 | [ 39, 53] | 8 | [ 14, 3] | .4929 | [ .4252, .5601] | ✓ |
Table 10: All 29 strata of the constrained grid. Each row is one training history evaluated on one task and critic and, for stacking, one decoy color; the two stacking colors of a training share its history. Entries are critic-minus-uniform changes after optimization with paired 95% intervals: wrong-object failures and successes as counts out of evaluation executions, and the original critic score in its native units (Robometer success probability; LIV cosine). A check marks the 11 strata whose wrong-object interval lies above zero.
| Training | Success (pp) | Wrong-object (pp) | Robometer | |||
|---|---|---|---|---|---|---|
| Critic 1 | +8.0 | [3.7, 12.3] | +6.6 | [2.5, 10.7] | +.092 | [.060, .124] |
| Critic 2 | +13.1 | [8.4, 18.0] | +4.9 | [0.8, 9.0] | +.097 | [.062, .132] |
| Critic 3 | +10.4 | [5.9, 14.8] | +7.2 | [3.1, 11.3] | +.075 | [.042, .108] |
| Critic 4 | +13.3 | [8.8, 17.8] | +1.2 | [ 2.7, 5.1] | +.085 | [.052, .118] |
| Critic 5 | +9.6 | [5.5, 13.7] | +5.1 | [1.2, 9.0] | +.069 | [.038, .101] |
| Critic, pooled | +10.9 | [6.6, 15.2] | +5.0 | [1.0, 8.9] | +.084 | [.054, .114] |
Table 11: Critic-only policies under the policy’s native sampler. The shared initialization and the five iteration-40 critic policies of Table 1 , decoded with the checkpoint’s 100-step DDPM sampler (standard-deviation floor .05 at the final step) instead of the five-step fine-tuning sampler, on the same 512 seeds, renderer and scorer. The initialization succeeds in 31.4% and fails on the wrong object in 19.3% of these executions.
| Family and checkpoint | Trainings | KL per decision | Success (pp) | Wrong-object (pp) |
|---|---|---|---|---|
| Warm start, critic, iteration 10 | 5 | 0.46 [0.42, 0.50] | +1.1 | +0.7 |
| Warm start, critic, iteration 20 | 5 | 1.01 [0.98, 1.05] | +3.1 | +2.0 |
| Warm start, critic, iteration 30 | 5 | 1.62 [1.57, 1.67] | +6.2 | +2.7 |
| Warm start, critic, iteration 40 | 5 | 2.15 [2.03, 2.25] | +8.4 | +3.8 |
| Warm start, product, iteration 10 | 5 | 0.55 [0.51, 0.60] | +3.2 | 0.4 |
| Warm start, product, iteration 20 | 5 | 1.20 [1.17, 1.23] | +8.6 | 0.7 |
Table 12: Distance from the initial policy and behavior. The KL divergence of each policy’s five-step denoising chain from the initial policy’s, per decision, estimated on that policy’s own 512 evaluation episodes (mean over trainings, range in brackets), beside the changes in success and wrong-object failure on the same seeds.
| Threshold | Warm-start critic | Critic product | Supervised Robometer | Robometer completion | ||||
|---|---|---|---|---|---|---|---|---|
| 5 mm | +3.9 | [0.7, 7.0] | +5.2 | [2.6, 8.0] | +12.1 | [8.4, 15.9] | +10.9 | [7.0, 14.9] |
| 10 mm | +3.7 | [0.6, 6.8] | +5.0 | [2.4, 7.7] | +10.9 | [7.1, 14.7] | +9.6 | [5.9, 13.6] |
| 15 mm | +4.1 | [1.0, 7.2] | +4.8 | [2.2, 7.6] | +10.4 | [6.7, 14.1] | +9.6 | [6.0, 13.4] |
| 20 mm | +3.8 | [0.8, 6.8] | +4.4 | [1.8, 7.1] | +10.9 | [7.3, 14.4] | +9.2 | [5.5, 13.0] |
| 25 mm | +3.9 | [1.2, 6.8] | +4.2 | [1.8, 6.8] | +10.9 | [7.5, 14.5] | +9.0 | [5.5, 12.5] |
| 30 mm | +4.3 | [1.5, 7.1] | +4.5 | [1.9, 7.1] | +10.7 | [7.3, 14.3] | +8.8 | [5.3, 12.5] |
Table 13: Wrong-object changes across distractor-opening thresholds. Each entry is the pooled change (pp) in failures that leave the target closed while opening the distractor by more than the threshold, with crossed bootstrap intervals. The paper uses 20 mm.
| Family | Reward gain | Predicted S | Measured | Predicted W | Measured | ||
|---|---|---|---|---|---|---|---|
| Supervised, Robometer | +.124 | +9.7 | [0.1, 0.1] | +10.0 | +7.2 | [0.1, 0.1] | +10.2 |
| Supervised, completion | +.163 | +14.3 | [0.1, 0.2] | +14.5 | 2.5 | [ 0.0, 0.0] | +1.9 |
| Warm start, critic | +.061 | +4.1 | [0.0, 0.0] | +8.4 | +3.9 | [0.0, 0.0] | +3.8 |
| Warm start, product | +.203 | +27.7 | [0.3, 0.3] | +15.5 | 5.0 | [ 0.1, 0.0] | 0.6 |
| Warm start, verifier only | +.339 | +33.3 | [0.3, 0.4] | +15.9 | 5.8 | [ 0.1, 0.0] | 1.1 |
Table 14: Tilt predictions for the full-denoiser studies. The initial policy’s 512 evaluation episodes are reweighted by with chosen so that the reweighted mean of the optimized reward equals the gain each family reached at iteration 40. Intervals resample the initial episodes. Changes are in percentage points.
| Success | Wrong-object | Other failure | |
| Executions | 2,009 | 1,130 | 1,325 |
| Detected as success (probability .6) | 73.4% | 89.4% | 1.6% |
| Mean detection step (of 300) | 295 | 236 | 297 |
| AUROC: S vs. W / other / all failures | |||
| Terminal success probability | 0.195 / 0.951 / 0.603 | ||
| Dense progress with success termination | 0.125 / 0.935 / 0.562 | ||
Table 15: Robometer’s published online-RL recipe on the drawer selector. All 4,464 executed drawer episodes of the five Robometer selector trainings, rescored with the recipe: every step pays plus the predicted progress, the step whose success probability exceeds .6 pays and ends the episode. The last block tilts the uniform selector by each reward to the KL that the trained terminal-reward selectors reached.
Figure 11: R82 fixed final PPO evaluation: the prespecified first paired seed, all 32 paired scores and complete physical behavior counts. Wrong-object counts include failures regardless of original reward.
Figure 12: All eight R82 PPO feedback batches, 24 executions per objective before each update, sharing the first uniform batch. Reward always denotes original Robometer, including on the product-trained trajectory. Training curves are not substituted for the new-seed final evaluation.
| Run / condition | [95% CI] | |||||
|---|---|---|---|---|---|---|
| Drawer / Robometer / PPO | ||||||
| R82 | 32 | 4/9 | 24/2 | 3/24 | 25/5 | |
| R87 | 32 | 11/4 | 19/3 | 2/28 | 25/3 | |
| R88 | 32 | 6/6 | 18/4 | 2/28 | 25/5 | |
| Drawer / Robometer / GRPO | ||||||
| R114 | 64 | 12/8 | 54/1 | 4/54 | 56/0 | |
Table 16: Physical repair in drawer Robometer and stacking LIV. Each policy entry is success/wrong-object count out of ; is original native reward change. denote uniform/oracle/critic/product. Drawer uses 384 labels; stacking uses 2,536 (2,080 beyond the smaller head). Both trained colors and all nine main trainings are retained. The verifier-only ablation appears in Section M.1 ; all budgets and frozen physical intervals appear in Section K .
Figure 13: All 29 task/color strata of the constrained-policy grid, grouped as main repair panels, Robometer stacking and drawer LIV. Orange: critic minus uniform; blue: product minus critic; frozen 95% paired intervals. Robometer stacking and drawer LIV largely lack induction, and product does not uniformly help there (for example the distinct-color R166 row).
Figure 14: Absolute outcomes of every arm in all 29 constrained-policy strata. Each bar partitions all fresh executions (32 for R82/87/88, 64 otherwise) into success, wrong-object failure and other failure; widths are proportions, not uncertainty. N/D: near/distinct trained color; *: normalized PPO; NA: no matched verifier-only control. The last column marks whether the critic-minus-uniform native-reward interval is strictly positive (+) or unresolved (?); it is not a hacking verdict.
Figure 15: Product minus verifier-only in the 26 strata with a matched verifier-only control, with frozen 95% paired intervals. Retaining the original reward raises success in several distinct-color Robometer-stacking strata, lowers it in R166 near and in drawer LIV R182 and R178, and is indistinguishable elsewhere; NA marks the three drawer PPO trainings without that control.
| Run / condition | Labels | ||||||
|---|---|---|---|---|---|---|---|
| Drawer / Robometer / PPO | |||||||
| R82 | 32 | 384 | 4/9 | 24/2 | 3/24 | 25/5 | – |
| R87 | 32 | 384 | 11/4 | 19/3 | 2/28 | 25/3 | – |
| R88 | 32 | 384 | 6/6 | 18/4 | 2/28 | 25/5 | – |
| Drawer / Robometer / GRPO | |||||||
| R114 | 64 | 384 | 12/8 | 54/1 | 4/54 | 56/0 | 55/0 |
Table 17: Every final common-protocol endpoint and supervision budget. Cells are success / total wrong-object counts. N/D denote near/distinct cube colors, both trained. is the identical-head verifier-only control; – denotes a missing matched control. Expanded cube heads add 2,080 labels.
Figure 16: All Robometer physical contrasts with their original paired 95% intervals. Success and total wrong-object changes are separate; indicates a native-reward-gain interval above zero. NA denotes a missing identical-budget only control. Both cube colors are trained. The rows retain near-color failures and both signs of distinct-color GRPO effects; row count is not independent-training count.
Figure 17: All LIV physical contrasts and original paired 95% intervals, including both cube label budgets and all six drawer panels. Distinct-color cube repair repeats, but an added original-LIV contribution over the identical verifier is not established. Near-color and drawer negative inductions remain visible. Drawer wrong-opening excludes upper closing; continuous movement is reported in Section L.3 .
| Static metric (initial policy) | Full panel | Strata | Disjoint halves (combined) | Strata (A B / B A) |
|---|---|---|---|---|
| S1 AUROC, success vs all failures | 29 | 28 / 28 | ||
| S2 AUROC, success vs wrong-object | 29 | 25 / 27 | ||
| S3 standardized margin | 29 | 29 / 29 | ||
| S4 top-quartile enrichment | 29 | 29 / 29 | ||
| (S2) (S1), common strata | 29 | – |
Table 18: Spearman correlation between each initial-policy static metric and the wrong-object change after critic optimization, across induction strata, with 95% bootstrap intervals over 20 training clusters. The full panel uses the same initial-policy episodes for the metric and the change; the disjoint-halves column computes them on label-blind halves of each panel in both directions and averages the two. The difference row is not pre-declared in the original analysis plan.
Figure 18: Wrong-object change after critic optimization against the initial-policy generic AUROC (left) and semantic AUROC (right), one point per stratum, marked by task and critic.
Figure 19: Training feedback of the whole-network arms: raw values of each 126-episode batch, without smoothing. Grey lines are the four oracle histories and appear only in the success panel, because oracle feedback was neither scored by the critic nor fully logged for wrong-object failures; the grey column marks the two value-only iterations.
| Arm | Iter. | success (pp) [95% CI] | completion [95% CI] |
|---|---|---|---|
| History A | 010 | ||
| 020 | |||
| 030 | |||
| 040 | |||
| History B | 010 | ||
| 020 |
Table 19: Physical-feedback arms of the whole-network fine-tuning study, each iteration against that arm’s own iteration-0 weights on its 128 paired evaluation seeds. Only the iteration-40 row enters the pre-declared verdict.
Figure 20: Every execution in the separate checkpoint-diagnostic panel: the shared initial-policy set and iterations 10–40 of each learned arm, 128 seeds each. No execution is dropped or reordered by outcome.
| Training | Interpretation | ||||
|---|---|---|---|---|---|
| R85 | 30 | 94 | 0 | 53 | Induction and partial repair |
| R93 | 31 | 94 | 4 | 3 | Induction; repair fails |
| R94 | 28 | 69 | 67 | 84 | Rewarded errors coexist with task learning |
Table 20: All three initial RoboReward NPG trainings, success counts out of 96. Two improve under the product; the failure is retained. These are separate protocols from the common PPO/GRPO matrix.
| Run / condition | [95% CI] | |||||
|---|---|---|---|---|---|---|
| Drawer / LIV / PPO | ||||||
| R182 / original | 64 | 14/11 | 58/2 | 17/10 | 53/5 | |
| R178 / original | 64 | 12/13 | 45/0 | 11/15 | 48/5 | |
| R187 / norm. | 64 | 14/10 | 60/2 | 7/0 | 63/0 | |
| R188 / norm. | 64 | 14/6 | 43/3 | 15/1 | 58/4 | |
| Drawer / LIV / GRPO | ||||||
Table 21: All six drawer LIV panels, using the same 384-label verifier. Entries are success/wrong-opening counts out of 64. R178 has no established native gain. Normalized PPO (norm.) differs from original-scale PPO. Closing is excluded from wrong-opening counts; continuous motion and exact intervals are retained in Section L.3 .
| Run | Protocol | /64 | (mm) | (mm) |
|---|---|---|---|---|
| R182 | PPO | |||
| R178 | PPO | |||
| R187 | Norm. PPO | |||
| R188 | Norm. PPO | |||
| R183 | GRPO | |||
| R179 | GRPO |
Table 22: All six drawer LIV movement contrasts and original paired 95% intervals. Positive target change is more opening; negative upper net-motion change is more closing, which is excluded from the wrong-opening event. R183 target-opening uncertainty crosses zero despite its positive mean.
| Development family | Outcome and boundary |
|---|---|
| Reward-weighted denoiser regression | Initial mitigation does not persist across the second training. It does not establish neural-decoder RL repair. |
| Noise-policy SAC | Oracle also degrades; the control does not validate a learnable family in which learned reward is specifically to blame. |
| RoboReward cube PPO/GRPO | Complete later panels do not induce the targeted original high-score failures. Product task improvement is not hacking repair. |
| Terminal-feature replacement | Does not reliably repair identity errors; late policy observations and final critic images are different inputs. |
| Unbalanced-color training | Negative transfer and supervision effects are retained as development boundaries. Final balanced-color evidence does not establish unseen-color or unseen-task transfer. |
Table 23: Negative findings relevant to interpreting the final method. The current manuscript does not substitute these protocols for the complete common PPO/GRPO endpoints.
| Run / color | ||||||
|---|---|---|---|---|---|---|
| Stacking / Robometer / PPO | ||||||
| R164 / near | 20/19 | 38/21 | 26/32 | 38/18 | 30/25 | |
| R164 / distinct | 20/17 | 59/3 | 60/1 | 61/0 | 48/16 | |
| R169 / near | 27/23 | 43/17 | 37/25 | 41/19 | 36/20 | |
| R169 / distinct | 27/22 | 63/1 | 61/3 | 62/2 | 45/18 | |
| R175 / near | 24/15 | 40/13 | 30/28 | 32/20 | 39/17 | |
Table 24: Robometer stacking with the 456-label head: success/wrong-object counts out of 64. The last column is the paired product-minus-only success-count difference and its 95% interval. Both colors participate in each training. Only controls for R164/R166/R184 come from matched R168/R170/R189 follow-ups; they do not add independent replications. Expanded-head results are retained in Section K .
| Window | Test AUROC | Brier | TP / 24 | FP / 88 |
|---|---|---|---|---|
| Full episode | 1.000 | .0022 | 24 | 0 |
| Final quarter | .718 | .2225 | 9 | 18 |
Table 25: Ballot visual-window confirmation on new object positions and terminal postures. Each head is fitted only on the 112 training executions; the acceptance threshold is .5.
Figure 21: Original ballot-control frames from the first test position/posture group, using release cells selected by training-only critic (top) and product (bottom) reward. By withdrawal, the successful ball is inside; failure leaves a visible terminal ball. Final-quarter verification misses this success. This pair illustrates window-dependent prediction, not terminal unobservability or online PPO performance; Tables 25 and 26 report the complete panels.
| Training 1 | Training 2 | |||
|---|---|---|---|---|
| Objective | Critic reward | Success / 64 | Critic reward | Success / 64 |
| Uniform | .0390 | 12 | .0383 | 12 |
| Oracle | .0471 | 63 | .0467 | 64 |
| Original reward | .0647 | 2 | .0681 | 5 |
| Full-history product | .0468 | 63 | .0466 | 64 |
| Verifier-only | .0468 | 63 | .0469 | 64 |
Table 26: New-seed endpoints for online ballot-release PPO. This controlled extension trains a 14-action scripted release selector, with two independent histories per objective; it is separate from the common frozen-decoder noise-policy matrix. The last row is a paired follow-up changing only the verifier window and its corresponding frozen head.
| Parent | Measure | ||||||
|---|---|---|---|---|---|---|---|
| ID opening .475 | |||||||
| R82 | Success / wrong | 10/10 | 4/58 | 3/60 | 61/0 | 61/0 | 54/6 |
| Original reward | 0.326 | 0.838 | 0.856 | 0.689 | 0.684 | 0.662 | |
| R87 | Success / wrong | 15/10 | 4/57 | 3/57 | 52/4 | 49/5 | 54/1 |
| Original reward | 0.330 | 0.813 | 0.809 | 0.613 | 0.602 | 0.602 | |
| R88 | Success / wrong | 10/16 | 3/56 | 4/54 | 16/30 | 22/20 | 59/2 |
Table 27: Checkpoint continuation: all three prespecified parents, both test openings. is uniform, the frozen hacked parent, and the original-reward/product/verifier/oracle continuations. Success/wrong are success and wrong-object counts out of 64 paired new seeds; rewards are original Robometer scores. Both conditions use the same final checkpoint 008; no labels or policy updates are added for the held-out opening.
Figure 22: Repair after exploitation and the value of process evidence. (a) All three exploited parents, with fixed endpoints at training opening .475 and opening .375 held out from continuation; no new verifier labels. (b) Two ballot PPO histories, using the same 112 labels and 64-frame count. Full-history verification improves repair over a final-quarter head. Verifier-only is retained in both panels; full controls and paired intervals are in Sections M.5 and M.4 .
| Setting | Training ID | Original run ID |
| Drawer / Robometer | PPO 1 | R82 |
| Drawer / Robometer | PPO 2 | R87 |
| Drawer / Robometer | PPO 3 | R88 |
| Drawer / Robometer | GRPO 1 | R114 |
| Drawer / Robometer | GRPO 2 | R151 |
| Stacking / LIV | PPO 1 | R167 |
Table 28: Reader-facing identifiers and original experiment records. Training numbers are local to each task–critic–optimizer group. Both trained stacking colors and all verifier budgets from one training share its ID; they are not independent replications. The two development runs supply data for the expanded stacking verifier.
More in this topic
RL for Robotics
Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.
See all 241 papers