Update-Free On-Policy Steering via Verifiers

Maria Attarian; Ian Vyse; Jasper Gerigk; Evgenii Opryshko; Yifan Ruan; Anas Almasri; Sumeet Singh; Yilun Du

cs.ROMar 10, 2026

Update-Free On-Policy Steering via Verifiers

Authors: Maria Attarian, Ian Vyse, Jasper Gerigk, Evgenii Opryshko, Yifan Ruan, Anas Almasri, Sumeet Singh, Yilun Du, +2 more

Organizations: University of Toronto · Google DeepMind · University of Alberta · Harvard University · University of Texas at Austin

Abstract

In recent years, Behavior Cloning (BC) has become one of the most prevalent methods for learning manipulation from human demonstrations. Despite their successes, BC policies are often brittle and struggle with precise manipulation. To overcome these issues, we propose UF-OPS, an Update-Free On-Policy Steering method that enables the robot to predict the success likelihood of its actions and adapt its strategy at execution time. We accomplish this by training verifier functions using policy rollout data obtained during an initial evaluation of the policy. These verifiers are subsequently used to steer the base policy toward actions with a higher likelihood of success. Our method improves the performance of black-box diffusion policies, without changing the base parameters, making it lightweight and flexible. We present results from both simulation and real-world data and achieve an average 49% improvement in success rate over the base policy across 5 real tasks.

Figures & tables

Fig. 1 : Our method relies on a policy’s own evaluation data to improve its performance. Training small verifiers and subsequently, utilizing them via inference-time steering, allows for improved policy performance without costly data collections and resource extensive fine-tuning.

Fig. 2 : Method overview: Given a base policy trained on a dataset of expert demonstrations, policy evaluation provides successful and failed rollouts. These are used for training a verifier function that scores a transition

(s,a)

in terms of its success likelihood. Finally, the verifier function is used in combination with a steering strategy, to improve the policy performance.

Fig. 3 : Real tasks on the Aloha bimanual system [ 57 ] used for evaluation. From left to right, a) pick and place the block on the cardboard, b) pick and place the ball in the bowl, c) pick up the hammer with the right hand, hand it over to the left hand and drop it in the box, d) pick up the pen cap with the right hand and the cap with the left and insert it on the pen, and e) pick up the green cup to stack it on the purple one.

Panel of Figure 4 — Fig. 4 : 2D Pedagogical Example.

Task	Base	SAILOR [ 24 ]	DSRL [ 51 ]	V-GPS [ 41 ]	Q $\times$ BoN	C $\times$ BoN	Q $\times$ CG	C $\times$ CG
Transport (low dim)	56.6 $\pm$ 3.07	-	24.8 $\pm$ 9.0	-	59.6 $\pm$ 3.04	62.7 $\pm$ 3.00	66.9 $\pm$ 2.92	64.7 $\pm$ 3.0
Square (low dim)	78.2 $\pm$ 2.56	-	74 $\pm$ 6.1	-	85.1 $\pm$ 2.2	86.0 $\pm$ 2.2	81.7 $\pm$ 2.4	85.5 $\pm$ 2.2
Transport (image)	58.1 $\pm$ 3.06	5.9 $\pm$ 1.46	-	-	65.7 $\pm$ 2.94	71.9 $\pm$ 2.79	62.5 $\pm$ 3.0	60.7 $\pm$ 3.03
Square (image)	70.1 $\pm$ 2.84	45.1 $\pm$ 3.08	-	53.2 $\pm$ 3.09	75.9 $\pm$ 2.7	83.5 $\pm$ 2.3	76.4 $\pm$ 2.63	77.6 $\pm$ 2.6

TABLE I : Comparison of Performance (%) on Robomimic SIM tasks for the base policy, four combinations of our method, Q function with Best-of-N (Q x BoN), classifier with Best-of-N (C x BoN), Q function with classifier guidance (Q x CG), and classifier with classifier guidance (C x CG), and the three baselines, V-GPS [ 41 ] , SAILOR [ 24 ] , and DSRL [ 51 ] .

Task	Base	$\lambda=0.1$	$\lambda=0.0$ )
Transport (low dim)	56.6 $\pm$ 3.07	62.7 $\pm$ 3.00	56.0 $\pm$ 3.08
Square (low dim)	78.2 $\pm$ 2.56	86.0 $\pm$ 2.2	85.5 $\pm$ 2.2

TABLE II : Contrastive loss ablation. Best-of-N policy steering using classifiers trained with and without the contrastive loss.

Task	Base	Q x BoN	C x BoN	Q x CG	C x CG
PH $\rightarrow$ MH Transport (low dim)	56.6 $\pm$ 3.07	54.7 $\pm$ 3.09	54.3 $\pm$ 3.09	42.3 $\pm$ 3.06	13.6 $\pm$ 2.12
PH $\rightarrow$ MH Square (low dim)	78.2 $\pm$ 2.56	78.6 $\pm$ 2.54	79.8 $\pm$ 2.49	82.1 $\pm$ 2.37	56.4 $\pm$ 3.07
MH $\rightarrow$ PH Transport (low dim)	81.1 $\pm$ 2.43	79.5 $\pm$ 2.5	79.4 $\pm$ 2.51	75.1 $\pm$ 2.68	27.9 $\pm$ 2.78
MH $\rightarrow$ PH Square (low dim)	90.7 $\pm$ 1.8	90.7 $\pm$ 1.8	93.7 $\pm$ 1.51	92.8 $\pm$ 1.6	89.4 $\pm$ 1.91

TABLE III : Comparison of steering via verifiers trained on off-policy, yet task-related, data reveals that leveraging on-policy rollouts is crucial for the efficacy of the method.

Task	Base	Q x BoN	C x BoN	Q x CG	C x CG
PH $\rightarrow$ PH Square (low dim)	90.7 $\pm$ 1.8	94.3 $\pm$ 1.8	94.4 $\pm$ 1.43	93.4 $\pm$ 1.54	92.5 $\pm$ 1.63

TABLE IV : Steering a highly performant policy with an on-policy verifier still improves upon baseline.

Method

task12

task26

task28

task31

task38

task43

task59

task60

task64

task77

task79

Mean

Base

9.9

\pm

4.14

5.9

\pm

3.26

41.2

\pm

13.8

\pm

4.78

31.3

\pm

4.78

9.9

\pm

4.14

18.8

\pm

5.41

49.5

\pm

6.93

35.9

\pm

6.65

30.4

\pm

6.37

33.6

\pm

6.55

25.5

V-GPS [ 41 ]

31.5

\pm

6.44

1.5

\pm

1.68

18.0

\pm

5.32

6.5

\pm

3.41

10.5

\pm

4.25

2.0

\pm

1.94

10.0

\pm

4.16

41.5

\pm

6.83

28.0

\pm

6.22

13.5

\pm

4.74

26.0

\pm

6.08

17.2

\times

BoN

25.0

\pm

19.5

\pm

5.49

72.0

\pm

6.22

38.0

\pm

6.73

67.5

\pm

6.49

35.5

\pm

6.63

47.5

\pm

6.92

90.5

\pm

4.06

78.5

\pm

5.69

62.0

\pm

6.73

83.0

\pm

5.21

56.3

\times

78.0

\pm

5.74

20.0

\pm

5.54

81.0

\pm

5.44

58.0

\pm

6.84

73.0

\pm

6.15

84.5

\pm

5.02

77.5

\pm

5.79

97.0

\pm

2.36

86.5

\pm

4.74

98.5

\pm

1.5

74.0

\pm

6.08

75.3

TABLE V: Performance (%) on 11 unseen Libero tasks: Comparison of the base policy against our method with time-to-success estimation and both steering alternatives.

Dataset Size	1000	2000	3000
Performance (%)	65.0%	69.8%	75.3%

TABLE VI : Performance (%) of UF-OPS as a function of dataset size on Libero using

\pi_{0.5}

Panel of Figure 11 — Fig. 5 : Performance on 5 real Aloha tasks. (a) Absolute success rate. (b) Improvement of each guided variant over the base policy, with 95% Newcombe confidence intervals for the difference of binomial proportions [ 42 , Method 10] .

	Low-dim.		Image			Low-dim.		Image
Hyperparameter	Square	Transport	Square	Transport	Hyperparameter	Square	Transport	Square	Transport
Problem setup					Diffusion process
Demonstration source	ph , mh	ph , mh	mh	mh	Scheduler	DDPM
RGB camera views	—	—	2	4	Train / inference timesteps	100 / 100
Image resolution	—	—	$3\times 84\times 84$		$\beta$ schedule	squaredcos_cap_v2
Low-dim. obs. dim. per step	23	59	9	18	$\beta_{\mathrm{start}}$ , $\beta_{\mathrm{end}}$	$10^{-4}$ , $2\times 10^{-2}$

TABLE VII : Hyperparameters of the base (unguided) diffusion policy for all four experimental settings. All settings use absolute-pose actions with a 6D rotation representation and the corresponding Robomimic dataset. The low-dimensional settings train a separate policy per demonstration source ( ph and mh ); the image settings use mh only. In the image settings the observation encoder is trained end-to-end with the denoising network.

	Low-dim.		Image			Low-dim.		Image
Hyperparameter	Square	Transport	Square	Transport	Hyperparameter	Square	Transport	Square	Transport
Guidance model: inputs					Guidance model: optimization
Observation representation	raw state		encoder features		Optimizer	AdamW
Obs. input dim. $d_{\mathrm{obs}}$	46	118	2066	4132	Learning rate	$10^{-3}$		$5\times 10^{-5}$
Action input dim. $d_{\mathrm{act}}$	160	320	160	320	Weight decay	$10^{-2}$		$10^{-4}$
Progress-time input	env. step index				Betas	$(0.9,0.999)$

TABLE VIII : Guidance-model, inference-time guidance, and evaluation hyperparameters for all four settings. t2s ( Time2Success ) regresses the discounted return and con ( Contrastive ) classifies success; rows marked t2s or con apply to that model only. Both take the flattened observation history, standardized with dataset statistics, concatenated with the flattened action chunk (

d_{\mathrm{act}}=T\cdot d_{a}

) in the policy’s normalized space — the raw state for low-dim. (

d_{\mathrm{obs}}=T_{o}\cdot d_{o}

), the frozen observation-encoder output for image. A single

\lambda=0.1

transfers across both low-dimensional tasks and both guidance models, whereas each image setting requires its own.

Explore similar work

Proxy Policy Steering
Sep 8, 2026Chuanruo Ning, Tianrui Wang, Wei-Chiu Ma +1Robot Policy Adaptation Robot Foundation Models
Cornell University
PPGuide: Steering Diffusion Policies with Performance Predictive Guidance
Mar 11, 2026Zixing Wang, Devesh K. Jha, Ahmed H. Qureshi +1Diffusion Guidance Diffusion Policy
Department of Computer Science at Purdue University, IN 47907, USA. · Contribution was conducted while the author was at Mitsubishi Electric Research Laboratories. · Mitsubishi Electric Research Laboratories, Cambridge, MA 02139 USA.
Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
Aug 21, 2026Varun Giridhar, Anant Khandelwal, Jeremy A. Collins +2Robot Policy Adaptation Robotic RL
Georgia Institute of Technology

Sep 8, 2026J/K move · Enter open · S save

Proxy Policy Steering

Chuanruo Ning, Tianrui Wang, Wei-Chiu Ma, Kuan Fang

Cornell University

Robot Policy Adaptation Robot Foundation Models Robotic Manipulation Robot Imitation Learning Policy Learning

Open paper AskPDF arXiv

Generalist robot policies carry broad manipulation priors from large-scale data, but specializing them to a new task remains the deployment bottleneck. This requires eliciting task-specific behavior from limited demonstrations without degrading their broad capabilities. We introduce Proxy Policy Steering (PPS), an inference-time adaptation method that resolves this challenge by training two lightweight proxy policies whose calibrated velocity-space difference steers the frozen base sampler. A reference proxy models the frozen base's behavior on target-task observations, and a task proxy, initialized from the reference, captures how this behavior changes under task supervision. Their difference forms a calibrated velocity-space residual that steers the frozen base sampler at every denoising step. We identify the conditions under which this residual isolates the change induced by task supervision, and validate them empirically. Because the base is never directly modified, its broad capabilities remain available at inference, including behaviors such as recovery from failure that the demonstrations themselves do not exercise. Adaptation requires only forward velocity predictions from the base, making PPS lightweight to train and applicable even without access to the base's parameters. On 8 real-world and 4 simulation manipulation tasks, PPS lifts the state-of-the-art pi 0.5 base policy by 53% absolute success rate on average, with zero-to-one gains on tasks the base never solves, while preserving the base's broad capabilities. PPS outperforms LoRA fine-tuning, from-scratch specialists, residual policies, and prior inference-time steering methods.

Abstract

Explore similar work

Proxy Policy Steering

PPGuide: Steering Diffusion Policies with Performance Predictive Guidance

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning