Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Varun Giridhar; Anant Khandelwal; Jeremy A. Collins; Ignat Georgiev; Animesh Garg

cs.ROAug 21, 2026

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Authors: Varun Giridhar, Anant Khandelwal, Jeremy A. Collins, Ignat Georgiev, Animesh Garg

Organizations: Georgia Institute of Technology

Abstract

Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.

Figures & tables

Figure 1: Q-Planning overview. Left: at inference, the frozen BC policy samples $N$ candidate action chunks; the Q-function scores them and the executed chunk is a single-step Q-weighted average. Right: rollouts (successful and failed) are appended to a replay buffer and used to fine-tune only the Q-function. The updated $Q_{\phi}$ returns to the next inference iteration; the BC policy is never updated.

Method	BC frozen	Plug-and-play large VLA	Learn from failures	Offline-to- online $Q$	No aux. actor
V-GPS [ 29 ]	✓	✓	✗	✗	✓
DSRL [ 33 ]	✓	✓	✓	✓	✗
IBRL [ 41 ]	✓	✗	✓	✗	✗
DAWR [ 42 ]	✗	✗	✓	✗	✓
Q-Planning (ours)	✓	✓	✓	✓	✓

Table 1: Positioning against the closest methods. Q-Planning is the only method that combines all five axes.

Action source	Success $\uparrow$	Mean ep. length $\downarrow$	Latency (ms) $\downarrow$
FastWAM (reference, no Q)	90.0	274	646
MPPI	89.5	270	1114
MPPI + temporal smoothing	93.0	261	1114
Ours (3-step FM draws + Q-weighted avg.)	93.0	261	640

Table 2: Action-source comparison on LIBERO-10 (

10

tasks,

20

episodes/task). Rows below the reference share the same

Q_{\phi}

and differ only in how candidates are drawn. Latency is median wall-clock time per planning step on an NVIDIA L40S under LIBERO’s eval-parity settings (App. E ). Temporally-smoothed MPPI and our single-step Q-weighted average match on success (

93.0\%

), but ours runs

1.7\times

faster and has fewer hyper-parameters.

	FastWAM [ 5 ]		Q-Planning (offline)		Q-Planning (self-improved)
Benchmark	Succ.	Len.	Succ.	Len.	Succ.	Len.
LIBERO-Spatial	90.5	106	91.5	115	98.5	107
LIBERO-Object	100.0	138	99.5	139	100.0	120
LIBERO-Goal	97.0	107	99.0	110	99.0	99
LIBERO-10	90.0	274	93.0	261	99.0	224
RoboTwin ( $47$ tasks)	83.2	220	83.8	231	91.4	232

Table 3: Main results on the four LIBERO suites and RoboTwin. Per-row values are means of per-task success rates (

\uparrow

) and per-task mean successful episode length (

\downarrow

). Bold marks the best success per row. Q-Planning (offline) improves over the BC policy on

4

5

benchmarks by

+1.3

pp on average;

10

iterations of Q-only self-improvement then lift every benchmark, reaching

97.6\%

mean success (

+5.5

pp over the frozen BC).

Figure 2: Q-Planning self-improvement in simulation. (a) LIBERO-10 vs. five online baselines under an identical rollout budget: Q-Planning lifts success from

93\%

\mathbf{99\%}

while every baseline plateaus, oscillates, or collapses. (b) Per-iteration trajectories across every LIBERO suite and bimanual RoboTwin, visualising the endpoints reported in Table 3 . The BC policy is frozen throughout; only

Q_{\phi}

is updated.

Figure 3: Q-value over time on two representative LIBERO-10 episodes. Solid lines show

Q_{\phi}

of the executed (Q-weighted average) chunk; faint scatter shows the

N{=}64

BC candidates scored per planning step.

Panel of Figure 4 — Figure 4: Real-robot self-improvement on two contact-rich bimanual tasks. From $100$ base demos and $20$ online episodes per iteration, with the BC frozen and no human intervention, Q-Planning improves from its own failures: stack-cups $40\to 90\%$ and insert-wallet $25\to 80\%$ in five iterations (Q-weighted selection alone starts the curves at $65\%$ and $40\%$ ). SFT on successes alone (dotted) stalls at $55\%$ and $30\%$ .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Q-function architecture. The Q-function $Q_{\phi}$ has its own DinoV2 visual encoder and T5 language encoder (parameter-disjoint from the BC policy). The transformer decoder cross-attends to visual tokens $\mathbf{z}^{\text{vis}}_{t}$ and language tokens $\mathbf{z}^{\text{txt}}$ (keys/values) and takes the candidate action chunk $\mathbf{a}_{t:t+H}$ as a sequence of query tokens. The HL-Gauss head outputs $B$ bin logits over discounted returns and the scalar $Q_{\phi}$ is the expectation of the softmaxed bin distribution.

Configuration	Q-evals	LIBERO strict	%	RoboTwin strict	%
BC only, $3$ denoise steps	$0$	$273$ ( $275$ )	$82$	$273$ ( $275$ )	$28$
BC only, $10$ steps (eval baseline)	$0$	$646$ ( $647$ )	$194$	$640$ ( $643$ )	$67$
Q-Planning, $N{=}16$ , $3$ steps	$16$	$337$ ( $338$ )	$101$	$352$ ( $354$ )	$37$
Q-Planning, $N{=}32$ , $3$ steps $\blacktriangleleft$ RoboTwin	$32$	$371$ ( $372$ )	$111$	$400$ ( $402$ )	$42$
Q-Planning, $N{=}64$ , $3$ steps $\blacktriangleleft$ LIBERO	$64$	$640$ ( $641$ )	$192$	$716$ ( $725$ )	$75$
Earlier iterative sample-based variant ( $3$ iters, $N{=}64$ )	$192$	$1114$ ( $1117$ )	$334$	$1276$ ( $1282$ )	$133$

Appendix

Table 4: Planning-step latency (ms) on an NVIDIA L40S. Median over

60

timed steps; parenthesised numbers are p95. Percentages express the fraction of the replan budget (LIBERO

333

ms, RoboTwin

960

ms). Strict = eval parity settings. Rows marked

\blacktriangleleft

are the deployed configurations. The final row is an earlier iterative sample-based variant we prototyped, retained here for comparison.

Explore similar work

May 6, 2026

When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning

Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning. In this work, we propose Q2RL, Q-Estimation and Q-Gating from BC for Reinforcement Learning, an algorithm for efficient offline-to-online learning. Our method consists of two parts: (1) Q-Estimation extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL with (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training. Across manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence. Q2RL is efficient enough to be applied in an on-robot RL setting, learning robust policies for contact-rich and high precision manipulation tasks such as pipe assembly and kitting, in 1-2 hours of online interaction, achieving success rates of up to 100% and up to 3.75x improvement against the original BC policy. Code and video are available at https://pages.rai-inst.com/q2rl_website/

Lakshita Dodeja, Ondrej Biza, Shivam Vats +5

1Robotics and AI Institute · 2Brown University · 3Northeastern University

Q-Learning Robotic RL Robotic Manipulation Robot Imitation Learning Offline-To-Online RL

Ask PDF

Mar 10, 2026

Update-Free On-Policy Steering via Verifiers

In recent years, Behavior Cloning (BC) has become one of the most prevalent methods for learning manipulation from human demonstrations. Despite their successes, BC policies are often brittle and struggle with precise manipulation. To overcome these issues, we propose UF-OPS, an Update-Free On-Policy Steering method that enables the robot to predict the success likelihood of its actions and adapt its strategy at execution time. We accomplish this by training verifier functions using policy rollout data obtained during an initial evaluation of the policy. These verifiers are subsequently used to steer the base policy toward actions with a higher likelihood of success. Our method improves the performance of black-box diffusion policies, without changing the base parameters, making it lightweight and flexible. We present results from both simulation and real-world data and achieve an average 49% improvement in success rate over the base policy across 5 real tasks.

Maria Attarian, Ian Vyse, Jasper Gerigk +7

University of Toronto · Google DeepMind · University of Alberta +2

Robot Policy Adaptation Robot Imitation Learning Robotic Policy Evaluation Diffusion Policy

Ask PDF

Sep 24, 2026

Learning from Mixed-Quality Deployment Experience for Robot Manipulation

Robot policies deployed in real environments naturally accumulate mixed-quality experience, including successful executions, partial progress, and failures. Although these rollouts provide valuable information for further learning, directly incorporating them into imitation learning may reinforce undesirable behaviors, while offline reinforcement learning often suffers from unreliable value estimation under sparse rewards and limited data coverage. We consider a practical post-deployment setting where learning relies only on naturally accumulated autonomous rollouts, without additional human corrections or exploratory interaction. To effectively exploit such experience, we propose Predictive Action Chunk Learning (PACL). PACL first learns a predictive chunk-level critic that evaluates temporally extended action sequences and augments temporal difference learning with future latent prediction, providing richer supervision for long-horizon value estimation. The learned critic then converts chunk-level Q-values into discrete quality conditions, which guide a diffusion actor to learn jointly from these mixed-quality experiences without treating all behaviors as equivalent supervision. At inference, the actor generates multiple action chunks and the critic selects the highest valued candidate. Experiments across simulated and real-world robot manipulation tasks show that PACL consistently improves the pretrained policy and outperforms strong imitation learning and offline reinforcement learning baselines.

Yangang Ren, Yujie Yan, Zirui Li +6

School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798 · Chongqing Changan Automobile Co., Ltd, Chongqing 400023, China · Guangzhou Cloudbutterfly Technology Co., Ltd., Guangzhou 510220, China

Robot Policy Learning Offline RL Robotic Manipulation Policy Learning

Ask PDF