Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Varun Giridhar; Anant Khandelwal; Jeremy A. Collins; Ignat Georgiev; Animesh Garg

cs.ROAug 21, 2026

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Authors: Varun Giridhar, Anant Khandelwal, Jeremy A. Collins, Ignat Georgiev, Animesh Garg

Organizations: Georgia Institute of Technology

Abstract

Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.

Figures & tables

Figure 1: Q-Planning overview. Left: at inference, the frozen BC policy samples $N$ candidate action chunks; the Q-function scores them and the executed chunk is a single-step Q-weighted average. Right: rollouts (successful and failed) are appended to a replay buffer and used to fine-tune only the Q-function. The updated $Q_{\phi}$ returns to the next inference iteration; the BC policy is never updated.

Method	BC frozen	Plug-and-play large VLA	Learn from failures	Offline-to- online $Q$	No aux. actor
V-GPS [ 29 ]	✓	✓	✗	✗	✓
DSRL [ 33 ]	✓	✓	✓	✓	✗
IBRL [ 41 ]	✓	✗	✓	✗	✗
DAWR [ 42 ]	✗	✗	✓	✗	✓
Q-Planning (ours)	✓	✓	✓	✓	✓

Table 1: Positioning against the closest methods. Q-Planning is the only method that combines all five axes.

Action source	Success $\uparrow$	Mean ep. length $\downarrow$	Latency (ms) $\downarrow$
FastWAM (reference, no Q)	90.0	274	646
MPPI	89.5	270	1114
MPPI + temporal smoothing	93.0	261	1114
Ours (3-step FM draws + Q-weighted avg.)	93.0	261	640

Table 2: Action-source comparison on LIBERO-10 (

10

tasks,

20

episodes/task). Rows below the reference share the same

Q_{\phi}

and differ only in how candidates are drawn. Latency is median wall-clock time per planning step on an NVIDIA L40S under LIBERO’s eval-parity settings (App. E ). Temporally-smoothed MPPI and our single-step Q-weighted average match on success (

93.0\%

), but ours runs

1.7\times

faster and has fewer hyper-parameters.

	FastWAM [ 5 ]		Q-Planning (offline)		Q-Planning (self-improved)
Benchmark	Succ.	Len.	Succ.	Len.	Succ.	Len.
LIBERO-Spatial	90.5	106	91.5	115	98.5	107
LIBERO-Object	100.0	138	99.5	139	100.0	120
LIBERO-Goal	97.0	107	99.0	110	99.0	99
LIBERO-10	90.0	274	93.0	261	99.0	224
RoboTwin ( $47$ tasks)	83.2	220	83.8	231	91.4	232

Table 3: Main results on the four LIBERO suites and RoboTwin. Per-row values are means of per-task success rates (

\uparrow

) and per-task mean successful episode length (

\downarrow

). Bold marks the best success per row. Q-Planning (offline) improves over the BC policy on

4

5

benchmarks by

+1.3

pp on average;

10

iterations of Q-only self-improvement then lift every benchmark, reaching

97.6\%

mean success (

+5.5

pp over the frozen BC).

Figure 2: Q-Planning self-improvement in simulation. (a) LIBERO-10 vs. five online baselines under an identical rollout budget: Q-Planning lifts success from

93\%

\mathbf{99\%}

while every baseline plateaus, oscillates, or collapses. (b) Per-iteration trajectories across every LIBERO suite and bimanual RoboTwin, visualising the endpoints reported in Table 3 . The BC policy is frozen throughout; only

Q_{\phi}

is updated.

Figure 3: Q-value over time on two representative LIBERO-10 episodes. Solid lines show

Q_{\phi}

of the executed (Q-weighted average) chunk; faint scatter shows the

N{=}64

BC candidates scored per planning step.

Panel of Figure 4 — Figure 4: Real-robot self-improvement on two contact-rich bimanual tasks. From $100$ base demos and $20$ online episodes per iteration, with the BC frozen and no human intervention, Q-Planning improves from its own failures: stack-cups $40\to 90\%$ and insert-wallet $25\to 80\%$ in five iterations (Q-weighted selection alone starts the curves at $65\%$ and $40\%$ ). SFT on successes alone (dotted) stalls at $55\%$ and $30\%$ .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Q-function architecture. The Q-function $Q_{\phi}$ has its own DinoV2 visual encoder and T5 language encoder (parameter-disjoint from the BC policy). The transformer decoder cross-attends to visual tokens $\mathbf{z}^{\text{vis}}_{t}$ and language tokens $\mathbf{z}^{\text{txt}}$ (keys/values) and takes the candidate action chunk $\mathbf{a}_{t:t+H}$ as a sequence of query tokens. The HL-Gauss head outputs $B$ bin logits over discounted returns and the scalar $Q_{\phi}$ is the expectation of the softmaxed bin distribution.

Configuration	Q-evals	LIBERO strict	%	RoboTwin strict	%
BC only, $3$ denoise steps	$0$	$273$ ( $275$ )	$82$	$273$ ( $275$ )	$28$
BC only, $10$ steps (eval baseline)	$0$	$646$ ( $647$ )	$194$	$640$ ( $643$ )	$67$
Q-Planning, $N{=}16$ , $3$ steps	$16$	$337$ ( $338$ )	$101$	$352$ ( $354$ )	$37$
Q-Planning, $N{=}32$ , $3$ steps $\blacktriangleleft$ RoboTwin	$32$	$371$ ( $372$ )	$111$	$400$ ( $402$ )	$42$
Q-Planning, $N{=}64$ , $3$ steps $\blacktriangleleft$ LIBERO	$64$	$640$ ( $641$ )	$192$	$716$ ( $725$ )	$75$
Earlier iterative sample-based variant ( $3$ iters, $N{=}64$ )	$192$	$1114$ ( $1117$ )	$334$	$1276$ ( $1282$ )	$133$

Appendix

Table 4: Planning-step latency (ms) on an NVIDIA L40S. Median over

60

timed steps; parenthesised numbers are p95. Percentages express the fraction of the replan budget (LIBERO

333

ms, RoboTwin

960

ms). Strict = eval parity settings. Rows marked

\blacktriangleleft

are the deployed configurations. The final row is an earlier iterative sample-based variant we prototyped, retained here for comparison.

Explore similar work

When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning
May 6, 2026Lakshita Dodeja, Ondrej Biza, Shivam Vats +5Q-Learning Robotic RL
1Robotics and AI Institute · 2Brown University · 3Northeastern University
Update-Free On-Policy Steering via Verifiers
Mar 10, 2026Maria Attarian, Ian Vyse, Jasper Gerigk +7Robot Policy Adaptation Robot Imitation Learning
University of Toronto · Google DeepMind · University of Alberta +2
Learning from Mixed-Quality Deployment Experience for Robot Manipulation
Sep 24, 2026Yangang Ren, Yujie Yan, Zirui Li +6Robot Policy Learning Offline RL
School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798 · Chongqing Changan Automobile Co., Ltd, Chongqing 400023, China · Guangzhou Cloudbutterfly Technology Co., Ltd., Guangzhou 510220, China

May 6, 2026J/K move · Enter open · S save

When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning

Lakshita Dodeja, Ondrej Biza, Shivam Vats, Stephen Hart+4

1Robotics and AI Institute · 2Brown University · 3Northeastern University

Q-Learning Robotic RL Robotic Manipulation Robot Imitation Learning Offline-To-Online RL

Open paper AskPDF arXiv

Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning. In this work, we propose Q2RL, Q-Estimation and Q-Gating from BC for Reinforcement Learning, an algorithm for efficient offline-to-online learning. Our method consists of two parts: (1) Q-Estimation extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL with (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training. Across manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence. Q2RL is efficient enough to be applied in an on-robot RL setting, learning robust policies for contact-rich and high precision manipulation tasks such as pipe assembly and kitting, in 1-2 hours of online interaction, achieving success rates of up to 100% and up to 3.75x improvement against the original BC policy. Code and video are available at https://pages.rai-inst.com/q2rl_website/

Abstract

Explore similar work

When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning

Update-Free On-Policy Steering via Verifiers

Learning from Mixed-Quality Deployment Experience for Robot Manipulation