Learning Quality-Controllable Robot Policies with Iterative Hindsight Relabeling

Sinjae Kang; Dong Kyu Shin; Jaehong Kim; Kamin Lee; Hyoeun Kim; Kaixin Wang; Li Zhao; Seonghyeon Ye

cs.ROOct 27, 2026

Learning Quality-Controllable Robot Policies with Iterative Hindsight Relabeling

Authors: Sinjae Kang, Dong Kyu Shin, Jaehong Kim, Kamin Lee, Hyoeun Kim, Kaixin Wang, Li Zhao, Seonghyeon Ye,

Organizations: KAIST · LGE · Microsoft Research · Config

Abstract

Robot policy evaluation often relies on a binary success signal for task completion. In real-world deployment, however, desired execution behaviors are also defined by quality attributes such as execution speed and motion smoothness, which need to be dynamically controlled according to task requirements. To address this, we introduce Quality-Controllable Policy (QCP), a robot policy that enables inference-time control over both individual quality attributes and their combinations. To extend the controllable quality range beyond initial demonstrations, we further propose an iterative hindsight relabeling loop: QCP collects online rollouts under target quality labels, adds successful trajectories to the dataset, relabels the accumulated trajectories in hindsight according to their measured quality, and fine-tunes the policy. Experiments on Robomimic and RoboEval demonstrate that QCP consistently improves both task execution quality and success rates across iterations, producing successful rollouts that exceed the quality of the initial demonstrations. Finally, we validate QCP on real-world manipulation tasks, where controlling speed increases throughput in conveyor operations and controlling transport stability improves success when carrying wine glasses, demonstrating that quality-controllable policies yield practical gains in physical deployment.

Figures & tables

Figure 1: Overview of QCP and iterative hindsight relabeling. Left: QCP controls execution quality at inference time through conditioning on quantized quality tokens. Right: Iterative hindsight relabeling shifts the quality distribution of successful rollouts toward higher quality, expanding the controllable range beyond the initial demonstrations.

Figure 2: Comparison of QCP with BC baselines (BC

{}_{\text{filter}}

, BC

{}_{\text{acc}}

) and behavioral quality learning methods (SORL, SCIQL) across iterations on (a) Robomimic and (b) RoboEval. Each plot shows task success rate against a quality metric, with the y-axis oriented so that higher is better. Darker and larger markers correspond to later iterations. QCP improves quality and task success together more effectively than the baselines. Results are averaged over three random seeds.

	Sq+Speed		Sq+Smoothness		Tr+Speed		Tr+Smoothness		Pack+Speed		Pack+JPL		Book+Speed		Book+JPL
Method	SR	Speed	SR	Smoothness	SR	Speed	SR	Smoothness	SR	Speed	SR	JPL	SR	Speed	SR	JPL
	(%)	(steps)	(%)	( $\times 10^{-3}$ )	(%)	(steps)	(%)	( $\times 10^{-3}$ )	(%)	(s)	(%)	(rad)	(%)	(s)	(%)	(rad)
QCP (ours)	82.2	120.3	79.7	8.5	22.2	432.8	12.3	22.2	45.8	4.82	35.3	6.42	47.0	2.43	37.9	4.63
SORL	53.2	156.0	48.4	11.5	4.6	466.8	5.7	34.9	39.7	5.64	41.8	7.11	48.9	3.37	41.4	5.15
SCIQL	68.7	206.5	83.0	12.8	10.6	540.8	7.9	47.2	39.9	9.24	37.4	9.06	48.4	4.75	47.0	6.40
DSRL	37.0	208.8	43.0	46.9	4.0	549.5	7.0	79.4	60.7	6.35	50.7	66.05	0.0	—	0.0	—

Table 1: QCP versus RL baselines. Results at the final iteration (iteration 5, i.e., after five rounds of rollout collection) are averaged across three random seeds. Higher success rates and lower quality metric values are better.

Figure 3: Quality achieved at each requested bin across iterations on Robomimic Square. The ordering of the bins is preserved while the whole range moves toward higher quality (arrows). Mean

\pm

one standard deviation over three seeds.

Figure 4: Quality distributions across iterations. Starting from the initial demonstration distribution, QCP shifts the quality distribution of successful rollouts toward higher quality, extending beyond the initial demonstration range. Arrows point toward higher quality.

Figure 5: QCP jointly adjusts execution speed and lift amplitude in the requested directions over iterations.

Figure 6: Joint improvement of six quality attributes on RoboEval. Final QCP and BC

{}_{\text{acc}}

policies are compared with initial BC on four tasks. Each axis shows the initial BC score divided by the method’s score, so the initial BC polygon lies on the

1\times

ring and outward indicates better quality. Success rates of initial BC and final QCP are shown above each task.

Figure 7: Quality control on real robots. QCP achieves higher task success than BC baselines while (a) reducing the tray tilt angle in wine serving and (b) increasing end-effector speed in conveyor work. Values are means over three seeds; error bars and the shaded band around BC denote $\pm$ 1 standard deviation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Main results (Figs. 2 , 4 , Tab. 1 )			Fig. 3	Fig. 5	Fig. 6	Fig. 9	Fig. 9
	Square	Transport	RoboEval	Square	Square	RoboEval	Square	Square
Number of iterations	6	6	6	10	10	7	6	7
Steps per iteration	100k	100k	30k	30k	30k	30k	30k	30k
Episode limit (steps)	400	700	400	400	800	400	800	400
Action chunk	6 / 4	10 / 8	10 / 8	10 / 8	10 / 8	10 / 8	10 / 8	10 / 8

Appendix

Table 2: Per-experiment settings. Steps are gradient steps per iteration. “Number of iterations” counts iterations including iteration 0, which is trained on the demonstrations alone. The action chunk is written as chunk length / executed steps.

	SORL		SCIQL
$\beta$	Speed	Smoothness	Speed	Smoothness
0	34.0 $\pm$ 3.5 / 213.8 $\pm$ 17.0	36.7 $\pm$ 5.3 / 0.0118 $\pm$ 0.0010	77.1 $\pm$ 9.0 / 251.2 $\pm$ 25.6	72.3 $\pm$ 6.0 / 0.0205 $\pm$ 0.0017
1	39.0 $\pm$ 6.0 / 206.5 $\pm$ 18.8	40.3 $\pm$ 8.3 / 0.0114 $\pm$ 0.0012	78.0 $\pm$ 8.7 / 231.0 $\pm$ 26.7	79.3 $\pm$ 6.2 / 0.0187 $\pm$ 0.0018
2	45.0 $\pm$ 9.5 / 189.4 $\pm$ 9.8	51.3 $\pm$ 3.7 / 0.0126 $\pm$ 0.0007	74.7 $\pm$ 4.3 / 218.9 $\pm$ 18.6	79.7 $\pm$ 9.8 / 0.0176 $\pm$ 0.0018
3	53.2 $\pm$ 7.8 / 156.0 $\pm$ 18.1	48.4 $\pm$ 6.5 / 0.0115 $\pm$ 0.0010	68.7 $\pm$ 5.2 / 206.5 $\pm$ 23.2	83.0 $\pm$ 6.0 / 0.0128 $\pm$ 0.0007
4	41.7 $\pm$ 5.7 / 188.1 $\pm$ 18.0	43.3 $\pm$ 4.9 / 0.0142 $\pm$ 0.0008	70.7 $\pm$ 8.4 / 212.5 $\pm$ 14.6	85.0 $\pm$ 4.7 / 0.0185 $\pm$ 0.0022

Appendix

Table 3: Coefficient sweep on Robomimic Square. Each cell is success rate (%) / quality of successful episodes (steps to success for Speed, action jerk for Smoothness; lower is better), mean

\pm

standard deviation over three seeds. Left: SORL and SCIQL over the advantage temperature

\beta

. Right: DSRL and EXPO over the quality reward weight

\lambda

. Bold rows are the settings used in the main results (

\beta=3

\lambda=4

) and reproduce Table 1 .

Task	Conditioned	Dropped (reason)
Lift Pot	JPL, OPL, CT, MJJ, BAVD, BGVD	CPL (a fixed vertical lift leaves little Cartesian path to vary), MCJ (redundant with MJJ)
Stack	CPL, JPL, CT, OPL, MJJ, MCJ	BAVD (one-arm task, equal arm velocities are not a quality), BGVD (large by design of the task)
Handover	CPL, OPL, MJJ, MCJ, BAVD, BGVD	CT (in conflict with the smoothness attributes), JPL (redundant with CPL and OPL)
Valve	JPL, OPL, CT, MJJ, BAVD, BGVD	MCJ (nearly constant for a wrist rotation), CPL (path fixed by the valve geometry)

Appendix

Table 4: Conditioned attributes per task. The two dropped attributes and the reason.

Lift Pot	SR	JPL	OPL	CT	MJJ	BAVD	BGVD
Initial BC	0.92 $\pm$ 0.01	4.37 $\pm$ 0.06	3.65 $\pm$ 0.04	2.04 $\pm$ 0.02	228.8 $\pm$ 2.6	0.418 $\pm$ 0.004	0.0052 $\pm$ 0.0003
$\textsc{BC}_{\text{acc}}$	0.99 $\pm$ 0.00	3.59 $\pm$ 0.07	3.08 $\pm$ 0.06	1.95 $\pm$ 0.03	176.6 $\pm$ 2.6	0.278 $\pm$ 0.015	0.0027 $\pm$ 0.0003
QCP	1.00 $\pm$ 0.00	2.82 $\pm$ 0.09	1.98 $\pm$ 0.09	1.72 $\pm$ 0.00	130.5 $\pm$ 6.4	0.173 $\pm$ 0.019	0.0022 $\pm$ 0.0002
Stack	SR	CPL	JPL	CT	OPL	MJJ	MCJ
Initial BC	0.46 $\pm$ 0.02	1.98 $\pm$ 0.03	8.16 $\pm$ 0.24	6.64 $\pm$ 0.05	5.72 $\pm$ 0.11	153.2 $\pm$ 3.0	39.2 $\pm$ 0.6
$\textsc{BC}_{\text{acc}}$	0.46 $\pm$ 0.02	1.68 $\pm$ 0.05	7.59 $\pm$ 0.22	8.62 $\pm$ 0.86	5.05 $\pm$ 0.09	80.4 $\pm$ 6.7	19.2 $\pm$ 1.2

Appendix

Table 5: Six-attribute quality on RoboEval. Numerical values of Figure 6 : success rate and the six conditioned attributes at iteration 6 for QCP and

\textsc{BC}_{\text{acc}}

, and for the initial BC. Mean

\pm

standard deviation over seeds; lower is better for every attribute; units as in Appendix A .

Figure 10: Quality labels for shorter segments. Success rate versus mean end-effector (EEF) displacement per step over successful rollouts on Square and Transport. We compare complete trajectory labels (QCP), segment labels, hybrid labels, and 10% filtered BC over iterations 0–9; darker points indicate later iterations. Transport 10% filtered BC has no successful rollouts, so its quality values are undefined and omitted.

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 377 papers

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
Aug 21, 2026Varun Giridhar, Anant Khandelwal, Jeremy A. Collins +2Robot Policy Adaptation Robotic RL
Georgia Institute of Technology
Learning from Mixed-Quality Deployment Experience for Robot Manipulation
Sep 24, 2026Yangang Ren, Yujie Yan, Zirui Li +6Robot Policy Learning Offline RL
School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798 · Chongqing Changan Automobile Co., Ltd, Chongqing 400023, China · Guangzhou Cloudbutterfly Technology Co., Ltd., Guangzhou 510220, China
RoboFQE: Rollout-Free Evaluation of Visual Robot Policies from Offline Trajectories
Oct 13, 2026Yunfu Deng, Sangwoo Shin, Brahma S. Pavse +1Robotic Policy Evaluation Off-Policy Evaluation
Department of Computer Sciences, University of Wisconsin–Madison

Aug 21, 2026J/K move · Enter open · S save

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Varun Giridhar, Anant Khandelwal, Jeremy A. Collins, Ignat Georgiev+1

Georgia Institute of Technology

Robot Policy Adaptation Robotic RL Robotic Manipulation Robot Imitation Learning Off-Policy RL

Open paper AskPDF arXiv

Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.

Abstract

Robot Policy Learning

Explore similar work

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Learning from Mixed-Quality Deployment Experience for Robot Manipulation

RoboFQE: Rollout-Free Evaluation of Visual Robot Policies from Offline Trajectories