Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning

Zihang Wang; Yishan Wang

cs.ROAug 15, 2026

Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning

Authors: Zihang Wang, Yishan Wang

Abstract

Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior. We present a training method for this setting based on two components. First, an MC Q-chunk critic regresses chunk-level action values onto Monte Carlo returns from the replay buffer, performing sample-average (behavior) policy evaluation so that intervention trajectories are credited directly rather than diluted by current-policy TD backups. Second, max-Q selective imitation updates the actor by imitating, at each state, the higher- $Q$ action between the current policy action and a buffer sample under a hard winner-take-all rule. This rule automatically switches between learning from interventions and on-policy self-improvement: when the autonomous policy is stronger, targets align with the policy distribution, reducing the policy--target-sample gap that otherwise induces execution-time distribution shift. In practice we score candidates with a standard critic ensemble mean to reduce comparison noise, without softening targets or introducing score-gap thresholds. On a real USB pick-and-insertion task with 20 demonstrations, ACT QChunk-MCBC attains 99% success within 30 minutes of HIL training, whereas HIL-SERL requires about 5 hours to converge. In simulation on Peg Insertion and Square, ACT/Flow Q-chunk variants similarly reach $\ge$ 96% success within roughly half an hour of effective training, outperforming HIL-SERL, EXPO, and E2HiL on the success--time frontier.

Figures & tables

Figure 1: Max-Q selective imitation in action / chunk space. (1) The replay buffer induces a human/past sample distribution

p_{b}

while the learner maintains a current policy

\pi

. (2) An MC Q-chunk critic provides

Q(s,a)

; the estimated

\hat{Q}

is reliable on support but overestimates out of distribution. Hard

\max

over candidates from

\{p_{b},\pi\}

selects winners that define a target

p^{\star}

. (3) The updated policy

\pi^{\prime}

lies between

\pi

and

p^{\star}

(

\pi^{\prime}=\pi\!\rightarrow\!p^{\star}

). (4) On-policy learning near

\pi^{\prime}

further concentrates mass toward

\pi^{\star}=\mathrm{greedy}(Q)

Figure 2: Peg Insertion: success versus real training time (left; HIL-SERL uses recorded time, others start at the first checkpoint) and versus environment steps (right). Curves stop at each method’s peak checkpoint.

Figure 3: Peg Insertion: best-checkpoint success with Wilson 95% confidence intervals.

Method	Steps	Env	Hours †	Success
HIL-SERL	100 000	200 196	3.19	100%
MCBC	10 112	20 906	0.67	96%
ACT QChunk-MCBC	4 386	8 781	0.32	99%
Flow QChunk-MCBC	5 075	10 160	0.50	96%
EXPO	26 000	54 782	1.55	30%
E2HiL	46 000	93 315	2.36	90%

Table 1: Peg Insertion: peak checkpoint summary

Figure 4: Square: success versus training time since the first checkpoint (left) and best-checkpoint success (right).

Method	Steps	Env	Hours †	Success
HIL-SERL	43 350	339 626	3.85	100%
MCBC	4 384	35 466	0.80	96%
ACT QChunk-MCBC	1 901	14 896	0.38	99%
Flow QChunk-MCBC	2 200	17 236	0.60	96%
EXPO	32 000	72 031	2.07	10%
E2HiL	46 000	93 315	2.36	10%

Table 2: Square: peak checkpoint summary

Panel of Figure 5 — Figure 5: Real-robot USB pick-and-insertion setup. Left: workspace with dual LED illumination and a Franka-class arm holding a USB connector. Right: close-up of alignment into a motherboard rear I/O port.

Figure 6: Real USB insertion: success versus robot training time (left; curves truncated at each method’s peak) and best-checkpoint success (right).

Method	Time to peak	Success
HIL-SERL	5.0 h	100%
MCBC	1.0 h	96%
ACT QChunk-MCBC	0.5 h (30 min)	99%
Flow QChunk-MCBC	0.8 h	96%
EXPO	2.4 h	30%
E2HiL	3.7 h	90%

Table 3: Real USB insertion: peak training time and success

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 252 papers

Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
Sep 24, 2026Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht +1Contact-Rich Robotic Manipulation Human-in-the-Loop AI
Siemens AG, Research and Predevelopment, Germany. · Technical University of Munich, Germany.
ReF-HIL: Shaping the Critic around Human Action Neighborhoods for Efficient Human-in-the-Loop Reinforcement Learning
Sep 29, 2026Shaoyin Luo, Song Wang, Shibo Xia +5RL for Robotics Reward Shaping
Department of Mechanical Engineering, Tsinghua University, Beijing 100084, China
Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
Jun 2, 2026Zeyi Liu, Guangyao Liu, Yinuo Qu +6Credit Assignment in RL Robotic RL
Central South University · Nanyang Technological University · Zhejiang University

Sep 24, 2026J/K move · Enter open · S save

Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation

Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht, Majid Khadiv

Siemens AG, Research and Predevelopment, Germany. · Technical University of Munich, Germany.

Contact-Rich Robotic Manipulation Human-in-the-Loop AI Robotic Manipulation Data-Efficient Robot Learning Robot Imitation Learning Residual Policy Learning

Open paper AskPDF arXiv

Imitation learning enables robots to acquire manipulation skills from demonstrations, but the resulting policies can fail outside the training data, while collecting more demonstrations requires substantial human effort. Human-in-the-loop reinforcement learning uses corrective feedback during online training, but typically learns the complete task policy rather than refining a pretrained imitation policy. We introduce Res-HIL, a human-in-the-loop residual reinforcement learning framework that learns corrective actions on top of a frozen imitation policy. Each human intervention provides two complementary learning signals: direct supervision of the residual policy and reward shaping of preceding autonomous behavior. Res-HIL combines these signals with zero initialization of the residual policy to stabilize and accelerate online learning. We evaluate Res-HIL on five contact-rich manipulation tasks spanning high-precision and long-horizon behaviors. With only 20 initial demonstrations, Res-HIL outperforms state-of-the-art full-policy human-in-the-loop reinforcement learning and residual fine-tuning without human guidance on every task after ten minutes of online training. Res-HIL improves its pretrained base policies and outperforms imitation policies trained with five times more demonstrations. An ablation study shows that direct residual supervision is critical to performance, while intervention-aware reward shaping substantially improves training efficiency.

Abstract

RL for Robotics

Explore similar work

Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation

ReF-HIL: Shaping the Critic around Human Action Neighborhoods for Efficient Human-in-the-Loop Reinforcement Learning

Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation