I3L: Iterative Interactive Imitation Learning for Whole-Body Mobile Manipulation

Wensi Ai; Christopher Luey; Karthik Dharmarajan; Evans Han; Yunfan Jiang; Stef Ren; Jianwen Xie; Jiajun Wu

cs.ROOct 13, 2026

I3L: Iterative Interactive Imitation Learning for Whole-Body Mobile Manipulation

Authors: Wensi Ai, Christopher Luey, Karthik Dharmarajan, Evans Han, Yunfan Jiang, Stef Ren, Jianwen Xie, Jiajun Wu, +2 more

Organizations: Stanford University · Lambda Inc.

Abstract

Policies trained offline from teleoperated demonstrations can degrade outside their training distribution. This gap is particularly acute in long-horizon whole-body mobile manipulation, where long horizons amplify compounding errors and coordinated arm, torso, and base motion makes exhaustive demonstration coverage impractical. We introduce I3L, an integrated hardware and learning system that combines whole-body correction with iterative policy refinement to address this last-mile reliability gap. Its low-cost, open-source JoyLo+ interface uses policy-tracking leader arms and touch-triggered takeover to enable coordinated corrections across the body parts, while iterative deployment and retraining allow operators to target failures of the updated policy. We evaluate I3L across three robot embodiments and eight simulated and real-world tasks, including four long-horizon whole-body mobile manipulation tasks. Controlled simulation experiments show that four-round HG-DAgger achieves the highest final success on both tasks. Under this protocol, I3L improves autonomous success over the base policies by an average of 68.3 percentage points and achieves at least 90% success on every task, outperforming demonstration-only training by 35.6 percentage points on average with the same number of training trajectories. Averaged across tasks, takeover duration per episode decreases by 72% over four rounds. Together, these results show that I3L helps close the last-mile gap between offline training and reliable deployment and provides practical insights into collecting and using corrections for long-horizon whole-body mobile manipulation. Videos, visualizations, and appendix are at https://i-3l.github.io/.

Figures & tables

Fig. 1: I3L combines interactive correction with iterative policy refinement. Left: 1) the policy approaches a failure, prompting the operator to take over with JoyLo+ and provide targeted corrections, 2) the operator uses JoyLo+ to take over during autonomous execution. Center: corrections are aggregated with prior demonstrations for policy retraining and redeployment. Right: eight evaluation tasks across three embodiments.

Fig. 2: JoyLo+ correction interface. Motorized arm leaders track policy targets during autonomous execution and switch to compliant current control during operator takeover. Capacitive touch sensors trigger takeover, while JoyCon inputs control the torso, mobile base, and grippers. The interface supports both single-arm and dual-arm configurations.

System	Mobile	Bimanual	Multi-embodiment	Actuated	Correction
ALOHA/2 [ 2 , 51 ]	–	$\checkmark$	–	–	–
GELLO [ 20 ]	–	$\checkmark$	$\checkmark$	–	–
Mobile ALOHA [ 31 ]	$\checkmark$	$\checkmark$	–	–	–
TeleMoMa [ 32 ]	$\checkmark$	$\checkmark$	$\checkmark$	–	–
JoyLo [ 21 ]	$\checkmark$	$\checkmark$	–	$\checkmark$	–
UME [ 52 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–

TABLE I: Teleoperation and correction interfaces.

\checkmark

: demonstrated; –: not demonstrated in the cited work.

Fig. 3: Evaluation tasks and initialization randomization. Representative images show initial configurations and successful task outcomes. The eight tasks span 29 annotated subtasks (2–6 per task). Across the four real-world R1 Pro tasks, task executions average approximately 83 seconds, with book storage and breakfast preparation averaging close to two minutes each. Yellow mats indicate the R1 Pro base initialization regions as well as where some task-relevant objects are randomized at the start of each episode.

Fig. 4: Intervention burden and task completion across correction rounds. In (a–c), the dark line shows the equal-weight mean across eight tasks; (b) measures the fraction of execution under intervention, and (c) measures time under intervention per episode. In (d), rounds 1–4 show takeover-free collection episodes. The final bar and dashed demo-only baseline (58.3%) compare autonomous evaluation success under the same trajectory budgets.

Simulation
Task	Base	100 Demos	100 Corr. (Ours)
Pick-and-Place Strawberries	53.2%	55.6%	92.8%
Open Fridge and Pick Up Radio	39.6%	30.4%	90.0%
Real world (Franka, Diffusion Policy)
Mug Hanging	6/25 (24%)	17/25 (68%)	23/25 ( 0 92%)
Peg Insertion	7/25 (28%)	12/25 (48%)	23/25 ( 0 92%)

TABLE II: Autonomous full-task success rates before and after additional data collection. Base policies receive 100 additional demonstrations or 100 corrective trajectories through four HG-DAgger rounds. Simulation rates are means across five evaluation seeds.

Fig. 5: Task-stage intervention patterns across correction rounds. Orange curves show intervention rate over normalized episode progress, where the color and line style distinguish Rounds 1–4. Each y-axis tick interval represents 10 percentage points. Red annotations indicate failures and green annotations indicate successful behaviors.

Open Fridge and Pick Up Radio (VLA)
Setting	Residual	BC (all data)	SIRIUS	HG-DAgger
Single round (100)	$21.6\pm 17.5$	$79.2\pm 13.2$	$73.2\pm 12.5$	$80.0\pm 14.8$
Round 1 (25)	$16.0\pm 2.0$	$70.8\pm 4.6$	$54.8\pm 20.9$	$51.6\pm 12.3$
Round 2 (50)	$24.0\pm 5.1$	$86.4\pm 3.8$	$68.4\pm 13.1$	$72.4\pm 6.2$
Round 3 (75)	$30.0\pm 12.2$	$87.2\pm 6.6$	$70.4\pm 16.9$	$82.0\pm 13.6$
Round 4 (100)	$19.6\pm 6.2$	$88.0\pm 3.7$	$71.6\pm 7.9$	$\boldsymbol{90.0\pm 2.4}$

TABLE III: Correction-learning comparison in simulation. Entries report mean

\pm

sample standard deviation of success rates (%) across five evaluation seeds. Parentheses indicate cumulative corrective trajectories. The entry with the best mean for each task is bold.

Fig. 6: Body-part activation shares during JoyLo+ corrections. Bars pool four 25-episode rounds; each body part receives one vote per takeover segment after three consecutive active frames. Thresholds are 0.003 m/s for base translation and 0.003 rad/s for base rotation commands, 0.01 rad for torso joint differences, 0.02 rad for arm joint differences, and 0.75 for normalized gripper differences. Grippers are assigned to their respective arms.

Fig. 7: Operator study on Mug Hanging. (a) Cumulative intervention share. (b) Cumulative takeover-free episode count. Both metrics aggregate episodes through the indicated round for each operator.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Command	Unit	Lower	Upper
Base translation $x$	m/s	$-0.3$	$0.3$
Base translation $y$	m/s	$-0.3$	$0.3$
Base yaw	rad/s	$-0.4$	$0.4$
Torso joint 1	rad	$-1.1345$	$1.8326$
Torso joint 2	rad	$-2.7925$	$2.5307$
Torso joint 3	rad	$-1.8326$	$1.5708$

Appendix

TABLE A1: Real-R1 adapter limits. Base bounds apply independently to each component; torso rows follow the robot’s joint ordering.

Joint	Sign	Leader offset	Home angle
1	$+1$	$-1.3962$	$0$
2	$+1$	$1.6046$	$0$
3	$+1$	$3.1416$	$0$
4	$-1$	$-2.3000$	$-\pi/2$
5	$+1$	$-3.2681$	$0$
6	$-1$	$1.6402$	$\pi/2$

Appendix

TABLE A2: Franka mapping and nominal home pose. Offsets and home angles are in radians; the insertion adjustment is applied separately.

View	Camera	Crop width	Offset $(x,y)$
External	ZED 2i	600	$(-30,-60)$
Wrist	ZED Mini	720	$(0,0)$

Appendix

TABLE A3: Recorded Franka camera profile. Crop widths and offsets are in pixels, before resizing to

240\times 240

View	Translation	Quaternion
External	$(0.7,0,0.6)$	$(0.255,0.255,0.66,0.66)$
Wrist	$(0.086,0,-0.066)$	$(0.652,0.652,0.274,0.274)$

Appendix

TABLE A4: A1 simulation camera poses. Translation is in metres and quaternions use

(x,y,z,w)

order. The external pose is relative to its scene parent and the wrist pose to the end-effector link.

Process	Configured rate
Real Franka/R1 Pro environment loop	30 Hz
Franka camera capture	30 Hz
Simulated action / rendering	30 / 30 Hz
Simulated physics	120 Hz
JoyLo+ leader control	100 Hz
Touch-state serial publication	200 Hz

Appendix

TABLE A5: Configured operating rates for the real and simulated setups.

Control	Function
R	Close gripper
ZR	Open gripper
X	Start recording
A	Accept and save trial
Y	Discard trial
Home	Home robot and leader

Appendix

TABLE A6: Franka Joy-Con controls for teleoperation and trial recording.

Control	Function
Left stick, unpressed	Base translation (up to 0.3 m/s per axis)
Left stick, pressed	Base yaw via horizontal deflection (up to 0.4 rad/s)
Right stick, vertical	Torso pitch (joint 3)
Right stick, horizontal	Torso yaw (joint 4)
Left D-pad, up / down	Move torso joints 1–2 toward stand $(0,0)$ / squat $(1.74,-2.70)$
ZL / ZR	Toggle left / right gripper

Appendix

TABLE A7: R1 Pro Joy-Con controls for teleoperation and trial recording. Torso angles are in radians.

Panel of Figure 18 — Fig. A1: R1 Pro touch wiring and command flow. Both grip electrodes connect to input 11 of one MPR121 and share one ESP32-S3; touching either grip activates takeover.

Setting	Value
Enabled electrodes	All 12 channels
Thresholds (touch / release)	195 / 117
Charge current	63 $\mu$ A
Sample interval	16 ms
Debounce (touch / release)	Three consecutive qualifying detections
Baseline calibration	Automatic configuration and tracking

Appendix

TABLE A8: MPR121 touch-sensor settings. Thresholds are in sensor register units.

		Quantity
Component	Unit cost	Franka	R1 Pro
XM430-W350-T servo	$220	3	4
XC330-M288-T servo	$50	4	3
U2D2 USB adapter	$30	2	2
Power supply (12 V/5 V)	$25	2	2
Printed links (set)	$15	1	1

Appendix

TABLE A9: Estimated bills of materials for one Franka or R1 Pro leader and the listed accessories. Costs are in USD. Robot, tax, shipping, cabling, fasteners, bearings, and mounting hardware are excluded.

Setting	Value
Initialization	ImageNet ResNet-18
Trainable params.	$\approx$ 27.8M
Image input	2 views
Image resolution	$224\times 224$
Optimizer	AdamW, wd 0
Batch size	128

Appendix

TABLE A10: Diffusion policy settings for A1 and Franka.

Setting	Value
Initialization	$\pi_{0.5}$ base checkpoint
Trainable params.	$\approx$ 3.35B (full model)
Image input	3 views
Observation horizon	1 (current observation)
Image resolution	$224\times 224$
Optimizer	AdamW, wd $10^{-10}$

Appendix

TABLE A11:

\pi_{0.5}

settings for R1 Pro.

Task	Language instruction
Open Fridge and Pick Up Radio	Open the fridge, then pick up and turn on the radio.
Boxing Books up for Storage	Place the basket onto the table, and put the two books from the shelf into the basket.
Make Microwave Popcorn	Pick up the red popcorn bag from the shelf and place it into the microwave.
Prepare Make-Ahead Breakfast Bowls	Retrieve the bowl from the drawer, put the two bread in from the plate, and place the bowl on the right side of the countertop.
Turning on Radio	Pick up the radio on the table and poke the orange button.

Appendix

TABLE A12: Language instructions used for the VLA tasks.

Method	Setting	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4	Mean
Demo-only	Base (100 demos)	36.0	48.0	38.0	36.0	40.0	39.6
Demo-only	Additional demos (200 total)	30.0	28.0	42.0	26.0	26.0	30.4
Residual	Single round (100)	10.0	12.0	46.0	6.0	34.0	21.6
	Round 1 (25)	16.0	14.0	18.0	14.0	18.0	16.0
	Round 2 (50)	18.0	30.0	28.0	24.0	20.0	24.0
	Round 3 (75)	20.0	20.0	40.0	24.0	46.0	30.0

Appendix

TABLE A13: Open Fridge and Pick Up Radio simulation success rates (%) across five evaluation seeds. For correction-learning methods, parentheses indicate cumulative corrective trajectories.

Method	Setting	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4	Mean
Demo-only	Base (75 demos)	50.0	42.0	58.0	48.0	68.0	53.2
Demo-only	Additional demos (175 total)	52.0	54.0	62.0	54.0	56.0	55.6
Residual	Single round (100)	48.0	44.0	48.0	42.0	44.0	45.2
	Round 1 (25)	56.0	46.0	50.0	46.0	48.0	49.2
	Round 2 (50)	48.0	46.0	46.0	44.0	44.0	45.6
	Round 3 (75)	50.0	36.0	44.0	30.0	40.0	40.0

Appendix

TABLE A14: Pick-and-Place Strawberries simulation success rates (%) across five evaluation seeds. For correction-learning methods, parentheses indicate cumulative corrective trajectories.

Explore similar work

Apr 14, 2026

Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllers

Mobile Manipulation (MoMa) of articulated objects, such as opening doors, drawers, and cupboards, demands simultaneous, whole-body coordination between a robot's base and arms. Classical whole-body controllers (WBCs) can solve such problems via hierarchical optimization, but require extensive hand-tuned optimization and remain brittle. Learning-based methods, on the other hand, show strong generalization capabilities but typically rely on expensive whole-body teleoperation data or heavy reward engineering. We observe that even a sub-optimal WBC is a powerful structural prior: it can be used to collect data in a constrained, task-relevant region of the state-action space, and its behavior can still be improved upon using offline reinforcement learning. Building on this, we propose WHOLE-MoMa, a two-stage pipeline that first generates diverse demonstrations by randomizing a lightweight WBC, and then applies offline RL to identify and stitch together improved behaviors via a reward signal. To support the expressive action-chunked diffusion policies needed for complex coordination tasks, we extend offline implicit Q-learning with Q-chunking for chunk-level critic evaluation and advantage-weighted policy extraction. On three tasks of increasing difficulty using a TIAGo++ mobile manipulator in simulation, WHOLE-MoMa significantly outperforms WBC, behavior cloning, and several offline RL baselines. Policies transfer directly to the real robot without finetuning, achieving 80% success in bimanual drawer manipulation and 68% in simultaneous cupboard opening and object placement, all without any teleoperated or real-world training data.

Snehal Jauhri, Vignesh Prasad, Georgia Chalvatzaki

PEARL Lab, Computer Science Department, Technische Universität Darmstadt, Germany · Hessian.AI, Darmstadt, Germany · Robotics Institute, Germany

Mobile Manipulation Whole-Body Robotic Manipulation Offline RL Bimanual Robotic Manipulation

Ask PDF

Apr 17, 2026

Rewind-IL: Online Failure Detection and State Respawning for Imitation Learning

Imitation learning has enabled robots to acquire complex visuomotor manipulation skills from demonstrations, but deployment failures remain a major obstacle, especially for long-horizon action-chunked policies. Once execution drifts off the demonstration manifold, these policies often continue producing locally plausible actions without recovering from the failure. Existing runtime monitors either require failure data, over-trigger under benign feature drift, or stop at failure detection without providing a recovery mechanism. We present Rewind-IL, a training-free online safeguard framework for generative action-chunked imitation policies. Rewind-IL combines a zero-shot failure detector based on Temporal Inter-chunk Discrepancy Estimate (TIDE), calibrated with split conformal prediction, with a state-respawning mechanism that returns the robot to a semantically verified safe intermediate state. Offline, a vision-language model identifies recovery checkpoints in demonstrations, and the frozen policy encoder is used to construct a compact checkpoint feature database. Online, Rewind-IL monitors self-consistency in overlapping action chunks, tracks similarity to the checkpoint library, and, upon failure, rewinds execution to the latest verified safe state before restarting inference from a clean policy state. Experiments on real-world and simulated long-horizon manipulation tasks, including transfer to flow-matching action-chunked policies, demonstrate that policy-internal consistency coupled with semantically grounded respawning offers a practical route to improved reliability in imitation learning. Supplemental materials are available at https://sjay05.github.io/rewind-il

Gehan Zheng, Sanjay Seenivasan, Matthew Johnson-Roberson +1

College of Connected Computing, Vanderbilt University, USA. · School of Computer Science, University of Waterloo, Canada. · School of Computer Science, The University of Sydney, Australia. +1

Robot Failure Detection Robot Failure Recovery Robotic Manipulation Robot Imitation Learning Anomaly Detection

Ask PDF

Mar 3, 2026

HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations

We present Whole-Body Mobile Manipulation Interface (HoMMI), a data collection and policy learning framework that learns whole-body mobile manipulation directly from robot-free human demonstrations. We augment UMI interfaces with egocentric sensing to capture the global context required for mobile manipulation, enabling portable, robot-free, and scalable data collection. However, naively incorporating egocentric sensing introduces a larger human-to-robot embodiment gap in both observation and action spaces, making policy transfer difficult. We explicitly bridge this gap with a cross-embodiment hand-eye policy design, including an embodiment agnostic visual representation; a relaxed head action representation; and a whole-body controller that realizes hand-eye trajectories through coordinated whole-body motion under robot-specific physical constraints. Together, these enable long-horizon mobile manipulation tasks requiring bimanual and whole-body coordination, navigation, and active perception. Results are best viewed on: https://hommi-robot.github.io

Xiaomeng Xu, Jisang Park, Han Zhang +6

Stanford University · Toyota Research Institute

Mobile Manipulation Whole-Body Robotic Manipulation Robot Imitation Learning Long-Horizon Robotic Manipulation Cross-Embodiment Robot Learning Robotic Data Collection

Ask PDF