Online World Modeling Enables Real-World Inverse Reinforcement Learning from Observation

Tyler Han; Bat Nemekhbold; Siyang Shen; Rohan Baijal; Richard Ebock; Harine Ravichandiran; Sanghun Jung; Kevin Huang

cs.ROFeb 27, 2026

Online World Modeling Enables Real-World Inverse Reinforcement Learning from Observation

Authors: Tyler Han, Bat Nemekhbold, Siyang Shen, Rohan Baijal, Richard Ebock, Harine Ravichandiran, Sanghun Jung, Kevin Huang, +1 more

Organizations: University of Washington

Abstract

Current methods in robot learning are fundamentally bottlenecked by one or more of: hand-designed rewards, simulation modeling, or action supervision (e.g. teleoperation) each requiring significant domain expertise, engineering effort, and robot-operator labor. Towards eliminating these bottlenecks, this work pursues observational learning via Inverse Reinforcement Learning from Observation (IRLfO) in which only access to task observations (e.g. video) is assumed. Due to the challenging setting and limitations of RL methods, IRLfO has thus far remained impractical for real-world robot learning. Here, we present the first IRL method to learn visual manipulation in the real world from scratch, and the first real-world demonstration of positive online transfer across visual manipulation tasks from scratch. In under 40 minutes, MPAIL2 learns pick-and-place from scratch to 82% success, where RL and BC with equal interaction and demonstration budgets reach only 0% and 12% despite their reward and action supervision. Interactive project page with training videos: https://uwrobotlearning.github.io/mpail2/

Figures & tables

Figure 1: Overview of MPAIL2. (0) Only task observations are provided to the learner. (1) The learner makes and encodes observations into a latent state representation. The policy predicts reactive actions which seed online planning. Performing online planning, the learner samples actions, predicts their future states , and evaluates plans using the online reward and the value . Plans with higher returns are more likely to contribute to the final decision. (2) After executing, the interaction $(o,a,o^{\prime})$ where $o^{\prime}$ is the posterior observation is accumulated in the learner’s experience. All components are updated online.

Task	Observations	Object dim. (reset region)	Demos
Simulation — IsaacLab, Franka arm; 5 seeds
Block Push (state)	state	$4{\times}4{\times}4$ cm ( $10{\times}6$ cm)	27
Block Push	images + proprioception	$4{\times}4{\times}4$ cm ( $10{\times}6$ cm)	27
Pick-and-Place	images + proprioception	$4{\times}4{\times}4$ cm ( $10{\times}6$ cm)	30
Real — $64{\times}64$ RGB (external + wrist); Franka: Push, Kinova Gen3: PnP; 3 seeds
Block Push	images + proprioception	$5{\times}5{\times}5$ cm ( $18{\times}18$ cm)	10 (1,451 trans.)

Table 1: Experiment Setup Summary. Block Push : push cube past a target line. Pick-and-Place : grasp, lift

15

cm, place past a target line. Full details in Sections B.2 and B.3 .

Figure 2: Cumulative successes over training across simulated and real experiments. The onset and slope of each curve correspond to the learning time and approximate success rate of the method. Evaluations by checkpoint reported in Table 3 . [

-

PMO] (AIRL) and [

-

O] (MPAIL) are not evaluated in real due to their negligible success rates across simulation tasks.

Supervision

Components

Transfer

Scratch

Video

Method

Reward

Actions

Plan.

Model

Off- Policy

PnP

MoP

BP (rt.)

PnP (rt.)

BP (rt.)

PnP (rt.)

MPAIL2

✔

100

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

[

-

P] (MAIRL)

✔

\pm

\pm

3.9

\pm

2.8

\pm

\pm

\pm

\pm

\pm

[

-

PM] (DAC)

✔

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

BC (Diffusion)

✔

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

RLPD

✔*

✔

\pm

\pm

\pm

\pm

\pm

\pm

\pm

\pm

Table 2: Real-world task evaluations summarizing over 96 total training runs from scratch. (3 seeds/task; 50 trials/task): Success rate (%, mean

\pm

std across seeds) at best checkpoint (full checkpoint evaluations in Section B.1 ). [

-

PMO] (AIRL) and [

-

O] (MPAIL) are not evaluated in real experiments due to impractically poor sample efficiency. Transfer : new push/place direction, initialized from last checkpoint of initial task. Scratch : new task trained from scratch. BP: Block Push, PnP: Pick-and-Place, MoP: Mug-on-Plate, rt.: right (opposite direction). *RLPD given known dense reward where available, otherwise classifier-based [ 30 ] reward.

Figure 3: Cumulative successes in real-world transfer experiments compared to training from scratch. Methods are trained on the initial task (e.g. Block Push), then weights are transferred to training on a new task (e.g. Block Push right). Dynamics Transfer reflects transfer of only encoder and dynamics model weights. Numerical results are in Table 4 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

\displaystyle\mathcal{B}\leftarrow\mathcal{B}\cup\{(o_{t},a_{t},o_{t+1})\}_{t=1}^{T}

(7)

Appendix

Algorithm 1 MPAIL2

Figure 5: MPAIL2 $\lambda$ -Return. Components are unrolled over horizon $H$ . The encoder $e_{\omega}$ produces $z_{t}$ ; the policy produces action sequences $\hat{a}_{t},\ldots,\hat{a}_{t+H}$ ; dynamics $f_{\psi}$ predicts next latent states; reward $r_{\theta}$ scores transitions; value $Q$ estimates returns. With the correct coefficients, these predictions compute the $\lambda$ -return $G_{t}^{\lambda}$ for actor-critic updates ( Equation 6 )

Sim: Block Push (State)
Ckpt	MPAIL2	MPAIL2	MPAIL2	MPAIL2	MPAIL2	RLPD	BC
(# steps)		[-P] (MAIRL)	[-PM] (DAC)	[-PMO] (AIRL)	[-O] (MPAIL)		(Diffusion)
100	$64.0$ 13.0	$21.0$ 14.7	$14.0$ 14.0	$0.0$ 0.0	$0.0$	$4.0$ 5.9	$60.2$ 1.6
200	$76.0$ 8.1	$6.0$ 6.0	$43.0$ 21.0	$0.0$ 0.0	$0.0$	$35.4$ 4.1	$60.2$ 1.6
300	$71.0$ 13.7	$4.0$ 4.0	$36.0$ 22.1	$8.0$ 8.0	$4.0$ 8.9	$54.8$ 11.5	$60.2$ 1.6
400	$87.0$ 6.4	$0.0$ 0.0	$24.0$ 19.4	$0.0$ 0.0	$22.4$ 32.9	$58.2$ 9.0	$60.2$ 1.6

Appendix

Table 3: Full Checkpoint History by Environment Steps in Simulation. BC (Diffusion) is offline; its performance is constant across interaction budgets.

Real: Block Push
Ckpt	MPAIL2	[-P] (MAIRL)	[-PM] (DAC)	RLPD	BC (Diff.)
Last	$61.8\pm 13.5$	$34.0\pm 17.7$	$0.0\pm 0.0$	$0.0\pm 0.0$	$\mathbf{94.1\pm 5.9}$
Best (#)	$\mathbf{100\pm 0}$ (100)	$64.0\pm 17.7$ (130)	$0.0\pm 0.0$ (—)	$0.0\pm 0.0$ (—)	$\mathbf{94.1\pm 5.9}$ (—)
Real: Pick-and-Place
Last	$\mathbf{68.0\pm 13.6}$	$16\pm 14.8$	$0.0\pm 0.0$	$0.0\pm 0.0$	$12.0\pm 5.8$
Best (#)	$\mathbf{82.0\pm 12.2}$ (140)	$16.0\pm 14.8$ (150)	$0.0\pm 0.0$ (—)	$0.0\pm 0.0$ (—)	$12.0\pm 5.8$ (—)

Appendix

Table 4: Real-World Results. Success rate (%)

\pm

std across seeds by task across methods. Evaluations performed over 50 trials/task over 3 seeds. Last : final-checkpoint success. Best (#) : best-checkpoint success with corresponding average environment-step count.

Component	Hyperparameter	Value
	Optimizer	Adam
	Learning Rate (LR)	3e-4
	Return Parameter ( $\lambda$ )	0.95
	Discount ( $\gamma$ )	0.99
	Planning Horizon ( $H$ )	7
	Replay Size	$\infty$

Appendix

Table 5: MPAIL2 Hyperparameters . Used across all experiments unless specified otherwise. Only the Reward model does not employ LayerNorm.

Panel of Figure 9 — Figure 9: Overview of evaluation tasks. The Pick and Place tasks (a,c) involve reaching the block (or mug), grasping it, lifting and placing it beyond a target line (or on the plate). The Push tasks (b,d) involve reaching the block and pushing it beyond a target line.

Figure 10: Results in Gymnasium environments [ 51 ] . All methods are provided 50 synthetic demonstration observations. RLPD is additionally provided dense reward and demonstration actions. Reward is normalized such that 1.0 reflects the average reward across the demonstrations.

Figure 11: MPAIL2 ablations over number of demonstrations (left), horizon lengths (middle), and gradient penalty coefficients (right) in Sim: Block Push.

Figure 12: Time efficiency of MPAIL2 compared to other methods . We plot

t_{\text{algo}}(c)/t_{\text{MPAIL2}}(c)

where

t(c)

is the learning iteration at which the algorithm achieves a cumulative success percentage. With respect to a given baseline, the value reflects how many times faster MPAIL2 achieves a given success percentage. An endpoint indicates where the corresponding baseline achieves its maximum success percentage.

Figure 13: Success rate stability . We plot the average success rate of each method over a moving window of 25 episodes. A value of 1 indicates success on all episodes across 25 updates. This graph captures a method’s stability across updates. In Sim: Block Push (State) and Sim: Pick and Place , MPAIL2 is able to maintain a consistent success rate of roughly 0.8 and 0.6 respectively while the decreasing curve in Sim: Block Push indicates instability as witnessed by the flattening of the cumulative success curve in Figure 2 .

Figure 14: Policy influence on planning. Influence is computed as the fraction of total MPPI score attributable to policy-seeded rollouts among elite samples at the final planning iteration (see Algorithm 2 ), averaged over all timesteps in an episode.

Figure 15: Top. MPAIL2 on Sim: Block Push (state) at three timesteps. Bottom. Predicted plans in the XY plane for the next one second. End-effector plans in green ; block plans in orange . Block trajectories are static until contact, illustrating causal world modeling. Planning is in latent space; a separately trained decoder is used for visualization.

Figure 16: Real-world planning visualization on the Mug-on-Plate task. Each row shows five frames at training iterations 0, 5, 15, 30, 60, and 100. Red lines show the top- $k$ end-effector plans; the green line is the optimal plan selected by the planner. Early episodes show unstructured, unmodeled, and exploratory behavior; by episode 100 the robot reaches, grasps, and places the mug onto the plate.

Explore similar work

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models
Sep 25, 2026Yuan Fang, Zechu Li, Haolei Tong +2Robotic RL Agentic RL
TU Darmstadt · University of Augsburg · Tongji University +1
Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation
Sep 8, 2026Weiwei Gu, Anmol Gupta, Anant Sah +5Long-Horizon Robotic Manipulation Robot Skill Learning
School of Augmented Intelligence, Arizona State University · NVIDIA
In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks
Sep 29, 2026Minxing Li, Minghao Han, Weizhi Zhao +10ICL for Robotics Robotic Manipulation
NLPR, Institute of Automation, Chinese Academy of Sciences (CASIA) · Amap, Alibaba Group

Sep 25, 2026J/K move · Enter open · S save

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models

Yuan Fang, Zechu Li, Haolei Tong, Puze Liu+1

TU Darmstadt · University of Augsburg · Tongji University+1

Robotic RL Agentic RL RL for VLMs Robotic Manipulation

Open paper AskPDF arXiv

Vision--language--action (VLA) models provide strong priors for robotic manipulation but are typically deployed as frozen policies, unable to improve from their own failures. Real-world reinforcement learning (RL) offers a path to continued improvement, yet manual environment resets and task-success supervision hinder autonomous learning. We introduce FIND, an agentic real-world RL framework that closes the loop between scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace. FIND reframes autonomous practice as a scene-conditioned, performance-aware task-selection problem: instead of restoring a predefined scene after each rollout, it uses the resulting scene to determine what to practice next. A vision--language agent identifies feasible tasks from a predefined library, prioritizes those with lower recent success rates, and evaluates outcomes using paired pre- and post-execution observations. We instantiate FIND with a frozen

π_{0.5}

VLA and residual off-policy RL. Across eight real-world manipulation tasks, the independent human-assessed success rate improves from

55\%

71.9\%

. A representative run completes 456 autonomous episodes within 6 hours of interaction, requiring 30 scene-recovery interventions and no human-provided reward labels during online learning. Ablations and systematic evaluations further examine key design choices, agent evaluation accuracy, and human intervention requirements. Our website is made publicly available at: FIND.github.io.

Abstract

Explore similar work

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models

Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation

In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks