Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics

Peiyan Tu; Cong Wang; Hanxin Zhu; Xin Jin; Zhibo Chen

cs.ROOct 9, 2026

Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics

Authors: Peiyan Tu, Cong Wang, Hanxin Zhu, Xin Jin, Zhibo Chen

Organizations: Zhejiang University · Beijing Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences · University of Science and Technology of China ·

Abstract

Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning. Meanwhile, the predominance of successful demonstrations leaves failure dynamics substantially underrepresented, limiting the ability of world models to anticipate unsuccessful executions. To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training. To bridge the embodiment gap, we develop a dual-branch framework that couples video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and robot interactions while suppressing egocentric camera motion. To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth videos. Extensive experiments demonstrate that HF-WM achieves state-of-the-art performance in action-conditioned video generation and exhibits strong utility for policy evaluation in downstream robotic tasks.

Figures & tables

Figure 1: Overview of HF-WM architecture. HF-WM extends learning beyond successful robot demonstrations in two stages. Stage 1 uses a dual-branch architecture with stabilized pointflow supervision to learn interaction dynamics from egocentric human interactions (Sec. 3.2 ). Stage 2 applies GRPO to counterfactual action perturbations, guided by action alignment and physical plausibility rewards, to model failure dynamics without paired failure-video supervision (Sec. 3.3 ).

Figure 2: Dual-branch architecture of HF-WM with stabilized pointflow. The auxiliary pointflow branch predicts interaction motion from intermediate video representations, while pointflow stabilization suppresses egocentric camera motion so that supervision is better focused on interaction-induced dynamics. The pointflow branch is used only during training and introduces no additional inference overhead.

Figure 3: Qualitative comparison of action-conditioned video generation. Evaluated on both successful and failed execution samples, our method is compared against six models and achieves much better visual quality with precise action following.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	FID $\downarrow$	$\mathbf{L2}_{\mathrm{latent}}\downarrow$	IDM $\downarrow$
Cosmos-Predict2.5 ( Ali et al., 2025 )	18.874	0.746	0.266	60.119	54.108	0.486	1.268
TesserAct ( Zhen et al., 2025 )	17.383	0.757	0.263	60.563	59.889	0.523	1.663
Ctrl-World ( Guo et al., 2025 )	21.683	0.770	0.305	36.916	64.665	0.418	0.949
Cosmos 3 ( Agarwal et al., 2026 )	20.233	0.766	0.178	42.987	36.780	0.365	1.108
OSCAR ( Wu and Gao, 2026 )	23.357	0.840	0.142	31.815	28.241	0.292	0.579
Kinema4D ( Xu et al., 2026 )	20.015	0.804	0.157	30.511	30.890	0.326	0.914

Table 1: Quantitative comparison with baselines. Best / second-best results are bolded / underlined.

Figure 4: Qualitative ablation of the key components in HF-WM. Large-scale human interaction data broadens the diversity of object interactions, while unified pointflow supervision provides a shared motion representation that facilitates cross-embodiment transfer. Building on this foundation, GRPO post-training improves the modeling of failure dynamics across diverse execution outcomes.

Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	FID $\downarrow$	$\mathbf{L2}_{\mathrm{latent}}\downarrow$	IDM $\downarrow$
baseline(Cosmos3)	20.233	0.766	0.178	42.987	36.780	0.365	1.108
embodied data only	21.448	0.833	0.120	23.650	20.574	0.332	0.424
+ human	24.997	0.876	0.093	15.229	16.980	0.303	0.391
$\Delta$ gain $\uparrow$	3.549	0.043	0.027	8.421	3.594	0.029	0.033
embodied data + pointflow	21.573	0.835	0.117	22.993	20.011	0.329	0.406
+ human + pointflow	25.294	0.880	0.090	15.327	16.972	0.227	0.376

Table 2: Ablation study on the effectiveness of human interaction data and pointflow constraints.

Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	FID $\downarrow$	$\mathbf{L2}_{\mathrm{latent}}\downarrow$	IDM $\downarrow$
w/o GRPO	25.828	0.887	0.086	14.993	15.991	0.198	0.355
failure-aware GRPO + action alignment	25.826	0.887	0.086	14.957	15.957	0.198	0.348
failure-aware GRPO + physical plausibility	25.574	0.862	0.098	15.307	15.454	0.209	0.381
failure-aware GRPO + full rewards	26.077	0.889	0.082	14.449	14.291	0.193	0.332

Table 3: Quantitative ablation of GRPO for learning from failure cases.

Figure 5: Real-world and HF-WM estimated success rates across six generalist robot policies.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: HF-WM architecture and attention visibility. Left: attention mask. Right: accessible keys/values and masking constraints for AR, DM, and PF queries.

Figure 7: Visualization of pointflow stabilization on egocentric interactions. Camera-motion compensation suppresses coherent background drift while preserving hand-object interaction motion.

Event class	Gripper event pattern	Target share	None	Gripper perturbation	Geometric total
close_hold	Open $\rightarrow$ Closed (hold)	18.75%	50%	reopen: 35%	15%
close_reopen	Open $\rightarrow$ Closed $\rightarrow$ Open	18.75%	50%	force_closed: 35%	15%
open_hold	Closed $\rightarrow$ Open (hold)	18.75%	50%	reclose: 8.75% suppress: 26.25%	15%
open_close	Closed $\rightarrow$ Open $\rightarrow$ Closed	18.75%	50%	force_open: 35%	15%
remain_open	Open (hold)	12%	50%	Not applied	50%
partial_hold	Indeterminate	8%	50%	Not applied	50%

Appendix

Table 4: Gripper event patterns, target sampling shares, and perturbation routing probabilities. Target shares are relative to all training samples; routing probabilities are conditional on the event class and sum to 100% per row.

Figure 8: Qualitative examples for the action alignment reward. Each row compares video-action pairs with higher (left) and lower (right) alignment. Action-skeleton overlays in full-frame views and end-effector crops highlight differences in consistency between the specified actions and observed robot motion.

Figure 9: Prompt template for GT Describer and Future-Event Contract Generator.

Method (Success/Failure)	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	FID $\downarrow$	$\mathbf{L2}_{\mathrm{latent}}\downarrow$	IDM $\downarrow$
Cosmos-Predict2.5 ( Ali et al., 2025 )	19.869/17.878	0.755/0.736	0.268/0.263	42.001/78.237	47.525/60.691	0.449/0.523	1.191/1.344
TesserAct ( Zhen et al., 2025 )	18.192/16.574	0.762/0.751	0.245/0.280	59.452/61.673	59.053/60.724	0.517/0.528	1.522/1.804
Ctrl-World ( Guo et al., 2025 )	21.746/21.620	0.788/0.752	0.301/0.309	32.247/41.585	61.721/67.608	0.406/0.429	0.887/1.011
Cosmos 3 ( Agarwal et al., 2026 )	21.172/19.293	0.815/0.716	0.142/0.213	38.512/47.461	34.205/39.354	0.349/0.381	1.101/1.114
OSCAR ( Wu and Gao, 2026 )	23.674/23.040	0.860/0.820	0.118/ 0.165	29.638/33.992	21.297/35.185	0.243/0.340	0.445/0.712
Kinema4D ( Xu et al., 2026 )	20.630/19.400	0.834/0.773	0.117 /0.197	27.155/33.866	23.495/38.285	0.270/0.381	0.904/0.923

Appendix

Table 5: Quantitative comparison with baselines on successful and failed executions. Each entry reports Success / Failure. Bold / underline denote the best / second-best results.

Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	FID $\downarrow$	$\mathbf{L2}_{\mathrm{latent}}\downarrow$	IDM $\downarrow$
mid [12,14,16,18]	25.756	0.886	0.087	15.163	16.031	0.201	0.377
back [20,22,24,26]	25.689	0.885	0.087	15.326	16.361	0.202	0.393
uniform [8,14,20,26]	25.761	0.886	0.086	15.047	16.211	0.201	0.357
front [4,6,8,10]	25.828	0.887	0.086	14.993	15.991	0.198	0.355

Appendix

Table 6: Ablation study on the feature extraction layers of the pointflow branch.(SFT)

Figure 10: Intermediate rollout examples during GRPO training with individual reward components. Each panel shows corresponding frames from a ground-truth video and four candidate rollouts. Top: With only the action alignment reward, some candidates exhibit physically implausible interaction dynamics and degraded background quality. Bottom: With only the physical plausibility reward, the generated motion appears physically plausible but deviates from the conditioning actions. ( Note. Variation across candidates arises from stochastic exploration during GRPO training. Their relatively lower visual fidelity is a consequence of the reduced denoising steps adopted for training efficiency.)

Figure 11: Qualitative case of GRPO post-training under action perturbations. Starting from the original action skeleton, we introduce two counterfactual perturbations, Up and Reopen . Before GRPO, the model fails to faithfully reflect the perturbed actions and produces physically implausible outcomes, such as the cup remaining unnaturally suspended instead of falling after release. After GRPO, the generated candidates better follow the perturbed actions and exhibit more physically plausible object dynamics.

Pearson $r\uparrow$	Spearman $\rho\uparrow$	Kendall $\tau_{b}\uparrow$	$\mathrm{SISR}_{\Delta}\downarrow$ (pp)	MMRV $\downarrow$
$0.9826$	$0.9559$	$0.9286$	$2.41$	$0.002$

Appendix

Table 8: HF-WM policy evaluation on 90 matched sessions.

Figure 14: Additional qualitative results for HF-WM.

Panel of Figure 15 — Figure 15: Additional qualitative comparison results

Figure 16: Uncover the wooden block. Each strip shows the world-model rollout (top) and the real-robot video (bottom) at six time steps. The real column is the RoboArena human success label; the WM column is the world-model verdict (Sec. 4.4 ). A red cell marks disagreement. On this task 1 of 6 policies succeeds on the real robot.

Figure 17: Put the food on the plate. Each strip shows the world-model rollout (top) and the real-robot video (bottom) at six time steps. The real column is the RoboArena human success label; the WM column is the world-model verdict (Sec. 4.4 ). A red cell marks disagreement. On this task 5 of 6 policies succeed on the real robot.

Action-Conditioned World Models

Action-conditioned world models predict how environments change under actions to support planning, control, and simulation.

See all 346 papers

Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training
Apr 23, 2026Yaxuan Li, Zhongyi Zhou, Yefei Chen +5Robot Policy Learning Robot Policy Adaptation
Current Robotics · Tsinghua University · Peking University +1
AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization
Aug 29, 2026Cheng Chen, Jerry Bai, Jiacheng Wei +11Robot Skill Learning World Models for Robotics
Nanyang Technological University · XPENG Robotics · Zhejiang University +2
GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions
Aug 6, 2026Chenghao Gu, Hanyang Yu, Jingbo Zhang +7Action-Conditioned World Models Robotic Policy Evaluation
Shenzhen International Graduate School, Tsinghua University · Tencent Robotics X · The Hong Kong University of Science and Technology +1

Apr 23, 2026J/K move · Enter open · S save

Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training

Yaxuan Li, Zhongyi Zhou, Yefei Chen, Yanjiang Guo+4

Current Robotics · Tsinghua University · Peking University+1

Robot Policy Learning Robot Policy Adaptation Robot Skill Learning World Models for Robotics Action-Conditioned World Models Human-in-the-Loop AI+1

Open paper AskPDF arXiv

Post-training is essential for turning pretrained generalist robot policies into reliable task-specific controllers, but existing human-in-the-loop pipelines remain tied to physical execution: each correction requires robot time, scene setup, resets, and operator supervision in the real world. Meanwhile, action-conditioned world models have been studied mainly for imagination, synthetic data generation, and policy evaluation. We propose Human-in-the-World-Model (Hi-WM), a post-training framework that uses a learned world model as a reusable corrective substrate for failure-targeted policy improvement. A policy is first rolled out in closed loop inside the world model; when the rollout becomes incorrect or failure-prone, a human intervenes directly in the model to provide short corrective actions. Hi-WM caches intermediate states and supports rollback and branching, allowing a single failure state to be reused for multiple corrective continuations and yielding dense supervision around behaviors that the base policy handles poorly. The resulting corrective trajectories are then added back to the training set for post-training. We evaluate Hi-WM on three real-world manipulation tasks spanning both rigid and deformable object interaction, and on two policy backbones. Hi-WM improves real-world success by 37.9 points on average over the base policy and by 19.0 points over a world-model closed-loop baseline, while world-model evaluation correlates strongly with real-world performance (r = 0.953). These results suggest that world models can serve not only as generators or evaluators, but also as effective corrective substrates for scalable robot post-training.

Abstract

Action-Conditioned World Models

Explore similar work

Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training

AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization

GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions