MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models

Yi Qiao; Maolin Zheng; Haoran Sun; Lujia Dong; Wenzhao Lian

cs.ROOct 15, 2026

MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models

Authors: Yi Qiao, Maolin Zheng, Haoran Sun, Lujia Dong, Wenzhao Lian

Organizations: School of Artificial Intelligence, Shanghai Jiao Tong University

Abstract

Reliable contact-rich manipulation requires detecting execution deviations early enough for corrective action. Yet most monitoring methods are reactive, identifying anomalies only after they become observable. Anticipating such anomalies is challenging because visual observations only partially reveal physical interaction, while planned actions describe intended motion rather than their actual physical consequences. We address this problem with MAIA, a multimodal framework for anomaly anticipation. Its vision-action-force world model learns nominal interaction dynamics and predicts the visual and force responses of planned actions, while a lightweight anomaly head predicts impending interaction anomalies. To enable systematic evaluation, we introduce CRIA, a real-world multimodal dataset of 2,852 trajectories spanning 18 contact-rich manipulation tasks, with frame-level annotations of anomaly onset and evolution. Experiments on both seen and novel tasks show that MAIA achieves more accurate anomaly discrimination than representative baselines while providing earlier warnings, with bACC scores of 0.765 and 0.710 and median anticipation lead times of 28.0 and 25.1 steps, respectively. Moreover, using these predictions for anomaly-aware action selection improves end-to-end task success from 52.6% to 61.5% without retraining the underlying manipulation policy. These results highlight the value of modeling future physical consequences for proactive anomaly anticipation.

Figures & tables

Fig. 2: CRIA overview. The central ring presents all 18 tasks grouped by interaction category, while the surrounding panels illustrate category-specific interaction characteristics and typical anomaly modes. The hardware setup used for data collection is shown on the right.

Dataset	Physical Sensing	Anomaly Source	Real Anomalous Traj.	Anomaly Supervision	Granularity
RH20T [ 29 ]	✓	Human	$\sim$ 10k	Outcome	Episode
REASSEMBLE [ 30 ]	✓	Human	516	Outcome	Segment
Hoi! [ 31 ]	✓	–	–	–	–
FreeTacMan [ 32 ]	✓	–	–	–	–
SOAR [ 33 ]	$\times$	Policy	20,564	Outcome	Episode
RoboFAC [ 34 ]	$\times$	Human + Injected	480	Outcome + Type + Explanation	Subtask

TABLE I: Comparison with representative multimodal manipulation and failure/anomaly datasets.

Fig. 3: MAIA Overview. The framework consists of a Vision-Action-Force World Model (VAF-WM) and a predictive anomaly head. VAF-WM extends GE-Base with gated action-force adapters to forecast future visual and force dynamics. The anomaly head fuses these predictions with interaction tokens and uses a temporal transformer to predict anomaly occurrence, temporal prevalence, and localization within the future window.

Method	Seen Tasks				Novel Tasks
Method	AUPRC $\uparrow$	bACC $\uparrow$	AR $\uparrow$	Med. ALT $\uparrow$	AUPRC $\uparrow$	bACC $\uparrow$	AR $\uparrow$	Med. ALT $\uparrow$
PCA-KMeans [ 38 ]	0.488	0.576	0.113	14.3	0.589	0.508	0.142	16.4
LogpZO [ 6 ]	0.551	0.619	0.145	16.0	0.556	0.504	0.161	24.6
SAFE-LSTM [ 7 ]	0.583	0.724	0.182	18.8	0.426	0.495	0.103	22.2
Robo-Dopamine [ 39 ]	0.411	0.523	0.352	18.9	0.500	0.546	0.361	17.4
Robometer [ 40 ]	0.356	0.512	0.320	12.0	0.469	0.524	0.377	15.3

TABLE II: Anomaly anticipation performance on seen and novel tasks, macro-averaged across tasks.

Variant	Insertion		Wiping		Pressing		Path-Constrained		Articulated-Object		Average
Variant	Seen	Novel	Seen	Novel	Seen	Novel	Seen	Novel	Seen	Novel	Seen	Novel
MAIA-V	0.709	0.766	0.684	0.615	0.655	0.668	0.724	0.654	0.674	0.624	0.689	0.665
MAIA-VA	0.764	0.815	0.604	0.616	0.663	0.615	0.669	0.558	0.623	0.637	0.665	0.648
MAIA	0.868	0.832	0.743	0.683	0.702	0.662	0.763	0.742	0.748	0.631	0.765	0.710

TABLE III: Ablation of visual, action, and force modeling across task categories.

Fig. 4: Qualitative comparison of anomaly predictions. Shaded regions indicate correct predictions (green), prediction errors (red), and early warnings (yellow) prior to the true anomaly onset (dashed line). (a) Nominal execution: our method produces fewer false alarms than the baselines. (b) Anomalous execution: our method provides an earlier sustained warning than WCM, while reward-model warnings are intermittent.

Fig. 5: Qualitative modality ablation on representative anomaly cases. The examples illustrate how incorporating physical feedback enables MAIA to distinguish anomalous interactions that remain ambiguous when only visual and action cues are available.

Fig. 6: Novel-task adaptation with varying amounts of task-specific data.

Method	Insertion	Wiping	Pressing	Path.	Articulated	Overall
$\pi_{0.5}$	63.3	46.7	48.3	48.9	53.3	52.6
$\pi_{0.5}$ + MAIA	63.3	66.7	54.2	62.2	62.5	61.5

TABLE IV: Real-robot task success rates (%).

Category / Task	Traj.	Anomalous Traj.	Anomaly Events	Anomalous Frames
Insertion
Insert-Plug-2	150	128	221	35.13%
Insert-Plug-3	150	133	171	44.11%
Insert-Type-C	150	139	156	38.83%
Insert-NI	150	140	161	50.30%
Subtotal	600	540	709	42.11%

TABLE V: Statistics of the CRIA dataset.

Insertion
Task: Insert-Plug-2 Instruction: Insert a two-prong plug. Description: Insert the two-prong power plug into its socket with precise alignment.	Task: Insert-Plug-3 Instruction: Insert a three-prong plug. Description: Insert the three-prong power plug into its socket with precise alignment.	Task: Insert-Type-C Instruction: Insert the typec. Description: Insert the Type-C connector into its port with correct orientation and fine alignment.	Task: Insert-NI Instruction: Insert the ethernet port. Description: Insert the RJ45 plug into the ethernet port with precise alignment.
Wiping
Task: Wipe-Board Instruction: Erase the whiteboard. Description: Bring the eraser into contact with the whiteboard, then move it across the board along a tangential wiping path while maintaining normal contact to remove the markings.	Task: Wipe-Bowl Instruction: Wipe the bowl from left to right. Description: Use the gripper-held wiping tool to contact the bowl’s inner surface, then move it along the inner rim while maintaining normal contact to clean the bowl.	Task: Wipe-Vase Instruction: Wipe the vase. Description: Bring the eraser into contact with the vase’s outer surface, then move it along the vase’s outer contour while maintaining normal contact to remove the markings.
Pressing
Task: Press-Seal Instruction: Press the seal on the white paper. Description: Press the seal onto the white paper with adequate axial force along the pressing axis to ensure a clear impression.	Task: Press-Switch-1 Instruction: Press the switch. Description: Contact the rocker switch, then press it with localized force at the correct position and angle to toggle its state.	Task: Press-Switch-2 Instruction: Press the red button. Description: Contact the red push-button, then press it with sufficient normal force and a short, controlled stroke to light up the LED strip.	Task: Press-Bottle Instruction: Press the pump to dispense liquid. Description: Contact the pump, then press it with adequate axial force along the pressing axis to press out liquid while keeping the bottle stable.
Path-Constrained Manipulation

TABLE VI: Task specifications in CRIA.

Stage	Steps	Batch (per-GPU)	LR	Warmup	Loss weights
Stage 1: VAF-WM	30k	8	$3{\times}10^{-5}$	1k	$\lambda_{\mathrm{v}}=1.0$ , $\lambda_{\mathrm{f}}=1.0$ , $\lambda_{\Delta\mathrm{f}}=0.2$ , $\lambda_{\mathrm{smooth}}=0.1$ , $\lambda_{\mathrm{sem}}=0.05$
Stage 2: Anomaly Head	5k	32	$5{\times}10^{-5}$	500	$\lambda_{\mathrm{y}}=1.0$ , $\lambda_{\mathrm{p}}=0.5$ , $\lambda_{\mathrm{a}}=0.5$
Stage 3: Novel-Task Adaptation	2k	8	$1{\times}10^{-5}$	500	$\lambda_{\mathrm{y}}=1.0$ , $\lambda_{\mathrm{p}}=0.5$ , $\lambda_{\mathrm{a}}=0.5$ , $\lambda_{\mathrm{force}}^{\mathrm{nom}}=0.1$

TABLE VII: Training hyperparameters for MAIA.

Category / Task	AUPRC $\uparrow$	bACC $\uparrow$	AR $\uparrow$	Med. ALT $\uparrow$
Insertion
Insert-Plug-2	0.949	0.887	0.581	27.0
Insert-Plug-3	0.963	0.912	0.567	39.5
Insert-Type-C	0.973	0.804	0.412	20.0
Insert-NI ∗	0.962	0.832	0.470	25.0
Wiping

TABLE VIII: Task-level anomaly anticipation performance across all 18 tasks.

Fig. 7: Comparison of the original

\pi_{0.5}

policy and

\pi_{0.5}

with MAIA across 18 contact-rich manipulation tasks. The rightmost column shows the success-rate difference in percentage points.

Anomaly Detection

Anomaly detection finds rare or unexpected patterns in data, from defective products and system faults to fraud and security threats.

See all 654 papers

FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation
Jun 7, 2026Haotian He, Zeyu Yan, Qipeng Liu +2Robotic Control World Models for Robotics
School of Mathematical Sciences, Peking University · School of Artificial Intelligence, Shanghai Jiao Tong University.
PSR: Predictive Sensorimotor Representation Learning for Contact-Rich Manipulation
Sep 18, 2026Shengbao Li, Peng Xu, Chao Tang +8Contact-Rich Robotic Manipulation Predictive Representation Learning
Samsung R&D Institute China-Beijing
ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
Aug 13, 2026Gehan Zheng, Matthew Johnson-Roberson, Weiming ZhiRobot Failure Detection Contact-Rich Robotic Manipulation
College of Connected Computing, Vanderbilt University, USA. · School of Computer Science, The University of Sydney, Australia. · 3Australian Centre for Robotics, The University of Sydney, Australia.

Jun 7, 2026J/K move · Enter open · S save

FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation

Haotian He, Zeyu Yan, Qipeng Liu, Ning Guo+1

School of Mathematical Sciences, Peking University · School of Artificial Intelligence, Shanghai Jiao Tong University.

Robotic Control World Models for Robotics Contact-Rich Robotic Manipulation Robotic Manipulation Force-Controlled Robotic Manipulation

Open paper AskPDF arXiv

Force signals provide critical interaction cues for contact-rich robotic manipulation. However, existing methods mostly use force as an additional observation modality, without fully exploiting its role in modeling future interaction dynamics or guiding execution-time feedback correction. In this paper, we propose FAWAM, a force-aware world action model that incorporates force information at three levels: perception, prediction, and closed-loop execution. FAWAM first encodes historical 6-axis force/torque signals to modulate action generation, then jointly predicts future actions and end-effector wrenches to explicitly model contact evolution. It further introduces a residual correction module that uses the predicted wrench trajectory as an execution-time reference to refine actions online based on real-time force feedback. Real-world experiments across multiple contact-rich tasks show that FAWAM improves the average success rate by 36.25% over vision-only baselines and 21.25% over existing force-aware baselines, demonstrating the effectiveness of our force-aware framework for robust contact-rich manipulation.

Abstract

Anomaly Detection

Explore similar work

FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation

PSR: Predictive Sensorimotor Representation Learning for Contact-Rich Manipulation

ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models