MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models

Yi Qiao; Maolin Zheng; Haoran Sun; Lujia Dong; Wenzhao Lian

cs.ROOct 15, 2026

MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models

Authors: Yi Qiao, Maolin Zheng, Haoran Sun, Lujia Dong, Wenzhao Lian

Organizations: School of Artificial Intelligence, Shanghai Jiao Tong University

Abstract

Reliable contact-rich manipulation requires detecting execution deviations early enough for corrective action. Yet most monitoring methods are reactive, identifying anomalies only after they become observable. Anticipating such anomalies is challenging because visual observations only partially reveal physical interaction, while planned actions describe intended motion rather than their actual physical consequences. We address this problem with MAIA, a multimodal framework for anomaly anticipation. Its vision-action-force world model learns nominal interaction dynamics and predicts the visual and force responses of planned actions, while a lightweight anomaly head predicts impending interaction anomalies. To enable systematic evaluation, we introduce CRIA, a real-world multimodal dataset of 2,852 trajectories spanning 18 contact-rich manipulation tasks, with frame-level annotations of anomaly onset and evolution. Experiments on both seen and novel tasks show that MAIA achieves more accurate anomaly discrimination than representative baselines while providing earlier warnings, with bACC scores of 0.765 and 0.710 and median anticipation lead times of 28.0 and 25.1 steps, respectively. Moreover, using these predictions for anomaly-aware action selection improves end-to-end task success from 52.6% to 61.5% without retraining the underlying manipulation policy. These results highlight the value of modeling future physical consequences for proactive anomaly anticipation.

Figures & tables

Fig. 2: CRIA overview. The central ring presents all 18 tasks grouped by interaction category, while the surrounding panels illustrate category-specific interaction characteristics and typical anomaly modes. The hardware setup used for data collection is shown on the right.

Dataset	Physical Sensing	Anomaly Source	Real Anomalous Traj.	Anomaly Supervision	Granularity
RH20T [ 29 ]	✓	Human	$\sim$ 10k	Outcome	Episode
REASSEMBLE [ 30 ]	✓	Human	516	Outcome	Segment
Hoi! [ 31 ]	✓	–	–	–	–
FreeTacMan [ 32 ]	✓	–	–	–	–
SOAR [ 33 ]	$\times$	Policy	20,564	Outcome	Episode
RoboFAC [ 34 ]	$\times$	Human + Injected	480	Outcome + Type + Explanation	Subtask

TABLE I: Comparison with representative multimodal manipulation and failure/anomaly datasets.

Fig. 3: MAIA Overview. The framework consists of a Vision-Action-Force World Model (VAF-WM) and a predictive anomaly head. VAF-WM extends GE-Base with gated action-force adapters to forecast future visual and force dynamics. The anomaly head fuses these predictions with interaction tokens and uses a temporal transformer to predict anomaly occurrence, temporal prevalence, and localization within the future window.

Method	Seen Tasks				Novel Tasks
Method	AUPRC $\uparrow$	bACC $\uparrow$	AR $\uparrow$	Med. ALT $\uparrow$	AUPRC $\uparrow$	bACC $\uparrow$	AR $\uparrow$	Med. ALT $\uparrow$
PCA-KMeans [ 38 ]	0.488	0.576	0.113	14.3	0.589	0.508	0.142	16.4
LogpZO [ 6 ]	0.551	0.619	0.145	16.0	0.556	0.504	0.161	24.6
SAFE-LSTM [ 7 ]	0.583	0.724	0.182	18.8	0.426	0.495	0.103	22.2
Robo-Dopamine [ 39 ]	0.411	0.523	0.352	18.9	0.500	0.546	0.361	17.4
Robometer [ 40 ]	0.356	0.512	0.320	12.0	0.469	0.524	0.377	15.3

TABLE II: Anomaly anticipation performance on seen and novel tasks, macro-averaged across tasks.

Variant	Insertion		Wiping		Pressing		Path-Constrained		Articulated-Object		Average
Variant	Seen	Novel	Seen	Novel	Seen	Novel	Seen	Novel	Seen	Novel	Seen	Novel
MAIA-V	0.709	0.766	0.684	0.615	0.655	0.668	0.724	0.654	0.674	0.624	0.689	0.665
MAIA-VA	0.764	0.815	0.604	0.616	0.663	0.615	0.669	0.558	0.623	0.637	0.665	0.648
MAIA	0.868	0.832	0.743	0.683	0.702	0.662	0.763	0.742	0.748	0.631	0.765	0.710

TABLE III: Ablation of visual, action, and force modeling across task categories.

Fig. 4: Qualitative comparison of anomaly predictions. Shaded regions indicate correct predictions (green), prediction errors (red), and early warnings (yellow) prior to the true anomaly onset (dashed line). (a) Nominal execution: our method produces fewer false alarms than the baselines. (b) Anomalous execution: our method provides an earlier sustained warning than WCM, while reward-model warnings are intermittent.

Fig. 5: Qualitative modality ablation on representative anomaly cases. The examples illustrate how incorporating physical feedback enables MAIA to distinguish anomalous interactions that remain ambiguous when only visual and action cues are available.

Fig. 6: Novel-task adaptation with varying amounts of task-specific data.

Method	Insertion	Wiping	Pressing	Path.	Articulated	Overall
$\pi_{0.5}$	63.3	46.7	48.3	48.9	53.3	52.6
$\pi_{0.5}$ + MAIA	63.3	66.7	54.2	62.2	62.5	61.5

TABLE IV: Real-robot task success rates (%).

Category / Task	Traj.	Anomalous Traj.	Anomaly Events	Anomalous Frames
Insertion
Insert-Plug-2	150	128	221	35.13%
Insert-Plug-3	150	133	171	44.11%
Insert-Type-C	150	139	156	38.83%
Insert-NI	150	140	161	50.30%
Subtotal	600	540	709	42.11%

TABLE V: Statistics of the CRIA dataset.

Insertion
Task: Insert-Plug-2 Instruction: Insert a two-prong plug. Description: Insert the two-prong power plug into its socket with precise alignment.	Task: Insert-Plug-3 Instruction: Insert a three-prong plug. Description: Insert the three-prong power plug into its socket with precise alignment.	Task: Insert-Type-C Instruction: Insert the typec. Description: Insert the Type-C connector into its port with correct orientation and fine alignment.	Task: Insert-NI Instruction: Insert the ethernet port. Description: Insert the RJ45 plug into the ethernet port with precise alignment.
Wiping
Task: Wipe-Board Instruction: Erase the whiteboard. Description: Bring the eraser into contact with the whiteboard, then move it across the board along a tangential wiping path while maintaining normal contact to remove the markings.	Task: Wipe-Bowl Instruction: Wipe the bowl from left to right. Description: Use the gripper-held wiping tool to contact the bowl’s inner surface, then move it along the inner rim while maintaining normal contact to clean the bowl.	Task: Wipe-Vase Instruction: Wipe the vase. Description: Bring the eraser into contact with the vase’s outer surface, then move it along the vase’s outer contour while maintaining normal contact to remove the markings.
Pressing
Task: Press-Seal Instruction: Press the seal on the white paper. Description: Press the seal onto the white paper with adequate axial force along the pressing axis to ensure a clear impression.	Task: Press-Switch-1 Instruction: Press the switch. Description: Contact the rocker switch, then press it with localized force at the correct position and angle to toggle its state.	Task: Press-Switch-2 Instruction: Press the red button. Description: Contact the red push-button, then press it with sufficient normal force and a short, controlled stroke to light up the LED strip.	Task: Press-Bottle Instruction: Press the pump to dispense liquid. Description: Contact the pump, then press it with adequate axial force along the pressing axis to press out liquid while keeping the bottle stable.
Path-Constrained Manipulation

TABLE VI: Task specifications in CRIA.

Stage	Steps	Batch (per-GPU)	LR	Warmup	Loss weights
Stage 1: VAF-WM	30k	8	$3{\times}10^{-5}$	1k	$\lambda_{\mathrm{v}}=1.0$ , $\lambda_{\mathrm{f}}=1.0$ , $\lambda_{\Delta\mathrm{f}}=0.2$ , $\lambda_{\mathrm{smooth}}=0.1$ , $\lambda_{\mathrm{sem}}=0.05$
Stage 2: Anomaly Head	5k	32	$5{\times}10^{-5}$	500	$\lambda_{\mathrm{y}}=1.0$ , $\lambda_{\mathrm{p}}=0.5$ , $\lambda_{\mathrm{a}}=0.5$
Stage 3: Novel-Task Adaptation	2k	8	$1{\times}10^{-5}$	500	$\lambda_{\mathrm{y}}=1.0$ , $\lambda_{\mathrm{p}}=0.5$ , $\lambda_{\mathrm{a}}=0.5$ , $\lambda_{\mathrm{force}}^{\mathrm{nom}}=0.1$

TABLE VII: Training hyperparameters for MAIA.

Category / Task	AUPRC $\uparrow$	bACC $\uparrow$	AR $\uparrow$	Med. ALT $\uparrow$
Insertion
Insert-Plug-2	0.949	0.887	0.581	27.0
Insert-Plug-3	0.963	0.912	0.567	39.5
Insert-Type-C	0.973	0.804	0.412	20.0
Insert-NI ∗	0.962	0.832	0.470	25.0
Wiping

TABLE VIII: Task-level anomaly anticipation performance across all 18 tasks.

Fig. 7: Comparison of the original

\pi_{0.5}

policy and

\pi_{0.5}

with MAIA across 18 contact-rich manipulation tasks. The rightmost column shows the success-rate difference in percentage points.

Anomaly Detection

Anomaly detection finds rare or unexpected patterns in data, from defective products and system faults to fraud and security threats.

See all 654 papers

Jun 7, 2026

FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation

Force signals provide critical interaction cues for contact-rich robotic manipulation. However, existing methods mostly use force as an additional observation modality, without fully exploiting its role in modeling future interaction dynamics or guiding execution-time feedback correction. In this paper, we propose FAWAM, a force-aware world action model that incorporates force information at three levels: perception, prediction, and closed-loop execution. FAWAM first encodes historical 6-axis force/torque signals to modulate action generation, then jointly predicts future actions and end-effector wrenches to explicitly model contact evolution. It further introduces a residual correction module that uses the predicted wrench trajectory as an execution-time reference to refine actions online based on real-time force feedback. Real-world experiments across multiple contact-rich tasks show that FAWAM improves the average success rate by 36.25% over vision-only baselines and 21.25% over existing force-aware baselines, demonstrating the effectiveness of our force-aware framework for robust contact-rich manipulation.

Haotian He, Zeyu Yan, Qipeng Liu +2

School of Mathematical Sciences, Peking University · School of Artificial Intelligence, Shanghai Jiao Tong University.

Robotic Control World Models for Robotics Contact-Rich Robotic Manipulation Robotic Manipulation Force-Controlled Robotic Manipulation

Ask PDF

Sep 18, 2026

PSR: Predictive Sensorimotor Representation Learning for Contact-Rich Manipulation

Contact-rich manipulation requires policies to generate precise actions by reasoning over contact forces, robot configurations, and interaction histories beyond visual observations. Existing methods passively condition on force feedback rather than actively predicting future contact dynamics, limiting their ability to generate high-precision actions. To address this problem, we introduce Predictive Sensorimotor Representation (PSR) learning, a framework that learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy. Specifically, during a pretraining stage, a multimodal Transformer is trained to learn a hierarchy of predictive representations by jointly forecasting future interaction dynamics. The learned hierarchy subsequently augments the action stream, enabling the resulting policy to exploit contact-relevant cues at multiple depths. We further instantiate PSR within a Vision-Language-Action (VLA) model, resulting in PSR-VLA, and evaluate it on six real-world contact-rich manipulation tasks. Experimental results show that PSR-VLA achieves 91.7% overall success, improving over

π_{0.5}

, ForceVLA-

π_{0.5}

, and ForceVLA2-

π_{0.5}

by 30.0, 22.5, and 19.2 percentage points, respectively. These results demonstrate the effectiveness of the proposed PSR for force-aware, contact-rich manipulation. Videos of the tasks and stability tests are available at https://psr-vla.pages.dev/.

Shengbao Li, Peng Xu, Chao Tang +8

Samsung R&D Institute China-Beijing

Contact-Rich Robotic Manipulation Predictive Representation Learning Visuomotor Policy Learning VLMs for Robotics

Ask PDF

Aug 13, 2026

ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models

Contact-rich manipulation failures are often detected only after the robot has committed to contact. This is especially limiting in wrist-camera setups: close gripper--object views help observe contact, but a poor approach may already push, miss, slip, or disturb the object before conventional detectors react. We introduce ContactGuard, a pre-contact execution monitor for chunked visuomotor policies. Given the policy's planned action chunk, ContactGuard predicts its short-horizon consequence in latent visual space and aborts if the predicted future latent indicates likely failure. Its latent world model is trained from unlabelled robot trajectories to predict compact multi-view visual embeddings under planned actions, avoiding pixel-level video prediction. A lightweight failure probe is then trained from a small labelled set of pre-contact clips. At deployment, ContactGuard anchors prediction before an imminent contact event, rolls the model forward under the policy's own actions, and verifies the predicted post-contact latent. Across real-world contact-rich manipulation tasks, ContactGuard predicts failure more accurately than direct and corrupted-action ablations, and transfers to live robot as a pre-contact abort signal without modifying the underlying policy.

Gehan Zheng, Matthew Johnson-Roberson, Weiming Zhi

College of Connected Computing, Vanderbilt University, USA. · School of Computer Science, The University of Sydney, Australia. · 3Australian Centre for Robotics, The University of Sydney, Australia.

Robot Failure Detection Contact-Rich Robotic Manipulation Action-Conditioned World Models Robotic Manipulation

Ask PDF

Abstract

Anomaly Detection

Explore similar work

FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation

PSR: Predictive Sensorimotor Representation Learning for Contact-Rich Manipulation

ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models