Belief Dynamics for Detecting Behavioral Shifts in Safe Collaborative Manipulation

Devashri Naik; Divake Kumar; Nastaran Darabi; Amit Ranjan Trivedi

cs.ROApr 4, 2026

Belief Dynamics for Detecting Behavioral Shifts in Safe Collaborative Manipulation

Authors: Devashri Naik, Divake Kumar, Nastaran Darabi, Amit Ranjan Trivedi

Organizations: University of Illinois Chicago

Abstract

Robots operating in shared workspaces must maintain safe coordination with other agents whose behavior may change during task execution. When a collaborating agent switches strategy mid-episode, continuing under outdated assumptions can lead to unsafe actions and increased collision risk. Reliable detection of such behavioral regime changes is therefore critical. We study regime-switch detection under controlled non-stationarity in ManiSkill shared-workspace manipulation tasks. Across ten detection methods and five random seeds, enabling detection reduces post-switch collisions by 52%. However, average performance hides significant reliability differences: under a realistic tolerance of +-3 steps, detection ranges from 86% to 30%, while under +-5 steps all methods achieve 100%. We introduce UA-TOM, a lightweight belief-tracking module that augments frozen vision-language-action (VLA) control backbones using selective state-space dynamics, causal attention, and prediction-error signals. Across five seeds and 1200 episodes, UA-TOM achieves the highest detection rate among unassisted methods (85.7% at +-3) and the lowest close-range time (4.8 steps), outperforming an Oracle (5.3 steps). Analysis shows hidden-state update magnitude increases by 17x at regime switches and decays over roughly 10 timesteps, while the discretization step converges to a near-constant value (Delta_t approx 0.78), indicating sensitivity driven by learned dynamics rather than input-dependent gating. Cross-domain experiments in Overcooked show complementary roles of causal attention and prediction-error signals. UA-TOM introduces 7.4 ms inference overhead (14.8% of a 50 ms control budget), enabling reliable regime-switch detection without modifying the base policy.

Figures & tables

Fig. 1 : Shared-workspace regime-switch detection. Top : a collaborating agent switches from cooperative (Helper) to adversarial (Blocker), creating collision risk. Bottom : enabling detection reduces post-switch collisions by 52% across all methods (No-Detect: $2.34$ vs. detection mean: ${\sim}1.1$ ). Among detectors, collision rates are comparable (range 0.99–1.29), motivating evaluation of detection reliability and adaptation quality.

Fig. 2 : UA-ToM computation flow. A frozen VLA backbone extracts features $\mathbf{z}_{t}$ . The UA-ToM module (992K parameters, 0.014% of backbone) processes these through four pathways: a selective SSM maintains persistent belief $\mathbf{b}_{t}$ via learned dynamics $\mathbf{A},\mathbf{B}$ (discretization $\Delta_{t}$ converges to 0.78); causal attention re-anchors belief to recent observations; prediction error $e_{t}$ provides an explicit change signal; and contrastive memory supplies categorical type cues via EMA prototypes. A gated fusion layer integrates all signals to produce type $\hat{\theta}_{t}$ , switch probability $\hat{s}_{t}$ , and predicted action $\hat{\mathbf{a}}^{c}_{t}$ . Internal dynamics around the regime switch are shown in Fig. 6 .

Condition	Post-Switch Collisions
No detection	$2.34\pm 0.40$
Detection (mean across methods)	$1.11\pm 0.12$
Reduction	52%

TABLE I : Effect of regime-switch detection on post-switch collision rate (5 seeds). Lower is better.

	Hard Detection (%)
Method	S0	S1	S2	S3	S4	Mean	$R$
Ctx-Cond. †	86.8	87.8	85.5	87.5	82.8	86.1	0.94
UA-ToM	85.3	83.5	89.2	87.3	83.0	85.7	0.93
Mamba	85.5	85.0	87.3	85.2	81.8	85.0	0.94
Transformer	83.3	83.3	82.8	83.8	79.2	82.5	0.95
LIAM	62.3	56.8	61.0	63.8	68.2	62.4	0.83

TABLE II : Hard detection rate (%) per seed at

\pm 3

step tolerance (150 ms at 50 ms control rate). At

\pm 5

, all methods achieve 100%. † Receives ground-truth type.

Fig. 3 : Detection reliability heatmap at $\pm 3$ step tolerance. Hard detection rate (%) per method and transition type across 12 behavioral transitions (e.g. Helper $\to$ Blocker, Competitor $\to$ Passive). Rows are sorted by mean detection rate. Methods separate into three tiers: fast detectors ( $>$ 82%, blue), agent-modeling (58–62%, yellow), and slow recurrent models ( $<$ 34%, red). At the standard $\pm 5$ tolerance, all cells would be 100%, masking these operationally relevant differences. Transition difficulty varies: Helper $\to$ Competitor is consistently hardest across methods due to similar spatial approach patterns.

Method	Latency $\downarrow$	Post-Sw. Coll. $\downarrow$	Total Coll. $\downarrow$
Oracle	$1.0\pm 0.0$	$1.07\pm 0.58$	$2.69\pm 1.02$
UA-ToM	$\mathbf{9.9\pm 0.8}$	$1.01\pm 0.39$	$2.66\pm 0.52$
Ctx-Cond. †	$9.4\pm 1.1$	$1.10\pm 0.61$	$2.64\pm 0.73$
Transformer	$10.5\pm 1.3$	$1.29\pm 0.48$	$3.35\pm 1.10$
Mamba	$10.6\pm 1.0$	$1.11\pm 0.24$	$3.20\pm 0.51$
LIAM	$11.0\pm 0.6$	$0.99\pm 0.28$	$2.46\pm 0.36$

TABLE III : Closed-loop performance over five seeds (1200 episodes). † Privileged.

Method	Det% $\uparrow$	Post-Coll. $\downarrow$	CRT $\downarrow$	Total Coll. $\downarrow$
Oracle	$100.0$	$2.6\pm 0.7$	$5.3\pm 0.9$	$7.1\pm 1.0$
Ctx-Cond. †	$98.6\pm 0.4$	$\mathbf{2.1\pm 0.6}$	$4.9\pm 1.0$	$\mathbf{5.2\pm 1.3}$
UA-ToM (MS)	$97.5\pm 0.9$	$2.6\pm 0.7$	$\mathbf{4.8\pm 1.1}$	$5.9\pm 1.4$
Mamba	$98.5\pm 0.4$	$3.5\pm 0.9$	$7.0\pm 0.7$	$7.0\pm 1.4$
LIAM	$97.3\pm 0.7$	$3.4\pm 0.5$	$13.4\pm 1.5$	$7.1\pm 0.9$
UA-ToM	$95.1\pm 0.8$	$2.7\pm 0.6$	$9.2\pm 1.8$	$6.4\pm 1.1$

TABLE IV : Extended evaluation (300 steps, 10 seeds). Detection saturates; the differentiator is adaptation quality. CRT = close-range time (steps at

<

15 cm post-switch). † Privileged.

Fig. 4 : Close-range time (CRT) under extended episodes (300 steps, 10 seeds). CRT measures post-switch timesteps at $<$ 15 cm end-effector distance. UA-ToM (MS) achieves the lowest CRT (4.8), including lower than Oracle (5.3), indicating that smooth belief revision maintains safer clearance than instantaneous strategy switching. LIAM detects at 97.3% yet accumulates $2.8\times$ more danger time (13.4 steps). † Privileged.

Fig. 5 : Post-switch adaptation dynamics (300-step episodes, 10 seeds). (A) : Post-switch collision rate over time following $t_{s}$ , showing instantaneous risk at each timestep. UA-ToM (MS) exhibits a smooth, monotonic decrease consistent with the SSM’s gradual belief revision (Fig. 6 ). Oracle shows a sharp transient from instantaneous strategy switching. LIAM remains elevated for an extended window despite successful detection. (B) : Cumulative post-switch collisions over time. The separation between methods grows over the post-switch window, confirming that sustained proximity behavior, not isolated collision events, drives the safety differences captured by CRT.

	Cramped-1		Asym.-1		Cramped-2		Asym.-2
Method	Fl.	CV	Fl.	CV	Fl.	CV	Fl.	CV
UA-ToM	87	.048	99	.007	99	.005	97	.017
Transformer	83	.049	99	.006	97	.013	97	.013
GRU	57	.208	86	.061	92	.033	90	.033

TABLE V : Detection reliability across Overcooked layouts. Floor = worst-seed detection rate (

\mu-\sigma

). CV = coefficient of variation across five seeds.

Model	Min Dist.	Mean Dist.	Silhouette	Hardest Pair
BOCPD	38.1	322.2	0.40	Help.–Comp.
GRU	34.5	127.5	0.40	Help.–Comp.
LIAM	22.5	89.0	0.47	Help.–Comp.
Ctx-Cond.	31.5	77.1	0.38	Help.–Block.
Mamba	33.8	76.2	0.38	Help.–Block.
UA-ToM	31.9	74.6	0.39	Help.–Block.

TABLE VI : Representation analysis. BOCPD has the highest mean separation yet 0% switch F1. CM = contrastive memory.

Fig. 6 : Internal dynamics around the regime switch (24 episodes, ManiSkill). Top : SSM state update magnitude

\|\mathbf{b}_{t}-\mathbf{b}_{t-1}\|

spikes

17\times

t_{s}

(from

9.4\times 10^{-5}

5.4\times 10^{-3}

) and decays over roughly 10 timesteps. Bottom : UA-ToM belief change overshoots the Oracle step function (1.58 vs. 1.0) then settles gradually over 5–10 steps. The SSM discretization

\Delta_{t}

remains near-constant at

{\approx}0.78

throughout, confirming that detection sensitivity resides in the learned dynamics matrices

\mathbf{A},\mathbf{B}

rather than input-dependent gating.

	ManiSkill		Overcooked
Configuration	F1	$\Delta$	Lat.	$\Delta$
Full UA-ToM	0.558	—	13.3	—
$-$ Hier. Pred. Error	0.121	$-$ 78%	14.9	+1.6
$-$ Causal Attention	0.412	$-$ 26%	26.4	+13.1
$-$ Selective SSM	0.479	$-$ 14%	—	—
$-$ Contrastive Memory	0.502	$-$ 10%	17.5	+4.2

TABLE VII : Ablation: ManiSkill (open-loop F1) and Overcooked (closed-loop latency, Cramped Room).

Method	Params	Mean (ms)	P99 (ms)	Budget
GRU	119K	0.40	1.94	0.8%
LIAM	215K	0.62	2.16	1.2%
Transformer	496K	0.97	2.78	1.9%
UA-ToM	992K	7.41	10.36	14.8%
Ctx-Cond.	259K	8.54	11.10	17.1%
Mamba	261K	9.27	11.50	18.5%

TABLE VIII : Inference latency (

T{=}50

window, RTX 6000 Ada). All methods fit within the 50 ms control budget.

Explore similar work

Sep 18, 2026

MAAP: Multi-Agent Active Perception for Collaborative Manipulation

Multi-agent manipulation naturally produces multiple task-driven viewpoints: every arm carries a wrist camera and moves through the scene while acting. Yet these observations are typically underutilized, and active perception in manipulation is still often treated as requiring a dedicated sensing agent. We introduce MAAP (Multi-Agent Active Perception), in which every arm is dual-purpose: it executes manipulation actions and, through the wrist camera it carries, simultaneously serves as a moving viewpoint for the team. We pair this with RAIL (Role-Aware Imitation Learning), a controller that predicts each arm's current role alongside its action chunk and conditions action generation on it, representing role-dependent actions within one network. Across four simulated tasks, widening the perception regime lifts average success from 56.5% with a fixed camera to 62.5% with one active wrist view and 70.0% with all of them, while MAAP+RAIL reaches 79.2%. RAIL's additional gain is concentrated on the three-arm Microwave task, where success rises from 47% to 82% on identical multi-wrist inputs. On a dual-arm platform, MAAP+RAIL succeeds in 14 of 20 placement trials compared with 0 of 20 for fixed-view ACT. Collaborative manipulation can thus serve as an active perception mechanism in its own right.

Bruno N. Y. Chen, Li Kang, Heng Zhou +4

Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China +3

Active Vision for Robot Manipulation Robotic Manipulation Multi-Agent Systems Imitation Learning Robot Imitation Learning Multi-Robot Coordination

Ask PDF

Mar 7, 2026

Foundational World Models Accurately Detect Bimanual Manipulator Failures

Deploying visuomotor robots at scale is challenging due to the potential for anomalous failures to degrade performance, cause damage, or endanger human life. Bimanual manipulators are no exception; these robots have vast state spaces comprised of high-dimensional images and proprioceptive signals. Explicitly defining failure modes within such state spaces is infeasible. In this work, we overcome these challenges by training a probabilistic, history informed, world model within the compressed latent space of a pretrained vision foundation model (NVIDIA's Cosmos Tokenizer). The model outputs uncertainty estimates alongside its predictions that serve as non-conformity scores within a conformal prediction framework. We use these scores to develop a runtime monitor, correlating periods of high uncertainty with anomalous failures. To test these methods, we use the simulated Push-T environment and the Bimanual Cable Manipulation dataset, the latter of which we introduce in this work. This new dataset features trajectories with multiple synchronized camera views, proprioceptive signals, and annotated failures from a challenging data center maintenance task. We benchmark our methods against baselines from the anomaly detection and out-of-distribution detection literature, and show that our approach considerably outperforms statistical techniques. Furthermore, we show that our approach requires approximately one twentieth of the trainable parameters as the next-best learning-based approach, yet outperforms it by 3.8% in terms of failure detection rate, paving the way toward safely deploying manipulator robots in real-world environments where reliability is non-negotiable.

Isaac R. Ward, Michelle Ho, Houjun Liu +7

Stanford University, Stanford, California, 94305, USA. · Watney Robotics, San Francisco, California, 94110, USA.

Robotic Manipulation Bimanual Robotic Manipulation Multimodal Anomaly Detection World Models

Ask PDF

Sep 16, 2026

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation

Detecting failures during execution is essential for reliable robotic manipulation. Vision-language models (VLMs) can assess task outcomes semantically but add runtime computation, whereas out-of-distribution (OOD) detectors may respond to harmless scene variations rather than failure-relevant deviations. We introduce RAFAIL, a framework for detecting execution failures during robotic manipulation. RAFAIL identifies failures by detecting anomalies in task-relevant relationships between entities, such as a gripper and an object or an object and its target. By focusing OOD detection on relevant parts of the observation, RAFAIL reduces sensitivity to task-irrelevant scene variation. Offline, a VLM annotates successful demonstrations with task progress and relationship importance, which are used to learn point-cloud-based relationship representations without relying on policy-internal features. At runtime, relationship-specific OOD detectors evaluate these representations while relationship importance and task progress are predicted without VLM inference. RAFAIL requires no failure data and achieves 73.4% balanced accuracy across three real-world robotic manipulation tasks, outperforming the strongest evaluated OOD- and uncertainty-based baselines.

Loris Schneider, Edgar Welte, Rania Rayyes

Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, 76131, Germany.

Robot Failure Detection Fault Detection Robotic Manipulation OOD Detection

Ask PDF