Belief Dynamics for Detecting Behavioral Shifts in Safe Collaborative Manipulation

Devashri Naik; Divake Kumar; Nastaran Darabi; Amit Ranjan Trivedi

cs.ROApr 4, 2026

Belief Dynamics for Detecting Behavioral Shifts in Safe Collaborative Manipulation

Authors: Devashri Naik, Divake Kumar, Nastaran Darabi, Amit Ranjan Trivedi

Organizations: University of Illinois Chicago

Abstract

Robots operating in shared workspaces must maintain safe coordination with other agents whose behavior may change during task execution. When a collaborating agent switches strategy mid-episode, continuing under outdated assumptions can lead to unsafe actions and increased collision risk. Reliable detection of such behavioral regime changes is therefore critical. We study regime-switch detection under controlled non-stationarity in ManiSkill shared-workspace manipulation tasks. Across ten detection methods and five random seeds, enabling detection reduces post-switch collisions by 52%. However, average performance hides significant reliability differences: under a realistic tolerance of +-3 steps, detection ranges from 86% to 30%, while under +-5 steps all methods achieve 100%. We introduce UA-TOM, a lightweight belief-tracking module that augments frozen vision-language-action (VLA) control backbones using selective state-space dynamics, causal attention, and prediction-error signals. Across five seeds and 1200 episodes, UA-TOM achieves the highest detection rate among unassisted methods (85.7% at +-3) and the lowest close-range time (4.8 steps), outperforming an Oracle (5.3 steps). Analysis shows hidden-state update magnitude increases by 17x at regime switches and decays over roughly 10 timesteps, while the discretization step converges to a near-constant value (Delta_t approx 0.78), indicating sensitivity driven by learned dynamics rather than input-dependent gating. Cross-domain experiments in Overcooked show complementary roles of causal attention and prediction-error signals. UA-TOM introduces 7.4 ms inference overhead (14.8% of a 50 ms control budget), enabling reliable regime-switch detection without modifying the base policy.

Figures & tables

Fig. 1 : Shared-workspace regime-switch detection. Top : a collaborating agent switches from cooperative (Helper) to adversarial (Blocker), creating collision risk. Bottom : enabling detection reduces post-switch collisions by 52% across all methods (No-Detect: $2.34$ vs. detection mean: ${\sim}1.1$ ). Among detectors, collision rates are comparable (range 0.99–1.29), motivating evaluation of detection reliability and adaptation quality.

Fig. 2 : UA-ToM computation flow. A frozen VLA backbone extracts features $\mathbf{z}_{t}$ . The UA-ToM module (992K parameters, 0.014% of backbone) processes these through four pathways: a selective SSM maintains persistent belief $\mathbf{b}_{t}$ via learned dynamics $\mathbf{A},\mathbf{B}$ (discretization $\Delta_{t}$ converges to 0.78); causal attention re-anchors belief to recent observations; prediction error $e_{t}$ provides an explicit change signal; and contrastive memory supplies categorical type cues via EMA prototypes. A gated fusion layer integrates all signals to produce type $\hat{\theta}_{t}$ , switch probability $\hat{s}_{t}$ , and predicted action $\hat{\mathbf{a}}^{c}_{t}$ . Internal dynamics around the regime switch are shown in Fig. 6 .

Condition	Post-Switch Collisions
No detection	$2.34\pm 0.40$
Detection (mean across methods)	$1.11\pm 0.12$
Reduction	52%

TABLE I : Effect of regime-switch detection on post-switch collision rate (5 seeds). Lower is better.

	Hard Detection (%)
Method	S0	S1	S2	S3	S4	Mean	$R$
Ctx-Cond. †	86.8	87.8	85.5	87.5	82.8	86.1	0.94
UA-ToM	85.3	83.5	89.2	87.3	83.0	85.7	0.93
Mamba	85.5	85.0	87.3	85.2	81.8	85.0	0.94
Transformer	83.3	83.3	82.8	83.8	79.2	82.5	0.95
LIAM	62.3	56.8	61.0	63.8	68.2	62.4	0.83

TABLE II : Hard detection rate (%) per seed at

\pm 3

step tolerance (150 ms at 50 ms control rate). At

\pm 5

, all methods achieve 100%. † Receives ground-truth type.

Fig. 3 : Detection reliability heatmap at $\pm 3$ step tolerance. Hard detection rate (%) per method and transition type across 12 behavioral transitions (e.g. Helper $\to$ Blocker, Competitor $\to$ Passive). Rows are sorted by mean detection rate. Methods separate into three tiers: fast detectors ( $>$ 82%, blue), agent-modeling (58–62%, yellow), and slow recurrent models ( $<$ 34%, red). At the standard $\pm 5$ tolerance, all cells would be 100%, masking these operationally relevant differences. Transition difficulty varies: Helper $\to$ Competitor is consistently hardest across methods due to similar spatial approach patterns.

Method	Latency $\downarrow$	Post-Sw. Coll. $\downarrow$	Total Coll. $\downarrow$
Oracle	$1.0\pm 0.0$	$1.07\pm 0.58$	$2.69\pm 1.02$
UA-ToM	$\mathbf{9.9\pm 0.8}$	$1.01\pm 0.39$	$2.66\pm 0.52$
Ctx-Cond. †	$9.4\pm 1.1$	$1.10\pm 0.61$	$2.64\pm 0.73$
Transformer	$10.5\pm 1.3$	$1.29\pm 0.48$	$3.35\pm 1.10$
Mamba	$10.6\pm 1.0$	$1.11\pm 0.24$	$3.20\pm 0.51$
LIAM	$11.0\pm 0.6$	$0.99\pm 0.28$	$2.46\pm 0.36$

TABLE III : Closed-loop performance over five seeds (1200 episodes). † Privileged.

Method	Det% $\uparrow$	Post-Coll. $\downarrow$	CRT $\downarrow$	Total Coll. $\downarrow$
Oracle	$100.0$	$2.6\pm 0.7$	$5.3\pm 0.9$	$7.1\pm 1.0$
Ctx-Cond. †	$98.6\pm 0.4$	$\mathbf{2.1\pm 0.6}$	$4.9\pm 1.0$	$\mathbf{5.2\pm 1.3}$
UA-ToM (MS)	$97.5\pm 0.9$	$2.6\pm 0.7$	$\mathbf{4.8\pm 1.1}$	$5.9\pm 1.4$
Mamba	$98.5\pm 0.4$	$3.5\pm 0.9$	$7.0\pm 0.7$	$7.0\pm 1.4$
LIAM	$97.3\pm 0.7$	$3.4\pm 0.5$	$13.4\pm 1.5$	$7.1\pm 0.9$
UA-ToM	$95.1\pm 0.8$	$2.7\pm 0.6$	$9.2\pm 1.8$	$6.4\pm 1.1$

TABLE IV : Extended evaluation (300 steps, 10 seeds). Detection saturates; the differentiator is adaptation quality. CRT = close-range time (steps at

<

15 cm post-switch). † Privileged.

Fig. 4 : Close-range time (CRT) under extended episodes (300 steps, 10 seeds). CRT measures post-switch timesteps at $<$ 15 cm end-effector distance. UA-ToM (MS) achieves the lowest CRT (4.8), including lower than Oracle (5.3), indicating that smooth belief revision maintains safer clearance than instantaneous strategy switching. LIAM detects at 97.3% yet accumulates $2.8\times$ more danger time (13.4 steps). † Privileged.

Fig. 5 : Post-switch adaptation dynamics (300-step episodes, 10 seeds). (A) : Post-switch collision rate over time following $t_{s}$ , showing instantaneous risk at each timestep. UA-ToM (MS) exhibits a smooth, monotonic decrease consistent with the SSM’s gradual belief revision (Fig. 6 ). Oracle shows a sharp transient from instantaneous strategy switching. LIAM remains elevated for an extended window despite successful detection. (B) : Cumulative post-switch collisions over time. The separation between methods grows over the post-switch window, confirming that sustained proximity behavior, not isolated collision events, drives the safety differences captured by CRT.

	Cramped-1		Asym.-1		Cramped-2		Asym.-2
Method	Fl.	CV	Fl.	CV	Fl.	CV	Fl.	CV
UA-ToM	87	.048	99	.007	99	.005	97	.017
Transformer	83	.049	99	.006	97	.013	97	.013
GRU	57	.208	86	.061	92	.033	90	.033

TABLE V : Detection reliability across Overcooked layouts. Floor = worst-seed detection rate (

\mu-\sigma

). CV = coefficient of variation across five seeds.

Model	Min Dist.	Mean Dist.	Silhouette	Hardest Pair
BOCPD	38.1	322.2	0.40	Help.–Comp.
GRU	34.5	127.5	0.40	Help.–Comp.
LIAM	22.5	89.0	0.47	Help.–Comp.
Ctx-Cond.	31.5	77.1	0.38	Help.–Block.
Mamba	33.8	76.2	0.38	Help.–Block.
UA-ToM	31.9	74.6	0.39	Help.–Block.

TABLE VI : Representation analysis. BOCPD has the highest mean separation yet 0% switch F1. CM = contrastive memory.

Fig. 6 : Internal dynamics around the regime switch (24 episodes, ManiSkill). Top : SSM state update magnitude

\|\mathbf{b}_{t}-\mathbf{b}_{t-1}\|

spikes

17\times

t_{s}

(from

9.4\times 10^{-5}

5.4\times 10^{-3}

) and decays over roughly 10 timesteps. Bottom : UA-ToM belief change overshoots the Oracle step function (1.58 vs. 1.0) then settles gradually over 5–10 steps. The SSM discretization

\Delta_{t}

remains near-constant at

{\approx}0.78

throughout, confirming that detection sensitivity resides in the learned dynamics matrices

\mathbf{A},\mathbf{B}

rather than input-dependent gating.

	ManiSkill		Overcooked
Configuration	F1	$\Delta$	Lat.	$\Delta$
Full UA-ToM	0.558	—	13.3	—
$-$ Hier. Pred. Error	0.121	$-$ 78%	14.9	+1.6
$-$ Causal Attention	0.412	$-$ 26%	26.4	+13.1
$-$ Selective SSM	0.479	$-$ 14%	—	—
$-$ Contrastive Memory	0.502	$-$ 10%	17.5	+4.2

TABLE VII : Ablation: ManiSkill (open-loop F1) and Overcooked (closed-loop latency, Cramped Room).

Method	Params	Mean (ms)	P99 (ms)	Budget
GRU	119K	0.40	1.94	0.8%
LIAM	215K	0.62	2.16	1.2%
Transformer	496K	0.97	2.78	1.9%
UA-ToM	992K	7.41	10.36	14.8%
Ctx-Cond.	259K	8.54	11.10	17.1%
Mamba	261K	9.27	11.50	18.5%

TABLE VIII : Inference latency (

T{=}50

window, RTX 6000 Ada). All methods fit within the 50 ms control budget.

Explore similar work

MAAP: Multi-Agent Active Perception for Collaborative Manipulation
Sep 18, 2026Bruno N. Y. Chen, Li Kang, Heng Zhou +4Active Vision for Robot Manipulation Robotic Manipulation
Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China +3
Foundational World Models Accurately Detect Bimanual Manipulator Failures
Mar 7, 2026Isaac R. Ward, Michelle Ho, Houjun Liu +7Robotic Manipulation Bimanual Robotic Manipulation
Stanford University, Stanford, California, 94305, USA. · Watney Robotics, San Francisco, California, 94110, USA.
RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation
Sep 16, 2026Loris Schneider, Edgar Welte, Rania RayyesRobot Failure Detection Fault Detection
Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, 76131, Germany.

Sep 18, 2026J/K move · Enter open · S save

MAAP: Multi-Agent Active Perception for Collaborative Manipulation

Bruno N. Y. Chen, Li Kang, Heng Zhou, Xiufeng Song+3

Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China+3

Active Vision for Robot Manipulation Robotic Manipulation Multi-Agent Systems Imitation Learning Robot Imitation Learning Multi-Robot Coordination

Open paper AskPDF arXiv

Multi-agent manipulation naturally produces multiple task-driven viewpoints: every arm carries a wrist camera and moves through the scene while acting. Yet these observations are typically underutilized, and active perception in manipulation is still often treated as requiring a dedicated sensing agent. We introduce MAAP (Multi-Agent Active Perception), in which every arm is dual-purpose: it executes manipulation actions and, through the wrist camera it carries, simultaneously serves as a moving viewpoint for the team. We pair this with RAIL (Role-Aware Imitation Learning), a controller that predicts each arm's current role alongside its action chunk and conditions action generation on it, representing role-dependent actions within one network. Across four simulated tasks, widening the perception regime lifts average success from 56.5% with a fixed camera to 62.5% with one active wrist view and 70.0% with all of them, while MAAP+RAIL reaches 79.2%. RAIL's additional gain is concentrated on the three-arm Microwave task, where success rises from 47% to 82% on identical multi-wrist inputs. On a dual-arm platform, MAAP+RAIL succeeds in 14 of 20 placement trials compared with 0 of 20 for fixed-view ACT. Collaborative manipulation can thus serve as an active perception mechanism in its own right.

Abstract

Explore similar work

MAAP: Multi-Agent Active Perception for Collaborative Manipulation

Foundational World Models Accurately Detect Bimanual Manipulator Failures

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation