Audit Before You Commit: Locating Belief Failures in Active Identification for One-Shot Manipulation

Mohamed Abouagour; Byung-Cheol Min

cs.ROSep 24, 2026

Audit Before You Commit: Locating Belief Failures in Active Identification for One-Shot Manipulation

Authors: Mohamed Abouagour, Byung-Cheol Min

Organizations: SMART Lab, Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, USA

Abstract

A robot that probes a few times before one irreversible action, such as tapping a surface before inserting a peg, must decide when the evidence is enough to commit. We argue that this decision rests on two conditions that existing methods do not separate: the belief must still cover the truth in the coordinate that decides the action, and the failure model that scores actions must track realized failure. We audit both conditions separately, offline and with ground truth, on a deployed probe-then-commit pipeline: a particle belief, a scenario failure score, and one commit. On simulated insertion, more taps sharpen the belief while the truth leaves its support on 16.9% of episodes and the failure score turns optimistic by 0.31. Conformal calibration restores coverage but not the decision: confidently wrong instances still pass a confidence gate. The audit's signatures instead point at the observation model, where a hand scan finds a 2.1 mm error in the tap boundary. Correcting that one number cuts failure from 0.354 to 0.112 on untouched instances and transfers unrefitted to a second engine, while in a third engine the same audit suggests an execution-model mismatch instead. Across seven task families in three engines, a few probes at a fixed executor reduce miss or failure. On a physical arm inserting a tool into a rigid pocket by touch, the gain and the audit's two conditions reproduce, and replaying the recorded taps under an injected model error shows the audit's signature on real data. Additional materials are available at https://sites.google.com/view/auditbeforeyoucommit.

Figures & tables

Fig. 1: Probe, believe, commit. One simulated insertion episode: (a) surface-height probes; (b) peak-normalized particle beliefs after taps 1, 3, and 5, with the true hole position and candidate aims; (c) insertion at the fifth aim.

Fig. 2: Probe-then-commit execution and offline audit. Probes update the belief; retained scenarios select an action and failure score. Ground-truth audits assess decision-coordinate coverage and mean score optimism separately. Coverage failures prompt belief-model inspection; optimism prompts score and execution-model validation. Neither uniquely identifies the cause. Corrections apply between deployments, not within an episode.

family	engine	latents	probe / action	$\sigma$	budgets $k$	$n$ ; metric
insertion	all three	$x_{\rm hole},c$	tap / aim	$0.5$ mm	1–8, cover	$96{\times}4$ ; failure
1-D push	analytic	$\mu,m,k_{d}$	pulse / impulse	$5$ cm	0, 2, 5	$81{\times}6$ ; miss
2-D push	MuJoCo	$\mu,m,k_{d}$	pulse / impulse	$5$ cm	0, 2, 5	$18{\times}6$ ; miss
darts	MuJoCo	$k_{d},m,g$	throw / launch	$2$ cm	0, 2, 5	$18{\times}8$ ; miss
hinge	MuJoCo	$f,b,m$	coast / torque	$0.02$ rad	0, 2, 5	$18{\times}16$ ; miss
shuffleboard	SAPIEN	$\mu,m,k_{d}$	strike / strike	$5$ cm	0, 2, 5	$18{\times}6$ ; miss

TABLE I: Experimental setup. Seven task families; simulation engines are MuJoCo, SAPIEN (PhysX), and PyBullet.

\sigma

denotes probe-noise standard deviation;

n

gives instances

\times

seeds or insertion repeats, per method and budget. The 2-D pushing transfer uses PyBullet (

k=5

18{\times}6

Fig. 3: The simulated scenes, with their engines in Table I , and the hardware rig from the front camera: the probe over the machined pocket. The 1-D push is closed form and grasp is a force-balance model with a MuJoCo lift check, so neither has a scene.

family	Blind-Bayes	Blind-Robust	Probe-Robust( $k$ )	paired $\Delta$ [95%]
insertion, A	$0.820$	–	$0.234$	$0.586\,[0.492,0.674]$
1-D push	$0.491$	–	$0.062$	–
2-D push, MJ	$0.252$	$0.307$	$0.221$	$0.031\,[0.007,0.054]$
2-D push, PB	$0.243$	–	$0.141$	–
darts	$0.026$	$0.035$	$0.010$	$0.016\,[0.009,0.024]$
hinge	$0.334$	–	$0.031$	$-91\%$

TABLE II: Every family, blind against probed , uncalibrated, five probes (two for shuffleboard, eight for grasp). Mean miss (m; rad for the hinge; mean / P90 for shuffleboard), failure for insertion on Split A, success for grasp on its feasible subset; MJ and PB are MuJoCo and PyBullet. From Blind-Bayes to Blind-Robust only the risk functional changes, from Blind-Robust to Probe-Robust(k) only the information. Last column: paired reduction against the matched blind baseline, cluster-bootstrap interval. PyBullet uses the MuJoCo-fitted analytic model; a residual model reaches

0.115

	Cluster coverage		$e_{k}$
$k$	original	calibrated	original	calibrated
1	$0.79$	$0.81$	$+0.115$	$\mathbf{-0.003}$
2	$0.69$	$0.82$	$+0.176$	$+0.022$
3	$0.55$	$0.78$	$+0.218$	$+0.078$
5	$0.42$	$0.76$	$+0.263$	$+0.047$
8	$0.46$	$0.77$	$+0.306$	$+0.091$

TABLE III: Insertion audit. Split B, random probes. Cluster coverage requires all

R=4

repeats to be covered;

e_{k}=R_{k}-\widehat{R}_{k}>0

indicates optimism. The bottom row averages

|\mathrm{coverage}-0.80|

and

|e_{k}|

over budgets.

Split	Method	Set cal.	Margin	Failure
A	no probe	–	–	$0.820\,[0.779,0.859]$
A	cover, $k{=}5$	no	no	$0.234\,[0.159,0.312]$
B	Blind-Robust	–	–	$0.870\,[0.836,0.901]$
B	cover, $k{=}5$	no	no	$0.255\,[0.211,0.299]$
B	cover, $k{=}5$	yes	no	$0.260\,[0.182,0.341]$
B	cover, $k{=}5$	yes	yes	$0.302\,[0.221,0.385]$

TABLE IV: Insertion failure by split and calibration setting. Each row uses

n=384

episodes; probed rows use the five-tap cover schedule. Instance sets are defined in Section V . Brackets show truth-cluster bootstrap intervals.

Fig. 4: The failure the audit found. (a) Two taps as scale-accurate sections with the probe’s simulated path. The analytic model reads pocket only inside the dashed edge; the simulator’s finger slides down the chamfer and reads pocket out to the solid one, $2.1$ mm further. (b) After the $+4.5$ mm reading the analytic model’s hypotheses exclude the true hole; the corrected model’s keep it.

	old	corr.	old $-$ corr. [95%]
escape, random $k{=}8$	$0.164$	$0.000$	$+0.164\,[+0.125,+0.203]$
episode cov., random $k{=}8$	$0.768$	$1.000$	$-0.232\,[-0.279,-0.188]$
optimism $e_{8}$ , random	$+0.274$	$-0.015$	$+0.289\,[+0.234,+0.344]$
failure, random $k{=}8$	$0.354$	$0.112$	$+0.242\,[+0.188,+0.297]$
failure, cover $k{=}5$	$0.326$	$0.159$	$+0.167\,[+0.091,+0.247]$
failure, cover $k{=}1$	$0.721$	$0.703$	$+0.018\,[-0.016,+0.052]$

TABLE V: Confirmation on untouched instances (Test C). Drawn after the correction froze (

96

per protocol, four repeats); both models see identical taps, differences paired and cluster-bootstrapped.

	PyBullet			PhysX
	deployed	MJ offset	own scan	deployed	MJ offset	own scan
support escape	$0.167$	$0.023$	$0.031$	$0.039$	$0.065$	$0.000$
episode coverage	$0.763$	$0.951$	$0.948$	$0.914$	$0.872$	$0.995$
optimism $e_{8}$	$+0.232$	$+0.002$	$-0.002$	$+0.369$	$+0.419$	$+0.338$
failure	$0.307$	$0.109$	$0.107$	$0.474$	$0.534$	$0.469$

TABLE VI: The audit across engines , random probes,

k=8

: the analytic model as deployed, with the MuJoCo-fitted offset (not refitted), and with the boundary scanned in that engine.

taps	episodes	in window	entered	predicted	realized	optimism
1	$26$	$6$	$19$	$0.657$	$0.769$	$+0.112$
2	$21$	$13$	$18$	$0.474$	$0.381$	$-0.093$
3	$26$	$19$	$24$	$0.315$	$0.269$	$-0.046$
4	$21$	$15$	$21$	$0.225$	$0.286$	$+0.061$
5	$24$	$22$	$24$	$0.084$	$0.083$	$-0.001$

TABLE VII: The run on the physical arm , by number of taps. Realized failure is the aim outside the model’s

6.57

mm window of the measured centre; optimism is realized minus predicted. Coverage was

118

118

Fig. 5: The run on the physical arm and its replay. (a) The arm at the rig, and one five-tap episode from the front camera, pocket at $-9.8$ mm: hole read at $-23$ and $0$ mm, plate elsewhere; the commit at $-11.6$ mm enters. (b) Aim error against taps, medians joined; filled points entered, open did not; band: the model’s window; dashed: the opening’s half-width. (c) The recorded taps under an injected boundary error: optimism of the faulted model minus the deployed one, paired over positions, $95\%$ band, filled where resolved; dotted: the simulation’s $2.1$ mm error at rig scale. (d) The same taps under a narrowed prior; at $\pm 12$ mm both conditions fire.

Fig. 6: Commit-decision audit on Split B. (a) Cluster coverage versus nominal level under random probing (

k=8

). (b) Failure among committed episodes versus commit rate: cover schedule, budgets

1

–

5

, budget-simultaneous calibration, and first-passage commitment. Curves compare scoring and execution jointly; areas are point estimates over each method’s attainable range. The non-deployable hindsight oracle achieves zero failure at

89\%

commitment. (c) Under the same cover schedule at

60\%

commitment, mean predicted score (pale) versus realized failure (solid); posterior variance makes no probability claim.

fault	fail	surprise	temp. $T{=}4$	known fix
none (baseline)	$0.10$	–	–	–
noise	$0.10$	–	$0.10$	$0.10$
boundary	$0.38$	$0.64$	$0.38$	$0.10$
prior	$0.29$	$0.59$	$0.29$	$0.18$
particles	$0.13$	$0.67$	$0.13$	$0.10$
frame	$0.26$	–	$0.26$	$0.10$

TABLE VIII: Controlled mismatch tests. Random probes,

k=8

, Split B. Fig. 7 shows the corresponding audit metrics. Noise at

4\times

has no observed effect, the two readings being

30\sigma

apart. Surprise is the prequential-surprise AUC for escape; other numeric columns report failure before and after tempering (

T=4

) or known-fault correction. Cluster-bootstrap half-widths reach

0.077

; paired differences are reported in the text.

Fig. 7: The five classes over the probe budget , random probes, corrected model,

n=384

per point. Zero-mean noise matches no fault at every budget, so they share a line; both, and the frame offset, escape on at most one episode in

384

Robotics

Robotics studies the design and control of machines that perceive, move, and act in physical environments.

See all 751 papers

Sep 17, 2026

CoreSense: Traceable Failure Recall and Conflict-Aware Belief Gating for Auditable Robot Decisions

Robots can recall prior failures without knowing whether recalled evidence remains valid, conflicts with current observations, or is sufficient to guide a decision. We present CoreSense, a robot-system integration architecture that combines traceable episodic evidence with a conflict-aware belief gate and bounded, auditable recommendations. The gate checks scope, provenance, time, contradiction, and support before it permits PROCEED, requests re-observation, abstains, or escalates. Evaluation follows three complementary layers without commanding a physical robot: offline public real-robot data, a frozen signal-level simulation, and a live cloud deployment path. On CableTrace-120 and BotFails-200, belief gating reduces protocol-defined unsafe proceeds from 20% and 40% to 0%. A disjointly calibrated raw-video policy also reaches 0% unsafe proceed, but overblocks every nominal episode. On public data, a ViFailback-BotFails visual detector reaches 0.778 AUROC yet remains all-blocking, whereas cycle-disjoint UR3 telemetry for protective stops yields 0% unsafe proceed, 36.1% overblocking, and 61.9% coverage; grip-loss transfer remains a negative result. Controlled physical corroboration yields 3.3%, 0%, and 42.0%, while conflict-aware fusion yields 4.7%, 0%, and 42.8%. Finally, 20/20 cloud recalls validate a CockroachDB Cloud-Amazon Bedrock deployment path. The evidence supports an auditable integration pattern, not autonomous recovery or certified safety.

Zoe Li

Independent Researcher, Seattle, WA, USA

Robot Failure Detection Robot Safety

Ask PDF

Oct 8, 2026

PIER: An Evidence-Gated Execution Interface for Robotic Manipulation

Generating a plausible robot action does not establish that current observations justify its execution. Motivated by exploratory observations of high-confidence visual outputs under severe occlusion, we present PIER, an execution-authorization interface that separates evidence checks, decision provenance, and stage-scoped re-observation from hardware control. The deterministic gate evaluates declared visual and tactile inputs, while its caller maintains a budget of at most one re-observation per stage. We evaluate the implementation using 1,600 threshold-grid cases and 1,200 paired synthetic traces spanning score noise, missing tactile inputs, stale observations, and falsely reassuring scores. The finite grid yields zero declared invariant violations, and a matched Boolean baseline reproduces all non-recovery decisions. Under synthetic score noise, re-observation reduces valid-state denials from 57/120 to 18/120 while increasing invalid-state proceeds from 5/120 to 7/120. Stale and falsely reassuring inputs expose limitations that threshold checks alone cannot resolve. Exploratory visual, tactile, and robot setup records provide context but do not establish physical task performance. These results characterize an inspectable authorization interface and its input-contract limitations, without claiming superiority over equivalent rule logic, calibrated tactile accuracy, or certified physical safety.

Zoe Li, Anze Wang, Zhongyu Chen +1

University of Washington · Independent Researcher · ZJU-UIUC Institute

Robotic Manipulation

Ask PDF

Jun 2, 2026

How Visible Are Silent Manipulation Failures? An Observability Study of False-Success Detection in Simulated Robot Episodes

Imitation-learning policies for robot manipulation inherit the quality of the success labels attached to their training episodes, and those labels are usually produced by the robot's own success check. A particularly damaging error is the false success: an episode the robot logs as a success when the task outcome was actually wrong. We ask a narrow but practical question about these episodes. Once an episode has already been flagged as a success, how much of the information needed to overturn that label is present in proprioception, and how much requires vision? We build a simulated testbed on two bimanual ALOHA tasks, induce failures through environment perturbations rather than label edits, label every episode by privileged simulator state that the detector never sees, and keep only episodes the robot flagged as successful. We then compare detectors restricted to proprioception against a vision-based detector. We find that recoverability spans a wide range: in cube transfer the false successes are almost fully recoverable from joint data alone, while in peg insertion proprioception recovers only part of them and a vision detector closes most of the gap. We also show that the proprioceptive separability we measure rests on velocity differences far below any realistic sensor noise floor, so it is best read as an optimistic upper bound that a noiseless simulator inflates. We release the generation and evaluation pipeline.

Aarav Bedi

University of California, Berkeley · Department of Mechanical Engineering University of California, Berkeley Berkeley, California, USA

Robot Failure Detection Bimanual Robotic Manipulation Robotics Robotic Perception

Ask PDF

Abstract

Robotics

Explore similar work

CoreSense: Traceable Failure Recall and Conflict-Aware Belief Gating for Auditable Robot Decisions

PIER: An Evidence-Gated Execution Interface for Robotic Manipulation

How Visible Are Silent Manipulation Failures? An Observability Study of False-Success Detection in Simulated Robot Episodes