cs.ROOct 27, 2026

Which Hand Was Late? Diagnosis for Robot Failure Recovery in Bimanual Tool Use

Authors: Baptiste Bonin, Caro Strickland, Audrey Durand

Organizations: Universit´e Laval · Mila – Quebec AI Institute · CIFAR AI Chair

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Failure diagnosers for robot manipulation are increasingly vision--language models (VLMs), trained and evaluated on single-arm failures produced by perturbing a successful demonstration. Bimanual tool use adds failures of other kinds: a tool that slips out of its socket while still held, two correct gestures made at different times, and attempts that fail for two reasons at once. On a simulated bimanual tool-use task whose failure causes are read from the recorded trace, we measure what each sensing channel lets a diagnoser tell. The tool's slip is read on the tool hand; which hand was late is read on both arms, while a sensor on one hand reads its own hand's delay and at most part of the other's; and whether a late attempt was also pushed too weakly is not identifiable from passive sensing when the tool hand is late. Within two attempts, a repair policy that escalates when the cause is not identifiable repairs as many failures as one told the true cause. A VLM's internal state carries the late hand almost perfectly, along a direction no closer to the one its answer reads than a random one; whether the answer reports it depends on the wording of the question.

Figures & tables

Figure 1: Which hand was late? One hand turns a key-tool in the lock while the other pushes the door (top). Each attempt is labelled from its recorded trace (strips): a wrong gesture (W), a push too weak while both hands acted (F), right gestures at different times (T), or success (S). A late attempt whose push was also below the hidden threshold τ\tau has the same frames and two causes.
Figure 2: The task, three recorded episodes (front camera). One arm fetches the key-tool from its stand, seats it in the square socket and turns it; the other pushes the door. Top: both hold their targets at the same time and the door opens. Middle: the tool hand is late; the push is over before the key turns. Bottom: the push hand is late; the key has sprung back before the push arrives. In both failures each gesture, taken alone, is correct.
Figure 3: Injected fault (left) against the class read from the recorded trace (right), 758758 attempts; the same fault can end in several classes. Wrong : a gesture was not delivered. Timing : the push reached the hidden threshold τ\tau , but never while the key was held. Force : the push was on while the key was held, but stayed under τ\tau . Two causes : both at once, the push never on while the key was held and never reaching τ\tau .
     coordination 
InputKiBGFLOPsdetectdiagnosetool latepush latepooledcam-out
Wrist torque, tool hand1.10.00.6540.7461.0000.7530.8630.972
Tool–socket contact0.70.00.7190.8580.9490.6460.8530.986
Wrist joint 7, tool arm0.20.00.7030.5761.0000.8010.6310.919
Wrist force, push hand1.10.00.5680.6400.3980.9660.6780.519
Hand contact, push hand0.90.00.6410.5850.5640.9680.6970.512
Table 1: Input cost per episode and information separated (trial window; within-door AUC). KiB denotes the input size: all samples in the window for numerical channels (a probe reads two channels) and two frames for images; GFLOPs denotes feature-extraction cost. Bold indicates the best robot-accessible probe per column. Lock angle is a mechanism reading and latch state an oracle; neither is evidence. Coordination measures timing relative to force; cam-out measures the corresponding 106/134106/134 trial separation. The VLM uses the front-camera image pair.
Figure 4: Channels across the two halves of the coordination failure (trial window; within-door AUC with 95%95\% intervals over doors; timing relative to force). Colour indicates sensor location; solid and light lines denote tool-hand late and push-hand late , respectively; grey shows lock angle. One-hand channels primarily capture their own hand’s delay, whereas two-arm channels capture both. Top: interaction effects (difference-in-differences) across paired door resamples.
Figure 5: What Qwen3-VL 8B holds and what it answers (front-camera frames, corrected labels). (a) Within-door AUC of a linear probe on the residual stream at the position that predicts the answer, by layer (shaded: 95% intervals over doors), and the AUC of the answer under each of the three phrasings (diamonds). (b, c) One point per attempt: the probe’s out-of-fold score on the final-norm input, against the answer’s log⁡P(Yes)−log⁡P(No)\log P(\text{Yes})-\log P(\text{No}) under the phrasing whose state the probe reads.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

 repaired (of 386)two causesattempts
Repair policy1st attempt≤\leq 2 attemptsrepaired (of 119)pushed harder
One label, one correction at a time2062113261
Escalate when the cause is not identifiable20029781323
Oracle: true cause28929781312
Every correction at once (no diagnosis)29729781386
Appendix
Table 2: Repairing the 386386 blocked attempts, each replayed in the same world, at most twice. Corrections: re-time the late hand, push harder, or both. One label chooses between timing and force; escalate only asks whether the hands overlapped, applies the matching correction, then both.
More in this topic

Robot Failure Detection

Robot failure detection identifies when autonomous systems are failing, predicts emerging faults, and supports diagnosis or recovery.

See all 47 papers

Explore similar work

CardsList

  1. Foundational World Models Accurately Detect Bimanual Manipulator Failures

    Mar 7, 2026Isaac R. Ward, Michelle Ho, Houjun Liu +7Robotic Anomaly DetectionRobotic Manipulation

    Stanford University, Stanford, California, 94305, USA. · Watney Robotics, San Francisco, California, 94110, USA.

  2. FailBench: How Reliable are VLMs at Judging Robot Task Success?

    Sep 3, 2026Zaruhi Navasardyan, Tatul Danielyan, Hrant DavtyanVLM EvaluationRobot Failure Detection

    Metric AI Lab

  3. ProTracer: Proprioception-Guided Failure Diagnosis in Robot Manipulation

    Sep 18, 2026Chang Dong, Mehdi Hosseinzadeh, King Hang Wong +4Robot Failure DetectionRobotic Manipulation

    Australian Institute for Machine Learning, Adelaide University, Australia. · School of Allied Health and Human Performance, Adelaide University, Australia.