REBOOT: From Failure to Recovery - A Dataset and Benchmark for Precision Assembly

Nana Yaw Owusu Ofori-Ampofo; Samira Ebrahimi Kahou; Joseph Thekinen

cs.ROSep 18, 2026

REBOOT: From Failure to Recovery - A Dataset and Benchmark for Precision Assembly

Authors: Nana Yaw Owusu Ofori-Ampofo, Samira Ebrahimi Kahou, Joseph Thekinen

Organizations: University of Calgary · University of Calgary, Mila

Abstract

Robot learning policies fail in characteristic ways: they stall in uncertain states, drift during contact-rich alignment, and miss targets by millimetres in precision tasks. Yet training datasets consist largely of successful demonstrations, while real-world benchmarks often reduce performance to binary success. This limits both supervision for recovery and analysis of where failures occur. We introduce REBOOT (Recovery Episode Benchmark for Off-nominal Trajectories), the first robot manipulation benchmark designed around failure as a first-class signal. REBOOT contains 2,160 demonstrations across 18 precision assembly tasks, each decomposed into five shared phases: Align(pick), Engage(pick), Transport, Align(place), and Engage(place), enabling phase-level evaluation beyond terminal success. Failures are introduced across phases and paired with expert recovery trajectories that return the system to a valid continuation state. Tasks are annotated with rotational symmetry, engagement-clearance precision tier, and assembly direction through matched install-remove pairs. Failure episodes are labeled by phase and categorical failure mode, enabling attribution to kinematic stage and tolerance violation. Data includes synchronized RGB-D observations from four viewpoints and grounded natural-language descriptions of phase-level success and failure conditions. Half the dataset contains expert demonstrations; the other half contains recovery demonstrations sampled to reflect failures observed in imitation-learned policy rollouts. We benchmark action-chunked transformer, diffusion, and $π_0$ -FAST policies using phase-level completion rates, revealing model-specific failure points hidden by binary evaluation. Dataset and code: https://nanayawoa.github.io/REBOOT

Figures & tables

Figure 1: Overview of REBOOT. Each precision-assembly task is decomposed into five kinematic phases— Align(pick) , Engage(pick) , Transport , Align(place) , and Engage(place) —shown vertically on the left. The first column illustrates successful task execution with natural language annotations at each phase. The second column shows some failure modes that arise at each phase during autonomous policy rollouts, paired with the expert recovery trajectories that return the system to a valid continuation state. REBOOT treats both columns as first-class supervision signal.

Dataset	Domain	# of demos	% Failure	Recovery - Yes/No
REFLECT [ 8 ]	Sim+Real	130	100.0%	No
RoboFAC [ 7 ]	Sim+Real	10,722	88.0%	No
RaC [ 4 ]	Sim+Real	N/A †	N/A †	Yes
DROID [ 2 ]	Real	$\approx$ 92,000	$\approx$ 17.4%	Yes (3.68%)
REASSEMBLE [ 9 ]	Real	4,551	11.3%	No
FurnitureBench [ 10 ]	Sim+Real	5,100	0%	No

Table 1: Robot manipulation datasets reporting explicit failure or recovery episode counts. Counts are taken verbatim from the primary source (paper text and dataset cards). REBOOT is the only real-robot dataset with a 50% failure rate and structured per-phase recovery annotations.

Figure 2: Rotational symmetry groups for representative objects in the dataset. Each column shows a goal–object pair; from left to right: 16 mm cylinder ( Continuous ), RCA connector ( Continuous ), M12 fastener ( C2 ), 16 mm bar ( C2 ), USB-C connector ( C2 ), NEMA 1-15P power plug ( C1 ), RJ45 connector ( C1 ), USB-A connector ( C1 ), and Harting HAN10E connector ( C1 ).

Figure 3: Overview of the expert and recovery data collection pipeline. Policies trained on expert demonstrations are rolled out to identify phase-level failures, which are subsequently re-enacted by the teleoperator to collect successful recovery demonstrations.

Connector	Clearance (mm)
RCA	interference fit (0)
USB-C	0.09
RJ45	0.13
16mm Bar	0.20
16mm Cylinder	0.20
NEMA 1-15P power plug	0.5

Table 2: Connector clearance values.

Panel of Figure 4 — Figure 4: Distribution of last-completed phase across precision assembly and disassembly tasks. Each group represents one connector; each bar shows the fraction of policy rollouts whose last successfully completed phase was the indicated phase, with the rightmost bar ( green ) showing the fraction that completed all five phases (full task success). Aggregated across all three policy families ( $N{=}15$ rollouts per task per policy). .

Panel of Figure 5 — Figure 5: Phase-failure profiles for installation and removal tasks. Boxes show median and interquartile range with whiskers at $1.5\times$ IQR; markers show all nine per-task counts. The install/remove asymmetry is visible at the Success endpoint, where removal counts are several times higher.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Product used	Used for
Mini-vise	Tabletop drill-press vise with rubber soft jaws, $0\text{\,}\mathrm{mm}63\text{\,}\mathrm{mm}$ clamping range	Holding the female connector fixed
RCA connectors	2 RCA-to-2 RCA gold-plated audio extension cable (male and female ends), $0.9\text{\,}\mathrm{m}$	RCA install / remove
USB-C connectors	USB-C 3.2 Gen2 male-to-female extension cable, $20\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$	USB-C install / remove
NEMA 1-15P (male)	Two-prong polarized AC power cord, 18 AWG, $0.5\text{\,}\mathrm{m}$	NEMA 1-15P install / remove
NEMA 1-15P (female)	Polarized two-prong extension cord, 16 AWG, $3.0\text{\,}\mathrm{m}$	NEMA 1-15P install / remove
Mounting tape	Double-sided indoor mounting tape	Fixing the mini-vise to the work surface

Appendix

Table 3: Bill of materials for the six custom tasks extending the NIST Assembly Task Board #1. Fabrication files and ordering information for the task board itself are available from NIST [ 13 ] .

Figure 7: Hardware and teleoperation setup for REBOOT data collection. Two Trossen WidowX AI 6-DOF follower arms (lower) manipulate connectors on a NIST Assembly Task Board #1(centre); two matching leader arms (upper) provide kinesthetic teleoperation. The control workstation (foreground) runs the LeRobot recording interface, displaying live feeds from all four RGB-D cameras alongside task and episode controls.

Figure 9: Screenshot of the LeRobot online dataset visualizer showing the M12_fastener_install task. Four synchronized RGB camera streams are displayed: overhead ( cam_high ), front-facing ( cam_low ), left-wrist ( cam_left_wrist ), and right-wrist ( cam_right_wrist ). The language instruction, joint position time series, and episode list are also shown.

Figure 10: Screenshot of the Rerun SDK visualizer showing a recorded REBOOT episode. The interface displays synchronized RGB and depth streams from all four cameras ( cam_high , cam_low , cam_left_wrist , cam_right_wrist ), with depth rendered as false-colour maps alongside their corresponding RGB views. Action and state time series are shown in the left and right panels respectively, with joint indices colour-coded across the timeline.

Task	Sym.	Tier	Exp.	Rec.	Dur. (s)	Al(p)	En(p)	Tr.	Al(pl)	En(pl)
USB-A install	C1	Mod.	60	60	30	16	9	4	19	12
USB-A remove	C1	Mod.	60	60	15	16	8	8	20	8
USB-C install	C2	Tight	60	60	30	11	12	4	18	15
USB-C remove	C2	Tight	60	60	15	28	8	4	20	0
RJ45 install	C1	Tight	60	60	30	10	14	7	19	10
RJ45 remove	C1	Tight	60	60	15	3	20	24	8	5

Appendix

Table 4: Per-task episode counts and recovery-episode failure distributions. Phase columns sum to the recovery total per row. (Sym – Rotational symmetry, Tier – Clearances, Exp – Expert demonstrations, Dur – Duration, Al(p) – Align(Place), En(p) – Engage(Place), Tr. –Trasport, Al(pl) – Align(place), En(pl) – Engage(Place))

Panel of Table 5 — Table 5: Qualitative examples of successful and failed trajectories across the five phases of our task decomposition. Each phase exhibits a characteristic failure mode, and the language descriptions accompany each frame to provide grounded supervision for vision-language recovery policies.

Task (recovery, install)	Count	Episode indices
RJ45	23	2, 3, 4, 10, 11, 17, 23, 25, 26, 27, 28, 29, 30, 31, 34, 41, 42, 48, 49, 50, 53, 55, 58
16 mm bar	15	1, 8, 9, 10, 11, 12, 25, 27, 30, 32, 34, 37, 40, 45, 47
USB-A	11	18, 25, 26, 27, 28, 30, 44, 48, 50, 55, 58
16 mm cyl.	9	0, 17, 20, 22, 29, 30, 34, 37, 45
HAN 10E	9	1, 5, 6, 8, 31, 34, 37, 38, 59
NEMA 1-15P	8	15, 16, 17, 18, 19, 27, 51, 56

Appendix

Table 6: Episodes containing bi-manual interaction, by task. All occur in the recovery split of installation tasks. Episode indices are as released.

Object	Mechanism	Sym.	Tier	Clear. (mm)	Object description
Threaded fastener
M12 fastener	Threaded	$C_{2}$	Mod.	0.70	Steel hex nut, 12 mm thread
Electrical connectors
USB-A	Peg-in-hole	$C_{1}$	Mod.	0.62	Standard USB-A plug
USB-C	Peg-in-hole	$C_{2}$	Tight	0.09	Standard USB-C plug
RJ45	Peg-in-hole	$C_{1}$	Tight	0.13	RJ45 plug with retention latch

Appendix

Table 10: Task inventory for REBOOT. Each object appears as both an install and a remove task. Symmetry:

C_{1}

(one valid orientation),

C_{2}

(two valid orientations

180^{\circ}

apart), Cont. (any in-plane rotation valid). Tier: Int. = interference fit, Tight =

<

0.2 mm, Mod. = 0.2–1 mm, Loose =

>

1 mm.

Explore similar work

LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models
Sep 4, 2026Lin Liu, Zhicheng Bao, Lu Zhang +7Robot Failure Recovery Robotic Manipulation
Beta Infinity · Beijing Jiaotong University · School of Information and Communication Engineering Dalian University of Technology +1
AlphaNIST: Emergent Assembly Skills from Binary Task Rewards
Oct 9, 2026Octi Zhang, Iretiayo Akinola, Rosario Scalise +12Robotic Assembly Long-Horizon Robotic Manipulation
NVIDIA · University of Washington
Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?
Jun 24, 2026Tyler Ga Wei Lum, Kushal Kedia, C. Karen Liu +1RL for Robotics Robot Skill Learning
1Stanford University · 2Cornell University

Sep 4, 2026J/K move · Enter open · S save

LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models

Lin Liu, Zhicheng Bao, Lu Zhang, Ziying Song+6

Beta Infinity · Beijing Jiaotong University · School of Information and Communication Engineering Dalian University of Technology+1

Robot Failure Recovery Robotic Manipulation Robot Manipulation Benchmarks Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) or World Action (WAM) models have recently demonstrated remarkable performance in robotic manipulation. On LIBERO, SOTA method have achieved nearly 100% success rates, seemingly suggesting that the models are ready for deployment in real world. However, near perfect performance on existing benchmarks can be misleading: success under ideal conditions does not imply real world robustness. Existing benchmarks primarily evaluate task completion from predefined initial states, while real world interactions inevitably involve failures such as failed grasps, collisions, and unintended object movements. A robot must therefore not only execute tasks successfully, but also recognize and recover from failures to continue the task. Yet this capability remains largely unmeasured, revealing a critical gap between benchmark performance and real world reliability. To address this gap, we introduce LIBERO-Recover Benchmark, a large scale benchmark for failure recovery in robotic manipulation. Built upon LIBERO, we collect real execution failures from SOTA embodied models and construct 1,000+ scenarios across four recovery levels: (1) Action Retry, (2) Action Adaptation, (3) Object State Recovery, and (4) Environmental Recovery. We evaluate four core capabilities: spatial understanding, object structure reasoning, interaction understanding, and topological reasoning. As the first large-scale benchmark for embodied failure recovery, LIBERO-Recover shifts evaluation from Can the robot succeed?'' to Can the robot recover after failure?'', promoting robust and generalizable embodied agents. The project will be avaible in \textcolor{blue}{https://liulin815.github.io/LIBERO-Recovery/}.

Abstract

Explore similar work

LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models

AlphaNIST: Emergent Assembly Skills from Binary Task Rewards

Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?