REBOOT: From Failure to Recovery - A Dataset and Benchmark for Precision Assembly

Nana Yaw Owusu Ofori-Ampofo; Samira Ebrahimi Kahou; Joseph Thekinen

cs.ROSep 18, 2026

REBOOT: From Failure to Recovery - A Dataset and Benchmark for Precision Assembly

Authors: Nana Yaw Owusu Ofori-Ampofo, Samira Ebrahimi Kahou, Joseph Thekinen

Organizations: University of Calgary · University of Calgary, Mila

Abstract

Robot learning policies fail in characteristic ways: they stall in uncertain states, drift during contact-rich alignment, and miss targets by millimetres in precision tasks. Yet training datasets consist largely of successful demonstrations, while real-world benchmarks often reduce performance to binary success. This limits both supervision for recovery and analysis of where failures occur. We introduce REBOOT (Recovery Episode Benchmark for Off-nominal Trajectories), the first robot manipulation benchmark designed around failure as a first-class signal. REBOOT contains 2,160 demonstrations across 18 precision assembly tasks, each decomposed into five shared phases: Align(pick), Engage(pick), Transport, Align(place), and Engage(place), enabling phase-level evaluation beyond terminal success. Failures are introduced across phases and paired with expert recovery trajectories that return the system to a valid continuation state. Tasks are annotated with rotational symmetry, engagement-clearance precision tier, and assembly direction through matched install-remove pairs. Failure episodes are labeled by phase and categorical failure mode, enabling attribution to kinematic stage and tolerance violation. Data includes synchronized RGB-D observations from four viewpoints and grounded natural-language descriptions of phase-level success and failure conditions. Half the dataset contains expert demonstrations; the other half contains recovery demonstrations sampled to reflect failures observed in imitation-learned policy rollouts. We benchmark action-chunked transformer, diffusion, and $π_0$ -FAST policies using phase-level completion rates, revealing model-specific failure points hidden by binary evaluation. Dataset and code: https://nanayawoa.github.io/REBOOT

Figures & tables

Figure 1: Overview of REBOOT. Each precision-assembly task is decomposed into five kinematic phases— Align(pick) , Engage(pick) , Transport , Align(place) , and Engage(place) —shown vertically on the left. The first column illustrates successful task execution with natural language annotations at each phase. The second column shows some failure modes that arise at each phase during autonomous policy rollouts, paired with the expert recovery trajectories that return the system to a valid continuation state. REBOOT treats both columns as first-class supervision signal.

Dataset	Domain	# of demos	% Failure	Recovery - Yes/No
REFLECT [ 8 ]	Sim+Real	130	100.0%	No
RoboFAC [ 7 ]	Sim+Real	10,722	88.0%	No
RaC [ 4 ]	Sim+Real	N/A †	N/A †	Yes
DROID [ 2 ]	Real	$\approx$ 92,000	$\approx$ 17.4%	Yes (3.68%)
REASSEMBLE [ 9 ]	Real	4,551	11.3%	No
FurnitureBench [ 10 ]	Sim+Real	5,100	0%	No

Table 1: Robot manipulation datasets reporting explicit failure or recovery episode counts. Counts are taken verbatim from the primary source (paper text and dataset cards). REBOOT is the only real-robot dataset with a 50% failure rate and structured per-phase recovery annotations.

Figure 2: Rotational symmetry groups for representative objects in the dataset. Each column shows a goal–object pair; from left to right: 16 mm cylinder ( Continuous ), RCA connector ( Continuous ), M12 fastener ( C2 ), 16 mm bar ( C2 ), USB-C connector ( C2 ), NEMA 1-15P power plug ( C1 ), RJ45 connector ( C1 ), USB-A connector ( C1 ), and Harting HAN10E connector ( C1 ).

Figure 3: Overview of the expert and recovery data collection pipeline. Policies trained on expert demonstrations are rolled out to identify phase-level failures, which are subsequently re-enacted by the teleoperator to collect successful recovery demonstrations.

Connector	Clearance (mm)
RCA	interference fit (0)
USB-C	0.09
RJ45	0.13
16mm Bar	0.20
16mm Cylinder	0.20
NEMA 1-15P power plug	0.5

Table 2: Connector clearance values.

Panel of Figure 4 — Figure 4: Distribution of last-completed phase across precision assembly and disassembly tasks. Each group represents one connector; each bar shows the fraction of policy rollouts whose last successfully completed phase was the indicated phase, with the rightmost bar ( green ) showing the fraction that completed all five phases (full task success). Aggregated across all three policy families ( $N{=}15$ rollouts per task per policy). .

Panel of Figure 5 — Figure 5: Phase-failure profiles for installation and removal tasks. Boxes show median and interquartile range with whiskers at $1.5\times$ IQR; markers show all nine per-task counts. The install/remove asymmetry is visible at the Success endpoint, where removal counts are several times higher.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Product used	Used for
Mini-vise	Tabletop drill-press vise with rubber soft jaws, $0\text{\,}\mathrm{mm}63\text{\,}\mathrm{mm}$ clamping range	Holding the female connector fixed
RCA connectors	2 RCA-to-2 RCA gold-plated audio extension cable (male and female ends), $0.9\text{\,}\mathrm{m}$	RCA install / remove
USB-C connectors	USB-C 3.2 Gen2 male-to-female extension cable, $20\text{\,}\mathrm{G}\mathrm{b}\mathrm{p}\mathrm{s}$	USB-C install / remove
NEMA 1-15P (male)	Two-prong polarized AC power cord, 18 AWG, $0.5\text{\,}\mathrm{m}$	NEMA 1-15P install / remove
NEMA 1-15P (female)	Polarized two-prong extension cord, 16 AWG, $3.0\text{\,}\mathrm{m}$	NEMA 1-15P install / remove
Mounting tape	Double-sided indoor mounting tape	Fixing the mini-vise to the work surface

Appendix

Table 3: Bill of materials for the six custom tasks extending the NIST Assembly Task Board #1. Fabrication files and ordering information for the task board itself are available from NIST [ 13 ] .

Figure 7: Hardware and teleoperation setup for REBOOT data collection. Two Trossen WidowX AI 6-DOF follower arms (lower) manipulate connectors on a NIST Assembly Task Board #1(centre); two matching leader arms (upper) provide kinesthetic teleoperation. The control workstation (foreground) runs the LeRobot recording interface, displaying live feeds from all four RGB-D cameras alongside task and episode controls.

Figure 9: Screenshot of the LeRobot online dataset visualizer showing the M12_fastener_install task. Four synchronized RGB camera streams are displayed: overhead ( cam_high ), front-facing ( cam_low ), left-wrist ( cam_left_wrist ), and right-wrist ( cam_right_wrist ). The language instruction, joint position time series, and episode list are also shown.

Figure 10: Screenshot of the Rerun SDK visualizer showing a recorded REBOOT episode. The interface displays synchronized RGB and depth streams from all four cameras ( cam_high , cam_low , cam_left_wrist , cam_right_wrist ), with depth rendered as false-colour maps alongside their corresponding RGB views. Action and state time series are shown in the left and right panels respectively, with joint indices colour-coded across the timeline.

Task	Sym.	Tier	Exp.	Rec.	Dur. (s)	Al(p)	En(p)	Tr.	Al(pl)	En(pl)
USB-A install	C1	Mod.	60	60	30	16	9	4	19	12
USB-A remove	C1	Mod.	60	60	15	16	8	8	20	8
USB-C install	C2	Tight	60	60	30	11	12	4	18	15
USB-C remove	C2	Tight	60	60	15	28	8	4	20	0
RJ45 install	C1	Tight	60	60	30	10	14	7	19	10
RJ45 remove	C1	Tight	60	60	15	3	20	24	8	5

Appendix

Table 4: Per-task episode counts and recovery-episode failure distributions. Phase columns sum to the recovery total per row. (Sym – Rotational symmetry, Tier – Clearances, Exp – Expert demonstrations, Dur – Duration, Al(p) – Align(Place), En(p) – Engage(Place), Tr. –Trasport, Al(pl) – Align(place), En(pl) – Engage(Place))

Panel of Table 5 — Table 5: Qualitative examples of successful and failed trajectories across the five phases of our task decomposition. Each phase exhibits a characteristic failure mode, and the language descriptions accompany each frame to provide grounded supervision for vision-language recovery policies.

Task (recovery, install)	Count	Episode indices
RJ45	23	2, 3, 4, 10, 11, 17, 23, 25, 26, 27, 28, 29, 30, 31, 34, 41, 42, 48, 49, 50, 53, 55, 58
16 mm bar	15	1, 8, 9, 10, 11, 12, 25, 27, 30, 32, 34, 37, 40, 45, 47
USB-A	11	18, 25, 26, 27, 28, 30, 44, 48, 50, 55, 58
16 mm cyl.	9	0, 17, 20, 22, 29, 30, 34, 37, 45
HAN 10E	9	1, 5, 6, 8, 31, 34, 37, 38, 59
NEMA 1-15P	8	15, 16, 17, 18, 19, 27, 51, 56

Appendix

Table 6: Episodes containing bi-manual interaction, by task. All occur in the recovery split of installation tasks. Episode indices are as released.

Object	Mechanism	Sym.	Tier	Clear. (mm)	Object description
Threaded fastener
M12 fastener	Threaded	$C_{2}$	Mod.	0.70	Steel hex nut, 12 mm thread
Electrical connectors
USB-A	Peg-in-hole	$C_{1}$	Mod.	0.62	Standard USB-A plug
USB-C	Peg-in-hole	$C_{2}$	Tight	0.09	Standard USB-C plug
RJ45	Peg-in-hole	$C_{1}$	Tight	0.13	RJ45 plug with retention latch

Appendix

Table 10: Task inventory for REBOOT. Each object appears as both an install and a remove task. Symmetry:

C_{1}

(one valid orientation),

C_{2}

(two valid orientations

180^{\circ}

apart), Cont. (any in-plane rotation valid). Tier: Int. = interference fit, Tight =

<

0.2 mm, Mod. = 0.2–1 mm, Loose =

>

1 mm.

Explore similar work

Sep 4, 2026

LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models

Vision-Language-Action (VLA) or World Action (WAM) models have recently demonstrated remarkable performance in robotic manipulation. On LIBERO, SOTA method have achieved nearly 100% success rates, seemingly suggesting that the models are ready for deployment in real world. However, near perfect performance on existing benchmarks can be misleading: success under ideal conditions does not imply real world robustness. Existing benchmarks primarily evaluate task completion from predefined initial states, while real world interactions inevitably involve failures such as failed grasps, collisions, and unintended object movements. A robot must therefore not only execute tasks successfully, but also recognize and recover from failures to continue the task. Yet this capability remains largely unmeasured, revealing a critical gap between benchmark performance and real world reliability. To address this gap, we introduce LIBERO-Recover Benchmark, a large scale benchmark for failure recovery in robotic manipulation. Built upon LIBERO, we collect real execution failures from SOTA embodied models and construct 1,000+ scenarios across four recovery levels: (1) Action Retry, (2) Action Adaptation, (3) Object State Recovery, and (4) Environmental Recovery. We evaluate four core capabilities: spatial understanding, object structure reasoning, interaction understanding, and topological reasoning. As the first large-scale benchmark for embodied failure recovery, LIBERO-Recover shifts evaluation from Can the robot succeed?'' to Can the robot recover after failure?'', promoting robust and generalizable embodied agents. The project will be avaible in \textcolor{blue}{https://liulin815.github.io/LIBERO-Recovery/}.

Lin Liu, Zhicheng Bao, Lu Zhang +7

Beta Infinity · Beijing Jiaotong University · School of Information and Communication Engineering Dalian University of Technology +1

Robot Failure Recovery Robotic Manipulation Robot Manipulation Benchmarks Robustness of VLA Models

Ask PDF

Oct 9, 2026

AlphaNIST: Emergent Assembly Skills from Binary Task Rewards

The NIST Assembly Task Board #1 remains a challenging benchmark for learning-based methods in robotic manipulation. Over short horizons, it demands precise, contact-rich control of small, tight-fitting parts. Over long horizons, it requires recovery and coordination across extended action sequences. Demonstrations, task decomposition, and dense progress rewards have enabled research advances by providing guidance on how to assemble. But how much of the strategy must be specified? The progression from AlphaGo to AlphaGo Zero showed that RL at scale can discover superhuman strategies without prescribing how to win. AlphaNIST asks whether a sparse binary task reward and task diversity admitted by this task-agnostic reward can similarly let a robot discover how to assemble. In simulation, we train a single policy from scratch across all 20 assembly tasks on the NIST Assembly Task Board #1, reaching 98.2% overall training success without demonstrations, progress shaping, scripted primitives, or per-task tuning. The framework also trains policies to complete ten assembly pairs in a single episode without a prescribed order, with task success rewarded only when all ten pairs are simultaneously complete. The learned policies discover dexterous and planning-like behaviors: choosing grasps for later assembly, using surrounding geometry to reorient parts, and preparing one part, assembling another, then returning to finish the first without an explicit planner. For videos, see our website: https://alpha-nist.github.io/

Octi Zhang, Iretiayo Akinola, Rosario Scalise +12

NVIDIA · University of Washington

Robotic Assembly Long-Horizon Robotic Manipulation Multi-Task Robotic Manipulation Sparse-Reward RL Robot Manipulation Benchmarks

Ask PDF

Jun 24, 2026

Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?

Multi-fingered robots promise the speed and dexterity of human hands, yet challenging problems such as precise assembly have remained out of reach. These tasks are contact-rich, making data collection for imitation learning difficult, and sparse-reward, making direct exploration with reinforcement learning (RL) intractable. Consequently, prior work has made progress by structuring the problem with specialized grippers, tool attachments, and environment fixtures. In this work, we argue that before a robot can perfect precise assembly, it must first learn to play. We further ask the question: what factors in the process of learning to play matter for precise assembly? We propose Play2Perfect, an RL framework for task-agnostic pretraining through play on diverse objects and goals, which is then perfected on precise assembly. The goal of play is to acquire reusable manipulation priors, such as grasping, in-hand reorientation and pose reaching. Finetuning then adapts this general prior to assembly, focusing exploration on the final contact-rich, high-precision interactions needed for success. We systematically study key design choices in play pretraining, including object diversity, training objective, trajectory diversity, and goal precision. We show that our prior is 33x more sample-efficient than RL training from scratch, even when provided with dense, multi-stage rewards. We demonstrate zero-shot sim-to-real transfer, achieving 60% success on tight insertions with only 0.5 mm contact clearance, and over 50% success on long-horizon multi-part assembly and screwing.

Tyler Ga Wei Lum, Kushal Kedia, C. Karen Liu +1

1Stanford University · 2Cornell University

RL for Robotics Robot Skill Learning Contact-Rich Robotic Manipulation Robotic Manipulation Robotic Assembly Robot Learning

Ask PDF