HATCH: Hardware-Attested Task Chains for Verifiable Manipulation

Shiza Charania

cs.ROOct 20, 2026

HATCH: Hardware-Attested Task Chains for Verifiable Manipulation

Authors: Shiza Charania

Organizations: Midas Robotics Inc., San Francisco, USA · University of Waterloo, Waterloo, Canada

Abstract

Capability claims in robot manipulation are unverifiable, and existing leaderboards are each closed, simulated, or gameable. We propose HATCH, a robot escape room: an openly specified, physically replicable, sensor-scored, and unrehearsable benchmark. Tasks span five skill categories at three difficulty levels and are chained so that the sensor confirming one task's completion unlocks the next, turning a room into a single long-horizon evaluation. Success is confirmed by instrumentation rather than human judgment, scoring is embodiment-agnostic, and object placements are randomized by a run-day seed. Entrants submit attested self-runs (uncut video, sensor logs, and the seed), with occasional organized events for calibration. Task specifications, build guide, bill of materials, and scoring rubrics are released under an open license.

Figures & tables

Category	Easy	Medium	Hard
Spatial reasoning	Place 4 labeled objects on matching colored mats	Fit chunky wooden shapes into a shape-sorting board	Rearrange 4 objects onto colored circles per a reference-card diagram
Symbolic / diagrammatic	Arrange blocks per a symbol-to-color key; color sensors confirm	Move a checker along a colored-arrow sequence; pressure plate confirms final square	UV light reveals $n$ symbols; select only matching objects from $m>n$
Multi-step chain	Note directs robot to red shelf’s letter; open compartment; retrieve clue	Match 5 of 10 objects to pinned photos; arrange in board order	Marked fake book hides key $\to$ box $\to$ order 4 tokens by size $\to$ enter revealed code on tile rack
Physical / causal	Open sealed envelope, extract keycard, swipe reader	Lift jar lids, read numbered tokens, place on numbered mat	Swap mismatched pen caps so body/cap colors match; overhead camera confirms
Dexterous / force	Insert varied-size pegs flush into matching holes; seating sensor confirms	Pick up flat poker chips; deposit each through a narrow slot, counter confirms	Match 4 screws to subtly different-sized holes; insert and drive each flush

TABLE I: HATCH task matrix (v0.1), one task per cell; full specifications in the open repository.

Robot Manipulation Benchmarks

Robot manipulation benchmarks measure how reliably robots perceive, plan, and execute physical tasks across objects, settings, and embodiments.

See all 312 papers

RobotAgentBench: Towards Open-world Robot Autonomy
Oct 26, 2026Lingjun Mao, Jinglin Cao, Lukun He +22Embodied Agent Benchmarks Robot Manipulation Benchmarks
Aether AI · University of California San Diego · University of California Davis
WorkBenchMark: A LEGO-Based Assembly Benchmark with an Assembly-by-Disassembly Baseline for the Smart Manufacturing League
Jun 2, 2026Wenbo Ma, Daniel Swoboda, Matteo Tschesche +1Robotic Manipulation Robot Task Planning
Chair of Machine Learning and Reasoning (i6), RWTH Aachen University, Aachen, Germany · MASCOR Institute, FH Aachen University of Applied Sciences, Aachen, Germany
RoboPlayground: Democratizing Robotic Evaluation through Structured Physical Domains
Apr 6, 2026Yi Ru Wang, Carter Ung, Evan Gubarev +3Robotic Policy Evaluation Robotics
University of Washington, Seattle, WA, USA · Allen Institute for Artificial Intelligence, Seattle, WA, USA

Oct 26, 2026J/K move · Enter open · S save

RobotAgentBench: Towards Open-world Robot Autonomy

Lingjun Mao, Jinglin Cao, Lukun He, Wenpeng Xu+21

Aether AI · University of California San Diego · University of California Davis

Embodied Agent Benchmarks Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Recent advances in vision-language-action (VLA) and world-action models (WAMs) have significantly improved robotic manipulation, yet deploying robots in the open world demands a higher level of autonomy beyond executing predefined tasks. Robot agents offer a promising paradigm for this transition, integrating task discovery, contextual reasoning, physical execution, and adaptive recovery into a closed interaction loop. However, existing benchmarks largely evaluate isolated capabilities, failing to capture the complexity of open-world robot autonomy. We introduce RobotAgentBench (RAB), a comprehensive benchmark for evaluating robot agents in realistic, open-world household environments. Through an agent-assisted, human-in-the-loop real-to-sim pipeline, RAB constructs 50 diverse household layouts with interactive objects and physically grounded tasks, encompassing 90 atomic tasks, 18 short-horizon tasks, 8 long-horizon tasks, and 4 open-ended scenarios. RAB provides a multi-level evaluation of robot agent capabilities: (1) task-level assessments systematically examine five complementary dimensions: task discovery, context reasoning, manipulation, failure recovery, and robustness; (2) scenario-level evaluation assesses how agents integrate these capabilities to autonomously fulfill high-level user needs. Furthermore, RAB enables systematic comparisons of standalone VLA/WAM models and diverse robot agent architectures under a unified planning-execution-verification framework. Extensive experiments reveal substantial gaps between manipulation and autonomy, demonstrate the advantages of agentic architectures and recursive self-improvement, and highlight the remaining challenges toward open-world robot autonomy.

Abstract

Robot Manipulation Benchmarks

Explore similar work

RobotAgentBench: Towards Open-world Robot Autonomy

WorkBenchMark: A LEGO-Based Assembly Benchmark with an Assembly-by-Disassembly Baseline for the Smart Manufacturing League

RoboPlayground: Democratizing Robotic Evaluation through Structured Physical Domains