Fiatlux: A Long-Horizon Benchmark for Humanoid Ladder Climbing and Light-Bulb Replacement

Pavel Bushuyeu; Yujin Chen; Anton Nikolaev; Brian Shu; Igor Molybog

cs.ROSep 27, 2026

Fiatlux: A Long-Horizon Benchmark for Humanoid Ladder Climbing and Light-Bulb Replacement

Authors: Pavel Bushuyeu, Yujin Chen, Anton Nikolaev, Brian Shu, Igor Molybog

Organizations: HawAII, University of Hawai‘i at M¯anoa · Independent Researcher · Purdue University

Abstract

Existing benchmarks evaluate tabletop manipulation, flat-floor household activity, or humanoid locomotion and manipulation as separate task groups; none scores vertical mobility and dexterous work on a fragile payload in one long-horizon episode. We present Fiatlux, a light-bulb replacement benchmark built on NVIDIA Isaac Lab. In one episode, a Unitree G1 humanoid positions a step ladder under a ceiling or wall fixture, climbs it, exchanges a spent bulb in a socket for a fresh one, and leaves the spent one in a disposal crate. We decompose the episode into twelve subtask environments scored on difficulty-weighted gates. The goal is a successful replacement, with the fresh bulb seated, the spent one disposed of, neither dropped, and a fragility bound not crossed. Runs that fall short can earn partial credit. Observations are split into a standard mode (signals a physical robot could sense or estimate) and a privileged mode (exact simulator state). We specify the evaluation protocol and provide reference baseline implementations spanning RSL-RL PPO, zero-shot NVIDIA GR00T N1.7 Vision-Language-Action (VLA) models, and whole-body controllers. Additionally, we provide the teleoperated recordings used to specify and check the success gates. The benchmark code and the teleoperated recordings are available at fiatlux-bench.github.io.

Figures & tables

ID	Subtask Name	Control Mode	Quantitative Evaluation / Termination Criterion
$S_{01}$	Move Ladder	Flat Navigation	Ladder top within $0.67\text{ m}$ of the fixture, upright, ladder feet within $0.02\text{ m}$ of the floor, ladder at rest, robot standing (sustained $1.0\text{ s}$ )
$S_{02}$	Climb Ladder	Climbing	Pelvis no more than $0.15\text{ m}$ below the ladder’s live top point and within $0.6\text{ m}$ of it horizontally, speed $<1.5\text{ m/s}$ , robot standing, ladder vertical
$S_{03}$	Remove Old Bulb	Manipulation, Climbing	Old bulb released by the socket and $0.10\text{ m}$ clear of the seat, held, elevation $\geq 0.15\text{ m}$ , robot standing, ladder vertical (sustained $0.5\text{ s}$ )
$S_{04}$	Descend with Bulb	Climbing, Manipulation	Pelvis below floor-stance height, within $0.6\text{ m}$ of the ladder, speed $<1.5\text{ m/s}$ , bulb held and elevated, robot standing, ladder vertical
$S_{05}$	Carry Bulb to Disposal	Flat Navigation	Within $0.50\text{ m}$ of the crate’s footprint, facing it within $0.5\text{ rad}$ , speed $<1.0\text{ m/s}$ , bulb held
$S_{06}$	Dispose Bulb	Manipulation	Old bulb geometry fully inside the crate interior and below its rim, translating below $0.05\text{ m/s}$ , grip released, robot standing (sustained $1.0\text{ s}$ )

TABLE I: Taxonomy of the 12 atomic subtasks (

S_{01}

–

S_{12}

) in Fiatlux across the three control modes. Four subtasks run two modes at once, dominant first. Each evaluation criterion is a conjunction whose clauses must hold at the same instant; rows marked sustained must hold continuously for the stated window. Throughout, robot standing means pelvis height above

0.35\text{ m}

and tilt under

1.0\text{ rad}

, ladder vertical means tilt under

0.6\text{ rad}

, held and released mean grip contact force above and below

1.0\text{ N}

, and at rest means below

0.05\text{ m/s}

and

0.10\text{ rad/s}

Feasibility check	Passed/Tested
USD Structure	6/6 objects
Basic physics	18/18 trials
Bulb drop	14/16 trials
Bulb-socket interaction (standardized)	16/32 trials
Ladder load and push	16/16 trials
Scene fit	12/18 configurations

TABLE II: Feasibility validation over six generated objects.

ID	Subtask	Weight	Takes	Full credit
$S_{01}$	Move Ladder	2.25	11	10
$S_{02}$	Climb Ladder	3.00	–	–
$S_{03}$	Remove Old Bulb	4.20	12	10
$S_{04}$	Descend with Bulb	4.50	–	–
$S_{05}$	Carry Bulb to Disposal	1.50	10	10
$S_{06}$	Dispose Bulb	1.80	12	10

TABLE III: Subtask difficulty weights and teleoperated validation takes per subtask.

Panel of Figure 4 — Fig. 1: Frames from the recorded teleoperated takes, one per subtask that has one. Each frame comes from a take that fires the subtask’s gate. The four climbing subtasks are omitted: none has a take that fires its gate. The two stance frames are recorded with the robot placed on the top tread, and show that it holds the on-ladder stance with and without the bulb; the transitions to and from that stance are not demonstrated.

ID	Subtask	GR00T	Zero	Random	Teleoperation
$S_{01}$	Move Ladder	0.00	0.00	0.00	1.00
$S_{02}$	Climb Ladder	0.00	0.00	0.00	–
$S_{03}$	Remove Old Bulb	0.06 $\pm$ 0.12	0.00	0.06 $\pm$ 0.12	0.90 $\pm$ 0.25
$S_{04}$	Descend with Bulb	0.25	0.12 $\pm$ 0.14	0.00	–
$S_{05}$	Carry Bulb to Disposal	0.25	0.25	0.00	1.00
$S_{06}$	Dispose Bulb	0.00	0.12 $\pm$ 0.08	0.33	0.85 $\pm$ 0.34

TABLE IV: Per-subtask score of Zero-shot GR00T N1.7 against the zero and random floor baselines and teleoperation, twelve subtasks. Each cell is mean

\pm

sample standard deviation over the policy’s runs. GR00T/Zero/Random are averaged across 4 domain randomization seeds, Teleoperation is averaged across multiple (see Table III ) recorded takes with random seeds. The climb tasks did not result in successful teleoperation. ∗ Teleoperation’s weighted score covers only the 8 subtasks with an entry above (19.05 of 34.05 total weight); the 4 dashed subtasks are excluded from the sum, not scored zero.

Explore similar work

RoboFolDeX: A Physical-World Benchmark for Long-Horizon Robotic Manipulation of Deformable Objects
Sep 9, 2026Chenhuan Liu, Yi Xu, Feng Wu +7Long-Horizon Robotic Manipulation Bimanual Robotic Manipulation
Fudan University · AI Research Center, Midea Group (Shanghai) Co., Ltd. · Carnegie Mellon University
LADDER: A Long-Horizon Robot Manipulation Benchmark
Oct 13, 2026Petko Petkov, Giuliano Albanese, Nikolay Nikolov +3Long-Horizon Robotic Manipulation Robot Manipulation Benchmarks
INSAIT, Sofia University “St. Kliment Ohridski” · NVIDIA
HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution
Oct 1, 2026Kyochul Jang, Seohyeon Park, Ohchul Kwon +9Mobile Manipulation Robotic Tool Use
Seoul National University · University of Massachusetts Amherst · Google Research

Sep 9, 2026J/K move · Enter open · S save

RoboFolDeX: A Physical-World Benchmark for Long-Horizon Robotic Manipulation of Deformable Objects

Chenhuan Liu, Yi Xu, Feng Wu, Hanyang Wang+6

Fudan University · AI Research Center, Midea Group (Shanghai) Co., Ltd. · Carnegie Mellon University

Long-Horizon Robotic Manipulation Bimanual Robotic Manipulation Robotic Data Collection Deformable Object Manipulation Cross-Embodiment Robot Learning Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Embodied AI, including vision-language-action and world-action models, must operate reliably in the physical world. Yet methods that perform well in simulation can degrade substantially on real robots, especially in long-horizon deformable-object manipulation, where policies must track changing states and execute reliable multi-stage bimanual interactions. Existing real-robot benchmarks mainly focus on short-horizon rigid-object tasks and offer limited coverage of long-horizon deformable manipulation. We introduce RoboFolDeX, a physical-world benchmark built entirely from real-robot data, with garment folding as its primary task. Since real-robot data collection is costly, RoboFolDeX studies how heterogeneous physical experience can be reused efficiently. The benchmark is organized around four research axes: leveraging human intervention and recovery data collected during deployment; transferring data across tasks, including across garment categories and from rigid to deformable-object manipulation; reusing data across scenes with changes in lighting, background, and layout; and transferring data across robotic embodiments. RoboFolDeX provides 2,000+ hours of real-robot data spanning 20+ tasks and 10+ embodiments. We also establish a fair real-robot evaluation platform for externally submitted policies, with standardized tasks, held-out physical objects, controlled initializations, and a unified execution protocol. The platform is publicly accessible at https://ai.midea.com/#/fold-challenge. We hope RoboFolDeX will serve as a unified testbed for heterogeneous real-robot data reuse and reliable long-horizon deformable manipulation.

Abstract

Explore similar work

RoboFolDeX: A Physical-World Benchmark for Long-Horizon Robotic Manipulation of Deformable Objects

LADDER: A Long-Horizon Robot Manipulation Benchmark

HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution