Existing benchmarks evaluate tabletop manipulation, flat-floor household activity, or humanoid locomotion and manipulation as separate task groups; none scores vertical mobility and dexterous work on a fragile payload in one long-horizon episode. We present Fiatlux, a light-bulb replacement benchmark built on NVIDIA Isaac Lab. In one episode, a Unitree G1 humanoid positions a step ladder under a ceiling or wall fixture, climbs it, exchanges a spent bulb in a socket for a fresh one, and leaves the spent one in a disposal crate. We decompose the episode into twelve subtask environments scored on difficulty-weighted gates. The goal is a successful replacement, with the fresh bulb seated, the spent one disposed of, neither dropped, and a fragility bound not crossed. Runs that fall short can earn partial credit. Observations are split into a standard mode (signals a physical robot could sense or estimate) and a privileged mode (exact simulator state). We specify the evaluation protocol and provide reference baseline implementations spanning RSL-RL PPO, zero-shot NVIDIA GR00T N1.7 Vision-Language-Action (VLA) models, and whole-body controllers. Additionally, we provide the teleoperated recordings used to specify and check the success gates. The benchmark code and the teleoperated recordings are available at fiatlux-bench.github.io.
Figures & tables
ID
Subtask Name
Control Mode
Quantitative Evaluation / Termination Criterion
S01
Move Ladder
Flat Navigation
Ladder top within 0.67 m of the fixture, upright, ladder feet within 0.02 m of the floor, ladder at rest, robot standing (sustained 1.0 s )
S02
Climb Ladder
Climbing
Pelvis no more than 0.15 m below the ladder’s live top point and within 0.6 m of it horizontally, speed <1.5 m/s , robot standing, ladder vertical
S03
Remove Old Bulb
Manipulation, Climbing
Old bulb released by the socket and 0.10 m clear of the seat, held, elevation ≥0.15 m , robot standing, ladder vertical (sustained 0.5 s )
S04
Descend with Bulb
Climbing, Manipulation
Pelvis below floor-stance height, within 0.6 m of the ladder, speed <1.5 m/s , bulb held and elevated, robot standing, ladder vertical
S05
Carry Bulb to Disposal
Flat Navigation
Within 0.50 m of the crate’s footprint, facing it within 0.5 rad , speed <1.0 m/s , bulb held
S06
Dispose Bulb
Manipulation
Old bulb geometry fully inside the crate interior and below its rim, translating below 0.05 m/s , grip released, robot standing (sustained 1.0 s )
TABLE I: Taxonomy of the 12 atomic subtasks ( S01 – S12 ) in Fiatlux across the three control modes. Four subtasks run two modes at once, dominant first. Each evaluation criterion is a conjunction whose clauses must hold at the same instant; rows marked sustained must hold continuously for the stated window. Throughout, robot standing means pelvis height above 0.35 m and tilt under 1.0 rad , ladder vertical means tilt under 0.6 rad , held and released mean grip contact force above and below 1.0 N , and at rest means below 0.05 m/s and 0.10 rad/s .
Feasibility check
Passed/Tested
USD Structure
6/6 objects
Basic physics
18/18 trials
Bulb drop
14/16 trials
Bulb-socket interaction (standardized)
16/32 trials
Ladder load and push
16/16 trials
Scene fit
12/18 configurations
TABLE II: Feasibility validation over six generated objects.
ID
Subtask
Weight
Takes
Full credit
S01
Move Ladder
2.25
11
10
S02
Climb Ladder
3.00
–
–
S03
Remove Old Bulb
4.20
12
10
S04
Descend with Bulb
4.50
–
–
S05
Carry Bulb to Disposal
1.50
10
10
S06
Dispose Bulb
1.80
12
10
TABLE III: Subtask difficulty weights and teleoperated validation takes per subtask.
Fig. 1: Frames from the recorded teleoperated takes, one per subtask that has one. Each frame comes from a take that fires the subtask’s gate. The four climbing subtasks are omitted: none has a take that fires its gate. The two stance frames are recorded with the robot placed on the top tread, and show that it holds the on-ladder stance with and without the bulb; the transitions to and from that stance are not demonstrated.
ID
Subtask
GR00T
Zero
Random
Teleoperation
S01
Move Ladder
0.00
0.00
0.00
1.00
S02
Climb Ladder
0.00
0.00
0.00
–
S03
Remove Old Bulb
0.06 ± 0.12
0.00
0.06 ± 0.12
0.90 ± 0.25
S04
Descend with Bulb
0.25
0.12 ± 0.14
0.00
–
S05
Carry Bulb to Disposal
0.25
0.25
0.00
1.00
S06
Dispose Bulb
0.00
0.12 ± 0.08
0.33
0.85 ± 0.34
TABLE IV: Per-subtask score of Zero-shot GR00T N1.7 against the zero and random floor baselines and teleoperation, twelve subtasks. Each cell is mean ± sample standard deviation over the policy’s runs. GR00T/Zero/Random are averaged across 4 domain randomization seeds, Teleoperation is averaged across multiple (see Table III ) recorded takes with random seeds. The climb tasks did not result in successful teleoperation. ∗ Teleoperation’s weighted score covers only the 8 subtasks with an entry above (19.05 of 34.05 total weight); the 4 dashed subtasks are excluded from the sum, not scored zero.
Embodied AI, including vision-language-action and world-action models, must operate reliably in the physical world. Yet methods that perform well in simulation can degrade substantially on real robots, especially in long-horizon deformable-object manipulation, where policies must track changing states and execute reliable multi-stage bimanual interactions. Existing real-robot benchmarks mainly focus on short-horizon rigid-object tasks and offer limited coverage of long-horizon deformable manipulation. We introduce RoboFolDeX, a physical-world benchmark built entirely from real-robot data, with garment folding as its primary task. Since real-robot data collection is costly, RoboFolDeX studies how heterogeneous physical experience can be reused efficiently. The benchmark is organized around four research axes: leveraging human intervention and recovery data collected during deployment; transferring data across tasks, including across garment categories and from rigid to deformable-object manipulation; reusing data across scenes with changes in lighting, background, and layout; and transferring data across robotic embodiments. RoboFolDeX provides 2,000+ hours of real-robot data spanning 20+ tasks and 10+ embodiments. We also establish a fair real-robot evaluation platform for externally submitted policies, with standardized tasks, held-out physical objects, controlled initializations, and a unified execution protocol. The platform is publicly accessible at https://ai.midea.com/#/fold-challenge. We hope RoboFolDeX will serve as a unified testbed for heterogeneous real-robot data reuse and reliable long-horizon deformable manipulation.
Chenhuan Liu, Yi Xu, Feng Wu +7
Fudan University · AI Research Center, Midea Group (Shanghai) Co., Ltd. · Carnegie Mellon University
General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon robot manipulation benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena-∞-Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only ~25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ~62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: https://ladder.insait.ai
Petko Petkov, Giuliano Albanese, Nikolay Nikolov +3
INSAIT, Sofia University “St. Kliment Ohridski” · NVIDIA
As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits. Successful tool use requires selecting a suitable tool and coordinating manipulation and, when needed, locomotion to complete the task. Existing benchmarks do not jointly evaluate these capabilities on a humanoid. We introduce HumanoidToolBench, an 18-task benchmark spanning three scenarios, three execution levels, and two tool-set modes, together with ToolBook, a dataset of 3.1k demonstrations collected in simulation and on a real Unitree G1. Evaluation of seven policies in simulation and three on the real robot reveals substantial gaps between selecting a suitable tool and completing the task. Focused GR00T N1.7 probes show reduced selection accuracy on unseen tools and continued task execution under unrelated instructions. Code and data are available at https://snu-pi.github.io/HumanoidToolBench/.
Kyochul Jang, Seohyeon Park, Ohchul Kwon +9
Seoul National University · University of Massachusetts Amherst · Google Research