Organizations: Department of Computer Science, Duke University, Durham, NC, USA · Department of Mechanical Engineering and Materials Science, Duke University, Durham, NC, USA
Large language model (LLM) planners can decompose natural-language instructions and select reusable robot skills, but choosing the correct skill does not guarantee successful physical execution. This gap is especially important in humanoid loco-manipulation, where errors during approach, grasping, transport, or placement can invalidate the remainder of a long-horizon plan. We present FRAMES, a failure-aware supervisory framework for the Unitree G1 humanoid that operates above the CEER whole-body controller. A Planner Agent selects subtasks through parameterized mid-level skills, while a vision-language-model-based Monitor Agent evaluates each skill using temporal multi-view observations and structured robot and contact evidence. Detected failures stop the active skill and provide grounded feedback to a Recovery Agent. The framework further includes a Memory Module for reusing prior skill experience, and geometric grounding via depth and segmentation. We independently evaluate the monitoring module of the framework in MuJoCo using 100 trials comprising 50 failed and 50 successful executions across five tasks. The monitor detects 48 of 50 failures, correctly accepts 46 of 50 successful executions, and achieves 94.0% overall accuracy. These results provide initial evidence for the monitoring component, while end-to-end evaluation of the complete recovery loop remains ongoing.
Figures & tables
Fig. 1: The planner selects the correct grasp subtask, yet the box can still slip during physical execution. Without failure-aware supervision, the robot proceeds from an invalid state and fails the task. FRAMES detects the failed grasp, invokes a centered grasp, and enables the robot to complete the turn while retaining the box.
Fig. 2: Overview of FRAMES. A natural-language instruction is handled by the Planner Agent , which selects a subtask from the skill library together with its API parameters and necessary preconditions and failure and success criteria. The selected subtask is executed through the fixed CEER whole-body controller in the MuJoCo environment. During execution, the Monitor Agent compares the current observation against the baseline observation using multi-view perception and structured state information to determine whether the subtask is succeeding or failing. The monitor verdict and supporting evidence are returned immediately to the decision layer, where failures inform the Recovery Agent . At the end of each run, completed skill executions are distilled into concise subtask-level records and appended to memory.
Category
API
Arguments
Function
Locomotion
move_to
(x,y,ψ)
Moves the robot base to an absolute world-frame pose.
turn
(ψ,T)
Rotates the robot toward an absolute world-frame yaw ψ over duration T .
move_backward
(x,y)
Moves the base toward an absolute world-frame position while preserving its final orientation.
approach_object
(target,dstand,Δψ)
Moves the base to a target-facing pose with the specified standoff distance and optional yaw offset.
Manipulation
move_hands
(pL,pR,Thold)
Moves both hands to robot-base-frame target positions pL,pR∈R3 and maintains them for Thold seconds, without moving the base.
grasp_object
(target,pL,pR,dlift)
Grasps the target using specified left- and right-hand contact positions, then lifts it by dlift .
TABLE I: Skill Library Available to the Planner and Recovery Agents
Fig. 3: Representative failure and success outcomes from the five Monitor Agent tasks. The top row shows failure conditions, and the bottom row shows matched successful executions: (a) failed versus successful grasp; (b) dropped object versus stable hold; (c) collision versus collision-free object approach; (d) grasping the blue distractor box versus the requested red box; and (e) placing the red box on the yellow distractor pad versus the blue target pad.
Benchmark
Failures Detected
Successful Runs Accepted
Overall Accuracy
Failed grasp
8/10
10/10
90%
Dropped object
10/10
10/10
100%
Collision during approach
10/10
7/10
85%
Wrong-object grasp
10/10
10/10
100%
Wrong placement
10/10
9/10
95%
Overall
48/50
46/50
94%
TABLE II: Monitor performance across five benchmarks, each containing 10 failed and 10 successful skill executions.
Robots are increasingly expected to execute open ended natural language requests in human environments, which demands reliable long horizon execution under partial observability. This is especially challenging for humanoids because locomotion and manipulation are tightly coupled through stance, reachability, and balance. We present a humanoid agent framework that turns VLM plans into verifiable task programs and closes the loop with multi object 3D geometric supervision. A VLM planner compiles each instruction into a typed JSON sequence of subtasks with explicit predicate based preconditions and success conditions. Using SAM3 and RGB-D, we ground all task relevant entities in 3D, estimate object centroids and extents, and evaluate predicates over stable frames to obtain condition level diagnostics. The supervisor uses these diagnostics to verify subtask completion and to provide condition-level feedback for progression and replanning. We execute each subtask by coordinating humanoid locomotion and whole-body manipulation, selecting feasible motion primitives under reachability and balance constraints. Experiments on tabletop manipulation and long horizon humanoid loco manipulation tasks show improved robustness from multi object grounding, temporal stability, and recovery driven replanning.
Peng Ren, Haoyang Ge, Chuan Qi +5
Beihang University, Beijing, China · Zhongguancun Academy, Beijing, China · Tianjin University, Tianjin, China +3
Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making. Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies, restricting their ability to handle complex task sequences. To address this limitation, we propose LUCID, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model. LUCID first trains a structured latent-conditioned low-level policy via adversarial imitation and then freezes it while jointly learning a high-level policy and macro-dynamics world model. The world model predicts the temporally extended state transitions induced by latent decisions, enabling high-level policy optimization through imagined rollouts. We evaluate our framework across various simulated multi-object rearrangement scenarios. Experimental results show that LUCID improves the full-task success and partial-completion rates compared to prior baseline methods, demonstrating its effectiveness in complex sequential loco-manipulation tasks.
Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy
Humanoid loco-manipulation requires robots to interpret task instructions and scene semantics while executing coordinated whole-body motions. We propose a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control. Each keyframe specifies a target whole-body robot pose and, when applicable, an object pose. Given a language instruction, scene observations, and execution feedback, the VLM selects successive task-relevant keyframes from a predefined library. The selected keyframes are retargeted to the current scene to account for object poses and dimensions. A keyframe-conditioned whole-body policy then generates joint-level actions to reach these goals. We introduce a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate from 44% to 92% when using sparse VLM keyframes. We evaluate our framework on object pickup, transport, and placement tasks in simulation and on a Unitree G1 humanoid. The system successfully performs both one- and two-handed manipulation and generalises to placement locations beyond the training reference data.