FRAMES: Failure Recovery And Monitoring of Embodied Skills for Humanoid Loco-Manipulation

Ajay Vikram Periasami; Xinyuan Luo; Haoyu Li; Xianyi Cheng

cs.ROSep 18, 2026

FRAMES: Failure Recovery And Monitoring of Embodied Skills for Humanoid Loco-Manipulation

Authors: Ajay Vikram Periasami, Xinyuan Luo, Haoyu Li, Xianyi Cheng

Organizations: Department of Computer Science, Duke University, Durham, NC, USA · Department of Mechanical Engineering and Materials Science, Duke University, Durham, NC, USA

Abstract

Large language model (LLM) planners can decompose natural-language instructions and select reusable robot skills, but choosing the correct skill does not guarantee successful physical execution. This gap is especially important in humanoid loco-manipulation, where errors during approach, grasping, transport, or placement can invalidate the remainder of a long-horizon plan. We present FRAMES, a failure-aware supervisory framework for the Unitree G1 humanoid that operates above the CEER whole-body controller. A Planner Agent selects subtasks through parameterized mid-level skills, while a vision-language-model-based Monitor Agent evaluates each skill using temporal multi-view observations and structured robot and contact evidence. Detected failures stop the active skill and provide grounded feedback to a Recovery Agent. The framework further includes a Memory Module for reusing prior skill experience, and geometric grounding via depth and segmentation. We independently evaluate the monitoring module of the framework in MuJoCo using 100 trials comprising 50 failed and 50 successful executions across five tasks. The monitor detects 48 of 50 failures, correctly accepts 46 of 50 successful executions, and achieves 94.0% overall accuracy. These results provide initial evidence for the monitoring component, while end-to-end evaluation of the complete recovery loop remains ongoing.

Figures & tables

Fig. 1: The planner selects the correct grasp subtask, yet the box can still slip during physical execution. Without failure-aware supervision, the robot proceeds from an invalid state and fails the task. FRAMES detects the failed grasp, invokes a centered grasp, and enables the robot to complete the turn while retaining the box.

Fig. 2: Overview of FRAMES. A natural-language instruction is handled by the Planner Agent , which selects a subtask from the skill library together with its API parameters and necessary preconditions and failure and success criteria. The selected subtask is executed through the fixed CEER whole-body controller in the MuJoCo environment. During execution, the Monitor Agent compares the current observation against the baseline observation using multi-view perception and structured state information to determine whether the subtask is succeeding or failing. The monitor verdict and supporting evidence are returned immediately to the decision layer, where failures inform the Recovery Agent . At the end of each run, completed skill executions are distilled into concise subtask-level records and appended to memory.

Category	API	Arguments	Function
Locomotion	move_to	$(x,y,\psi)$	Moves the robot base to an absolute world-frame pose.
	turn	$(\psi,T)$	Rotates the robot toward an absolute world-frame yaw $\psi$ over duration $T$ .
	move_backward	$(x,y)$	Moves the base toward an absolute world-frame position while preserving its final orientation.
	approach_object	$(\textit{target},d_{\mathrm{stand}},\Delta\psi)$	Moves the base to a target-facing pose with the specified standoff distance and optional yaw offset.
Manipulation	move_hands	$(p_{L},p_{R},T_{\mathrm{hold}})$	Moves both hands to robot-base-frame target positions $p_{L},p_{R}\in\mathbb{R}^{3}$ and maintains them for $T_{\mathrm{hold}}$ seconds, without moving the base.
	grasp_object	$(\textit{target},p_{L},p_{R},d_{\mathrm{lift}})$	Grasps the target using specified left- and right-hand contact positions, then lifts it by $d_{\mathrm{lift}}$ .

TABLE I: Skill Library Available to the Planner and Recovery Agents

Panel of Figure 4 — Fig. 3: Representative failure and success outcomes from the five Monitor Agent tasks. The top row shows failure conditions, and the bottom row shows matched successful executions: (a) failed versus successful grasp; (b) dropped object versus stable hold; (c) collision versus collision-free object approach; (d) grasping the blue distractor box versus the requested red box; and (e) placing the red box on the yellow distractor pad versus the blue target pad.

Benchmark	Failures Detected	Successful Runs Accepted	Overall Accuracy
Failed grasp	8/10	10/10	90%
Dropped object	10/10	10/10	100%
Collision during approach	10/10	7/10	85%
Wrong-object grasp	10/10	10/10	100%
Wrong placement	10/10	9/10	95%
Overall	48/50	46/50	94%

TABLE II: Monitor performance across five benchmarks, each containing 10 failed and 10 successful skill executions.

Humanoid Loco-Manipulation

Humanoid loco-manipulation studies how robots coordinate walking, balance, and whole-body interaction to move and handle objects.

See all 93 papers

Cybo-Waiter: A Physical Agentic Framework for Humanoid Whole-Body Locomotion-Manipulation
Mar 11, 2026Peng Ren, Haoyang Ge, Chuan Qi +5Humanoid Loco-Manipulation Long-Horizon Robotic Manipulation
Beihang University, Beijing, China · Zhongguancun Academy, Beijing, China · Tianjin University, Tianjin, China +3
LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation
Aug 7, 2026Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1Humanoid Loco-Manipulation Latent Action Learning
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy
KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation
Sep 16, 2026Sitong Chen, Fatemeh Zargarbashi, Jin Cheng +2Robotic Control RL for Robotics
ETH Zürich, Switzerland

Mar 11, 2026J/K move · Enter open · S save

Cybo-Waiter: A Physical Agentic Framework for Humanoid Whole-Body Locomotion-Manipulation

Peng Ren, Haoyang Ge, Chuan Qi, Cong Huang+4

Beihang University, Beijing, China · Zhongguancun Academy, Beijing, China · Tianjin University, Tianjin, China+3

Humanoid Loco-Manipulation Long-Horizon Robotic Manipulation Whole-Body Robotic Manipulation Humanoid Robot Locomotion VLMs for Robotics

Open paper AskPDF arXiv

Robots are increasingly expected to execute open ended natural language requests in human environments, which demands reliable long horizon execution under partial observability. This is especially challenging for humanoids because locomotion and manipulation are tightly coupled through stance, reachability, and balance. We present a humanoid agent framework that turns VLM plans into verifiable task programs and closes the loop with multi object 3D geometric supervision. A VLM planner compiles each instruction into a typed JSON sequence of subtasks with explicit predicate based preconditions and success conditions. Using SAM3 and RGB-D, we ground all task relevant entities in 3D, estimate object centroids and extents, and evaluate predicates over stable frames to obtain condition level diagnostics. The supervisor uses these diagnostics to verify subtask completion and to provide condition-level feedback for progression and replanning. We execute each subtask by coordinating humanoid locomotion and whole-body manipulation, selecting feasible motion primitives under reachability and balance constraints. Experiments on tabletop manipulation and long horizon humanoid loco manipulation tasks show improved robustness from multi object grounding, temporal stability, and recovery driven replanning.

Abstract

Humanoid Loco-Manipulation

Explore similar work

Cybo-Waiter: A Physical Agentic Framework for Humanoid Whole-Body Locomotion-Manipulation

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation

KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation