UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs
Authors: Haichao Liu, Yuanjiang Xue, Yuheng Zhou, Li Yang, Haoyuan Deng, Yinan Liang, Lihua Xie, Ziwei Wang
Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Automation, Tsinghua University, China
Achieving general-purpose robotic manipulation requires robots to bridge high-level semantic intent with low-level physical interaction in unstructured environments. However, zero-shot generalization remains challenging: end-to-end Vision-Language-Action (VLA) models often lack the precision required for long-horizon tasks, while traditional hierarchical planners suffer from semantic rigidity under open-world variations. We present UniManip, a unified framework centered on an Agentic Operational Graph (AOG), an explicit cognitive and operational backbone for task orchestration, state tracking, execution verification, and closed-loop failure recovery. High-level semantic intents are grounded into actionable primitives via a physically grounded scene graph that provides geometric affordances, contact-aware interaction parameters, and collision-free trajectory synthesis. By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, UniManip continuously aligns abstract planning with geometric constraints, enabling robust execution. Unlike static pipelines, UniManip operates as a dynamic agentic loop: it instantiates object-centric scene graphs from unstructured perception, parameterizes these representations into collision-free trajectories via a safety-aware local planner, and exploits structured memory to autonomously diagnose and recover from execution failures. UniManip achieves state-of-the-art performance on LIBERO-Pro, while extensive real-world experiments demonstrate robust zero-shot generalization across diverse tasks, embodiments, and dexterous hands, outperforming leading VLA models and hierarchical baselines by 22.5% and 25.0% in task success rate, respectively. Open-source project page: https://henryhcliu.github.io/unimanip.
Figures & tables
Fig. 2: Overview of the UniManip framework. The system integrates high-level task planning with low-level motion execution through an Agentic Operational Graph (AOG), illustrated at the agent level. The VLM interprets human commands to generate an operational graph, which guides the robot’s actions. A reflective recovery mechanism allows the system to diagnose and adapt to execution failures.
Fig. 3: Bidirectional coupling between the Agentic Logic Graph (ALG) and Semantic-Operational Scene Graph (SOSG). ALG operation nodes query SOSG geometry, affordances, and verified preconditions to ground tool parameters; execution verification writes outcomes and updated states back to SOSG, enabling local repair or global replanning.
Fig. 4: Dual-memory mechanism. Episodic Memory stores task history, object states, action outcomes, and failure diagnostics for recovery. Procedural Memory stores executable skill primitives, affordance graphs, tool-selection logic, and recovery branches. Procedural Memory reads the current episodic state to select the next operation, while verification updates Episodic Memory after every execution.
Fig. 5: Demonstration of the spatial operations of the robot, with an instance of opening a drawer. The task is decomposed into several tool invocations, and each tool has its specific spatial operational formats for the movement of the robotic manipulator.
Fig. 6: Visualization of the conservative volumetric occupancy grid Mfinal generated from a single-view RGB-D observation. The gravity-aligned completion over-approximates unknown space, improving safety under occlusion.
Fig. 7: The corresponding scene and a sliced ESDF illustrating clearance values from obstacles. The ESDF is derived from the conservative occupancy grid, which accounts for occlusions and ensures safe trajectory planning.
Fig. 8: The closed-loop failure reflection and recovery mechanism in UniManip. The system continuously monitors execution states, detects anomalies, and invokes high-level cognitive reasoning to diagnose and recover from failures, either through local corrections or global replanning.
Fig. 9: Physical experimental setups across diverse robotic embodiments. The suite consists of the base Galaxea A1 arm with a wrist-mounted RGB-D camera, an XSTD mobile manipulator carrying a RealMan RM65 arm, and a UR7e arm interfaced with both a multi-fingered dexterous hand and a two-finger parallel gripper.
Task Format
Task
π0 [ 7 ]
NORA [ 46 ]
NORA-1.5 [ 8 ]
UniManip (Ours)
PSR ↑
Dist. ↓
SR ↑
PSR ↑
Dist. ↓
SR ↑
PSR ↑
Dist. ↓
SR ↑
PSR ↑
Dist. ↓
SR ↑
Put S in S
Put carrot in yellow plate
1.00
–
1.00
1.00
–
1.00
1.00
–
1.00
1.00
–
1.00
put blue cube on green cube
0.90
–
0.80
0.90
–
0.90
1.00
–
0.90
1.00
–
0.90
put eggplant in blue bowl
1.00
–
0.90
0.90
–
0.90
0.90
–
0.90
1.00
–
1.00
Put mango in basket
0.90
–
0.90
0.90
–
0.90
0.90
–
0.90
1.00
–
0.90
Put U in U
Put eggplant in pink bowl
0.90
–
0.80
0.90
–
0.90
1.00
–
1.00
1.00
–
1.00
TABLE I: Comparison with VLA Methods on Diverse Manipulation Tasks
Task Format
Objects of Interest
Distractor(s)
Target
Container
Put U in S
strawberry
green plate
apple
grape
green plate
eggplant
orange
green plate
banana
mango
yellow plate
apple
orange
yellow plate
apple and grape
TABLE II: Detailed Distractor Configurations for Unseen Object Manipulation Scenarios
Fig. 10: The comparison scenarios with VLA models. The distractors (unseen objects during model training) are marked with red curves.
Task
ReKep [ 10 ]
MOKA [ 29 ]
VoxPoser [ 11 ]
RoboEXP [ 33 ]
Ours w/o Graph
Ours
PW-OB
0.60
0.60
0.50
0.40
0.20
0.80
PW-AP
0.60
0.60
0.70
0.60
0.70
0.90
PW-TB
0.20
0.80
0.70
0.40
0.00
0.90
PW-ST
0.40
0.20
0.60
0.00
0.20
0.60
PW-PP
0.40
0.60
0.40
0.50
0.40
0.70
PW-CR
0.80
0.60
0.60
0.50
0.40
0.90
TABLE III: Comparison with Hierarchical and Graph-Based Methods on Cluttered Desktop Tasks. Each entry is the success rate as a decimal over 10 randomized trials.
Objects of Interest
Distractor(s)
Target
Container
orange
blue bowl
strawberry, apple, tomato, green plate, azure plate
apple
beige plate
orange, strawberry, tomato, blue bowl, green plate
tomato
basket
carambola, orange, blue plate, blue bowl
strawberry
basket
carambola, orange, azure plate, blue plate, blue bowl
pepper
blue plate
carrot, azure plate, blue bowl, green plate
TABLE IV: Detailed Distractor Configurations for Cluttered Object Manipulation Scenarios
Object
Goal
Spatial
Long
Method
Pos.
Task
Pos.
Task
Pos.
Task
Pos.
Task
MolmoAct [ 48 ]
0.06
0.00
0.00
0.00
0.00
0.00
0.00
0.06
NORA [ 46 ]
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
X-VLA [ 49 ]
0.02
0.08
0.01
0.09
0.00
0.00
0.00
0.10
OpenVLA [ 6 ]
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
π0 [ 7 ]
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
TABLE V: LIBERO-Pro comparison. Success rates are reported as decimals for position (Pos.) and task (Task) perturbations in the Object, Goal, Spatial, and Long suites. These metrics respectively test geometric pose generalization and task-level reasoning.
Fig. 11: Qualitative case studies of UniManip’s AOG for closed-loop failure recovery and target verification. Top: UniManip detects the object slipping during lifting, autonomously triggers agentic replanning, and successfully executes a regrasp. Bottom: When an incorrect distractor is initially engaged, the agent verifies the object attributes against the task instruction, rectifies the target selection, and completes the placement task.
Fig. 12: Demonstration of UniManip executing various manipulation tasks in a cluttered desktop environment. The robot successfully performs tasks such as sorting trash, inserting objects, pouring water, preparing breakfast, and opening/closing drawers, showcasing its versatility and robustness.
Feature
V1
V2
V3
Parameter-relaxed Inverse Kinematics (RIK)
✓
✓
✓
Collision-Free Planning (CFP)
✗
✓
✓
Failure Detection & Recovery (FDR)
✗
✗
✓
TABLE VI: Ablation Settings. We evaluate three configurations (V1, V2, V3) to analyze the contribution of each module. (✓: Included, ✗: Excluded)
Category
Task
Objects
Variants of UniManip
V1
V2
V3
Rigid obj. manipulation
Sort trash to different container
Plastic bottle, batteries
0.50
0.40
0.90
Insert bricks
Green ring, cube, base with poles
0.40
0.45
0.70
Insert rod-shaped object
Eggplant, banana
0.50
0.40
0.80
Pour water
Bottle, bowl
0.20
0.10
0.60
Deformable obj. manipulation
Prepare breakfast
Bread, apple, banana, plate, bowl
0.60
0.40
0.90
TABLE VII: Ablation Results. Success rates are reported as decimals across different manipulation categories. The configurations V1, V2, and V3 are defined in Table VI .
Embodiment
Task
Success
Total time (s)
VLM time (s)
UR7e + RF-85
Pour water
0.65
41.22 ± 14.14
5.49 ± 1.15
UR7e + RF-85
Prepare meal
0.80
53.52 ± 17.98
7.87 ± 2.06
UR7e + Inspire
Block to drawer
0.53
109.48 ± 25.26
6.90 ± 0.61
TABLE VIII: Cross-embodiment hardware evaluation. Success rates are decimals; times are mean ± sample standard deviation over all recorded trials.
Fig. 13: Cross-embodiment evaluation on a UR7e arm with a gripper and an Inspire multi-fingered hand, respectively. The tasks need tilting, dexterous grasping, pulling, pushing, and precise placement. The same AOG is used across embodiments and tasks.
Fig. 14: Mobile-manipulation demonstration with a Realman RM65 arm on a mobile base. The AOG instantiates navigation and manipulation operations, while SOSG state updates close the loop across docking points in an office environment.
Target Object
Grasp SR
Place SR
Task SR
Exec. Time (s)
Meal Preparation
Banana
0.80
1.00
0.80
35.05±6.25
Carrot
0.80
1.00
0.80
24.91±2.34
Workspace Cleanup
Grape
0.90
0.89
0.80
32.53±5.81
Bottle
0.90
0.83
0.75
24.99±5.14
TABLE IX: Fine-grained mobile-manipulation results on the Realman RM65 platform. Grasp SR and Task SR are calculated over 20 trials per target object; Place SR is conditioned on successful grasps. Execution times are reported as mean ± standard deviation.
Fig. 15: Breakdown of failure modes observed across all evaluation trials. Physical interaction failures and kinematic constraints are the primary bottlenecks, while semantic grounding remains relatively robust.