UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs

Haichao Liu; Yuanjiang Xue; Yuheng Zhou; Li Yang; Haoyuan Deng; Yinan Liang; Lihua Xie; Ziwei Wang

cs.ROFeb 13, 2026

UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs

Authors: Haichao Liu, Yuanjiang Xue, Yuheng Zhou, Li Yang, Haoyuan Deng, Yinan Liang, Lihua Xie, Ziwei Wang

Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Automation, Tsinghua University, China

Abstract

Achieving general-purpose robotic manipulation requires robots to bridge high-level semantic intent with low-level physical interaction in unstructured environments. However, zero-shot generalization remains challenging: end-to-end Vision-Language-Action (VLA) models often lack the precision required for long-horizon tasks, while traditional hierarchical planners suffer from semantic rigidity under open-world variations. We present UniManip, a unified framework centered on an Agentic Operational Graph (AOG), an explicit cognitive and operational backbone for task orchestration, state tracking, execution verification, and closed-loop failure recovery. High-level semantic intents are grounded into actionable primitives via a physically grounded scene graph that provides geometric affordances, contact-aware interaction parameters, and collision-free trajectory synthesis. By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, UniManip continuously aligns abstract planning with geometric constraints, enabling robust execution. Unlike static pipelines, UniManip operates as a dynamic agentic loop: it instantiates object-centric scene graphs from unstructured perception, parameterizes these representations into collision-free trajectories via a safety-aware local planner, and exploits structured memory to autonomously diagnose and recover from execution failures. UniManip achieves state-of-the-art performance on LIBERO-Pro, while extensive real-world experiments demonstrate robust zero-shot generalization across diverse tasks, embodiments, and dexterous hands, outperforming leading VLA models and hierarchical baselines by 22.5% and 25.0% in task success rate, respectively. Open-source project page: https://henryhcliu.github.io/unimanip.

Figures & tables

Fig. 2: Overview of the UniManip framework. The system integrates high-level task planning with low-level motion execution through an Agentic Operational Graph (AOG), illustrated at the agent level. The VLM interprets human commands to generate an operational graph, which guides the robot’s actions. A reflective recovery mechanism allows the system to diagnose and adapt to execution failures.

Fig. 3: Bidirectional coupling between the Agentic Logic Graph (ALG) and Semantic-Operational Scene Graph (SOSG). ALG operation nodes query SOSG geometry, affordances, and verified preconditions to ground tool parameters; execution verification writes outcomes and updated states back to SOSG, enabling local repair or global replanning.

Fig. 4: Dual-memory mechanism. Episodic Memory stores task history, object states, action outcomes, and failure diagnostics for recovery. Procedural Memory stores executable skill primitives, affordance graphs, tool-selection logic, and recovery branches. Procedural Memory reads the current episodic state to select the next operation, while verification updates Episodic Memory after every execution.

Fig. 5: Demonstration of the spatial operations of the robot, with an instance of opening a drawer. The task is decomposed into several tool invocations, and each tool has its specific spatial operational formats for the movement of the robotic manipulator.

Fig. 6: Visualization of the conservative volumetric occupancy grid

\mathcal{M}_{final}

generated from a single-view RGB-D observation. The gravity-aligned completion over-approximates unknown space, improving safety under occlusion.

Panel of Figure 6 — Fig. 7: The corresponding scene and a sliced ESDF illustrating clearance values from obstacles. The ESDF is derived from the conservative occupancy grid, which accounts for occlusions and ensures safe trajectory planning.

Fig. 8: The closed-loop failure reflection and recovery mechanism in UniManip. The system continuously monitors execution states, detects anomalies, and invokes high-level cognitive reasoning to diagnose and recover from failures, either through local corrections or global replanning.

Fig. 9: Physical experimental setups across diverse robotic embodiments. The suite consists of the base Galaxea A1 arm with a wrist-mounted RGB-D camera, an XSTD mobile manipulator carrying a RealMan RM65 arm, and a UR7e arm interfaced with both a multi-fingered dexterous hand and a two-finger parallel gripper.

Task Format	Task	$\pi_{0}$ [ 7 ]			NORA [ 46 ]			NORA-1.5 [ 8 ]			UniManip (Ours)
Task Format	Task	PSR $\uparrow$	Dist. $\downarrow$	SR $\uparrow$	PSR $\uparrow$	Dist. $\downarrow$	SR $\uparrow$	PSR $\uparrow$	Dist. $\downarrow$	SR $\uparrow$	PSR $\uparrow$	Dist. $\downarrow$	SR $\uparrow$
Put S in S	Put carrot in yellow plate	1.00	–	1.00	1.00	–	1.00	1.00	–	1.00	1.00	–	1.00
	put blue cube on green cube	0.90	–	0.80	0.90	–	0.90	1.00	–	0.90	1.00	–	0.90
	put eggplant in blue bowl	1.00	–	0.90	0.90	–	0.90	0.90	–	0.90	1.00	–	1.00
	Put mango in basket	0.90	–	0.90	0.90	–	0.90	0.90	–	0.90	1.00	–	0.90
Put U in U	Put eggplant in pink bowl	0.90	–	0.80	0.90	–	0.90	1.00	–	1.00	1.00	–	1.00

TABLE I: Comparison with VLA Methods on Diverse Manipulation Tasks

Task Format	Objects of Interest		Distractor(s)
Task Format	Target	Container	Distractor(s)
Put U in S	strawberry	green plate	apple
	grape	green plate	eggplant
	orange	green plate	banana
	mango	yellow plate	apple
	orange	yellow plate	apple and grape

TABLE II: Detailed Distractor Configurations for Unseen Object Manipulation Scenarios

Fig. 10: The comparison scenarios with VLA models. The distractors (unseen objects during model training) are marked with red curves.

Task	ReKep [ 10 ]	MOKA [ 29 ]	VoxPoser [ 11 ]	RoboEXP [ 33 ]	Ours w/o Graph	Ours
PW-OB	0.60	0.60	0.50	0.40	0.20	0.80
PW-AP	0.60	0.60	0.70	0.60	0.70	0.90
PW-TB	0.20	0.80	0.70	0.40	0.00	0.90
PW-ST	0.40	0.20	0.60	0.00	0.20	0.60
PW-PP	0.40	0.60	0.40	0.50	0.40	0.70
PW-CR	0.80	0.60	0.60	0.50	0.40	0.90

TABLE III: Comparison with Hierarchical and Graph-Based Methods on Cluttered Desktop Tasks. Each entry is the success rate as a decimal over 10 randomized trials.

Objects of Interest		Distractor(s)
Target	Container	Distractor(s)
orange	blue bowl	strawberry, apple, tomato, green plate, azure plate
apple	beige plate	orange, strawberry, tomato, blue bowl, green plate
tomato	basket	carambola, orange, blue plate, blue bowl
strawberry	basket	carambola, orange, azure plate, blue plate, blue bowl
pepper	blue plate	carrot, azure plate, blue bowl, green plate

TABLE IV: Detailed Distractor Configurations for Cluttered Object Manipulation Scenarios

	Object		Goal		Spatial		Long
Method	Pos.	Task	Pos.	Task	Pos.	Task	Pos.	Task
MolmoAct [ 48 ]	0.06	0.00	0.00	0.00	0.00	0.00	0.00	0.06
NORA [ 46 ]	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
X-VLA [ 49 ]	0.02	0.08	0.01	0.09	0.00	0.00	0.00	0.10
OpenVLA [ 6 ]	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
$\pi_{0}$ [ 7 ]	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

TABLE V: LIBERO-Pro comparison. Success rates are reported as decimals for position (Pos.) and task (Task) perturbations in the Object, Goal, Spatial, and Long suites. These metrics respectively test geometric pose generalization and task-level reasoning.

Fig. 11: Qualitative case studies of UniManip’s AOG for closed-loop failure recovery and target verification. Top: UniManip detects the object slipping during lifting, autonomously triggers agentic replanning, and successfully executes a regrasp. Bottom: When an incorrect distractor is initially engaged, the agent verifies the object attributes against the task instruction, rectifies the target selection, and completes the placement task.

Fig. 12: Demonstration of UniManip executing various manipulation tasks in a cluttered desktop environment. The robot successfully performs tasks such as sorting trash, inserting objects, pouring water, preparing breakfast, and opening/closing drawers, showcasing its versatility and robustness.

Feature	V1	V2	V3
Parameter-relaxed Inverse Kinematics (RIK)	✓	✓	✓
Collision-Free Planning (CFP)	✗	✓	✓
Failure Detection & Recovery (FDR)	✗	✗	✓

TABLE VI: Ablation Settings. We evaluate three configurations (V1, V2, V3) to analyze the contribution of each module. (✓: Included, ✗: Excluded)

Category	Task	Objects	Variants of UniManip
Category	Task	Objects	V1	V2	V3
Rigid obj. manipulation	Sort trash to different container	Plastic bottle, batteries	0.50	0.40	0.90
	Insert bricks	Green ring, cube, base with poles	0.40	0.45	0.70
	Insert rod-shaped object	Eggplant, banana	0.50	0.40	0.80
	Pour water	Bottle, bowl	0.20	0.10	0.60
Deformable obj. manipulation	Prepare breakfast	Bread, apple, banana, plate, bowl	0.60	0.40	0.90

TABLE VII: Ablation Results. Success rates are reported as decimals across different manipulation categories. The configurations V1, V2, and V3 are defined in Table VI .

Embodiment	Task	Success	Total time (s)	VLM time (s)
UR7e + RF-85	Pour water	0.65	41.22 $\pm$ 14.14	5.49 $\pm$ 1.15
UR7e + RF-85	Prepare meal	0.80	53.52 $\pm$ 17.98	7.87 $\pm$ 2.06
UR7e + Inspire	Block to drawer	0.53	109.48 $\pm$ 25.26	6.90 $\pm$ 0.61

TABLE VIII: Cross-embodiment hardware evaluation. Success rates are decimals; times are mean

\pm

sample standard deviation over all recorded trials.

Fig. 13: Cross-embodiment evaluation on a UR7e arm with a gripper and an Inspire multi-fingered hand, respectively. The tasks need tilting, dexterous grasping, pulling, pushing, and precise placement. The same AOG is used across embodiments and tasks.

Fig. 14: Mobile-manipulation demonstration with a Realman RM65 arm on a mobile base. The AOG instantiates navigation and manipulation operations, while SOSG state updates close the loop across docking points in an office environment.

Target Object	Grasp SR	Place SR	Task SR	Exec. Time (s)
Meal Preparation
Banana	0.80	1.00	0.80	$35.05\pm 6.25$
Carrot	0.80	1.00	0.80	$24.91\pm 2.34$
Workspace Cleanup
Grape	0.90	0.89	0.80	$32.53\pm 5.81$
Bottle	0.90	0.83	0.75	$24.99\pm 5.14$

TABLE IX: Fine-grained mobile-manipulation results on the Realman RM65 platform. Grasp SR and Task SR are calculated over 20 trials per target object; Place SR is conditioned on successful grasps. Execution times are reported as mean

\pm

standard deviation.

Fig. 15: Breakdown of failure modes observed across all evaluation trials. Physical interaction failures and kinematic constraints are the primary bottlenecks, while semantic grounding remains relatively robust.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

Mar 14, 2026

Building Explicit World Model for Zero-Shot Open-World Object Manipulation

Open-world object manipulation remains a fundamental challenge in robotics. While Vision-Language-Action (VLA) models have demonstrated promising results, they rely heavily on large-scale robot action demonstrations, which are costly to collect and can hinder out-of-distribution generalization. In this paper, we propose an explicit-world-model-based framework for open-world manipulation that achieves zero-shot generalization by constructing a physically grounded digital twin of the environment. The framework integrates open-set perception, digital-twin reconstruction, sampling and evaluation of interaction strategies. By constructing a digital twin of the environment, our approach efficiently explores and evaluates manipulation strategies in physic-enabled simulator and reliably deploys the chosen strategy to the real world. Experimentally, the proposed framework is able to perform multiple open-set manipulation tasks without any task-specific action demonstrations, proving strong zero-shot generalization on both the task and object levels. Project Page: https://bojack-bj.github.io/projects/thesis/

Xiaotong Li, Gang Chen, Javier Alonso-Mora

Department of Cognitive Robotics (CoR), Delft University of Technology, 2628CD Delft, The Netherlands.

Zero-Shot Robotic Manipulation Robotic Manipulation World Models for Robotics

Ask PDF

Jul 9, 2026

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R. Real-world demonstrations on dual-Franka robots further show target redirection, grasp recovery, and new task compositions with the same frozen VLA. Code is available at https://github.com/RLinf/RPent.

Yixian Zhang, Huanming Zhang, Feng Gao +14

Tsinghua University · Striding AI · Purdue University +4

Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation Generalization in Robotic Manipulation Vision-Language-Action Models Test-Time Adaptation of VLA Models Robustness of VLA Models+1

Ask PDF

Feb 18, 2026

MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation

Task planning for robotic manipulation with large language models (LLMs) is an emerging area. Prior approaches rely on specialized models, fine tuning, or prompt tuning, and often operate in an open loop manner without robust environmental feedback, making them fragile in dynamic settings. MALLVI presents a Multi Agent Large Language and Vision framework that enables closed-loop feedback driven robotic manipulation. Given a natural language instruction and an image of the environment, MALLVI generates executable atomic actions for a robot manipulator. After action execution, a Vision Language Model (VLM) evaluates environmental feedback and decides whether to repeat the process or proceed to the next step. Rather than using a single model, MALLVI coordinates specialized agents, Decomposer, Localizer, Thinker, and Reflector, to manage perception, localization, reasoning, and high level planning. An optional Descriptor agent provides visual memory of the initial state. The Reflector supports targeted error detection and recovery by reactivating only relevant agents, avoiding full replanning. Experiments in simulation and real-world settings show that iterative closed loop multi agent coordination improves generalization and increases success rates in zero shot manipulation tasks. Code available at https://github.com/iman1234ahmadi/MALLVI .

Mehrshad Taji, Arad Mahdinezhad Kashani, Iman Ahmadi +3

Department of Electrical Engineering Sharif University of Technology

Large Language Model-Based Robot Planning Robotic Manipulation Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation Multi-Agent Planning VLMs for Robotics

Ask PDF

Abstract

Robotic Manipulation

Explore similar work

Building Explicit World Model for Zero-Shot Open-World Object Manipulation

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation