A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution

Guangming Wang; Pengfei Ye; Qizhen Ying; Yixiong Jing; Yuxiang Ma; Haonan Chen; Haibing Wu; Olaf Wysocki

cs.ROSep 5, 2026

A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution

Authors: Guangming Wang, Pengfei Ye, Qizhen Ying, Yixiong Jing, Yuxiang Ma, Haonan Chen, Haibing Wu, Olaf Wysocki,

Organizations: Department of Engineering, University of Cambridge, Cambridge, U.K. · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Mechanical and Aerospace Engineering, Hong Kong University of Science and Technology, Hong Kong, China · Department of Engineering Science, University of Oxford, Oxford, U.K. ·

Abstract

Robots that follow open-ended language instructions need to connect semantic intent to visual scene understanding, geometric feasibility, object states, and physical interaction conditions. End-to-end Vision-Language-Action policies have improved cross-task generalization, but they typically map visual and language inputs directly to robot actions, leaving limited explicit structure for long-horizon decomposition, physical verification, and recovery. We present \method, a zero-shot hierarchical framework functionally inspired by the division of roles in the human brain, comprising visual perception and state inference, language grounding and action-sequence generation from a shared atomic action library, cost-based plan selection, and real-robot execution and verification. The framework grounds commands in explicit object states, composes reusable atomic actions into task-conditioned sequences, ranks alternative sequences by execution cost, and verifies intermediate physical outcomes from refreshed observations. In the evaluation, \method{} completes 10/10 clean board trials, 10/10 pick-and-place trials, and 4/5 pyramid stacking trials for both the flat and irregular initial-layout conditions; the corresponding mean task progress is $99.03\%$ , $100.00\%$ , and $96.67\%$ respectively. Across all evaluated conditions, \method{} achieves higher success rates than ReKep, Dream2Flow, and $π_{0.5}$ benchmarks, demonstrating the effectiveness of combining explicit object-state reasoning, compositional atomic actions, cost-based plan selection, and closed-loop execution verification.

Figures & tables

Fig. 1: Overview of ActionReasoning 2.0. (A) A natural-language command specifies the robot task. (B) A brain-inspired hierarchy organizes perception and state inference, language grounding and atomic-action composition, and cost-based plan selection through functional analogies to the visual cortex, cerebellum, and prefrontal cortex. (C) The resulting task plans are executed on real robots across construction and surface-cleaning tasks, with updated observations supporting verification until task completion.

Fig. 2: Brick-construction data flow. RGB-D and language inputs feed object-state, action-generation, and decision roles. Unique assignments are dispatched directly, surplus alternatives are optimized, and placement ends with a checked Release .

Task	Grounded objective	Action-sequence composition	Feasibility and verification conditions
Pick-and-Place	Transfer the commanded object to the grounded target region or pose	Move (above object) $\rightarrow$ Descend $\rightarrow$ Close $\rightarrow$ Lift $\rightarrow$ Move (target pose) $\rightarrow$ Descend $\rightarrow$ Release	Reachability, collision, grasp success, and final pose or containment
Pyramid Stacking	Place each object at its grounded target while satisfying the required support and relative-pose relations	Shared pick-and-place sequence with state-conditioned Rotate and additional Move actions for reorientation and separation	Reachability, collision/contact, alignment, support relation, settling, and final placement
Clean Board	Remove the grounded residual region while maintaining the required interaction with the board	Move (approach) $\rightarrow$ Descend $\rightarrow$ repeated contact-constrained Move $\rightarrow$ Lift	Reachability, collision, contact condition, board boundary, residual mask, and coverage

TABLE I: Task-dependent action-sequence composition, feasibility checks, and execution verification using the shared atomic action library.

Task	Task progress $\mathrm{TP}_{r}$	Success criterion
Pick-and-Place	Verified blocks transferred to the bowl / required blocks	$\mathrm{TP}_{r}=100\%$
Pyramid Stacking	Verified bricks correctly placed / target bricks	$\mathrm{TP}_{r}=100\%$ and stable final structure
Clean Board	$100(1-\|D_{T}\|/\|D_{0}\|)$	$\mathrm{TP}_{r}>80\%$

TABLE II: Task-progress definitions and terminal success criteria.

Condition	Method	SR (%; $n_{s}/N$ )	Time (s)	TP (%)
Clean Board	ReKep	0; 0/10	–	15.22 $\pm$ 19.35
	Dream2Flow	10; 1/10	93.0	52.93 $\pm$ 26.50
	$\pi_{0.5}$	30; 3/10	124.0 $\pm$ 9.67	42.46 $\pm$ 42.27
	ActionReasoning 2.0 (ours)	100; 10/10	171.3 $\pm$ 28.65	99.03 $\pm$ 1.67
Pick-and-Place	ReKep	10; 1/10	279.0	53.33 $\pm$ 28.11
	Dream2Flow	0; 0/10	–	53.33 $\pm$ 17.21

TABLE III: Comparison with ReKep, Dream2Flow, and

\pi_{0.5}

across the three task families. SR reports success rate and successful trials; completion time is computed over successful trials only; TP reports mean

\pm

sample standard deviation over all trials.

Fig. 3: Selected Pick-and-Place rollouts (rows: ActionReasoning 2.0, $\pi_{0.5}$ , ReKep, and Dream2Flow). Frames progress from left to right. Purple highlights an externally introduced human disturbance, yellow a step-level execution error, green task completion, and red an incomplete terminal outcome.

Fig. 4: Selected Clean Board rollouts in the same row order as Fig. 3 . Yellow highlights the onset of ineffective wiping caused by a stall or missed region, green task completion, and red incomplete cleaning. Quantitative success is defined by $\mathrm{TP}>80\%$ .

Fig. 5: Selected Pyramid Stacking rollouts for nominal and recovery ActionReasoning 2.0 cases, $\pi_{0.5}$ , and ReKep. Yellow highlights the onset of a failed grasp, green task completion or repaired case, and red incomplete construction. The repaired case shows ActionReasoning 2.0 reorienting an irregularly placed brick to a flat state before continuing construction.

Condition	Variant	SR [-1pt] $n_{s}/N$ (%)	TP [-1pt] (%)
Clean Board	Full ActionReasoning 2.0	10/10 (100)	99.03 $\pm$ 1.67
	Single agent	8/10 (80)	91.55 $\pm$ 16.38
	Single LLM	0/10 (0)	0.00 $\pm$ 0.00
Pick-and-Place	Full ActionReasoning 2.0	10/10 (100)	100.00 $\pm$ 0.00
	Single agent	7/10 (70)	86.67 $\pm$ 23.31
	Single LLM	8/10 (80)	93.33 $\pm$ 14.05

TABLE IV: Component ablations. TP reports mean

\pm

sample standard deviation.

Robotics

Robotics studies the design and control of machines that perceive, move, and act in physical environments.

See all 735 papers

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation
Jun 22, 2026Ali Alabbas, Dipshikha Das, Camillo Murgia +3Zero-Shot Robotic Manipulation Language-Conditioned Robot Manipulation
Department of Electronic, Software and Advanced Manufacturing Engineering, Atlantic Technological University, Galway, Ireland · Atlantic Technological University, Galway, Ireland
VersualRL: Closed-Loop Verbal Reinforcement Learning with Visual Execution Feedback for Task-Level Robot Planning
Mar 23, 2026Dmitrii Plotnikov, Iaroslav Kolomiets, Dmitrii Maliukov +9Robot Policy Learning Large Language Model-Based Robot Planning
The authors are with the Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology.
MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation
Sep 29, 2026Bingxuan Li, Siqi Song, Yizhuo Wu +3Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation
University of Illinois Urbana-Champaign

Jun 22, 2026J/K move · Enter open · S save

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation

Ali Alabbas, Dipshikha Das, Camillo Murgia, Sainul Ansary+2

Department of Electronic, Software and Advanced Manufacturing Engineering, Atlantic Technological University, Galway, Ireland · Atlantic Technological University, Galway, Ireland

Zero-Shot Robotic Manipulation Language-Conditioned Robot Manipulation Large Language Model-Based Robot Planning Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.

Abstract

Robotics

Explore similar work

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation

VersualRL: Closed-Loop Verbal Reinforcement Learning with Visual Execution Feedback for Task-Level Robot Planning

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation