A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution
Authors: Guangming Wang, Pengfei Ye, Qizhen Ying, Yixiong Jing, Yuxiang Ma, Haonan Chen, Haibing Wu, Olaf Wysocki, Molong Duan, Brian Sheil
Organizations: Department of Engineering, University of Cambridge, Cambridge, U.K. · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Mechanical and Aerospace Engineering, Hong Kong University of Science and Technology, Hong Kong, China · Department of Engineering Science, University of Oxford, Oxford, U.K. · Computer Science and Kempner Institute, Harvard University, Cambridge, MA, USA
Robots that follow open-ended language instructions need to connect semantic intent to visual scene understanding, geometric feasibility, object states, and physical interaction conditions. End-to-end Vision-Language-Action policies have improved cross-task generalization, but they typically map visual and language inputs directly to robot actions, leaving limited explicit structure for long-horizon decomposition, physical verification, and recovery. We present \method, a zero-shot hierarchical framework functionally inspired by the division of roles in the human brain, comprising visual perception and state inference, language grounding and action-sequence generation from a shared atomic action library, cost-based plan selection, and real-robot execution and verification. The framework grounds commands in explicit object states, composes reusable atomic actions into task-conditioned sequences, ranks alternative sequences by execution cost, and verifies intermediate physical outcomes from refreshed observations. In the evaluation, \method{} completes 10/10 clean board trials, 10/10 pick-and-place trials, and 4/5 pyramid stacking trials for both the flat and irregular initial-layout conditions; the corresponding mean task progress is 99.03%, 100.00%, and 96.67% respectively. Across all evaluated conditions, \method{} achieves higher success rates than ReKep, Dream2Flow, and π0.5 benchmarks, demonstrating the effectiveness of combining explicit object-state reasoning, compositional atomic actions, cost-based plan selection, and closed-loop execution verification.
Figures & tables
Fig. 1: Overview of ActionReasoning 2.0. (A) A natural-language command specifies the robot task. (B) A brain-inspired hierarchy organizes perception and state inference, language grounding and atomic-action composition, and cost-based plan selection through functional analogies to the visual cortex, cerebellum, and prefrontal cortex. (C) The resulting task plans are executed on real robots across construction and surface-cleaning tasks, with updated observations supporting verification until task completion.
Fig. 2: Brick-construction data flow. RGB-D and language inputs feed object-state, action-generation, and decision roles. Unique assignments are dispatched directly, surplus alternatives are optimized, and placement ends with a checked Release .
Task
Grounded objective
Action-sequence composition
Feasibility and verification conditions
Pick-and-Place
Transfer the commanded object to the grounded target region or pose
Reachability, collision, contact condition, board boundary, residual mask, and coverage
TABLE I: Task-dependent action-sequence composition, feasibility checks, and execution verification using the shared atomic action library.
Task
Task progress TPr
Success criterion
Pick-and-Place
Verified blocks transferred to the bowl / required blocks
TPr=100%
Pyramid Stacking
Verified bricks correctly placed / target bricks
TPr=100% and stable final structure
Clean Board
100(1−∣DT∣/∣D0∣)
TPr>80%
TABLE II: Task-progress definitions and terminal success criteria.
Condition
Method
SR (%; ns/N )
Time (s)
TP (%)
Clean Board
ReKep
0; 0/10
–
15.22 ± 19.35
Dream2Flow
10; 1/10
93.0
52.93 ± 26.50
π0.5
30; 3/10
124.0 ± 9.67
42.46 ± 42.27
ActionReasoning 2.0 (ours)
100; 10/10
171.3 ± 28.65
99.03 ± 1.67
Pick-and-Place
ReKep
10; 1/10
279.0
53.33 ± 28.11
Dream2Flow
0; 0/10
–
53.33 ± 17.21
TABLE III: Comparison with ReKep, Dream2Flow, and π0.5 across the three task families. SR reports success rate and successful trials; completion time is computed over successful trials only; TP reports mean ± sample standard deviation over all trials.
Fig. 3: Selected Pick-and-Place rollouts (rows: ActionReasoning 2.0, π0.5 , ReKep, and Dream2Flow). Frames progress from left to right. Purple highlights an externally introduced human disturbance, yellow a step-level execution error, green task completion, and red an incomplete terminal outcome.
Fig. 4: Selected Clean Board rollouts in the same row order as Fig. 3 . Yellow highlights the onset of ineffective wiping caused by a stall or missed region, green task completion, and red incomplete cleaning. Quantitative success is defined by TP>80% .
Fig. 5: Selected Pyramid Stacking rollouts for nominal and recovery ActionReasoning 2.0 cases, π0.5 , and ReKep. Yellow highlights the onset of a failed grasp, green task completion or repaired case, and red incomplete construction. The repaired case shows ActionReasoning 2.0 reorienting an irregularly placed brick to a flat state before continuing construction.
Condition
Variant
SR [-1pt] ns/N (%)
TP [-1pt] (%)
Clean Board
Full ActionReasoning 2.0
10/10 (100)
99.03 ± 1.67
Single agent
8/10 (80)
91.55 ± 16.38
Single LLM
0/10 (0)
0.00 ± 0.00
Pick-and-Place
Full ActionReasoning 2.0
10/10 (100)
100.00 ± 0.00
Single agent
7/10 (70)
86.67 ± 23.31
Single LLM
8/10 (80)
93.33 ± 14.05
TABLE IV: Component ablations. TP reports mean ± sample standard deviation.
This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.
Ali Alabbas, Dipshikha Das, Camillo Murgia +3
Department of Electronic, Software and Advanced Manufacturing Engineering, Atlantic Technological University, Galway, Ireland · Atlantic Technological University, Galway, Ireland
We introduce VersualRL, a closed-loop framework for task-level robot planning that uses visual execution feedback to iteratively refine executable Behavior Trees through structured natural-language critique. VersualRL combines a vision-language model critic, which analyzes visual observations and Behavior Tree execution traces, with a large language model actor that performs discrete and interpretable policy updates. During physical deployment, both foundation models remain frozen, while the executed Behavior Tree is updated at the symbolic level without online gradient-based policy optimization. This enables transparent reasoning, explicit causal feedback, and human-interpretable policy evolution. We validate VersualRL on a real mobile robot performing a multi-stage navigation and manipulation task under execution uncertainty. Experimental results show that the framework supports explainable policy improvements, closed-loop adaptation to execution failures, and successful deployment on physical robotic systems.
Vision-language-action (VLA) models have advanced robotic manipulation, but their zero-shot generalization in new tasks and environments remains limited, and their reliance on specialized training keeps them from benefiting directly from rapidly advancing general-purpose vision-language models (VLMs). In parallel, recent agentic robotic systems leverage VLMs for high-level reasoning or coding agents for robot control, but often depend on extensive external models and tools, introducing additional complexity and cost. This motivates us to ask: Can a general-purpose VLM itself operate a robot more like the human teleoperator by reasoning directly from observations, issuing actions, and continuously adapting to execution feedback, without relying on external models such as learned action experts, coding agents or grounding tools like SAM3? In this work, we introduce MotorMind, a robot manipulation harness that connects VLM-proposed mid-level actions to deterministic robot control and feedback, with asynchronous monitoring and background memory updates. Without task-specific policy training, coding agents, or additional grounding tools such as SAM3, MotorMind achieves 66.7% success on the base LIBERO-PRO suites and 53.8% under perturbations, compared with at most 13.3% and 19.2%, respectively, for the prior zero-shot methods we evaluate. The same interface reaches 95% average success on a real xArm6 robot across direct manipulation and human-perturbation settings. Replacing the backbone with a stronger VLM further improves performance, while the remaining failures - primarily due to visual grounding, embodied reasoning, and action knowledge - decrease as VLM capability improves. These results show that a general-purpose VLM, when equipped with an appropriate mid-level action representation and asynchronous execution harness, can perform effective zero-shot robotic manipulation.