ActionReasoning: Robot Action Reasoning in 3D Space with LLM for Robotic Brick Stacking
Authors: Guangming Wang, Qizhen Ying, Yixiong Jing, Olaf Wysocki, Brian Sheil
Organizations: CV4DT, CSIC, Department of Engineering, University of Cambridge, Cambridge CB2 1PZ, U.K. · Department of Engineering, University of Oxford, Oxford, U.K.
Classical robotic systems typically rely on custom planners designed for constrained environments. While effective in restricted settings, these systems lack generalization capabilities, limiting the scalability of embodied AI and general-purpose robots. Recent data-driven Vision-Language-Action (VLA) approaches aim to learn policies from large-scale simulation and real-world data. However, the continuous action space of the physical world significantly exceeds the representational capacity of linguistic tokens, making it unclear if scaling data alone can yield general robotic intelligence. To address this gap, we propose ActionReasoning, an LLM-driven framework that performs explicit action reasoning to produce physics-consistent, prior-guided decisions for robotic manipulation. ActionReasoning leverages the physical priors and real-world knowledge already encoded in Large Language Models (LLMs) and structures them within a multi-agent architecture. We instantiate this framework on a tractable case study of brick stacking, where the environment states are assumed to be already accurately measured. The environmental states are then serialized and passed to a multi-agent LLM framework that generates physics-aware action plans. The experiments demonstrate that the proposed multi-agent LLM framework enables stable brick placement while shifting effort from low-level domain-specific coding to high-level tool invocation and prompting, highlighting its potential for broader generalization. This work introduces a promising approach to bridging perception and execution in robotic manipulation by integrating physical reasoning with LLMs.
Figures & tables
Fig. 1 : Three stages of the ActionReasoning pipeline. (1) Inputs & World Model: The world model of bricklaying is provided as input. (2) Multi-Agent Planning: Leveraging the world model input, an LLM orchestrator decomposes the task into specialized agents that generate actions and waypoints to plan the motion of a selected brick toward its target location. (3) Simulator & Robot Execution: The robot (a Kuka simulator in this case) executes the planned actions from multiple agents to control grasp and motion. Observations from changes in the 3D scene and robot arm state are used to update the world model, enabling continual re-planning as the task progresses.
Fig. 2 : Illustration of the six agents ( Ag1−Ag6 ) in the present ActionReasoning framework: (1) Pre-grasp positioning to guide the arm to approach the brick; (2) Opening and descent to position the gripper above the brick surface; (3) Grasp closure to secure the brick; (4) Safe lift to raise the brick from the ground; (5) Brick placement to stably move and accurately align the brick at the target location; and (6) Grasp release to land the brick. The corresponding execution of each agent is illustrated on the simulated KUKA robot arm at the bottom of this figure.
Fig. 3 : Detailed architecture of Agent 5 (Brick Placement) with six prompt-driven components. (1) Current environment state: provides the latest world model; (2) Memory information: explains task progress of previous agents and the current task status; (3) Role definition: specifies the agent’s function and responsibilities, including collision-avoidance tasks for stable brick placement. A comparison of the brick laying between specifying and not specifying is visualized aside; (4) Knowledge base: describes domain knowledge such as dynamics, gripper strategies, and safety constraints; (5) Thinking chain: outlines stepwise reasoning for placement and retreat; and (6) Output format: structured JSON commands for execution in the KUKA simulator.
Method
Pyramid-like stacking
Grid-like stacking
Average
Rot. Err ↓
Ctr. Off. ↓
IoU ↑
Rot. Err ↓
Ctr. Off. ↓
IoU ↑
Rot. Err ↓
Ctr. Off. ↓
IoU ↑
Classical Controller (baseline)
1.103
4.318
38.51
0.939
4.379
37.72
1.004
4.314
38.38
ActionReasoning (Ours)
0.583
0.561
89.03
0.822
0.712
87.02
0.703
0.637
88.03
TABLE I : Comparison with classical baseline on two stacking patterns. Report rotation error (°), center offset (cm), and 3D box IoU (%). For rotation error and center offset, the lower the better. For 3D box IoU, the higher the better.
Fig. 4 : Qualitative comparison across two stacking patterns. Each block shows the baseline (top row) and our method (bottom row). In each block of stacking pattern, columns show some stages of the placement cycle according to the timestamp from left to right. Our method achieves noticeably better brick alignment, as evidenced by the more neatly stacked bricks.
Fig. 5 : Ablation visualizations for single agent. Rows depict single-agent vs multi-agent. Columns show key stages of the placement cycle according to the timestamp from left to right. The figure shows that the single agent toppled the structure and failed to complete the whole wall.
Task planning for robotic manipulation with large language models (LLMs) is an emerging area. Prior approaches rely on specialized models, fine tuning, or prompt tuning, and often operate in an open loop manner without robust environmental feedback, making them fragile in dynamic settings. MALLVI presents a Multi Agent Large Language and Vision framework that enables closed-loop feedback driven robotic manipulation. Given a natural language instruction and an image of the environment, MALLVI generates executable atomic actions for a robot manipulator. After action execution, a Vision Language Model (VLM) evaluates environmental feedback and decides whether to repeat the process or proceed to the next step. Rather than using a single model, MALLVI coordinates specialized agents, Decomposer, Localizer, Thinker, and Reflector, to manage perception, localization, reasoning, and high level planning. An optional Descriptor agent provides visual memory of the initial state. The Reflector supports targeted error detection and recovery by reactivating only relevant agents, avoiding full replanning. Experiments in simulation and real-world settings show that iterative closed loop multi agent coordination improves generalization and increases success rates in zero shot manipulation tasks. Code available at https://github.com/iman1234ahmadi/MALLVI .
Mehrshad Taji, Arad Mahdinezhad Kashani, Iman Ahmadi +3
Department of Electrical Engineering Sharif University of Technology
The heterogeneity between high-level vision-language understanding and low-level action control remains a fundamental challenge in robotic manipulation. Although recent methods have advanced task-specific action alignment, they often struggle to generate robust and accurate actions for novel or semantically related tasks. To address this, we propose the Language-Grounded Decoupled Action Representation (LaDA) framework, which leverages natural language as a semantic bridge to connect perception and control. LaDA introduces a fine-grained intermediate layer of three interpretable action primitives--translation, rotation, and gripper control--providing explicit semantic structure for low-level actions. It further employs a semantic-guided soft-label contrastive learning objective to align similar action primitives across tasks, enhancing generalization and motion consistency. An adaptive weighting strategy, inspired by curriculum learning, dynamically balances contrastive and imitation objectives for stable and effective training. Extensive experiments on simulated benchmarks (LIBERO and MimicGen) and real-world demonstrations validate that LaDA achieves strong performance and generalizes effectively to unseen or related tasks.
Wuding Weng, Tongshu Wu, Liucheng Chen +5
School of Computer Science and Technology, Tongji University, China
We present a hierarchical language-driven framework for robotic task and motion planning to improve natural, intuitive human-robot interaction in service and assistance scenarios. The proposed system employs two large language model (LLM) modules: a high-level planning agent and a low-level spatial reasoning sub-module. The primary agent processes natural language commands and generates action sequences using a ReAct-style prompt, interacting with tools for object perception and manipulation (e.g., pick, place, release). For precise spatial placement, such as interpreting "place the mug next to the plate", a separate sub-prompting module handles 3D reasoning based on object geometry and scene layout. The system integrates YOLOX-GDRNet for object detection and pose estimation, along with a motion execution stub. We evaluated the system in 24 test scenarios, ranging from simple spatial commands to high-level instructions and infeasible requests. The system achieved an overall task success rate of 86%.
Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2
IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1