cs.ROAug 29, 2026

Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation

Authors: Wenhao Hong, Lan Wei, Dandan Zhang

Organizations: Imperial College London, London, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A language-guided robot operating in a real kitchen must do more than produce a plan that appears correct. It must also execute that plan safely in cluttered environments under imperfect perception. Large language models (LLM) can decompose instructions into action sequences, yet a language-action gap remains: a plan may appear valid linguistically while being physically infeasible under kinematic and collision constraints. We bridge this gap by formalizing the reasoning-execution boundary as a typed contract. From RGB-D observations, the system grounds perceived objects in an explicit, collision-aware scene model and constrains language-level decisions through schema-validated tool calls defined by the Model Context Protocol (MCP), rejecting malformed commands before they reach the robot. Each validated call is deterministically grounded in a MoveIt Task Constructor pipeline, where candidate motions are evaluated against the reconstructed planning scene in a verify-then-act step. Only trajectories that pass both kinematic and collision checks are sent to the robot. On a physical UFactory 850, the method achieves up to 80% success across ten trials per task on pouring tasks involving liquids, granular media, and discrete solids. It achieves 90% success on a grasp-and-place task using the same planning, protocol, and verification stack. Although a scripted policy slightly outperforms our method on the easiest task, its success rate falls to 10% on the hardest, compared with 60% for our method.

Figures & tables

Figure 1: Overview of the proposed framework. The Semantic Layer interprets the user instruction and produces a symbolic task state without generating poses or trajectories. The Protocol Layer validates MCP tool calls against a typed schema and binds valid calls to MTC templates. The Execution Layer follows a verify-then-act protocol, executing only motions that satisfy collision, kinematic, and task-specific constraints. The perception pipeline (bottom) grounds RGB-D observations into an explicit 3D MoveIt planning scene for collision-aware verification and execution.
Figure 2: Control flow through the three layers, complementing the contract view of Fig. 1 . A semantic resolver routes each instruction either to a constrained ReAct loop that only queries information, or to a deterministic task graph. The protocol layer decomposes the resulting plan into schema-validated tool calls. The execution layer monitors execution and re-plans transient failures, committing to hardware only after verification. The bottom row shows the canonical pick, move, pour, and place sequence executing on the real robot.
MCP tool interface (atomic)Underlying MTC stage pipeline (composite)
setup_scenePerception injection →\rightarrow collision-object spawning
pick(object)Grasp generation →\rightarrow IK filtering →\rightarrow approach →\rightarrow close →\rightarrow lift
move_to_pourCartesian planning →\rightarrow tilt-constraint check →\rightarrow velocity scaling →\rightarrow execute
place(target)Place-pose generation →\rightarrow approach →\rightarrow open →\rightarrow retreat
Table 1: Mapping from MCP tools to MTC task graphs. Each atomic tool call at the planner level is bound to a composite pipeline of MTC stages at the executor level.
Figure 3: (a) Real-world experimental setup using a UFactory 850 robot equipped with a two-finger gripper and an Intel RealSense D435i for perception. A separate camera records the experiments for evaluation. (b) Modular perception tools process RGB-D observations, transform detected objects into the robot base frame using calibrated extrinsics, and insert them into the planning scene through the setup_scene tool. (c) Detected objects and their estimated base-frame poses are shown on the colorized depth image (left), with the corresponding collision geometry in the planning scene (right).
Figure 4: World-frame object pose grounding errors across all scenarios. Both distributions exhibit non-negligible tails, motivating protocol-level verification prior to execution.
Figure 5: Real-robot task suite. Left: two dimensions of increasing difficulty, task complexity (single source, multiple sources, different materials, and integrated task) and material uncertainty (liquid, granular, discrete, and mixed). Right: representative executions of the four evaluated scenarios, T1-T4.
TaskOursFSM/BTLLM w/o task graph
T1: Single Pour80%90%60%
T2: Multi-Source Pour70%40%30%
T3: Multi-Material Pour80%60%60%
T4: Complex Scene60%10%40%
Mean72.5%50.0%47.5%
Table 2: Real-robot task performance. Each entry reports end-to-end success over 10 independent physical trials, with each trial corresponding to 10 percentage points. Per-task differences should therefore be interpreted cautiously. FSM/BT denotes a hand-crafted scripted policy, while LLM w/o task graph uses the same tool set without canonical action sequencing. Best result in each row is shown in bold.
Figure 6: Generalization to a fruit grasp-and-place task. Only the skill-specific tool schema and MTC template are changed. Perception grounds the fruit and bowl in the planning scene, while validated pick-and-place stages transfer the fruit to the bowl. The planner, protocol layer, and verification stack remain unchanged.
TaskOursw/o safe-posew/o retry
T1: Single Pour80%50%20%
T2: Multi-Source Pour70%20%40%
T3: Multi-Material Pour80%70%10%
T4: Complex Scene60%20%30%
Mean72.5%40.0%25.0%
Table 3: Ablations of our own system, each over 10 independent physical trials. Removing the safe-pose reset or the retry mechanism leaves schema validation and verify-then-act intact, so these isolate the protocol-level recovery mechanisms.

Explore similar work

CardsList

  1. Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation

    Oct 8, 2026Lonnie Tankersley, Noorbakhsh Amiri Golilarz, Soroush KorivandRobotic ManipulationRobot Task Planning

    dept. of Aerospace Engineering, Mississippi State University, Starkville, MS · dept. of Computer Science, The University of Alabama, Tuscaloosa, AL · dept. of Mechanical Engineering, Mississippi State University, Starkville, MS

  2. CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning

    Jun 5, 2024Xinrui Lin, Yangfan Wu, Huanyu Yang +3Large Language Model-Based Robot PlanningRobot Task Planning

    University of Science and Technology of China, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China

  3. Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning

    May 8, 2026Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2Large Language Model-Based Robot PlanningLLM Prompting

    IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1