cs.ROAug 29, 2026

Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation

Authors: Wenhao Hong, Lan Wei, Dandan Zhang

Organizations: Imperial College London, London, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A language-guided robot operating in a real kitchen must do more than produce a plan that appears correct. It must also execute that plan safely in cluttered environments under imperfect perception. Large language models (LLM) can decompose instructions into action sequences, yet a language-action gap remains: a plan may appear valid linguistically while being physically infeasible under kinematic and collision constraints. We bridge this gap by formalizing the reasoning-execution boundary as a typed contract. From RGB-D observations, the system grounds perceived objects in an explicit, collision-aware scene model and constrains language-level decisions through schema-validated tool calls defined by the Model Context Protocol (MCP), rejecting malformed commands before they reach the robot. Each validated call is deterministically grounded in a MoveIt Task Constructor pipeline, where candidate motions are evaluated against the reconstructed planning scene in a verify-then-act step. Only trajectories that pass both kinematic and collision checks are sent to the robot. On a physical UFactory 850, the method achieves up to 80% success across ten trials per task on pouring tasks involving liquids, granular media, and discrete solids. It achieves 90% success on a grasp-and-place task using the same planning, protocol, and verification stack. Although a scripted policy slightly outperforms our method on the easiest task, its success rate falls to 10% on the hardest, compared with 60% for our method.

Figures & tables

Figure 1: Overview of the proposed framework. The Semantic Layer interprets the user instruction and produces a symbolic task state without generating poses or trajectories. The Protocol Layer validates MCP tool calls against a typed schema and binds valid calls to MTC templates. The Execution Layer follows a verify-then-act protocol, executing only motions that satisfy collision, kinematic, and task-specific constraints. The perception pipeline (bottom) grounds RGB-D observations into an explicit 3D MoveIt planning scene for collision-aware verification and execution.
Figure 2: Control flow through the three layers, complementing the contract view of Fig. 1 . A semantic resolver routes each instruction either to a constrained ReAct loop that only queries information, or to a deterministic task graph. The protocol layer decomposes the resulting plan into schema-validated tool calls. The execution layer monitors execution and re-plans transient failures, committing to hardware only after verification. The bottom row shows the canonical pick, move, pour, and place sequence executing on the real robot.
MCP tool interface (atomic)Underlying MTC stage pipeline (composite)
setup_scenePerception injection →\rightarrow collision-object spawning
pick(object)Grasp generation →\rightarrow IK filtering →\rightarrow approach →\rightarrow close →\rightarrow lift
move_to_pourCartesian planning →\rightarrow tilt-constraint check →\rightarrow velocity scaling →\rightarrow execute
place(target)Place-pose generation →\rightarrow approach →\rightarrow open →\rightarrow retreat
Table 1: Mapping from MCP tools to MTC task graphs. Each atomic tool call at the planner level is bound to a composite pipeline of MTC stages at the executor level.
Figure 3: (a) Real-world experimental setup using a UFactory 850 robot equipped with a two-finger gripper and an Intel RealSense D435i for perception. A separate camera records the experiments for evaluation. (b) Modular perception tools process RGB-D observations, transform detected objects into the robot base frame using calibrated extrinsics, and insert them into the planning scene through the setup_scene tool. (c) Detected objects and their estimated base-frame poses are shown on the colorized depth image (left), with the corresponding collision geometry in the planning scene (right).
Figure 4: World-frame object pose grounding errors across all scenarios. Both distributions exhibit non-negligible tails, motivating protocol-level verification prior to execution.
Figure 5: Real-robot task suite. Left: two dimensions of increasing difficulty, task complexity (single source, multiple sources, different materials, and integrated task) and material uncertainty (liquid, granular, discrete, and mixed). Right: representative executions of the four evaluated scenarios, T1-T4.
TaskOursFSM/BTLLM w/o task graph
T1: Single Pour80%90%60%
T2: Multi-Source Pour70%40%30%
T3: Multi-Material Pour80%60%60%
T4: Complex Scene60%10%40%
Mean72.5%50.0%47.5%
Table 2: Real-robot task performance. Each entry reports end-to-end success over 10 independent physical trials, with each trial corresponding to 10 percentage points. Per-task differences should therefore be interpreted cautiously. FSM/BT denotes a hand-crafted scripted policy, while LLM w/o task graph uses the same tool set without canonical action sequencing. Best result in each row is shown in bold.
Figure 6: Generalization to a fruit grasp-and-place task. Only the skill-specific tool schema and MTC template are changed. Perception grounds the fruit and bowl in the planning scene, while validated pick-and-place stages transfer the fruit to the bowl. The planner, protocol layer, and verification stack remain unchanged.
TaskOursw/o safe-posew/o retry
T1: Single Pour80%50%20%
T2: Multi-Source Pour70%20%40%
T3: Multi-Material Pour80%70%10%
T4: Complex Scene60%20%30%
Mean72.5%40.0%25.0%
Table 3: Ablations of our own system, each over 10 independent physical trials. Removing the safe-pose reset or the retry mechanism leaves schema validation and verify-then-act intact, so these isolate the protocol-level recovery mechanisms.

Explore similar work

CardsList
Oct 8, 2026

Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation

Large language models can translate natural-language instructions into plausible manipulation plans, but robotic execution requires physically grounded decisions that respect robot embodiment, workspace geometry, reachability, collision constraints, and grasp feasibility. This paper presents a hardware-aware large language model-vision-language model (LLM-VLM) manipulation framework for a simulated Franka Research 3 robot in NVIDIA Isaac Sim. The framework separates symbolic reasoning from metric execution: an API-hosted LLM planner receives a structured scene summary and generates only symbolic task information, including the object to manipulate, target relation, approach preference, and objects to avoid. A local Qwen3-VL 8B VLM verifier supports semantic confirmation of ambiguous objects and open-container surrogates. Both the LLM planner and the VLM verifier are prohibited from generating robot coordinates, grasp poses, keepout dimensions, gripper commands, trajectory waypoints, or success labels. Instead, these decisions are handled by deterministic grounding and execution modules that use the robot model, scene geometry, MoveIt 2 collision checking, inverse kinematics, grasp validation, release verification, and final visual assessment. Execution feedback is returned only as symbolic context for replanning. The framework is evaluated through simulated digital-twin trials involving pickup, occluded-source access, keepout reasoning, relation placement, and sequential manipulation tasks.

Lonnie Tankersley, Noorbakhsh Amiri Golilarz, Soroush Korivand

dept. of Aerospace Engineering, Mississippi State University, Starkville, MS · dept. of Computer Science, The University of Alabama, Tuscaloosa, AL · dept. of Mechanical Engineering, Mississippi State University, Starkville, MS

Robotic ManipulationRobot Task PlanningLarge Language Model-Based Robot PlanningZero-Shot Robotic ManipulationVLMs for Robotics
AskPDF
Jun 5, 2024

CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning

Large Language Models (LLMs) possess extensive foundational knowledge and moderate reasoning abilities, making them suitable for general task planning in open-world scenarios. However, it is challenging to ground a LLM-generated plan to be executable for the specified robot with certain restrictions. This paper introduces CLMASP, an approach that couples LLMs with Answer Set Programming (ASP) to overcome the limitations, where ASP is a non-monotonic logic programming formalism renowned for its capacity to represent and reason about a robot's action knowledge. CLMASP initiates with a LLM generating a basic skeleton plan, which is subsequently tailored to the specific scenario using a vector database. This plan is then refined by an ASP program with a robot's action knowledge, which integrates implementation details into the skeleton, grounding the LLM's abstract outputs in practical robot contexts. Our experiments conducted on the VirtualHome platform demonstrate CLMASP's efficacy. Compared to the baseline executable rate of under 2% with LLM approaches, CLMASP significantly improves this to over 90%.

Xinrui Lin, Yangfan Wu, Huanyu Yang +3

University of Science and Technology of China, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China

Large Language Model-Based Robot PlanningRobot Task Planning
AskPDF
May 8, 2026

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning

We present a hierarchical language-driven framework for robotic task and motion planning to improve natural, intuitive human-robot interaction in service and assistance scenarios. The proposed system employs two large language model (LLM) modules: a high-level planning agent and a low-level spatial reasoning sub-module. The primary agent processes natural language commands and generates action sequences using a ReAct-style prompt, interacting with tools for object perception and manipulation (e.g., pick, place, release). For precise spatial placement, such as interpreting "place the mug next to the plate", a separate sub-prompting module handles 3D reasoning based on object geometry and scene layout. The system integrates YOLOX-GDRNet for object detection and pose estimation, along with a motion execution stub. We evaluated the system in 24 test scenarios, ranging from simple spatial commands to high-level instructions and infeasible requests. The system achieved an overall task success rate of 86%.

Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2

IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1

Large Language Model-Based Robot PlanningLLM PromptingHuman-Robot InteractionRobot Motion Planning
AskPDF