Physical Agentic AI: An Architecture for Orchestrating a Robot Crew with LLMs

Xinyuan Liu; Eren Sadikoglu; Riana Chatterjee; Ransalu Senanayake

cs.ROAug 23, 2026

Physical Agentic AI: An Architecture for Orchestrating a Robot Crew with LLMs

Authors: Xinyuan Liu, Eren Sadikoglu, Riana Chatterjee, Ransalu Senanayake

Organizations: School of Computing and Augmented Intelligence, Arizona State University, AZ, USA

Abstract

Agentic AI frameworks interpret open-ended task goals and decompose them into multi-step plans. Richer information about embodiment-specific capabilities, physical preconditions, and cross-robot coordination improves grounding, but does not eliminate infeasible, mistimed, or unsafe physical actions. Physical robot crews therefore require an explicit architectural interface between semantic planning and execution, where every planned action is verified against robot capabilities, system state, and workflow constraints before actuation. This paper introduces Physical Agentic AI, a framework for skill-grounded robot agent orchestration, in which each robot exposes a typed library of executable skills while a foundation model planner decomposes a task into phases and assigns each phase to a robot-skill pair. A Robot Orchestration layer exposes the skill library, robot state, named locations, and workflow contracts to a non-actuating Mission Planner, while a deterministic Robot Orchestrator validates and authorizes one skill at a time. We evaluate on a drone-UGV search-and-dispatch mission, where every mission in every condition is executed live in Gazebo, and on a humanoid-quadruped transportation task using hardware-equivalent skill interfaces plus two physical trials on a Unitree G1 and Go2. Varying planner knowledge and runtime enforcement independently, we find that retrieval raises skill grounding from 51% to 96% yet leaves informed planners dispatching 23-29% of faulted steps. Per-dispatch enforcement reduces false dispatch to 0% with no false blocks, and a held-plan ablation confirms that the gate, not plan variation, is responsible. Live execution makes the difference physical: without enforcement all eight injected faults crossed the orchestration boundary and six produced robot motion; with enforcement all eight were refused before motion.

Figures & tables

Fig. 1: The Physical Agentic AI pipeline. (a) Retrieval injects typed skills, locations, and workflow contracts into the planning context; (b) a non-actuating Mission Planner generates a structured plan; (c) the Robot Orchestrator validates the plan, permits one feedback replan, and re-authorizes each dispatch, rejecting invalid steps before actuation; and (d) authorized skills execute on heterogeneous platforms: a drone–UGV Gazebo team with dispatch-time state binding, and a physical humanoid–quadruped crew.

Fig. 2: Gazebo Search-and-Dispatch testbed: the Iris drone localizes the victim (flag) while the workflow contract holds the TurtleBot3 (foreground, lidar shown) staged until a validated detection is published.

Fig. 3: The three air–ground Search-and-Dispatch evaluation scenarios. Each panel shows the search zone (shaded), obstacles, the Iris drone and TurtleBot3 spawn poses, and the victim location: A (Baseline) places the victim at $(5,5)$ , B (Near-field) at $(-3,4)$ , and C (Far-field) at $(8,-4)$ , varying the distance and obstacle clutter between the rover spawn and the delivery goal. All configurations in Table I are evaluated on these three maps.

Metric	LLM-only	Skill-list	RO-prompt	RO
Planner behavior (what was proposed)
Workflow-selection acc. (%)	100 [84, 100]	100 [84, 100]	100 [84, 100]	100 [84, 100]
Skill grounding (%)	51 [37, 65]	96 [86, 99]	96 [86, 99]	96 [86, 99]
Plan executability (%)	25 [11, 47]	90 [70, 97]	90 [70, 97]	90 [70, 97] ‡
Contract violation (%) $\downarrow$	70 [48, 85]	0 [0, 16]	0 [0, 16]	0 [0, 16]
Execution outcomes (what reached the robots)

TABLE I: Air–ground Search-and-Dispatch evaluation across 20 scenarios (12 nominal, 8 fault-injection). Every mission in every condition was executed live in Gazebo against PX4 SITL and the ROS2 robot services. All conditions share the model, schema, parser, and dispatch interface. Brackets denote 95% Wilson confidence intervals; Fisher’s exact tests compare RO against each baseline. Darker bars denote RO on a

0

–

100\%

scale.

Fig. 4: Execution timeline (full system, Scenario A, means). The workflow gate (dashed) holds the rover idle until the drone’s detection is published, then releases navigation; cf. the hardware timeline in Fig. 6 .

Panel of Figure 6 — Fig. 5: Hardware demonstrations of distance-conditioned delivery, with representative frames extracted from the two recordings. Top (a): when the requester is near the humanoid, the planner selects the Handover workflow and the humanoid (Unitree G1) hands the red cube directly to the person. Bottom (b): when the requester is far from the humanoid, the planner selects the Carry workflow; the humanoid grasps the red cube and loads it onto the quadruped (Unitree Go2) payload bin, which then navigates across the room to deliver it. Approximate phase windows annotated from the clips are shown beneath each frame.

Metric	LLM-only	Skill-list	RO-prompt	RO
Planner behavior (what was proposed)
Workflow-selection acc. (%)	100 [84, 100]	100 [84, 100]	100 [84, 100]	100 [86, 100]
Skill grounding (%)	9 [3, 22]	91 [77, 97]	97 [85, 100]	97 [85, 100]
Plan executability (%)	15 [5, 36]	85 [64, 95]	95 [76, 99]	95 [76, 99]
Contract violation (%) $\downarrow$	70 [48, 85]	10 [3, 30]	15 [5, 36]	15 [5, 36]
Execution outcomes (what reached the robots)

TABLE II: Mocked humanoid–quadruped evaluation across 20 scenarios (12 nominal, 8 fault-injection). The setup uses hardware-equivalent skill interfaces with mock adapters. Brackets denote 95% Wilson confidence intervals. Fisher’s exact tests compare RO against each baseline. Darker bars denote RO on a

0

–

100\%

scale.

Panel of Figure 8 — Fig. 6: Execution timelines for the two hardware demonstrations. Handover uses only the G1, whereas Carry serializes G1 manipulation and Go2 transport under the load-before-depart constraint. Timings are annotated from the recorded demonstrations.

Explore similar work

Jul 23, 2026

Addressing the Orchestration Gap in Generalist Robots via Physical Agency

General-purpose robots need to reason about their actions, combining perception, world knowledge, planning, success detection, recovery, and low-level control. Today's state-of-the-art models attempt to combine all these capabilities into the learned policy via large-scale pre-training. Instead, we show that these capabilities can be decomposed into a general language-conditioned policy/control agent and a high-level agent manager/orchestrator. Rather than training policies to reason via pre-training, we build a closed-loop physical agent orchestrator that can do high-level planning, decompose the goal into achievable subgoals, command low-level motor commands, track and verify the outcome from low-level observations, and recover from failures. Our Physical Agency orchestrator (Pigey) can control existing vision-language-action (VLA) policies as well as parametrized skills to solve complex reasoning tasks in the real world, without any additional data collection or post-training. We evaluate Pigey extensively across simulation benchmarks and challenging real-world robotic manipulation tasks, and demonstrate significant performance improvements over existing generalist policies. On LIBERO-PRO, Pigey advances the state-of-the-art by over 4x (12.8% -> 53.3%) with no task-specific fine-tuning. On a real robot, Pigey lifts the frozen policy from near-zero to over 90% on reasoning-limited tasks. We call the difference between what frozen motor skills achieve alone and inside the agentic loop the orchestration gap.

Liane Galanti, Dhruv Shah, Tri Dao

1Princeton University · 2Together AI

Robotic Manipulation Robot Task Planning Language-Conditioned Robot Control

Ask PDF

Jun 24, 2026

HEART: Coordination of Heterogeneous Expert Agents for Physically Grounded Robotic Task Planning

Large Language Models (LLMs) can reason over complex instructions but often fail to satisfy the physical and spatial constraints required for robotic task planning. Recent LLM-based planners directly translate text into action sequences, yet they lack structured reasoning about feasibility, reachability, and logical order, resulting in invalid or incomplete plans. We present a heterogeneous multi-LLM framework that decomposes instructions into atomic reasoning tasks and allocates them to role-specialized expert agents under a token budget for real-world computational and communicational constraints. By combining role-oriented reasoning from heterogeneous agents followed by constraint-driven plan synthesis, HEART validates capability, reachability, and constraint conditions before planning and helps produce physically executable plans while maintaining efficiency. Experiments across different household benchmarks show that HEART consistently improves plan success compared to single-LLM and rule-based planners, demonstrating that heterogeneous LLM collaboration enables robust and scalable robotic task planning under resource constraints.

Junho Lee, Seabin Lee, Wonjong Lee +3

Dept. of Electronic Engineering, Sogang University, Seoul, Korea

Large Language Model-Based Robot Planning Robot Task Planning Multi-Agent Planning Multi-Agent LLMs LLM Agent Planning

Ask PDF

Sep 24, 2026

HarnessPAI: An Evolving Harness for Physical AI

Physical AI aims to build embodied agents that perceive the world, understand and reason about it, and decide how to act. Yet the field has focused primarily on the last component: the action model that maps observations to low-level controls. The prevailing training recipe can erode the perceptual and reasoning capabilities needed for robust behavior, leaving even strong action models vulnerable to scene perturbations and long-horizon tasks. We introduce HarnessPAI, a model- and embodiment-agnostic Harness framework for Physical AI that treats code as the executable and evolvable interface that organizes the underlying action primitive. The framework separates two timescales: within a rollout, it executes open-loop at the program level, with a fixed program guiding and checking execution; across rollouts, it evolves closed-loop, using execution feedback to revise the program and distill failures into reusable skills. Across desktop robot arms, household robots, a robot vacuum, and a legged walking agent, HarnessPAI improves on both pure action models and code-as-policy baselines without retraining the underlying model: a 61.6-point gain over

π_{0.5}

on LIBERO-PRO and a 27.2-point gain over WorldDreamer on RoboCasa atomic tasks. Once a program is selected, rollout execution requires no online high-level LLM deliberation. Beyond execution, the converged program is also a cheap and reliable expert-data collector, and fine-tuning

π_{0.5}

on collected expert data lifts success rate on LIBERO-PRO by 38.8 points. Our results suggest that the frontier of Physical AI depends not only on stronger action models, but also on executable harnesses that integrate perception, task understanding and reasoning, and action execution into a unified, verifiable, and feedback-driven system. Website: https://darwin-agent.github.io/HarnessPAI

Xin Wang, Wenhao Wu, Menghao Zhang +20

Program Synthesis Agent Harness Optimization Embodied AI Robot Learning

Ask PDF