cs.RONov 3, 2026

ActOR: Ambient AI-Based Agentic Orchestration for Humanoid Autonomy in Operating Rooms

Authors: Han Zhang, Sicheng Lu, Aleks Santari, Jeffrey K. Jopling, Masaru Ishii, Mathias Unberath

Organizations: Johns Hopkins University, Baltimore, MD, USA. · Johns Hopkins Medical Institutions, Baltimore, MD, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Operating rooms (ORs) increasingly contain ambient sensing infrastructure to enable AI-based optimization of workflows and utilization. We pursue an alternative path: using ambient AI to guide robot action and optimize utilization. Deploying humanoids in the OR without ambient AI is severely limited by cumbersome de novo orientation and planning due to partial onboard observations. We posit that this can be achieved by coupling an ambient AI-based agentic orchestrator with the humanoid robotic actor, providing shared perception and execution feedback for humanoid autonomy in the OR. ActOR fuses a wall-mounted ambient camera and the robot's head-mounted camera into a shared object-level digital twin (DT). A large language model agent plans over the DT using pre-programmed locomotion and manipulation skills without task-specific training. During execution, DT updates and execution outcomes inform retries, alternative approaches, or replanning. We evaluated ActOR in simulated and physical environments. Across three tasks in two simulated ORs reconstructed from real rooms, ambient perception reduced exploration from 1.05 to 0.12 Explore calls per trial when targets were initially outside the robot's view. Closed-loop feedback increased success from 46.7% to 83.3% under nominal execution and from 8.3% to 51.7% under injected navigation, object-relocation, and grasp perturbations. In a physical mock OR with a humanoid, ActOR completed 14 of 20 single-object disposal trials and demonstrated replanning after scene changes. Ambient AI-based observations provide scene state that improves robot action planning, increasing success and reducing exploration compared with robot-only observations. Feeding skill outcomes and DT updates back to the planner improves multi-step task success. Our results show the feasibility of humanoid task autonomy for multi-step disposal tasks in a mock OR.

Figures & tables

Figure 1: Overview of ActOR. (a) A humanoid executes a multi-object disposal task in a mock OR. (b) The shared scene representation integrates ambient and robot observations into an object-level DT and occupancy map. (c) An LLM agent uses the DT and robot skills for task planning, execution monitoring, and replanning.
SkillFunctionSuccess condition
Observe(target)Capture target and update the DT–
Explore()Navigate to likely target locations–
Navigate(target)Walk to a standoff pose facing targetdistxy(pbase,Btarget)≤dreach\mathrm{dist}_{xy}(\mathbf{p}_{\text{base}},\mathbf{B}_{\texttt{target}})\leq d_{\text{reach}}
Pick(target)Grasp targethtarget=in-handh_{\texttt{target}}=\texttt{in-hand}
Place(target,dest)Release target at destptargetxy∈Bdestxy\mathbf{p}_{\texttt{target}}^{xy}\in\mathbf{B}_{\texttt{dest}}^{xy} at release
Table 1: Skill library and the condition under which each skill reports success.
Figure 2: Experimental environments. (a, b) Simulated ORs reconstructed from real rooms for controlled studies. (c) Physical mock OR with a Unitree G1 humanoid.
Figure 3: Controlled simulation results across two ORs. (a) Task success with and without ambient perception. (b) Mean Explore calls pooled across tasks. (c) Task success with and without closed-loop feedback. Error bars: Wilson 95% confidence intervals.
Figure 4: Representative humanoid autonomy in the mock operating room. (a) Multi-step execution in a single-object disposal trial. (b) Replanning after a scene change.
MethodNavigation errorObject relocationGrasp misalignment
Open-loop2/243/180/18
ActOR15/2411/185/18
Table S1 : Successful trials by perturbation type, pooled over tasks and environments. Denominators are the trials assigned each type.
  OR-AOR-B
ConditionMethodRetr.Coll.ResetRetr.Coll.Reset
Ego-observableRobot-only776879
ActOR8891049
Partially observableRobot-only656887
ActOR888978
NominalOpen-loop732745
Table S2 : Successful trials by environment and task (out of 10).
ConditionMethodSuccessesSR [95% CI]
Ego-observableRobot-only44/6073.3 [61, 83]
ActOR48/6080.0 [68, 88]
Partially observableRobot-only40/6066.7 [54, 77]
ActOR48/6080.0 [68, 88]
NominalOpen-loop28/6046.7 [35, 59]
ActOR50/6083.3 [72, 91]
Table S3 : Pooled success rates with 95% Wilson confidence intervals.
TaskInstruction
Retrieval“Clean up: put the [appearance] into the biohazard box.”
Collection“Put all the bottles into the biohazard box.”
Room reset“Put all the used disposable items into the biohazard box.”
Table S4 : Example instructions.
More in this topic

Agentic AI for Robotics

See all 60 papers

Explore similar work

CardsList
Aug 23, 2026

Physical Agentic AI: An Architecture for Orchestrating a Robot Crew with LLMs

Agentic AI frameworks interpret open-ended task goals and decompose them into multi-step plans. Richer information about embodiment-specific capabilities, physical preconditions, and cross-robot coordination improves grounding, but does not eliminate infeasible, mistimed, or unsafe physical actions. Physical robot crews therefore require an explicit architectural interface between semantic planning and execution, where every planned action is verified against robot capabilities, system state, and workflow constraints before actuation. This paper introduces Physical Agentic AI, a framework for skill-grounded robot agent orchestration, in which each robot exposes a typed library of executable skills while a foundation model planner decomposes a task into phases and assigns each phase to a robot-skill pair. A Robot Orchestration layer exposes the skill library, robot state, named locations, and workflow contracts to a non-actuating Mission Planner, while a deterministic Robot Orchestrator validates and authorizes one skill at a time. We evaluate on a drone-UGV search-and-dispatch mission, where every mission in every condition is executed live in Gazebo, and on a humanoid-quadruped transportation task using hardware-equivalent skill interfaces plus two physical trials on a Unitree G1 and Go2. Varying planner knowledge and runtime enforcement independently, we find that retrieval raises skill grounding from 51% to 96% yet leaves informed planners dispatching 23-29% of faulted steps. Per-dispatch enforcement reduces false dispatch to 0% with no false blocks, and a held-plan ablation confirms that the gate, not plan variation, is responsible. Live execution makes the difference physical: without enforcement all eight injected faults crossed the orchestration boundary and six produced robot motion; with enforcement all eight were refused before motion.

Xinyuan Liu, Eren Sadikoglu, Riana Chatterjee +1

School of Computing and Augmented Intelligence, Arizona State University, AZ, USA

Agentic AI for RoboticsLLM Agent OrchestrationRobot Task PlanningRuntime Enforcement for AI Agents
AskPDF
Oct 26, 2026

RobotAgentBench: Towards Open-world Robot Autonomy

Recent advances in vision-language-action (VLA) and world-action models (WAMs) have significantly improved robotic manipulation, yet deploying robots in the open world demands a higher level of autonomy beyond executing predefined tasks. Robot agents offer a promising paradigm for this transition, integrating task discovery, contextual reasoning, physical execution, and adaptive recovery into a closed interaction loop. However, existing benchmarks largely evaluate isolated capabilities, failing to capture the complexity of open-world robot autonomy. We introduce RobotAgentBench (RAB), a comprehensive benchmark for evaluating robot agents in realistic, open-world household environments. Through an agent-assisted, human-in-the-loop real-to-sim pipeline, RAB constructs 50 diverse household layouts with interactive objects and physically grounded tasks, encompassing 90 atomic tasks, 18 short-horizon tasks, 8 long-horizon tasks, and 4 open-ended scenarios. RAB provides a multi-level evaluation of robot agent capabilities: (1) task-level assessments systematically examine five complementary dimensions: task discovery, context reasoning, manipulation, failure recovery, and robustness; (2) scenario-level evaluation assesses how agents integrate these capabilities to autonomously fulfill high-level user needs. Furthermore, RAB enables systematic comparisons of standalone VLA/WAM models and diverse robot agent architectures under a unified planning-execution-verification framework. Extensive experiments reveal substantial gaps between manipulation and autonomy, demonstrate the advantages of agentic architectures and recursive self-improvement, and highlight the remaining challenges toward open-world robot autonomy.

Lingjun Mao, Jinglin Cao, Lukun He +22

Aether AI · University of California San Diego · University of California Davis

Embodied Agent BenchmarksRobot Manipulation Benchmarks
AskPDF
May 8, 2026

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning

We present a hierarchical language-driven framework for robotic task and motion planning to improve natural, intuitive human-robot interaction in service and assistance scenarios. The proposed system employs two large language model (LLM) modules: a high-level planning agent and a low-level spatial reasoning sub-module. The primary agent processes natural language commands and generates action sequences using a ReAct-style prompt, interacting with tools for object perception and manipulation (e.g., pick, place, release). For precise spatial placement, such as interpreting "place the mug next to the plate", a separate sub-prompting module handles 3D reasoning based on object geometry and scene layout. The system integrates YOLOX-GDRNet for object detection and pose estimation, along with a motion execution stub. We evaluated the system in 24 test scenarios, ranging from simple spatial commands to high-level instructions and infeasible requests. The system achieved an overall task success rate of 86%.

Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2

IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1

Large Language Model-Based Robot PlanningLLM PromptingHuman-Robot InteractionRobot Motion Planning
AskPDF