RoboAssist: Interactive Human-Humanoid Planning for Long-Horizon Surgical Assistance

Jingwei Jia; Keyu Zhou; Jiewei Wang; Peisen Xu; Xingyuan Zhou; Liang Wang; Jiming Chen; Gaofeng Li

cs.ROSep 30, 2026

RoboAssist: Interactive Human-Humanoid Planning for Long-Horizon Surgical Assistance

Authors: Jingwei Jia, Keyu Zhou, Jiewei Wang, Peisen Xu, Xingyuan Zhou, Liang Wang, Jiming Chen, Gaofeng Li, +2 more

Organizations: Hangzhou Dianzi University, China. · Zhejiang University, China. · New York University, USA. · Italian Institute of Technology (IIT), Italy.

Abstract

Long-horizon surgical assistance requires humanoid robots to coordinate with evolving human activities while maintaining safety across planning and execution. We present RoboAssist, an agent-based framework for interactive human-humanoid planning that integrates workflow reasoning, task coordination, and cross-layer safety. At its core is an asymmetric dual-track representation that separates partially observed human process states from executable robot task sequences. By updating human-process estimates, scene context, and task dependencies online, RoboAssist revalidates the remaining task sequence and replans only the affected suffix when workflow requests change. A cross-layer safety architecture combines preventive navigation regulation, reactive regulation during close-range handover, and independent whole-body runtime supervision. This design couples online task coordination with safety constraints throughout execution. We demonstrate the framework on a Unitree G1 humanoid robot in long-horizon, multi-stage simulated surgical assistance scenarios encompassing multimodal interaction, instrument handling, medical material transport, navigation, and safe human-robot handover. Experiments show multi-stage task completion and adaptation to workflow-request changes. A targeted full-replanning ablation shows that residual replanning reduces plan-update latency and post-update token usage. Separate safety experiments demonstrate complementary protection across navigation, handover, and runtime supervision. Additional results and demonstrations are available online at https://roboassist.github.io.

Figures & tables

Fig. 2: The proposed framework integrates workflow knowledge, task memory, scene observations, and human requests for online task alignment and replanning. The agent predicts human–robot task progression, dispatches executable skills, and enforces runtime safety constraints during navigation, manipulation, and handover.

Fig. 3: Cross-layer safety architecture combining surgeon-aware navigation regulation, reactive handover regulation, and fail-safe whole-body runtime supervision with increasing intervention authority.

Fig. 4: Experimental setup. (a) Overall experimental environment, including the surgeon, Unitree G1, surgical, handover, and sterile tables, global ZED 2i camera, and LED lighting. (b) Three task stages: picking, handover, and surgical assistance. (c) Camera configuration and representative views from the global camera and the robot’s head and wrist cameras.

Fig. 5: Task execution performance. (a) Success rates over 15 single-object trials and seven multi-object task trials per method, with 95% Wilson confidence intervals. (b) Cumulative completion at four ordered subtask checkpoints in the multi-object task. Each point is the fraction of all seven trials that successfully complete every checkpoint up to and including the indicated subtask. The final point equals the corresponding multi-object task success rate in (a).

Fig. 6: Representative moments of the RoboAssist workflow: anticipating the surgeon’s next action, navigating to the object table, grasping and holding an object, handing it over to the surgeon, and placing an object on the table.

TABLE I: Request-order adaptation and planning overhead. Success rates use 16 trials per method; latency and tokens are mean $\pm$ sample SD over 10 logged trials per LLM-based method. $\mathrm{k}=10^{3}$ tokens; N/A denotes not applicable.

Fig. 8: Cross-layer safety evaluation. (a) Recorded navigation profile, where negative distance indicates boundary crossing. (b) Protective responses with 95% Wilson confidence intervals; L3 OFF disables the runtime supervisor. (c) Mean response times of the VLM and force-feedback pathways, measured from the corresponding hazard trigger to the protective response.

Explore similar work

Assistance Without Interruption: A Benchmark and LLM-based Framework for Non-Intrusive Human-Robot Assistance
May 2, 2026Yuedi Zhang, Shuanghao Bai, Wanqi Zhou +4Human-Robot Interaction Assistant
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, China · School of Electrical Engineering, Xi’an University of Technology, Xi’an, China
PACT: Proactive Asking for Continual Task Assistance in Human-Robot Collaboration
May 23, 2026Chengbo He, Sheng Li, Chenyang Ma +6Human-Robot Collaboration Proactive
University of Science and Technology Beijing · University of Oxford · 3Tsinghua University
Safe Embodied AI for Long-horizon Tasks: A Cross-layer Analysis of Robotic Manipulation
Jun 4, 2026Dabin Kim, Daemin Park, Sangyub Lee +4Robotic Manipulation Long-Horizon Task Planning
UNIST InnoCORE AI-Space Solar Initiative, Ulsan National Institute of Science and Technology (UNIST), 50 UNIST-gil, Ulsan, 44919, Republic of Korea. · Automation and Systems Research Institute, Seoul National University, 1 Gwanak-ro, Seoul, 08826, Republic of Korea. · Department of Electrical and Computer Engineering, Seoul National University, 1 Gwanak-ro, Seoul, 08826, Republic of Korea. +3

May 2, 2026, cs.ROJ/K move · Enter open · S save

Assistance Without Interruption: A Benchmark and LLM-based Framework for Non-Intrusive Human-Robot Assistance

Yuedi Zhang, Shuanghao Bai, Wanqi Zhou, Haoran Zhang+3

Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, China · School of Electrical Engineering, Xi’an University of Technology, Xi’an, China

Human-Robot Interaction Assistant Negotiation

Open paper AskPDF arXiv

Human-robot interaction (HRI) has long studied how agents and people coordinate to achieve shared goals. In this work, we formalize and benchmark the non-intrusive assistance as an independent paradigm of HRI, where a robot proactively supports a human's ongoing multi-step activities while strictly avoiding interruptions. Unlike conventional HRI tasks that rely on direct commands, explicit negotiation, or proactive interventions based on user habits and history, our task treats the human's plan as the primary process and formulates assistance as a joint decision over when to act and what to do. To systematically evaluate this problem, we establish a simulation benchmark, NIABench, along with new metrics tailored to the non-intrusive assistance task. We further propose a hybrid architecture that integrates an LLM with a scoring model. The scoring model first applies semantic retrieval to prune large candidate action sets, and then a ranker evaluates human-step and robot-action pairs, enabling reasoning over timing and cross-step dependencies. Comprehensive experiments on both NIABench and real-world scenarios demonstrate that our method achieves proactive, non-intrusive assistance that reduces human effort while preserving task effectiveness.

Abstract

Explore similar work

Assistance Without Interruption: A Benchmark and LLM-based Framework for Non-Intrusive Human-Robot Assistance

PACT: Proactive Asking for Continual Task Assistance in Human-Robot Collaboration

Safe Embodied AI for Long-horizon Tasks: A Cross-layer Analysis of Robotic Manipulation