On-Demand Human Assistance for Task Continuation under Physical Action Failures in LLM-based Planning
Authors: Shoichi Hasegawa, Akira Taniguchi, Lotfi El Hafi, Gustavo Alfonso Garcia Ricardez, Tadahiro Taniguchi
Organizations: Shoichi Hasegawa and Akira Taniguchi are with Ritsumeikan University; 2-150 Iwakura, Ibaraki, Osaka 567-8570, Japan. · Akira Taniguchi is with Kyushu Institute of Technology; 680-4 Kawadu, Iizuka, Fukuoka 820-0067, Japan. · Lotfi El Hafi is with Coarobo GK; 2-2-2 Hikaridai, Seika, Soura, Kyoto 619-0237, Japan. · Lotfi El Hafi and Tadahiro Taniguchi are with Ritsumeikan University Research Organization of Science and Technology; 1-1-1 Nojihigashi, Kusatsu, Shiga 525-8527, Japan. · Gustavo Alfonso Garcia Ricardez is with The University of Osaka; 1-3 Machikaneyama, Toyonaka, Osaka 560-8531, Japan. · Gustavo Alfonso Garcia Ricardez is with Ritsumeikan Global Innovation Research Organization; 1-1-1 Nojihigashi, Kusatsu, Shiga 525-8527, Japan. · Tadahiro Taniguchi is with Kyoto University; Yoshida Honmachi, Sakyo, Kyoto, Kyoto 606-8317, Japan.
While robot action planning based on large language models (LLMs) has advanced remarkably, continuing a task after a physical action failure during execution remains a key challenge. For example, after failing to grasp an object, the robot may not notice that it has fallen off the table and keep trying to detect and pick it up, stalling the task. Existing approaches either rely on closed-loop autonomous re-planning, which fails when errors exceed the robot's capabilities, or on human intervention without updating the LLM's plan, hindering subsequent planning. We present a system for LLM-based robot action planning that integrates remote human intervention with feedback-based replanning. When the LLM detects an action failure, the robot requests remote assistance; an operator resolves the failure through teleoperation and reports the outcome in natural language, which is fed back to the LLM to update its plan so that the task can continue. We evaluated the proposed system on a real-world trash-collection task, comparing it with a fully autonomous method and a teleoperation-only baseline. The proposed system improves task progress over the autonomous method in this case study. For easy-to-retrieve trash, the gap from the teleoperation reference was small, while a larger gap remained for harder-to-retrieve trash. These results suggest that, in this case study, combining remote human intervention with feedback-based replanning was associated with continued task execution even in the face of otherwise unrecoverable failures. The project website is https://emergentsystemlabstudent.github.io/REPAIR/.
Figures & tables
Fig. 1: An overview of our study. When a robot drops an object and subsequently fails to detect it, it falls into a loop state. In such cases, the robot requests assistance from an operator on demand, and the issue is resolved through remote error resolution.
Fig. 2: An overview of the proposed system. The LLM takes a user’s instruction, decomposes it based on robot capabilities and rules, and assigns tasks to each robot. After that, LLMs generate executable actions, which the robots perform. When failures occur, a robot requests operator assistance. The operator resolves the issue remotely and provides feedback, allowing the LLM to update its understanding and resume execution.
Fig. 3: Experimental Environment and Object Placements.
Fig. 4: Distribution of task progress and results of statistical comparisons for each condition. The higher the task progress, the better (↑).
Methods
Whole ↑
Level 1 Object ↑
Level 2 Object ↑
Teleop
2.7±2.2
1.2±1.2
1.5±1.0
Ours
1.6±1.2
1.0±0.60
0.66±0.77
Auto
0.0±0.0
0.0±0.0
0.0±0.0
TABLE I: Mean and standard deviation of task progress for each condition and each level of collection difficulty.
Fig. 5: Fig. 5 (a) shows the overall NASA-TLX score, while Fig. 5 (b)–(g) show the scores for each subscale. The gray lines in each figure represent changes in scores across conditions for the same subject. In Fig. 5 (b), (c), and (f), the Friedman test revealed significant differences among the three conditions ( p<0.05 ). A lower score indicates a better result (↓).
Fig. 6: Demonstration of the proposed system in operation. (a) Operator performing remote error resolution for an object dropped by a robot. (b) Inter-robot collision due to operator remote control error.
Anticipating and adapting to failures is a key capability robots need to collaborate effectively with humans in complex domains. This continues to be a challenge despite the impressive performance of state of the art AI planning systems and Large Language Models (LLMs) because of the uncertainty associated with the tasks and their outcomes. Toward addressing this challenge, we present a hybrid framework that integrates the generic prediction capabilities of an LLM with the probabilistic sequential decision-making capability of Relational Dynamic Influence Diagram Language. For any given task, the robot reasons about the task and the capabilities of the human attempting to complete it; predicts potential failures due to lack of ability (in the human) or lack of relevant domain objects; and executes actions to prevent such failures or recover from them. Experimental evaluation in the VirtualHome 3D simulation environment demonstrates substantial improvement in performance compared with state of the art baselines.
Nabanita Dash, Ayush Kaura, Shivam Singh +4
Robotics Research Center, IIIT Hyderabad, India · TCS Research, Tata Consultancy Services, India · School of Informatics, University of Edinburgh, UK
Large Language Models (LLMs) can reason over complex instructions but often fail to satisfy the physical and spatial constraints required for robotic task planning. Recent LLM-based planners directly translate text into action sequences, yet they lack structured reasoning about feasibility, reachability, and logical order, resulting in invalid or incomplete plans. We present a heterogeneous multi-LLM framework that decomposes instructions into atomic reasoning tasks and allocates them to role-specialized expert agents under a token budget for real-world computational and communicational constraints. By combining role-oriented reasoning from heterogeneous agents followed by constraint-driven plan synthesis, HEART validates capability, reachability, and constraint conditions before planning and helps produce physically executable plans while maintaining efficiency. Experiments across different household benchmarks show that HEART consistently improves plan success compared to single-LLM and rule-based planners, demonstrating that heterogeneous LLM collaboration enables robust and scalable robotic task planning under resource constraints.
Junho Lee, Seabin Lee, Wonjong Lee +3
Dept. of Electronic Engineering, Sogang University, Seoul, Korea
General Purpose Service Robot (GPSR) tasks, as defined in the RoboCup@Home benchmark, require robots to interpret diverse natural language commands and generate multi-step action sequences in real home environments. Conventional Single Prompt (SP) approaches suffer from context bloat and the "Lost in the Middle" phenomenon, leading to unreliable task planning. We propose an LLM chaining architecture that separates instruction classification and action generation into two specialized stages, reducing per-inference prompt length by approximately 45% while improving planning consistency. We evaluate our method using 100 randomly generated GPSR commands across three language models spanning local open-source and frontier cloud deployment contexts. Results show consistent planning improvements over SP across all models, with gains of up to +37 percentage points on local models. Further, real-robot execution experiments on the Toyota Human Support Robot (HSR) reveal that planning success alone does not guarantee task completion, with 6 of 10 tasks completing successfully and execution-layer failures identified as the primary remaining bottleneck.
Lucas Da Mota Bruno, Jiahao Sim, Yoshinobu Hagiwara
Grad. School of Science and Engineering Soka University Tokyo, Japan · Faculty of Science and Engineering Soka University Tokyo, Japan