Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8% in action executability and 14.8% in final-state error.
Figures & tables
Fig. 1 : STEP with robot system setup. Predicting structured state representations along with future actions allows STEP to quantitatively measure the distance of a state from the goal and deduce assistance parameters required for executing the predicted actions.
Fig. 2 : After a hand-over by the operator, the overall task and current system state are estimated. They are then used to iteratively rollout next actions and states multiple times. Each rollout selects the closest state to the goal to guide actions towards the estimated task.
Fig. 3 : For each of the blocks our structured state representation records the block location (table/another block with its number), block orientation (stand/lie on face/lie on side/NA), top face orientation (long-side parallel/perpendicular to camera/NA), specific location (left/right/center if the block is on another block, otherwise NA). The modular state design also allows extension to other scenarios by adding descriptors - ‘nearby objects’, ‘facing’, ‘containing objects’ etc.
Fig. 4 : The eight block assembly tasks.
TABLE I : Comparison of executability, final error, and LCS of STEP with the baseline, along with task correctness and current state error for multiple task completion percentages.
Rollouts
Exec. % ↑
Final Err. ↓
LCS % ↑
1
88.98
4.40
31.06
2
85.60
4.11
37.4
3
84.55
4.05
38.82
TABLE II : Ablation of rollouts for 50% task completion. Final error improves with rollouts showing that additional rollouts decrease the distance from goal.
Model
Exec. % ↑
Final Error ↓
Task Corr.
Base
STEP
Base
STEP
% ↑
GPT-4o
71.68
78.12
4.53
4.38
62.22
GPT-4o mini
82.60
82.16
4.49
4.93
26.67
GPT-4.1
82.28
92.02
4.80
4.11
71.11
GPT-5.1
69.24
82.70
4.80
3.71
73.33
GPT-5.2
76.64
84.44
4.73
4.49
71.11
TABLE III : Ablating MM-LLM to compare executability and final error of STEP with the baseline for 50% task completion. Results for 45 random dataset instances.
Task corr.
Exec. % ↑
Final Err. ↓
LCS % ↑
1
84.63
3.64
41.16
0
84.37
4.9
33.45
TABLE IV : Ablation of task correctness for 50% task completion. All metrics improve with correct task prediction. Improvement in earlier stages improves downstream performance.
Curr. st. err.
Exec. % ↑
Final Err. ↓
LCS % ↑
0
86.38
3.93
36.08
> 0
84.41
4.06
39.07
TABLE V : Ablation of current state error for 50% task completion. Executability and final error improve with no error in the current state estimation.
In modern industrial production, multiple robots often collaborate to complete complex manufacturing tasks. Large language models (LLMs), with their strong reasoning capabilities, have shown potential in coordinating robots for simple household and manipulation tasks. However, in industrial scenarios, stricter sequential constraints and more complex dependencies within tasks present new challenges for LLMs. To address this, we propose IMR-LLM, a novel LLM-driven Industrial Multi-Robot task planning and program generation framework. Specifically, we utilize LLMs to assist in constructing disjunctive graphs and employ deterministic solving methods to obtain a feasible and efficient high-level task plan. Based on this, we use a process tree to guide LLMs to generate executable low-level programs. Additionally, we create IMR-Bench, a challenging benchmark that encompasses multi-robot industrial tasks across three levels of complexity. Experimental results indicate that our method significantly surpasses existing methods across all evaluation metrics.
Large Language Models (LLMs) can reason over complex instructions but often fail to satisfy the physical and spatial constraints required for robotic task planning. Recent LLM-based planners directly translate text into action sequences, yet they lack structured reasoning about feasibility, reachability, and logical order, resulting in invalid or incomplete plans. We present a heterogeneous multi-LLM framework that decomposes instructions into atomic reasoning tasks and allocates them to role-specialized expert agents under a token budget for real-world computational and communicational constraints. By combining role-oriented reasoning from heterogeneous agents followed by constraint-driven plan synthesis, HEART validates capability, reachability, and constraint conditions before planning and helps produce physically executable plans while maintaining efficiency. Experiments across different household benchmarks show that HEART consistently improves plan success compared to single-LLM and rule-based planners, demonstrating that heterogeneous LLM collaboration enables robust and scalable robotic task planning under resource constraints.
Junho Lee, Seabin Lee, Wonjong Lee +3
Dept. of Electronic Engineering, Sogang University, Seoul, Korea
Approaches to incorporating human awareness into mobile robot decision-making mainly focus on collision avoidance in low-level motion planning, often overlooking the challenges posed by human presence and high-level behavior. To address this vacancy, we present HINT-Plan, a novel approach to integrate human intention prediction into robot task planning. HINT-Plan employs Vision Language Models (VLMs) to anticipate high-level human intentions from third-person image observations, convert them into goal states, and solve joint task-planning problems. To effectively enable scene awareness in context-rich environments, we use hierarchical Scene Graphs (SGs) as high-level representations of the environment, and translate environmental topology and actionable knowledge into formal planning language to ensure executable plans. Evaluated in a photorealistic simulation, HINT-Plan achieves an overall success rate of 69.71% in joint human-robot task planning, substantially outperforming the baselines by up to 35.29%, while also reducing functional conflicts. The results show the effectiveness of explicitly incorporating inferred human intentions into formal multi-agent task planning for proactive human-aware robot decision-making.
Yuchen Liu, Luigi Palmieri, Lujun Li +3
University of Stuttgart, Germany · University of Luxembourg, Luxembourg · Bosch Center for Artificial Intelligence, Germany