Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8% in action executability and 14.8% in final-state error.
Figures & tables
Fig. 1 : STEP with robot system setup. Predicting structured state representations along with future actions allows STEP to quantitatively measure the distance of a state from the goal and deduce assistance parameters required for executing the predicted actions.
Fig. 2 : After a hand-over by the operator, the overall task and current system state are estimated. They are then used to iteratively rollout next actions and states multiple times. Each rollout selects the closest state to the goal to guide actions towards the estimated task.
Fig. 3 : For each of the blocks our structured state representation records the block location (table/another block with its number), block orientation (stand/lie on face/lie on side/NA), top face orientation (long-side parallel/perpendicular to camera/NA), specific location (left/right/center if the block is on another block, otherwise NA). The modular state design also allows extension to other scenarios by adding descriptors - ‘nearby objects’, ‘facing’, ‘containing objects’ etc.
Fig. 4 : The eight block assembly tasks.
TABLE I : Comparison of executability, final error, and LCS of STEP with the baseline, along with task correctness and current state error for multiple task completion percentages.
Rollouts
Exec. % ↑
Final Err. ↓
LCS % ↑
1
88.98
4.40
31.06
2
85.60
4.11
37.4
3
84.55
4.05
38.82
TABLE II : Ablation of rollouts for 50% task completion. Final error improves with rollouts showing that additional rollouts decrease the distance from goal.
Model
Exec. % ↑
Final Error ↓
Task Corr.
Base
STEP
Base
STEP
% ↑
GPT-4o
71.68
78.12
4.53
4.38
62.22
GPT-4o mini
82.60
82.16
4.49
4.93
26.67
GPT-4.1
82.28
92.02
4.80
4.11
71.11
GPT-5.1
69.24
82.70
4.80
3.71
73.33
GPT-5.2
76.64
84.44
4.73
4.49
71.11
TABLE III : Ablating MM-LLM to compare executability and final error of STEP with the baseline for 50% task completion. Results for 45 random dataset instances.
Task corr.
Exec. % ↑
Final Err. ↓
LCS % ↑
1
84.63
3.64
41.16
0
84.37
4.9
33.45
TABLE IV : Ablation of task correctness for 50% task completion. All metrics improve with correct task prediction. Improvement in earlier stages improves downstream performance.
Curr. st. err.
Exec. % ↑
Final Err. ↓
LCS % ↑
0
86.38
3.93
36.08
> 0
84.41
4.06
39.07
TABLE V : Ablation of current state error for 50% task completion. Executability and final error improve with no error in the current state estimation.