IMR-LLM: Industrial Multi-Robot Task Planning and Program Generation using Large Language Models

Xiangyu Su; Juzhan Xu; Oliver van Kaick; Kai Xu; Ruizhen Hu

cs.ROMar 3, 2026

IMR-LLM: Industrial Multi-Robot Task Planning and Program Generation using Large Language Models

Authors: Xiangyu Su, Juzhan Xu, Oliver van Kaick, Kai Xu, Ruizhen Hu

Organizations: Shenzhen University. · SpeedBot Robotics Co., Ltd. · Carleton University. · Institute of AI for Industries, Chinese Academy of Sciences.

Abstract

In modern industrial production, multiple robots often collaborate to complete complex manufacturing tasks. Large language models (LLMs), with their strong reasoning capabilities, have shown potential in coordinating robots for simple household and manipulation tasks. However, in industrial scenarios, stricter sequential constraints and more complex dependencies within tasks present new challenges for LLMs. To address this, we propose IMR-LLM, a novel LLM-driven Industrial Multi-Robot task planning and program generation framework. Specifically, we utilize LLMs to assist in constructing disjunctive graphs and employ deterministic solving methods to obtain a feasible and efficient high-level task plan. Based on this, we use a process tree to guide LLMs to generate executable low-level programs. Additionally, we create IMR-Bench, a challenging benchmark that encompasses multi-robot industrial tasks across three levels of complexity. Experimental results indicate that our method significantly surpasses existing methods across all evaluation metrics.

Figures & tables

Fig. 1: A multi-robot industrial production line. Our method transforms manufacturing tasks described in natural language into high-level task plans and low-level execution programs, allowing multiple robots to collaborate efficiently in completing the tasks.

Fig. 2: An overview of our method. Given an instruction $I$ , an industrial scene $S$ , and program examples $\mathbb{E}$ , our method performs task planning (highlighted in green) to decompose operations, assign robots, and schedule operations using a disjunctive graph and a heuristic solver. This is followed by program generation (highlighted in blue) that translates the plan into executable Python code under the guidance of an operation process tree. The resulting high-level plan and low-level program enable collaborative execution by multiple robots.

Fig. 3: An overview of our dataset. Our tasks consist of (a) various scenes and (b) various machines and robots equipped with different end-effectors. (c) A pie chart showing the distribution of task types on the left and a bar chart showing the average number of operations per task type on the right.

Methods	Single Robot					Simple Multi-Robot					Complex Multi-Robot
Methods	OC $\uparrow$	SE $\uparrow$	Exe $\uparrow$	GCR $\uparrow$	SR $\uparrow$	OC	SE $\uparrow$	Exe $\uparrow$	GCR $\uparrow$	SR $\uparrow$	OC $\uparrow$	SE $\uparrow$	Exe $\uparrow$	GCR $\uparrow$	SR $\uparrow$
SMART-LLM [ 6 ]	0.83	0.70	0.50	0.50	0.50	0.67	0.46	0.46	0.32	0.20	0.50	0.04	0.00	0.03	0.00
LaMMA-S [ 9 ]	0.80	0.80	0.70	0.70	0.70	0.71	0.67	0.40	0.45	0.33	0.56	0.26	0.20	0.33	0.16
LaMMA-O [ 9 ]	0.80	0.80	0.80	0.80	0.80	0.71	0.67	0.53	0.55	0.46	0.56	0.26	0.28	0.37	0.20
LiP-O [ 7 ]	1.00	1.00	0.90	0.90	0.90	0.93	0.80	0.73	0.74	0.73	0.63	0.28	0.36	0.42	0.24
Ours (GPT-4o)	1.00	1.00	0.90	0.98	0.90	1.00	1.00	0.87	0.94	0.87	0.88	0.75	0.76	0.79	0.68

TABLE I: Evaluation of IMR-LLM and baselines on different categories of tasks in the IMR-Bench dataset.

Methods	Single Robot					Simple Multi-Robot					Complex Multi-Robot
Methods	OC $\uparrow$	SE $\uparrow$	Exe $\uparrow$	GCR $\uparrow$	SR $\uparrow$	OC $\uparrow$	SE $\uparrow$	Exe $\uparrow$	GCR $\uparrow$	SR $\uparrow$	OC $\uparrow$	SE $\uparrow$	Exe $\uparrow$	GCR $\uparrow$	SR $\uparrow$
Ours ( $w/order$ )	1.00	1.00	0.90	0.95	0.90	1.00	0.67	0.47	0.42	0.47	0.85	0.07	0.08	0.10	0.00
Ours ( $w/dependency$ )	1.00	1.00	0.90	0.98	0.90	1.00	0.73	0.80	0.80	0.60	0.83	0.39	0.44	0.47	0.36
Ours ( $w/o\ T$ )	1.00	1.00	0.80	0.80	0.80	1.00	1.00	0.64	0.74	0.64	0.88	0.75	0.48	0.53	0.44
Ours	1.00	1.00	0.90	0.98	0.90	1.00	1.00	0.87	0.94	0.87	0.88	0.75	0.76	0.79	0.68

TABLE II: Ablation study of different variations of IMR-LLM.

Fig. 4: Keyframes in the execution process. Workpiece 1 and 2 (highlighted in pink and green) are initially placed on the conveyor belt. The robots (highlighted in the yellow boxes) collaborate to complete the operations and finally place workpiece 2 on pallet 1 (keyframe 5) and workpiece 1 on pallet 2 (keyframe 6). Each keyframe contains textual annotations describing the operations performed by the robots.

Fig. 5: A real-world experiment. Three robots collaborate to complete a transportation task.

Explore similar work

STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration
Aug 27, 2026Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu +2Large Language Model-Based Robot Planning Robot Task Planning
University of North Carolina at Chapel Hill, United States · Honda Research Institute, San Jose, CA, USA
Precise Robot Command Understanding Using Grammar-Constrained Large Language Models
Apr 5, 2026Xinyun Huo, Raghav Gnanasambandam, Xinyao ZhangHuman-Robot Interaction Human-Robot Collaboration
Industrial and Manufacturing Engineering, Florida State University, Tallahassee, FL, USA
Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams
Mar 9, 2026Piyush Gupta, Sangjae Bae, Jiachen Li +1Large Language Model-Based Robot Planning Symbolic Planning
Honda Research Institute, San Jose, CA 95134, USA. · Georgia Institute of Technology, Atlanta, GA 30332, USA.

Aug 27, 2026J/K move · Enter open · S save

STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration

Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu, Songpo Li+1

University of North Carolina at Chapel Hill, United States · Honda Research Institute, San Jose, CA, USA

Large Language Model-Based Robot Planning Robot Task Planning Human-Robot Collaboration

Open paper AskPDF arXiv

Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8% in action executability and 14.8% in final-state error.

Abstract

Explore similar work

STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration

Precise Robot Command Understanding Using Grammar-Constrained Large Language Models

Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams