Task-Aware Positioning for Improvisational Tasks in Mobile Construction Robots via an AI Agent with Multi-LMM Modules
Authors: Seongju Jang, Francis Baek, SangHyun Lee
Organizations: Ph.D. Student, Department of Civil and Environmental Engineering, University of Michigan, 2350 Hayward St, Ann Arbor, MI 48109 · Assistant Professor, School of Civil and Environmental Engineering, Georgia Institute of Technology, North Avenue, Atlanta GA 30332 · Robert B. Harris Collegiate Professor, Department of Civil and Environmental Engineering, University of Michigan, 2350 Hayward St, Ann Arbor, MI 48109
Due to the ever-changing nature of construction, many tasks on sites occur in an improvisational manner. Existing mobile construction robot studies remain limited in addressing improvisational tasks, where task-required locations, timing of task occurrence, and contextual information required for task execution are not known in advance. We propose an agent that understands improvisational tasks given in natural language, identifies the task-required location, and positions itself. The agent's functionality was decomposed into three Large Multimodal Model (LMM) modules operating in parallel, enabling the application of LMMs for task interpretation and breakdown, construction drawing-based navigation, and visual reasoning to identify non-predefined task-required locations. The agent was implemented with a quadruped robot and achieved a 92.2% success rate for identifying and positioning at task-required locations across three tests designed to assess improvisational task handling. This study enables mobile construction robots to perform non-predefined tasks autonomously.
Figures & tables
Figure 1: Framework of the proposed AI agent with multiple modules
Figure 2: Conceptual overview of the proposed agent’s operation
Figure 3: Three components of the agent core and their processing
Figure 4: Prompts for agent core
Figure 5: Three-stage process of the navigation module
Figure 6: Prompts for the navigation module
Figure 7: Process for positioning module
Figure 8: Implemented system architecture, illustrating interactions among modules and tools
Test
Need to identify and move to the task-required location
Need to consider attribute and contextual conditions
Need to address mid-execution task assignments and adjustments
A
Yes
No
No
B
Yes
Yes
No
C
Yes
Yes
Yes
Table 1: Improvisational task challenges in each test
Test
No. of sessions
No. of tasks
A
15
35
B
15
35
C
15
33
Total
45
103
Table 2: Number of sessions and tasks in each test
Figure 9: Prompts for the navigation module
Test
Example task target setup
Example command
A
Move to the south hallway and inspect the fire extinguisher. Then move to the hallway intersection to inspect the safety vest. Then move to 1220 and inspect the brown box.
B
Move to 12C5 and inspect the open toolbox. Then move to 12C3 to inspect the fire extinguisher without a yellow tag.
C
Both settings with and without distractors are included.
(Mid-execution command) Actually, inspect the hand-pulled cart carrying fire extinguishers instead of the one carrying the brown boxes.
Table 3: Example task target setup and example commands for Test A, B, and C
Test
Success task
Failed task
Success session
Failed session
Task success rate
Session success rate
A
33
2
13
2
94.3%
86.7%
B
31
4
11
4
88.6%
73.3%
C
31
2
13
2
93.9%
86.7%
Total
95
8
37
8
92.2%
82.2%
Table 4: Measured success rates in each test
Target identification method
Success Task
Failed Task
Success Rate
Open-vocabulary model (Grounding DINO)
4
11
26.7%
Open-vocabulary model (Grounding DINO)
5
10
33.3%
Ours
13
2
86.7%
Table 5: Comparison of attribute and contextual condition handling in Test B setting
Figure 10: Example of a failure case in contextual condition handling using Grounding DINO
Figure 11: Example session on Test C
Failed Module
Reason
Frequency
Navigation
Wrong zone label selected
1
Positioning
Failed to find the target (searching until max iterations)
1
Positioning to other objects, including distractors
2
Final distance was longer than 1 m
3
The agent did not judge success as success
1
Total
8
Table 6: Task failure causes and responsible modules
Travel distance per session (m)
Travel distance per task (m)
Time per session (sec)
Time per task (sec)
Successed sessions
30.2
13.6
248
111
Failed sessions
32.6
13.9
425
175
All sessions
30.6
13.7
280
124
Table 7: Average travel distance and time per session and task
Department of Computer Science and Engineering, Mississippi State University, Starkville, MS, USA · 2Graduate School of Arts and Sciences, Columbia University, New York, NY, USA · 3Consumer and Community Banking, JPMorgan Chase, New York, NY, USA +2
Department of Civil and Environmental Engineering, Syracuse University, 158 Link Hall, Syracuse, NY 13244, USA · Department of Civil and Environmental Engineering, Syracuse University, 151J Link Hall, Syracuse, NY 13244, USA