Task-Aware Positioning for Improvisational Tasks in Mobile Construction Robots via an AI Agent with Multi-LMM Modules
Authors: Seongju Jang, Francis Baek, SangHyun Lee
Organizations: Ph.D. Student, Department of Civil and Environmental Engineering, University of Michigan, 2350 Hayward St, Ann Arbor, MI 48109 · Assistant Professor, School of Civil and Environmental Engineering, Georgia Institute of Technology, North Avenue, Atlanta GA 30332 · Robert B. Harris Collegiate Professor, Department of Civil and Environmental Engineering, University of Michigan, 2350 Hayward St, Ann Arbor, MI 48109
Due to the ever-changing nature of construction, many tasks on sites occur in an improvisational manner. Existing mobile construction robot studies remain limited in addressing improvisational tasks, where task-required locations, timing of task occurrence, and contextual information required for task execution are not known in advance. We propose an agent that understands improvisational tasks given in natural language, identifies the task-required location, and positions itself. The agent's functionality was decomposed into three Large Multimodal Model (LMM) modules operating in parallel, enabling the application of LMMs for task interpretation and breakdown, construction drawing-based navigation, and visual reasoning to identify non-predefined task-required locations. The agent was implemented with a quadruped robot and achieved a 92.2% success rate for identifying and positioning at task-required locations across three tests designed to assess improvisational task handling. This study enables mobile construction robots to perform non-predefined tasks autonomously.
Figures & tables
Figure 1: Framework of the proposed AI agent with multiple modules
Figure 2: Conceptual overview of the proposed agent’s operation
Figure 3: Three components of the agent core and their processing
Figure 4: Prompts for agent core
Figure 5: Three-stage process of the navigation module
Figure 6: Prompts for the navigation module
Figure 7: Process for positioning module
Figure 8: Implemented system architecture, illustrating interactions among modules and tools
Test
Need to identify and move to the task-required location
Need to consider attribute and contextual conditions
Need to address mid-execution task assignments and adjustments
A
Yes
No
No
B
Yes
Yes
No
C
Yes
Yes
Yes
Table 1: Improvisational task challenges in each test
Test
No. of sessions
No. of tasks
A
15
35
B
15
35
C
15
33
Total
45
103
Table 2: Number of sessions and tasks in each test
Figure 9: Prompts for the navigation module
Test
Example task target setup
Example command
A
Move to the south hallway and inspect the fire extinguisher. Then move to the hallway intersection to inspect the safety vest. Then move to 1220 and inspect the brown box.
B
Move to 12C5 and inspect the open toolbox. Then move to 12C3 to inspect the fire extinguisher without a yellow tag.
C
Both settings with and without distractors are included.
(Mid-execution command) Actually, inspect the hand-pulled cart carrying fire extinguishers instead of the one carrying the brown boxes.
Table 3: Example task target setup and example commands for Test A, B, and C
Test
Success task
Failed task
Success session
Failed session
Task success rate
Session success rate
A
33
2
13
2
94.3%
86.7%
B
31
4
11
4
88.6%
73.3%
C
31
2
13
2
93.9%
86.7%
Total
95
8
37
8
92.2%
82.2%
Table 4: Measured success rates in each test
Target identification method
Success Task
Failed Task
Success Rate
Open-vocabulary model (Grounding DINO)
4
11
26.7%
Open-vocabulary model (Grounding DINO)
5
10
33.3%
Ours
13
2
86.7%
Table 5: Comparison of attribute and contextual condition handling in Test B setting
Figure 10: Example of a failure case in contextual condition handling using Grounding DINO
Figure 11: Example session on Test C
Failed Module
Reason
Frequency
Navigation
Wrong zone label selected
1
Positioning
Failed to find the target (searching until max iterations)
1
Positioning to other objects, including distractors
2
Final distance was longer than 1 m
3
The agent did not judge success as success
1
Total
8
Table 6: Task failure causes and responsible modules
Travel distance per session (m)
Travel distance per task (m)
Time per session (sec)
Time per task (sec)
Successed sessions
30.2
13.6
248
111
Failed sessions
32.6
13.9
425
175
All sessions
30.6
13.7
280
124
Table 7: Average travel distance and time per session and task
This study introduces intelligent frameworks that use Large Language Models (LLMs) to improve task scheduling for construction robots. The LLM is fed with key data about the desired task, such as agent action abilities, and the desired end goal to be achieved. A well-balanced allocation strategy is developed, optimizing both time efficiency and resource utilization. Our system utilizes a Natural Language Processing interface to streamline communication with construction professionals and adapt in real-time to unexpected site conditions. We concurrently use two LLM agents, specifically generator (GPT-4) and supervisor (Gemma 3/Llama 4/Mistral 7b) LLM agents to provide a more precise task schedule. We evaluate the proposed methodology using a straightforward scenario and provide metric scores to prove the efficacy of the frameworks. Our results highlight that the implementation of LLMs is crucial in construction operational tasks including robots.
Swayamjit Saha, Subhabrata Das, Haonan Duan +1
Department of Computer Science and Engineering, Mississippi State University, Starkville, MS, USA · 2Graduate School of Arts and Sciences, Columbia University, New York, NY, USA · 3Consumer and Community Banking, JPMorgan Chase, New York, NY, USA +2
Deploying language-model agents onboard mobile robots requires task reasoning and responsive execution within limited compute and memory. We propose BT-Agent, an onboard architecture combining LLM-driven task decisions with predefined behavior-tree (BT) skills. These skills enforce execution conditions and return structured failure feedback. We demonstrate fully onboard operation with a 9B model on a quadruped across five task categories. Intervention tests show that local checks stop ongoing navigation without waiting for another model decision, while the agent remains active to retry or report constraints. To compare agent designs efficiently, we introduce a 161-task multi-fidelity benchmark, evaluated in text on all tasks and in Habitat on 39. Text supports 10.9× higher evaluation throughput, while simulation reveals how perception and motion failures change task outcomes. BT-Agent's largest success gains over the strongest Act/ReAct baseline occur at 4B and 9B: approximately 10 and 5 percentage points. Ablations identify structured tool calls and informative failure feedback as the most useful components.
Fabio Hübel, Adam Lilja, Yue Yin +5
Stanford University · University of California, Berkeley · NVIDIA
Humanoid robots, with their human-like shape and multi-tasking capabilities, are well-aligned with human-dominated workplaces, like those in civil and construction engineering, where they could collaborate with human workers or autonomously perform physically demanding and hazardous tasks. Despite this promise, limited research has explored how to endow these robots with the practical capabilities needed to perform construction tasks. To this end, this study proposes a novel perception-and-action system that enables humanoid robots to learn and perform construction tasks from worker demonstrations. This system contains two deep networks: Humanoid-PoseNet, which extracts human postures and translates them into mechanically feasible poses for a humanoid robot; and Humanoid-ActionNet, which learns robot-executable actions based on these translated poses. Experimental results demonstrate that the humanoid robot reliably executed eight construction-related actions, achieving an average motion-tracking error of 82.45 mm MPJPE (Mean Per Joint Position Error). This work provides an early step toward deploying humanoid collaborators in construction.
Yanxi Liu, Yizhi Liu
Department of Civil and Environmental Engineering, Syracuse University, 158 Link Hall, Syracuse, NY 13244, USA · Department of Civil and Environmental Engineering, Syracuse University, 151J Link Hall, Syracuse, NY 13244, USA